Autofocus Reality Part 4: Nikon Full Frame

I started this series to simply demonstrate a well-known fact: that contrast detection autofocus in live view was more accurate than standard phase detection autofocus for still subjects. It ended up being a bit more than I’d bargained for, just with the Canon cameras. After that series was done, I promised several people that I would take a similar look at Nikon cameras. I also got several pertinent suggestions and criticisms of the first series about gathering more data and presenting more examples. I promised to do that, too.

You ever notice how easy it is to promise to do some work later on? Only then do you realize that what you’ve promised is going to take about 30 hours to get done. That causes an immediate disturbance in the force, and your day job workload triples because there’s a holiday weekend coming up. Plus, some cool stuff that you really want to play with need to carefully evaluate comes in. So for the three or four people who were waiting for this, I apologize for taking so long.

(If you've not been following the series from the start, by the way, you'll want to read the earlier parts: Part 1: Center-Point, Single-Shot Accuracy, Part 2: One vs. Two, Old vs. New, Part 3A: Canon Lenses, and Part 3B: Canon Cameras.)

 

 

The Conclusion
Several people wanted to see more data presented, so I’ll try to show more this time. Most people, though, simply want to know what I found. A few, of course, want to know how they can work in what I found into some raving fanboy post or other. Sorry, fanboys, there really isn’t any Canon / Nikon forum fodder here. These tests are fairly blunt tools that are only looking at one tiny aspect of autofocus.

Anyway, if you just want the results, they are pretty straightforward:

1. The Nikon full-frame cameras all do well with center-point, single-shot AF.

2. The newer bodies seem a bit better than the older ones.

3. There don’t seem to be dramatic changes with a few camera-lens combinations like the newer Canon bodies and lenses showed.

There are a few rather interesting things we found this time around. Additionally, my “write as you go, not when you know” technique in this series has gotten several knowledgeable people to email more thoughts and information. That helped me accomplish my main purpose: find out more about how phase detection AF works. Which lets me speculate a bit more at the end of the article.

Data
We tested a larger number of lens / camera combinations this time around. As always, we used cameras and lenses that had recently been tested and found acceptable. One series of shots, discussed here, was removed and not included in these results since the lens used was found to be decentered.

I’ll show some example graphs of various lens / camera runs to give an idea of what the actual data points look like. This space is too small to try to give a graph of every data point, so there will still be some summary numbers. The graphs show resolution (MTF50 measured by Imatest) for different lens / camera combinations. One lens tested on one camera is a run--then we changed both lens and camera for the next run, etc. For each run I show the results for 10 shots taken with live view (contrast detection AF) as diamonds, and 10 shots with standard phase detection AF as crosses. The vertical axis is average resolution at 13 points on the sensor (corners, edges, four mid-points, and center) while the horizontal axis is resolution at the center point (all in line pairs per picture height from unsharpened raw files).

Each run (represented by one color) is one copy of the camera with one copy of the lens; no lens or camera repeats. For example, when we tested Nikon D800 bodies paired with AF-S Nikkor 28mm f/1.8 G lenses, we tested four different lenses on four different bodies. I’m only showing two runs in the graphs because they get too crowded and run over each other, but all four are used in determining the average numbers. I repeat this because tomorrow someone will be saying “they only tested two cameras” because, well, some people apparently find reading too hard and they just look at the pictures.

One other thing is worth mentioning. We look at the standard deviation as a convenient measure of variation between the sample sizes. That’s a fairly valid and convenient measurement when we’re comparing D700 contrast detection to D700 phase detection. When we start comparing D700 numbers to D800 numbers, though, the resolution numbers are quite different. The absolute numbers of the D800 are much higher, so the same perecentage of variation is going to make a slightly larger standard deviation.

Some Visual Examples
The range of the axis on the graphs below change depending on which camera is used. In the Nikon D700 graph, for example, the peak resolution range is 400 to 900 line pairs per picture height. With the D800 the range shown is 700 to 1,100 LP/PH.

For the D700 and D8000 I only graphed two of the four runs, to clearly show how contrast detection and phase detection varies. For the Nikon D4 and D800 I graphed three runs. It makes the graphs a little more crowded, but in both of those graphs the runs colored red make a point I want to use. They also show why I didn’t try to put four runs on one graph.

The D700 shows a difference between phase and contrast detection similar to what we’ve seen in the better Canon bodies. Contrast detection gives a very tight pattern, about as good as I could manage using careful live view manual focusing with magnification. Phase detection gives some shots every bit as good as contrast, some not quite as good numerically (but not different enough that you could easily tell in a photograph), and an occasional shot that has just missed focus.

The Nikon D3x gave us similar results. I should mention that out of the four runs with these two cameras, we had three completely miss focus with the D700--one shows in the graph above--and four with the D3x, out of 40 shots with each. That’s about what we’ve seen with most Canon cameras and at around 10% is similar to the missed focus rate others have reported with phase detection.

The D800 example above shows something worth demonstrating again, because so many people still don’t get it. For the green and blue runs phase detection is similar to the previous cameras: a bit wider spread (larger standard deviation) but generally close to the contrast detection results.

The red run, though, shows the phase detection average is clearly a bit lower than the contrast detection. In other words, the pattern is similar to the phase pattern of the other runs, but the average is further away from the best shots obtained with contrast detection. That’s because this lens required microfocus adjustment of -5 on this particular camera. When we made that adjustment and repeated the run, the red pattern looked similar to the others.

The point: microfocus adjustment brings the average up of phase detection AF up closer to the contrast detection range, but it doesn’t reduce the variation seen with phase detection shots. The D4 graph below shows a similar example. For purposes of the averaged numbers I’ll talk about below, we repeated those runs after making appropriate microfocus adjustments and used the adjusted runs for our actual calculations.

You’ve noticed in the graphs, I’m sure, that both the D4 and D800 seemed to have fewer ‘bad misses’ than the D3x and D700 did. The D800 had two of 40 missed shots, and the D4 just one of 40. That’s not clearly significant, but certainly it's an interesting trend.

Data Summary
The summary of multiple runs using a different camera and lens each time, for the 28mm f/1.8 lens shot at f/2.8 on the various cameras, is shown in the table below.

It doesn’t look like much, I know, but that table represents 320 Imatest shots. Before we start discussing it, let me be clear: the numbers we’re looking at aren’t a statistical analysis, and anyone who tries to say that “Camera X was 30 and camera Y was 17, so camera Y has a better autofocus system” is missing the point entirely. We’re looking at one tiny part of the AF system: how well does the phase detection system compare to contrast detection, looking only at the center point with a still target.

Before we started we all knew that contrast detection was more accurate. When we tested Canon cameras, most had a standard deviation about twice as large for phase detection as for contrast detection. The Canon 1D Mark IV was more accurate than that. When mounted with a select set of the newest lenses, the Canon 5D Mark III and Canon 1Dx were just about as accurate in phase detection as with contrast detection. With older lenses, though, they were about as accurate as the 1D Mark IV.

With the Nikon full-frame cameras we see that the D700 and D3x seem similar to the Canon cameras: phase detection has a standard deviation about twice as large as contrast detection. The newer cameras--the D800 and D4--seem clearly better, very similar to the newer Canon bodies on most lenses.

Of course, we did check other Nikon lenses to see if there was a subset, like the newest Canon lenses, that seemed as accurate with phase detection as with contrast on the D4 and D800. We did smaller runs (two lenses on two cameras) with the AF-S NIKKOR 70-200mm f/2.8G ED VR II, AF-S NIKKOR 35mm f/1.4G, and AF-S NIKKOR 85mm f/1.8G. Results were similar to the 28mm f/1.8. We’ll check back with the next new Nikon lens release to see if it has improved accuracy on these cameras, whenever that is.

Discussion
There’s not really any difference between Canon and Nikon cameras for the lenses we all use every day. Both do pretty well, and the newer cameras are a bit better than older models. Of course, that doesn’t mean there’s a huge difference in real life.

A better real life comparison would be “how many shots were badly missed”, like the ones you see in the D700 graphs. That degree of miss would be obvious in the photograph. The variation in standard deviation probably would only be detected by pixel-peeping. Unfortunately, it would take several hundred more shots with each camera to get large enough numbers to determine statistical significance for number of missed shots. At roughly an hour per Imatest run, there’s no way I’ll ever have enough time to do that.

These results don’t translate to outer focus points, servo focus, etc. They simply hint that the manufacturers are doing something that makes phase detection more accurate on still shots using the center points.

There does seem to be a small group of Canon lenses (24mm f/2.8 IS, 28mm f/2.8 IS, 40mm f/2.8, 70-300mm L IS and 300mm f/2.8 IS II) that are more accurate on just the newest bodies. We’ll have to wait for the next new Nikon lens release to see if there is a similar improvement with newer Nikon lenses.

On the other hand, there’s one thing worth considering about the “more accurate” group of new Canon lenses. All are f/2.8 prime lenses. It will be very interesting to see if the the Canon 24-70 f/2.8 II lens, and any future fast prime lenses, are also more accurate. And yes, I was able to get the Imatest shots to check that, though it will take me a while to get the numbers run.

I know some of you want to know how the crop sensor Nikons did. I’ll be honest; this is incredibly time consuming work and I just can’t get to more cameras right now. I’m comfortable that none of the crop sensors will be better, and some will probably be worse than the full-frames. That’s what we saw on Canon cameras and I don’t think it will be hugely different for Nikon. But it’s certainly possible that Nikon crop cameras are just as good as Nikon full-frames.

I’ve also had several people ask me to compare contrast detection accuracy between SLRs and mirrorless cameras. That’s kind of a fool’s errand. Contrast detection on SLRs is as accurate as I can measure. We can’t detect more accurate than we can measure.

I do think it would be worthwhile for someone to compare the speed of contrast detection in different systems. While I expect all contrast detection to be as accurate as we could measure, doing this little experiment made me very aware that the speed to lock on using contrast detection is very different with different cameras. The most practical way would probably be to record the AF motor sound and then use that to measure time.

Notice I’m giving this suggestion out for someone to do, because I’m not going to be able to get to it any time soon. If one of you wants to do it, we’d be happy to publish the results. Heck, if one of you wants to come to LensRentals HQ and do it with our gear, we’d be happy to set you up.

(Roger Cicala is the founder of LensRentals.com. Visit LensRentals.com to check out that cool lens you've been hankering for, and for some of the best customer service on the Internet!)