A "Mini-Shootout" of Pro Digital SLRs!

By: Dave Etchells, Imaging Resource
Additonal perspective by Rob Galbraith, see section at bottom!

As we alluded to in our special coverage of the new Fuji S1 Pro SLR digicam, we also managed to get our hands on several other pro cameras at PMA, and conducted an informal test of resolution and image noise with them. We felt this test would be particularly interesting in light of the controversy surrounding Fuji's S1 camera, which creates a ~6 million pixel file, using a ~3.2-2.5 (?) million pixel SuperCCD sensor. Fuji claims that the 6 million pixel final file size is entirely justified, due to the unique geometry of their SuperCCD sensor design, while detractors have maintained that this is mere hogwash and marketing hype, essentially saying "a pixel is a pixel", and "there ain't no free lunch." We suspect that both sides will find support for their positions in this test: We felt that the S1's resolution clearly outshown that of the pro cameras using conventional 3 megapixel sensors, but did in fact fall short of the Kodak DCS 660, which has a full 6 million pixel conventional CCD.

We also looked at noise performance from each of the cameras, at a variety of ISO ratings. Here, we felt we found a bigger difference in the S1's output, with noticeably less noise at a given ISO level (particularly very high ISOs) than the cameras with conventional CCDs. We felt this was particularly impressive for a product that's still in early prototype stages, as our experience has been that noise levels in production cameras are generally lower, due to higher levels of electronics integration, and the use of lower-power semicustom ICs.


What's Wrong With This Test?
Before going any further, or looking at any of the images we captured, we feel obligated to point out the limitations of this test. (There are a LOT of them!) Here's the short list:

• A 3-D subject, with no control over aperture, focusing distance, etc. - Working under restrictions of very short times with each camera, and no prior preparation, our choice of subjects was very limited. Since we were borrowing units that were intended to be used in the show booths, time was at an extreme premium, and our range severely limited. (Essentially, to whatever we could fine on a few minutes notice in the main hall itself.) We chose a "neutral" show booth (not belonging to any of the manufacturers in the test) that had some fine structural elements in it, some smallish text, and a flat gray area we could look at to evaluate noise performance. One of the big problems with the test though, was that we didn't have time to experiment with the various cameras' focusing systems, or try varying the focus to insure optimal, comparable results. Worse, variations in the different cameras' exposure systems meant that all the shots were taken at different aperture settings. Just to further confuse things, the lenses involved were all operating at different focal lengths, thanks to the need to shoot the subject at roughly the same scale, and from the same position, for the sake of consistency.
  
  
• A rickety little tabletop tripod, and the possibility of camera shake. - Again, thanks to the last-minute nature of our test, we just stuck the various cameras on our tiny "pocket" tripod that's really better suited to steadying the little Fuji MX-1700 digicam I use for my at-the-show product shots. We thus can't rule out the possibility that some of the cameras suffered in the resolution department due to camera shake. (We shot first with the Nikon D1, didn't notice until we went back for the second round with the S1 that the tripod actually had a pretty severe tilt when it was showing "level." Thus, the D1 shots are all tilted relative to the others, as well!)
  
  
• Bizarre color balance in the scene lighting. Fortunately, this isn't germane to the particular tests we're conducting here, but I mention it so people won't be inclined to give any of the cameras tests adverse marks for color balance. - We just shot with whatever the booth staff suggested we use for a white balance setting, and some were pretty far off the mark. No fault of the manufacturers though: The lighting in the convention center where PMA was held was provided some strange sort of arc-based lamps. While it at first glance looked fairly close to moderately high color-temperature incandescents (~3400-3800K?), it obviously had a very "spikey" spectrum, given how oddly various color were rendered under it, even to the naked eye. (Skin tones had a decidedly yellowish cast, while greens took on a somewhat electric hue.) Because of this, we ask our readers to *entirely ignore* white balance in the shots appearing here.
  
  
• A very limited view of "goodness" in pro SLR digicams! This is probably the biggest problem here: This is an admittedly one-dimensional (well, maybe two-dimensional) view of these cameras, completely ignoring issues such as color fidelity, tonal range, frame rate, shutter delay, systems-level capabilities, exposure control, etc. In short, just about all the parameters that enter into choosing a professional SLR system are completely missing from this discussion! (At least one parameter is constant: As it happens, all the cameras tested use Nikon lenses for their optics. - Obviously, Nikon won't be too concerned regardless of how this "shootout" turns out!)
  
  
• Intermediate processing and JPEG from multiple sources. Some cameras (the D1 and S1) produce JPEG files natively. With the Kodak cameras, we used the (excellent!) Kodak driver software for Photoshop on the Mac to convert the raw CCD-format TIFF files to standard JPEGs. (Saved at a quality level of "10" from Photoshop version 4.) The various images were thus handled somewhat differently, which could affect the outcome slightly. (We don't think this is a significant effect, but wanted to head off the inevitable questions on the topic.)

OK, enough babbling, let's see some results!
The shot at right shows a reduced view of the booth we used as a test subject. We'll show a few areas in detail, to illustrate both resolution and noise performance. The signage at front center provides a useful reference for resolution, as the various strokes and lines of the script text clearly show the resolving power of the various cameras.

Interesting, but probably less useful are the cross-members of the girder structure supporting the central sign on the booth, and the black sign hanging on the far wall that you can see peeking out behind and between the girders, just above and to the left of center in the image. The cables hanging the distant sign might have made an excellent test of fine detail handling, but their distance from the main focal point of the shot means that depth of field issues prevent us from drawing any firm conclusions.

Finally, the small gray box just below center in the image, to the right of the white chemical pails at the front of the booth provides an excellent uniform gray object to study image noise on.


The "Text" resolution test.
We liked this test the best for showing resolution and detail, because the strokes of the script in the sign make it easy to discern just how much each camera is actually resolving. Here are cropped results from each camera. (To make comparisons easier, we've done an "auto levels" operation in Photoshop to brighten up the images and give them a more uniform tonal range. We felt this was legitimate, as it wouldn't disturb the resolution, only the tonal range.)

Kodak DCS 620 (2.0 megapixels)	Kodak DCS 660 (6 megapixels)
Well, no question who wins this contest: The DCS 660 is obviously sharper than any of the others: The "no free lunch" people are at least somewhat correct, in that a conventional 6 megapixel CCD definitely appears to record more detail than a 3+ megapixel SuperCCD, with Fuji's clever interpolation. But don't write off the S1's larger file size as mere marketing hype just yet: Compare the details of the strokes and small elements of the script text between the S1 and either of the conventional 3 megapixel cameras. Look in particular at how open (or not) the centers of the letters "o", "a", "g", and even the small loop of the "e". It's hard to argue that there isn't more detail present than in either of the conventional 3 megapixel CCD images.	Fuji S1 (6 megapixel file, 3+ megapixel SuperCCD sensor)
Nikon D1 (3 megapixels)	Kodak DCS 330 (3 megapixels)

Further Manipulations (Added 2/12/00)
A number of readers wrote in, saying that the D1 image actually looked about as detailed as the S1 when it was upsampled to the same size in Photoshop(tm). This was an obvious thing we should have done in the first place, and so include the images below for your perusal. The top one is the D1 resampled to the same size as the S1 image. When we did so, we noticed that the D1 image had quite a bit of sharpening applied to it in-camera, so we applied some sharpening to the Fuji image in Photoshop, to try to get the best equivalence between the two. The results? Pretty interesting - While we still think we see a bit more detail in the S1 image, it's clearly not the blow-out you'd expect from "6" megapixels vs 2.7.

Nikon D1 image, upsampled 157%, bicubic interpolation

Fuji S1 image, unsharp masked in Photoshop(tm), radius 0.6 pixels, 200%

The "Fine Lines" Resolution Test
This one is a bit less conclusive/obvious, thanks to the problems of depth of field and focusing distance mentioned earlier. Still, it may have some value, so we're presenting it here:

Kodak DCS 620 (2.0 megapixels)	Kodak DCS 660 (6 megapixels)
While more equivocal (as noted), this is an interesting result: The DCS 660 still wins the resolution derby, but the S1 again does pretty well. The 3 megapixel cameras don't do too badly with the fine wires from the hanging sign, and the D1's picture retains a surprising delicacy in these fine lines.	Fuji S1 (6 megapixel file, 3+ megapixel SuperCCD sensor)
Nikon D1 (3 megapixels)	Kodak DCS 330 (3 megapixels)

Resolution Conclusion
I'm sure we'll get some argument/discussion on it, but the first-order results seem pretty clear to me: The S1 clearly bests conventional 3-megapixel cameras in the realm of resolution (at least within the limits of this test), but falls short of a "true" 6 megapixel design.


Comments on this article
Click here to go to a page where you can leave your own observations/comments, etc.


Image noise!
Given that this page is already painfully large, click here to go to a page containing the cropped selections from the camera files to evaluate image noise.

---

Special: Perspective from Rob Galbraith

As you might guess, this article elicited a LOT of comments and emails. Among the more interesting correspondence was an email thread between myself (Dave Etchells) and Rob Galbraith. Rob is one of the premier digital photojournalists worldwide, with vast experience in that market/application. (He maintains an exceptionally well put-together web site at www.robgalbraith.com.)

Rob graciously agreed to let me share our dialog here, so the rest of our readers could benefit from his perspective. Here's a direct excerpt from our email (my comments back to Rob are further indented below...):

At 9:37 AM -0700 2/8/00, Rob Galbraith wrote:
Hi Dave,

*The DCS 620 frame appears that it could be shot with the anti-aliasing filter installed, while the DCS 660 is clearly not. This may be making the DCS 620 frame look softer and less detailed than it actually is. Even so, my experience with the DCS 620 and a good lens like Nikon's 80-200mm f2.8 is that it will produce greater clarity than that, especially with the text test. If you shot the samples with a wide angle lens, then the DCS 620 results might be more in line with what I'm accustomed to seeing.

I believe the 220 did have the antialiasing filter in place, the 660 obviously did not. You're correct about the lens, they were all taken with a 28-105mm zoom (zoom, yuck!) I'll make note of that fact. (The Nikon D1 had the same lens on it, don't recall what the lens was on the S1.)

*It's possible that the differences in resolution only might be accounting
for the differences in apparent detail, at least between the 2MP DCS 620,
2.7MP Nikon D1 and the Fuji Super CCD at it's 3.x MP resolution.

I think there's more to it than that: I see a more pronounced difference between the S1 and the D1 than 2.7 - 3.2 or whatever MP would make. At the same time, the S1 looks to have more resolution than the DCS 330, which has a 3+ megapixel sensor as well.

*Perhaps the single biggest difference might be lens and focal length. You indicate that for the sake of consistency, different lenses and focal lengths were used. I understand the need for different focal lengths, but doesn't using different lenses just about kill the validity of the test itself? I think so.

You're right: As noted, the 620 and 660 had the identical lens on them, the D1 the same lens model, just a different unit. S1 is unknown. (I can try to find out - I remember it being a zoom with "one touch" zoom control) Definitely makes a difference, I don't think it kills the test entirely though.

*Sharpness differences could exacerbated by an underpowered tripod.

You've gone to great pains to explain the uncontrolled nature of the test, and I appreciate that, and why. But given that many of your readers are going to make lasting impressions of these cameras based solely on these frames, I'm concerned that the test is not up to the usual Imaging Resource standards and does more harm than good ultimately.

I agree about the "not up to the usual standards", but hope that the readers will actually read what I've written. I may be wrong (you're in a better position to say actually, given your focus and readership), but I tend to think that people considering spending $4-10K on a camera will be fairly sophisticated about the impact of these various factors. The other half of the test (image noise) of course isn't sensitive to optical issues like these, although it may be a bit unfair (as noted) to subject the S1 to such a high level of scrutiny so early in its product life...

What I'd obviously *really* like is to get all the cameras here to do a full-blown test, with identical lenses, locked-down massive tripod, standard res targets, with/without antialiasing filters, etc, etc. The ideal would be to use a single, very sharp fixed focal-length lens (maybe my Nikkor 50mm f/1.4, stopped down to around f/8 or so?), and just move the tripod in or out to adjust framing. This would eliminate the vagaries of lens focal length, or even zoom setting on the same lens. I'd also hope to get away from this one-dimensional view of the products as well, look at them in their fullness, considering and reporting on the myriad other features that should/would go into a photographer's decision to buy one or the other.

I'm going to work on the various mfrs involved to see if I can't set something like that up later this spring

>One more thing: sharpening in the camera. I didn't see any mention if that
>was on/off, and what setting. It could also have an impact.

Obviously, one more parameter for the full test, with huge potential impact. I used the default setting on cameras with an option. As I recall, the 330 and 620 were set to "medium". (Recall Mike DeLuca explicitly making that setting on one of the Kodak cameras at least.)

This was a very ad-hoc, uncontrolled test, but I think the results were/are still useful. There were some clear differences between units that I don't think can at all be attributed to just lens differences. (Although the DCS 620 clearly deserves another shot, minus anti-aliasing filter and with a lens more suited to it.)

Thanks again very much for your input, your perspective and experience is invaluable in this area. - Let me know if it'd be OK for me to share this conversation thread via the article itself

Original Image Files
Here's a list of links to the original test files from all the cameras:


Fuji S1, ISO 400 (1380k)
Fuji S1, ISO 800 (1476k)
Fuji S1, ISO 1600 (1680k)


Kodak DCS 330, ISO 125 (1000k)
Kodak DCS 330, ISO 250 (1168k)
Kodak DCS 330, ISO 400 (1296k)


Kodak DCS 620, ISO 200 (1112k)
Kodak DCS 620, ISO 400 (1492k)
Kodak DCS 620, ISO 1600 (2680k)


Kodak DCS 660, ISO 80 (3384k)


Nikon D1, ISO 200 (1092k)
Nikon D1, ISO 400 (1212k)
Nikon D1, ISO 800 (1168k)
Nikon D1, ISO 1600 (1048k)

Copyright © 2000 by Imaging Resource, Inc.