Sigma DP1 Imatest Results

Sigma DP1 Imatest Results

We routinely use Norman Koren's excellent "Imatest" analysis program for quantitative, thoroughly objective analysis of digicam test images. I highly recommend it to our technically-oriented readers, as it's far and away the best, most comprehensive analysis program I've found to date.

My comments below are just brief observations of what we see in the Imatest results. A full discussion of all the data Imatest produces is really beyond the scope of this review: Visit the Imatest web site for a full discussion of what the program measures, how it performs its computations, and how to interpret its output.

Here's some of the results produced by Imatest for the Sigma DP1:

As mentioned elsewhere in the review Sigma DP1 showed pretty good color accuracy, though its average saturation is quite a bit lower than that of its competition. Overall hue accuracy was very good, although small hue shifts occurred in many colors. Average saturation was 93.14% or 6.86% undersaturated (most cameras oversaturate somewhat) but average "delta-C" color error was 5.22 after correction for saturation, which is pretty good. (Delta-C is the same as the more commonly referred to delta-E, but delta-C takes into account only color differences, ignoring luminance variation.) Overall, a good (albeit somewhat muted) color response for a compact camera. Mouse over the links below the illustration above to compare results with other recent models.

While not part of our normal color-accuracy discussion, we feel we should mention a color issue that we observed in some of our test images. Particularly noticeable on our MULTI target shots, the white balance changes from the center of the target towards the corners, with a noticeable green cast appearing as you move from the center to the corners. Oddly, while still apparent, this effect was reduced in images processed into JPEGs from the DP1's X3F RAW-format files via Sigma's software. This looks like it might be a lens effect, but we seemed to see it more in some shots than others, even under identical lighting. (It was more apparent with our MULTI target than the Still Life, although it was still somewhat evident there as well.)

When using the Adobe RGB color space (which provides a much wider gamut, or range of colors that can be expressed), the Sigma DP1 delivers slightly more saturated color, with an average saturation of 95.46% and average saturation-corrected hue error of 5.76 "delta-C" units. Again, mouse over the links below the illustration above to compare results with recent dSLRs.

This image shows how the Sigma DP1 actually rendered the colors of the MacBeth chart, compared to a numerically ideal treatment. In each color swatch, the outer perimeter shows the color as actually captured by the camera, the inner square shows the numerically ideal color, after correcting for the luminance of the photographed chart (as determined by a second-order curve fit to the values of the gray swatches), and the small rectangle inside the inner square shows the numerically ideal color, without any luminance correction. This image shows the good hue accuracy, as well as a gamma curve that results in a slight underexposure of upper midtones and slight overexposure of deep shadows.

There's a lot in this particular graph, as usual a lot more than we have room to go into here. (This set of plots has also changed a little in the more recent versions of Imatest. Some of the plots that were shown here previously are now shown in other Imatest output. Since we largely focus on the Noise Spectrum plot, we'll only show the graphic above, which includes that plot.)

In comparing these graphs with those from competing cameras, I've found that the Noise Spectrum graph at lower right is the most important. Cameras that manage to shift their noise spectrum to higher frequencies have much finer-grained noise structures, making their noise less visually objectionable. In the graph above, this would show up as a noise spectrum curve that remained higher on the right side, representing higher noise frequencies. The champion at this was the Nikon EOS-1Ds Mark II, which produced remarkably fine-grained image noise, even at very high ISOs.

At low ISO settings, the Sigma DP-1's noise characteristic is quite unusual: It shows low luminance noise, and does a good job of keeping the luminance noise energy up in the higher frequency range, producing relatively fine-grained luminance noise. Unfortunately, chroma noise is high and has a lot of energy at low spatial frequencies, producing a more blotchy appearance. The overall noise levels at low ISO settings are low enough that the blotchiness may be entirely academic (you're really not likely to see it in any normal subjects), but the noise spectrum is different enough from most digicams and SLRs that we felt we should call attention to it.

Here's the same set of noise data at ISO 800, the DP1s maximum sensitivity. Here, the Noise Spectrum graphs for the color channels are shifted quite a bit toward the left-hand, lower-frequency side than it was at ISO 100, but the luminance noise still remains impressively flat. This indicates a coarsening of the "grain" of the image color noise patterns, and indeed that's what we see when inspecting the DP1's ISO800 images. There are large grains or blobs of green and magenta noise to be seen, particularly in the shadows.

This chart compares the Sigma DP1's luminance noise performance over a range of ISOs against that of other cameras. While I continue to show noise plots of this sort because readers ask for them, I each time point out that the noise magnitude is only a small part of the story, the grain pattern being much more important. Here, we can see that the Sigma DP1's luminance noise starts out very low, and remains below the SLRs in this group up to its maximum ISO of 800 (where it's more aggressive noise reduction reduces luminance noise below the ISO 400 level). What this graph doesn't show is the chroma noise that results in the very objectional green and magenta noise that plagues the shadows (and even lower midtones) in DP1 high ISO shots.

Sigma DP1 Dynamic Range Analysis

A key parameter in a digital camera is its Dynamic Range, the range of brightness that can be faithfully recorded. At the upper end of the tonal scale, dynamic range is dictated by the point at which the RGB data "saturates" at values of 255, 255, 255. At the lower end of the tonal scale, dynamic range is determined by the point at which there ceases to be any useful difference between adjacent tonal steps. Note the use of the qualifier "useful" in there: While it's tempting to evaluate dynamic range as the maximum number of tonal steps that can be discerned at all, that measure of dynamic range has very little relevance to real-world photography. What we care about as photographers is how much detail we can pull out of the shadows before image noise becomes too objectionable. This, of course, is a very subjective matter, and will vary with the application and even the subject matter in question. (Noise will be much more visible in subjects with large areas of flat tints and subtle shading than it would in subjects with strong, highly contrasting surface texture.)

What makes most sense then, is to specify useful dynamic range in terms of the point at which image noise reaches some agreed-upon threshold. To this end, Imatest computes a number of different dynamic range measurements, based on a variety of image noise thresholds. The noise thresholds are specified in terms of f-stops of equivalent luminance variation in the final image file, and dynamic range is computed for noise thresholds of 1.0 (low image quality), 0.5 (medium image quality), 0.25 (medium-high image quality) and 0.1 (high image quality). For most photographers and most applications, the noise thresholds of 0.5 and 0.25 f-stops are probably the most relevant to the production of acceptable-quality finished images, but many noise-sensitive shooters will insist on the 0.1 f-stop limit for their most critical work.

The image below shows the test results from Imatest for an in-camera JPEG file from the Sigma DP1 with a nominally-exposed density step target (Stouffer 4110), and the DP1's contrast and other settings at their default positions.

These results would be very respectable for a dSLR and simply amazing for a camera the size of the Sigma DP1.

Processing the Sigma DP1's RAW (.X3F) files through Sigma's Photo Pro 3.2 decreased total dynamic slightly from 10.1 to 9.92, and also decreased dynamic range slightly the highest quality level. These results were obtained by using the software's automatic settings; better results may be possible by tweaking manually, but we ran out of time.

Dynamic Range, the bottom line:

The net result was that the Sigma DP1 performed better than quite a few SLRs for JPEG images straight from the camera. Its JPEG results topped all the current SLRs with Four-Thirds sensors, as well as some older APS-C models. We may have been able to improve the RAW DR results we got by experimenting with Sigma's Photo Pro software, but the software is quite slow, limiting the amount of experimentation we could perform. It's too bad Adobe Camera Raw doesn't support the DP1, as it would have been nice to use it for a fair comparison of RAW results. Still, an amazing performance for such a compact camera.

To get some perspective, here's a summary of the Sigma DP1's dynamic range performance, and how it compares to digital SLRs that we also have Imatest dynamic range data for. (Results are arranged in order of decreasing dynamic range at the "High" quality level.):

The results shown in the table are interesting. One of the first things that struck me when I initially looked at test data for a wide range of dSLRs, was that here again, purely analytical measurements don't necessarily correlate all that well with actual photographic experience. There's no question that the Fuji S3 Pro deserves its place atop the list, as its unique "SR" technology does indeed deliver a very obvious improvement in tonal range in the highlight portion of the tonal scale. I was surprised to see the analytical results place the Olympus EVOLT as highly as they did, given that our sense of that camera's images was that they were in fact noisier than those of many other dSLRs that we looked at. In the other direction, I was quite surprised to see the Nikon D2x place as low on the listings as it did, given that we found that camera's shadow detail to be little short of amazing.

One thing that's going on here though, is that we tested each camera at its lowest ISO setting, which should produce best-case noise levels. This is in fact what many photographers will be most interested in, but it does perhaps place some of the Nikons (like the D40) at a disadvantage, as their lowest ISO setting is 200, as compared to the ISO 100 settings available on most other models.

Sigma DP1 Resolution Chart Test Results

The chart above shows consolidated results from spatial frequency response measurements in both the horizontal and vertical axes. The "MTF 50" numbers tend to correlate best with visual perceptions of sharpness, so those are what I focus on here. The uncorrected resolution figures are 1,080 line widths per picture height in the horizontal direction (corresponding to the vertically-oriented edge), and 1,187 lines along the vertical axis (corresponding to the horizontally-oriented edge), for a combined average of 1,133 LW/PH. Correcting to a "standardized" sharpening with a one-pixel radius increased both vertical and horizontal resolution by a bit, resulting in an average of 1,281 LW/PH. This would be considered a good performance for a conventional 5-megapixel camera. Where the Foveon X3 sensor really shines is in chrominance resolution, which this chart doesn't really show, nor which we normally test separately for.

(Technical side note: Interesting to note here, in comparing these plots against those for cameras with conventional sensors is how much higher the cycles/pixel numbers are (the left vertical axis).The DP-1's Foveon sensor gives it almost twice the cycles/pixel of a camera with a conventional sensor. This supports Foveon and Sigma's claims that the Foveon technology delivers much more detail from a given number of sensor pixels than conventional designs do. It isn't 3x the amount of information though, but more like 2x. (That's because, while conventional sensors have separate red, green, and blue pixels, they have twice as many green pixels as either of the other colors. - And human detail perception tends to follow the green channel quite strongly. Thus, conventional cameras could be said to get luminance information from about half their pixels, while Foveon sensors extract "detail" information from every pixel. Hence the roughly 2:1 ratio in cycles/pixel.)

To see what's going on, refer to the plots below, which show the actual edge profiles for both horizontal and vertical edges, in both their original and corrected forms. Here, you can see that very conservative in-camera sharpening is applied in the horizontal direction (undersharpened by 8.31%) , and also in the vertical direction (undersharpened by 6.24%). For the ultimate in sharpness, instead of turning up the in-camera sharpness, try processing the resulting images after the fact in Photoshop or other image-processing application to bring out the maximum detail without introducing sharpening artifacts.