Panasonic G1 Imatest Results

Panasonic G1 Imatest Results

We routinely use Norman Koren's excellent "Imatest" analysis program for quantitative, thoroughly objective analysis of digicam test images. I highly recommend it to our technically-oriented readers, as it's far and away the best, most comprehensive analysis program I've found to date.

My comments below are just brief observations of what we see in the Imatest results. A full discussion of all the data Imatest produces is really beyond the scope of this review: Visit the Imatest web site for a full discussion of what the program measures, how it performs its computations, and how to interpret its output.

Here's some of the results produced by Imatest for the Panasonic G1:

The Panasonic DMC-G1 showed good color accuracy, with only minor oversaturation of bright reds, deep blues and some greens. The G1 actually undersaturates some yellows and greens. Hue accuracy was also very good, with most of the hue shift occurring in the cyans, reds, and oranges. Average saturation was 104.9% (oversaturated by 4.9%, mostly in the reds and blues). Average "delta-C" color error was 6.47 after correction for saturation, which is quite good, just a small step down from the best cameras we've tested. (Delta-C is the same as the more commonly referred to delta-E, but delta-C takes into account only color differences, ignoring luminance variation.) All in all, a very good color response for this class of camera. Mouse over the links below the illustration above to compare results with recent entry-level SLRs.

As is true of most SLRs, when using the Adobe RGB color space (which provides a much wider gamut, or range of colors that can be expressed), the Panasonic G1 delivers more highly saturated color, though the increase compared to sRGB is not as stark as with most cameras. Average saturation was 105.6% which is only slightly greater, and average saturation-corrected hue error was 5.96 "delta-C" units, which is actually slightly more accurate than the sRGB result. Again, mouse over the links below the illustration above to compare results with recent entry-level SLRs.

This image shows how the Panasonic G1 actually rendered the colors of the MacBeth chart, compared to a numerically ideal treatment. In each color swatch, the outer perimeter shows the color as actually captured by the camera, the inner square shows the numerically correct color after correcting for the luminance of the photographed chart (as determined by a second-order curve fit to the values of the gray swatches), and the small rectangle inside the inner square shows the numerically correct color, without the luminance correction. This image shows the very good hue and saturation accuracy, as evidenced by the fact that the small rectangles are almost invisible against the luminance-corrected colors in the larger square swatches. The as-captured colors are a little bright for many blues, and a little dark for some warm tones, notably yellows and oranges. We did notice some yellows looked a little dull in some of our test images (see the yellow embroidery thread in our Still Life target), but felt that the overall visual impact of the G1's color was quite positive. If Panasonic could just clean up their handling of the range of spectrum from orange through yellow-green, the G1's color would be truly impressive.

There's a lot in this particular graph, a lot more than we have room to go into here. (This set of plots has also changed a little in the more recent versions of Imatest. Some of the plots that were shown here previously are now shown in other Imatest output. Since we largely focus on the Noise Spectrum plot, we'll only show the graphic above, which includes that plot.)

In comparing these graphs with those from competing cameras, I've found that the Noise Spectrum graph at lower right is the most important. Cameras that manage to shift their noise spectrum to higher frequencies have much finer-grained noise structures, making their noise less visually objectionable. In the graph above, this would show up as a noise spectrum curve that remained higher on the right side, representing higher noise frequencies. The champion at this was (and still is) the Canon EOS-1Ds Mark II, which produced remarkably fine-grained image noise, even at very high ISOs.

The Panasonic G1 does a pretty good job of keeping plenty of the luminance noise energy (indicated by the black line) at high frequencies. What little low-ISO luminance image noise that's there is quite fine-grained as a result. Chroma noise, though, is much higher, especially in the red and blue channels, some of the highest noise values we've seen in the DSLR category. When inspecting low ISO G1 images very closely, you can see blotches of chroma noise in neutral colors, such as the darker gray patches in the MacBeth chart (see the exaggerated example at right - NOTE that this is exaggerated 400% over the original file data). While the Panasonic G1 did quite well at high ISO shooting, we did see quite a bit of chroma noise in shadow areas. Cranking the noise reduction setting up to +2 alleviated a lot of this, but we'd have been happier if it wasn't there in the first place.

Above is the same set of noise data at ISO 1,600. Here, the Noise Spectrum graph is shifted quite a bit toward the left-hand, lower-frequency side than it was at ISO 100, coarsening the "grain" of the image noise patterns quite a bit. The red, blue and green channels still exhibit a bit more noise at the low end of the frequency spectrum, but, interestingly, are much lower relative to the luminance noise, tracking it closely at all but the lowest frequencies. We were a little surprised to see this; that the differential between chroma and luminance noise was actually lower at high ISOs than at low ones. - Particularly given that our perception was consistently that chroma noise was the biggest issue at high ISO settings. Perhaps this is because the luminance noise tended to be so fine-grained, while the red and blue-channel noise tended to be blotchy. (Even though the luminance curve in the figure above is shifted quite a bit toward the left, low-frequency side relative to the curve at ISO 100 we saw earlier, the G1 still spreads much more of its noise energy to higher frequencies than is common.)

Here's the same set of noise data at ISO 3,200. Here again, the Noise Spectrum graph is shifted even more toward the left-hand side, coarsening the "grain" of the image noise patterns further.

This chart compares the Panasonic G1's noise performance over a range of ISOs against that of other cameras. While we continue to show noise plots of this sort because readers ask for them, we each time point out that the noise magnitude is only a small part of the story, the grain pattern being much more important. Here, we can see that the Panasonic G1's luminance noise magnitude starts out about average relative to its competition, and increases at a fairly constant rate as ISO climes to 800, where it ends up close to most of the pack. However, at ISOs 1,600 and 3,200, G1 luminance noise levels are much higher than the rest. Do keep in mind these measurements are taken with each camera set at default settings, so the shape or position of the curve could be influenced by the settings you choose to use. The Panasonic G1 offers five levels of noise reduction, with the above data collected using the default, middle level. Also, while the G1's luminance noise at ISO 1,600 is higher than its competitors, the fine-grained structure of it meant that it was less visible in prints, at least up to 8.5x11 inches in size.

Panasonic G1 Dynamic Range Analysis

A key parameter in a digital camera is its Dynamic Range, the range of brightness that can be faithfully recorded. At the upper end of the tonal scale, dynamic range is dictated by the point at which the RGB data "saturates" at values of 255, 255, 255. At the lower end of the tonal scale, dynamic range is determined by the point at which there ceases to be any useful difference between adjacent tonal steps. Note the use of the qualifier "useful" in there: While it's tempting to evaluate dynamic range as the maximum number of tonal steps that can be discerned at all, that measure of dynamic range has very little relevance to real-world photography. What we care about as photographers is how much detail we can pull out of the shadows before image noise becomes too objectionable. This, of course, is a very subjective matter, and will vary with the application and even the subject matter in question. (Noise will be much more visible in subjects with large areas of flat tints and subtle shading than it would in subjects with strong, highly contrasting surface texture.)

What makes most sense then, is to specify useful dynamic range in terms of the point at which image noise reaches some agreed-upon threshold. To this end, Imatest computes a number of different dynamic range measurements, based on a variety of image noise thresholds. The noise thresholds are specified in terms of f-stops of equivalent luminance variation in the final image file, and dynamic range is computed for noise thresholds of 1.0 (low image quality), 0.5 (medium image quality), 0.25 (medium-high image quality) and 0.1 (high image quality). For most photographers and most applications, the noise thresholds of 0.5 and 0.25 f-stops are probably the most relevant to the production of acceptable-quality finished images, but many noise-sensitive shooters will insist on the 0.1 f-stop limit for their most critical work.

The image below shows the test results from Imatest for an in-camera JPEG file from the Panasonic G1 with a nominally-exposed density step target (Stouffer 4110), and the G1's settings such as Contrast (0) and iExposure (Off) at their default positions.

Here, we can see that he tone curve shows pretty gradation in highlights, but the shadow end trails off more abruptly. These are pretty good numbers for a Four-Thirds sensor, but lag behind results from the larger APS-C sensors the competition uses.

Above are the results with iExposure adjusted to the Low setting. It produced a slight increase at the highest quality, but a very subtle effect.

Above are the results with iExposure turned at the Standard setting. This time, actually a slight decrease at the highest quality. Again, a very subtle effect.

Above are the results with iExposure turned at the High setting. A pretty nice looking curve, and higher overall dynamic range (10.8 EV), but the noise in the darker patches of the step-chart that were boosted limits the Imatest score at the highest quality level.

Unfortunately, Adobe Camera Raw, our standard RAW converter does not yet support the Panasonic G1, so we couldn't test the results from RAW at the time of writing (early November 2008).

Turning to the Panasonic G1's RAW files, we opened RAW version of the iExposure Off file discussed above in SilkyPix (Adobe Camera Raw didn't support the G1 yet, as of this writing), and played with the highlight recovery and contrast controls to bring out more steps of the Stouffer 4110 target. With SilkyPix's default noise reduction settings, the result was about a third of a stop more dynamic range at the highest quality level.

Suspecting that noise in the shadows was the primary limiter of the G1's dynamic range, we processed the RAW file through SilkyPix again, this time with fairly strong noise reduction applied. The result was a dramatic jump in dynamic range, a full 1.2 stops improvement over the camera JPEG. This underscores that the G1 is capable of delivering excellent image quality and dynamic range, when working from RAW files, and when using good noise reduction software. We didn't list the SilkyPix results in the table below, as they can't really be compared with those from Adobe Camera Raw; the processing parameters were likely wildly different.

The illustration above shows the results from Adobe Camera Raw 5.2, with Auto settings (slightly better results are likely possible with manually tweaking). As can be seen, the score at the highest quality level increased over camera JPEGs to 7.32 f-stops, while total dynamic range increased to 10.9 f-stops.

Dynamic Range, the bottom line:

The net result was that the G1 performed well when compared against most current Four-Thirds models, but lags behind most DSLRs with APS-C size sensors. The G1 lags behind some other Four-Thirds cameras in RAW mode, because the smaller pixels from the higher-resolution 12-megapixel sensor results in increased noise, which knocked the Imatest score down at the highest quality. This is the same reason it lags behind APS-C models.

To get some perspective, here's a summary of the Panasonic G1's dynamic range performance, and how it compares to other digital SLRs that we also have Imatest dynamic range data for. (Results are arranged in order of decreasing dynamic range at the "High" quality level.):

The results shown in the table are interesting. One of the first things that struck me when I initially looked at test data for a wide range of DSLRs, was that here again, purely analytical measurements don't necessarily correlate all that well with actual photographic experience. There's no question that the Fuji S3 Pro deserved its place atop the list, as its unique "SR" technology did indeed deliver a very obvious improvement in tonal range in the highlight portion of the tonal scale relative to competing models of its day. (Amazing that it's now surpassed by even consumer-level models using today's technology.) I was also surprised to see the analytical results place the original Olympus E-300 as highly as they did, given that our sense of that camera's images was that they were in fact noisier than those of many other DSLRs that we looked at. In the other direction, I was quite surprised to see the Nikon D2x place as low on the listings as it did, given that we found that camera's shadow detail to be little short of amazing.

One thing that's going on here though, is that we tested each camera at its base ISO setting, which should produce best-case noise levels. This is in fact what many photographers will be most interested in, but it does perhaps place some of the Nikons (like the D40) at a disadvantage, as their lowest ISO setting is 200, as compared to the ISO 100 settings available on most other models.

Panasonic G1 Resolution Chart Test Results

The chart above shows consolidated results from spatial frequency response measurements in both the horizontal and vertical axes. The "MTF 50" numbers tend to correlate best with visual perceptions of sharpness, so those are what we focus on here. The uncorrected resolution figures are 1,832 line widths per picture height in the horizontal direction (corresponding to the vertically-oriented edge), and 1,716 lines along the vertical axis (corresponding to the horizontally-oriented edge), for a combined average of 1,774 LW/PH. Correcting to a "standardized" sharpening with a one-pixel radius increased vertical resolution by quite a bit, but horizontal resolution increased only slightly, resulting in an average of 2,000 LW/PH.

The Panasonic G1 delivers excellent resolution, very competitive with other 12-megapixel DSLRs currently on the market. Its images straight from the camera show a lot of detail, and even more can be extracted with careful RAW processing.

To see what's going on, refer to the plots below, which show the actual edge profiles for both horizontal and vertical edges, in both their original and corrected forms. Here, you can see that a nearly ideal amount of in-camera sharpening is applied in the horizontal direction (undersharpened by only 4.82%, explaining why standardized sharpening wasn't able to improve on the MTF 50 numbers much), while in-camera sharpening is not as aggressive in the vertical direction (12.5 % undersharpened). Despite the "undersharpening" reported by Imatest though, there is a noticeable "bump" visible in the horizontal edge profile. (A very slight one in the vertical edge profile, but probably negligible.) You should thus turn the camera's sharpening down a little for optimal results when sharpening in-camera JPEGs post-exposure in Adobe Photoshop or other image editing software.