The Imaging Resource Solar Simulator - A new "Outdoor" test shot.
| The
original "Outdoor Portrait" |
The
new "Harsh Portrait" |
 |
 |
| The original "Outdoor Portrait" shot did a good job of providing an acid test for cameras' tonal capabilities, but was subject to random variation from weather conditions. (Not to mention that it was a royal pain to deal with the limited shooting time that was possible with it.) | The new "Harsh Portrait" shot has been designed to duplicate the color temperature and contrast ratio of the local sunlight here on an "average" day, providing an equally challenging test of camera capabilities. Shot in the studio though, it will be 100% consistent and repeatable, as well as much more convenient to work with. (The color difference you see here is likely the result of different white balance response to the background.) |
After nearly six years of using the original test setup, I'm changing my "Outdoor Portrait" standard test shot, in order to improve the repeatability of that test, and elevate it to the level of a true test standard. This may not yet be the final form of this test, but any changes from this point on should be pretty minor.
This page is a brief explanation of the new "Outdoor Portrait" test setup, the lighting used, the rationale behind its configuration, and in part, the rationale behind my standardized tests in general. This won't by any means be a full treatment of the subject -- In response to reader emails over the last year or so, I'm working on a more complete discussion of my overall test philosophy and procedures in "background mode," in and around the perpetual backlog of tests and reviews waiting to be completed. Given that I've just changed the "Outdoor" test shots though, it seemed important to describe what was going on, what this particular subject is intended to test, how I've set up the new arrangement, and in the process also describe how to use this shot to reveal common digicam limitations. (It really does make a great "torture" test for cameras, able to reveal subtle noise problems, as well as more obvious ones of tone, contrast, and color.)
Why the change?
I'm sure that the first question many of you will have is "Why change?"
- The old test was long established on the site, and provided a basis for comparing
the capabilities of cameras stretching back over the last six years now. Why
take the drastic step of changing a standardized test setup?
The answer is twofold, but both issues come down to issues of practicality. This particular test has been a thorn in my side for years now, because there's only about an hour or so in the early afternoon each day when the sun is at the right angle for the test to be shot. Here in Atlanta, it turns out that the magical hour happens to be just at the time of day when afternoon thunderheads balloon up out of nowhere, making it a dicey affair to grab the necessary shots while the sun is shining through gaps in the clouds. This problem has been particularly severe over the last year. - Atlanta has come out of a multi-year drought, with the result that we've had solid weeks at a time with no sunshine at the right time of day for this shot to be taken.
Even when the sun was shining, its position in the sky varied quite a bit with the change of seasons, making exact comparisons difficult. Even more than the angle of the sun though, the color balance and contrast levels of the lighting varied all over the place, depending on atmospheric conditions. In the summertime, high humidity and haze typically made for more diffuse light, with a warmer color temperature. - Under these conditions, the sky ends up being almost white, rather than blue, and the proportion of light coming from the sky itself is much greater than when the air is clear and dry. Conversely, in the fall and winter time, crisp, dry air meant no haze providing unwanted "fill" illumination, and the intense blue sky shifted the color balance toward the blue end of the spectrum.
| More about our testing |
| New Indoor/Outdoor Portrait Tests (5/23/2008) |
| New Daylight Simulator (5/23/2008) |
| "Sunlit" Portrait - What to look for (5/23/2008) |
| Indoor Portrait - What to look for (5/23/2008) |
| Still Life and Multi Targets (Updated 11/29/2010) |
| HMI Studio Lighting (5/22/2006) |
| Performance Timing (5/22/2006) |
| New Review Format (5/22/2006) |
Bottom line, this "standard" wasn't a very true standard at all, despite my best efforts to control for sun angle and appropriate shooting conditions. It was still quite useful for evaluating how well cameras handled scenes with a wide range of light-to-dark (dynamic range), but it wasn't something that lent itself well to the sort of direct A/B comparisons that are one of the real strengths of this site. This is a tradeoff that's necessary any time you want to use a "real" subject for a test shot, vs a completely artificial one, constructed in the studio or lab. I've been willing to accept the compromises in this particular test in the past, wanting to have more "natural" subjects in my test suite. Now that I'm routinely shooting and posting "galleries" of random photos for most of the cameras I test though, the need to keep the standardized test subjects "natural" is less of a factor than it once was.
The straw that broke this particular camel's back was that we moved about 15 minutes away from our old house at the end of 2003, making it even more difficult to play hide-and-seek with the sun and clouds. (Even though the folks who bought our house have been very accommodating about our continued use of it as a test subject.) This was finally the impetus I needed to come up with an in-studio version of this test.
What is "Daylight?"
Looking to create a suitable substitute in the studio led me to look a bit more
closely at what constitutes "Daylight." My experience with the old
setup showed me that there's really no such thing as "Standard Daylight,"
the ISO 7589 specification notwithstanding. Color temperature and contrast ratio
obviously vary all over the place, depending on the season, latitude, and local
atmospheric conditions. So what's a reasonable average?
As noted, there's actually an ISO standard specification (7589) that sets out what "standard" daylight is, right down to a specific spectral curve. This standard was settled upon after much debate, by measuring the actual characteristics of sunlight at a particular location and time that was determined to be a reasonably representative middle ground relative to the range of variation across the face of the globe. Here's an example of what typical daylight color spectra look like. (From Testing linear models on spectral daylight measurements by Javier Hernandez-Andres, Javier Romero, Antonio Garc&Mac245;´a-Beltran, and Juan L. Nieves):
Boiled down to a single number, that spectrum comes out to a correlated color temperature of 5500K. While I'm sure that the ISO committee had very good reasons for coming up with the 5500K standard for daylight, it's worth noting that the widely accepted standard for daylight color temperature in the graphic arts industry is 5000K, a noticeably warmer shade of "white." In the interest of providing the greatest compatibility with my own prior tests though, the real question was, what was the color temperature of the light that I'd been shooting under myself all these years?
Taking my trusty Minolta color meter in hand, I ventured out to measure the color temperature of the local Atlanta-area sunlight over a period of several months. I found that most of the time, the color of the local sunlight during the 1-3 PM time window in which I shot this test was in the range of 4700-4900K, rather than the more blue-tinted 5500K of the ISO spec. (I should note that I don't know the absolute accuracy of my color meter, so it's possible that the light I was measuring was in fact closer to 5500K. - But what matters here isn't the absolute value, but rather that I match what I'd been seeing in previous test shots before adopting the new standard.)
So... As measured by my color meter, 4700-4900K is the color that I aimed for in setting up the lighting for my new test. - But there's more involved than just the sun itself, there's the sky, and reflection from surrounding objects as well. Here's what I did to mimic all three:
The Light Source(s)
While it's neither practical nor necessary to precisely match the lighting
of any given natural scene (overall color balance and repeatability are the
key criteria for a test standard), I did want the lighting for this test to
have some of the second-order characteristics of natural daylight. Thus, I've
arranged the lighting to include three different components: 1) Direct "sunlight",
2) Diffuse "sky light", and 3) Secondary "fill" light.
"Sunlight"
The "sunlight" for this test comes from an array of 24 Solux 4700K
incandescent lights in a 4x6 array, running at about 15 volts apiece, yielding
a color temperature of about 4850K, measured at the subject. Natural sunlight
is pretty collimated, which means that the light rays fall more or less parallel
to each other, as from a distant point source. Given the limits of my current
studio (ceiling height of 9 - 10 feet, and a practical limit for light mounting
of around 8 feet), I couldn't take the most obvious approach of simply hanging
a single, very bright light 20 or 30 feet above the subject to mimic the sun.
The solution I came up with was to construct a rectangular array of lights at
a height of about 8 feet, with a baffle (an aperture plate, really) positioned
in front of them to limit side-spill from the individual bulbs. This produced
reasonably well-collimated light, with a very high contrast ratio. (Limiting
side-spill also helped avoid the warmer-hued light that the Solux bulbs emit
off-axis.)
"Skylight"
For the diffuse "Sky" illumination, I used another array of 16 Solux
bulbs, running at ~16 volts, and gelled to produce a color temperature of about
7000K. This bounces off the face of the baffle surrounding the main light array,
mimicking a blue sky. This will produce a slight bluish component in shadows,
similar to that found in outdoor shots on clear days. The "sky" light
is about four stops down from the main "Sun" light, roughly matching
typical conditions here in Atlanta in the late spring and early fall. (A reasonable
compromise between summer and winter values.)
"Fill light"
The lights described so far produce about 3 KW of very high-contrast light,
with a color balance that pretty closely approximates that of late Spring and
early Fall sunlight here in Georgia. Shooting in the studio though, there wasn't
nearly as much "fill" illumination as there would be outside, where
the sunlight reflects off surrounding buildings, trees, grass, people, etc.
As a result, the shadows were a little darker than they'd be in a natural environment,
even under conditions of maximum contrast. (As seen in wintertime shots, when
the sky is a very dark blue, and there's no atmospheric haze to act as a diffuser.)
To open the shadows a bit, I set up two 1,000 watt PhotoFlex "StarLights"
in 2x3 foot softboxes, at about 45 degree angles from the subject, gelled to
match the roughly 3900K color temperature that I found typical of the fill light
bouncing off the surrounding trees, grass, houses, etc of our original location.
This "fill" light is about four and a half stops down from the main
"sun" light, approximating the typical main/fill ratios I found outdoors.
The net of all this is that, while it's by no means an exact duplication of the sunlight I've shot under in the past, the color temperatures and sun/sky/fill intensity ratios are a pretty good approximation of natural daylight, at least here in Georgia.
What's not the same, and how much does it matter?
Of course, there's no question that the light from the above setup isn't
identical to the light outdoors, at my prior shooting location. (But then, as
noted above, that light was also subject to variation with season and weather.)
So what's different about it, and what impact (if any) do those differences
have on the test shots themselves?
Overall Brightness
Despite the almost 5 KW of lighting, the light from my "Solar Simulator"
is still a good three and a half to four stops dimmer than typical sunlight.
Fortunately, this really doesn't matter, for the purposes of the test. The issue
isn't whether a camera has a fast enough shutter speed and/or small enough minimum
aperture to properly expose sunlit scenes (you can safely count on that capability,
as it'd be a pretty useless digicam that wasn't able to shoot outdoors in sunlight.).
Rather, the issue is how well a camera handles extreme contrast ratios between
highlights and shadows. - And for that, this test subject is ideal, and a fair
representation of what you'll encounter out in the "real world." The
verdict: No big deal.
The "Sun" is actually multiple light sources
This is probably the most visually distracting factor when looking at photos
shot with this setup. Because the "Sun" is actually a 4x6 array of
individual light bulbs, sharp edges in the subject cast multiple shadows. Depending
on the camera in use (and how it handles highlight detail), and the specific
position of the collar of Marti's shirt, you may see shadows that look like
they have "echoes" to them, with multiple fringes of decreasing shadow
depth as you move away from the edge that's casting them. While this is a little
distracting from an aesthetic standpoint though, it really doesn't have any
bearing on the primary purpose of the test. The verdict: No big deal.
No ultraviolet light in the "sunlight."
In most of their applications for general scene illumination, one of the advantages
of the Solux bulbs used here is that, while they have a spectrum that's pretty
close to typical sunlight, there's no ultraviolet component in their spectrum.
(This is generally an advantage, as UV will cause objects to fade over time.)
While digicams aren't generally sensitive to UV (as indeed they should not be),
there is some possibility that objects being photographed may fluoresce slightly,
effectively changing their visible characteristics as a function of how much
UV is present. It remains to be seen whether this is an issue for the particular
subject content of my "Outdoor Portrait" test. It won't likely affect
the rendering of Marti's hair or skin tones, but it could possibly affect the
colors of the flowers. (I've long wondered whether this could account for some
of the difficulty many digicams seem to have with the blue flowers in this shot,
as some organic dyes are known to be particularly sensitive to UV content in
photographic lighting.) Only time will tell, as I shoot this test with many
different digicams. Since I didn't control for UV content in the previous outdoor
shots though, the ultimate result will simply be that, once again, results from
the new test setup should be more consistent from one shooting session to the
next. The verdict: This light source may not show some color problems that
might occur under daylight, but it will be absolutely consistent from test to
test.
Different scene content = different white-balance response
This is visible in the example images I included at the top of this page. The
spectrum of light reflecting from Marti and (more significantly) the wall of
the house in the old shot tended to push cameras' auto white balance systems
toward a more blue-hued white balance than does the current setup. So images
of the new test may generally be a little warmer-toned than those from the old
one. Here again though, the main issue is how consistent the test is from shot
to shot, and the new setup is near-perfect in that regard. The verdict: Scene
content is different from the original, but has no impact on the validity of
the test. (Better repeatability with the new setup.)
Different main-light angle
The limited ceiling height in my current studio and practical limitations in
the size of the light array meant that I was pretty much restricted to having
the main light be above and slightly in front of Marti. This is more directly
overhead when compared to the typical position of the sun in the old shot, which
of course varied depending on the season of the year. (Closer to overhead in
the summertime, angled quite a bit in the winter, when the sun was lower in
the sky.) This is the one area in which the lighting for the current setup is
most visibly different from that of the original test. Here again though, while
this difference makes it more difficult to directly compare results from the
old setup with those from the new one, it really has no impact on the value
of the new test. While the shadow on the left side of Marti's face (the viewer's
left that is, her right) was a good place to look to see how cameras handled
detail in deep shadows, the new setup produces fairly deep shadows with lots
of detail in her eyes, and in the shadow areas beneath the bouquet. Looking
at the results from the first dozen or so cameras I've shot the new test with,
it looks like the results are all I could have hoped for, as there's a good
range of shadow density present. Having a range of shadow density will let us
evaluate the performance of cameras with a wide range of capabilities. The
verdict: The main-light angle is different from the original, but has no impact
on the validity of the test. (Better repeatability with the new setup.)
"Sky" illumination doesn't cover as wide an angle as it does outdoors
This is another case of a difference between the new and old test setups that,
while it makes it difficult to compare results from the two tests, has essentially
no effect on the value of the new test itself. With the old test, in the fall
and winter of the year, the "sky" light had a moderate effect on the
color balance of the image, particularly in the shadows. This is because the
sun angle is lower in the fall and winter (reducing its relative intensity,
as well as causing it to cast larger shadows), and because the lower humidity
means less atmospheric haze, resulting in skies that are much more intensely
blue than in the summertime. Given that shadow color isn't really a key component
of this test, I probably could have left out the "sky" illumination
entirely, simply relying on the fill illumination from the softboxes to open
the shadows and produce the desired contrast ratio. Aiming for the greatest
verisimilitude possible (or at least, practical) though, I went ahead and constructed
the light setup to provide at least some "sky" lighting, even though
it didn't cover as broad an area as in nature. The verdict: Different, but
no impact on the validity of the test. (Better repeatability with the new setup.)
Fill illumination comes primarily from two discrete sources
Finally, yet another difference that affects the similarity between the old
and new tests, but has no impact on the value of the new version. In the original
test scene, the "fill" illumination came from sunlight reflecting
from objects all around the subject (albeit not uniformly, as there were dark
and light areas in the surrounding buildings and landscaping). In the new setup,
while some light reflects from a white wall behind the camera, the majority
of the fill illumination comes from two large softboxes located about 4 feet
in front of the subject, at roughly 45 degree angles relative to the centerline
between the subject and the camera's lens. As a result, the fill lighting in
the new version is somewhat more concentrated. One possible effect of insufficiently
diffuse fill lighting could be that the fill lights themselves might cast secondary
shadows on the subject. Fortunately, this doesn't appear to remotely be the
case with the current setup, thanks to the relative size of the softboxes, their
relative proximity to the subject, and the fact that, taken separately, the
light they cast on the subject is a full five stops down from the level of the
main light. The verdict: Different, but no impact on the validity of the
test. (Better repeatability with the new setup.)
What does this test show? What should I look for?
This is the section that I'm most hoping to come back to update, but I wanted
to include at least some notes here in the meantime. This subject is actually
very revealing of a whole range of digicam failings and limitations,
so it's well worth studying.
Dynamic range - highlights and shadow detail.
This was really the primary aim of this test in the first place. One of
the first things all the how-to books tell novice photographers is to avoid
harsh direct sunlight of the sort this test is set up to simulate. This is because
the range of light to dark tones is well beyond what typical cameras (film or
digital) can reproduce, let alone that the harsh lighting is ugly and unflattering
to your subjects. Since most consumers judge a photo's exposure level based
on the midtones, I generally choose the exposure for this shot that puts Marti's
skin tones in more or less the right range. With most cameras (even those having
a "low contrast" option), this means that the strong highlights on
her shirt are going to be blown out, and that the deepest shadows beneath the
flowers will be pretty badly plugged. You can tell a lot about a camera's practical
dynamic range by looking at how good a job it does of holding onto detail at
both extremes of the tonal range. A lot of consumer digicams get their
bright, punchy color by jacking up the contrast. This makes bright, pretty-looking
(if not terribly accurate) images when you're shooting under relatively even
lighting, but loses horrific amounts of detail when faced with a contrasty scene.
How-to advice to the contrary, lots of amateurs shoot under harsh sunlight,
so this is an important test. Of course pros want cameras that hold onto highlight
detail, because it gives them more to work with in Photoshop, so this test is
equally important for them as well.
Color - skin tones, colors in flowers
It turns out that Caucasian skin tones are tough for cameras to get right.
For one thing, our eye is very accustomed to seeing these colors (at least,
here in North America), so we can tell immediately when something is amiss.
At the same time, even slight amounts of oversaturation can make your subjects'
complexions look very blotchy. Since skin tones are such an important part of
consumer (and professional) photography, this is an important subject. (Caucasian
skin also seems to be something of a worst-case subject as well. Black and Asian
skin is much more forgiving, in that it's much less prone to radical hue and
tonal shifts if the color hue or saturation is slightly awry.) The other colors
in this shot cover a useful range. Green foliage is a common element in landscape
photographs, so the green leaves in the bouquet will give a good idea of how
a camera will do with landscape shots. The blue flowers appear to be a very
difficult hue for digicams to get right, as many render them with varying degrees
of purple tints in them. The red flowers appear to be right on the edge of the
sRGB color gamut, so many cameras have a hard time holding "shape"
in them. All in all, there's a wealth of color-rendering data available here.
Fine detail - Marti's hair, fabric in flowers, "Peachfuzz" on
her cheeks.
This subject is also great for evaluating fine detail, in Marti's hair, the
texture of the fabric in the flowers and in the backdrop, right down to the
delicate "peachfuzz" on Marti's cheeks.
Image noise - look in shadows on Marti's face, underneath the bouquet
As camera makers have crammed ever-more pixels onto the same size chips, image
noise has steadily crept upward. This shot is great for seeing image noise in
shadows, particularly if you play with the brightness and contrast in your imaging
program. (If you have Adobe Photoshop or Photoshop Elements, the "Levels"
adjustment is great for playing with this. You can check detail in both highlights
and shadows, just by sliding the midtone (gamma) slider back and forth.)
Impact of anti-noise processing on fine detail - Look in Marti's hair
Faced with more and more noise coming from the image sensor itself, many manufacturers
have been cranking up their anti-noise processing in the camera to hide the
noise in the final images. The only problem is that this tactic tends to lose
a lot of image detail in the process. Most anti-noise algorithms look at how
much contrast there is in local areas of the image. Noise will be most noticeable
in areas where there's relatively little fine detail in the subject, and these
are generally also the areas of the image where people will be less aware of
the subject detail in the first place. Consequently, anti-noise algorithms tend
to set a local contrast threshold, and just flatten-out the detail whenever
the local contrast drops below that level. The extent to which this trick is
visible will depend a lot on the particular subject matter in a photo, but you'll
tend to see it a lot in grass, other foliage, and hair. Because hair
texture tends to be so adversely affected by anti-noise processing, this
is a great shot to use to look for its effects. Look in areas where the strands
of Marti's hair are close to the same color/shade, and see whether you can actually
see individual strands, or if they all blur together into an undifferentiated
mush. - The worst cameras produce hair and foliage images that look almost like
watercolors. Very revealing of a syndrome that's becoming all too common
in consumer digicams these days. (And is even prevalent in a few unfortunate
pro SLRs as well.)
Summary
Phew! This ended up being a lot more words than I intended, but there was
a lot of ground to cover, and hopefully the end result is that I've answered
the questions most people had. The bottom line is that the new "Solar Simulator"
lighting setup gives me an absolutely consistent test setup that closely mimics
the most important characteristics of natural daylight. While photos shot with
it can't be directly compared to those from the old arrangement, the test conditions
should be equally challenging for the cameras to deal with, and the lighting
will be absolutely invariant from day to day and month to month.
