High Speed and 360: Dave E dishes 2016’s hottest camera tech
posted Friday, December 30, 2016 at 6:00 AM EST
Over the past year in the newsroom and over the water cooler, one thing kept coming up when we talked about camera technology: cameras were getting faster. More frames per second, larger buffers, faster response times. And if you talk to Imaging-Resource founder and editor-in-chief Dave Etchells, he's not shy about telling you about it. We thought that the end of the year would be an excellent time to sit down with Dave and have him explain how the cameras are doing the extraordinary things they are doing.
Andrew Alexander: Thanks for sitting down with us again, Dave. You've mentioned that high-speed camera technology has advanced significantly this year: can you elaborate on what you mean?
Dave Etchells: Sure, it's been a huge move for the industry. If you think about it, there are three parts of the high-speed process: the first step is the A to D (analog to digital) conversion: measuring the signal that's coming from each pixel, and converting it to a digital number. Traditionally, that's been a bottleneck. Then there's the matter of getting the data from the A to D converters over to the processor, and finally, there's a question of what the processor does with it and how they handle it.
With the A to D conversion process, it comes down to a question of how much you can parallelize the conversion. Years ago there was one wire coming off the chip, it was an analog signal, amplified and then sent to an A to D converter. And then we went to multiple channels, but they were still analog. I think it was Sony that had the first patent for a per-column A to D conversion process, and it was quite an advance. It was quite clever; what they did was, at the bottom of each column of pixels, there was a counter and a comparator. They'd feed in a ramp signal from the edge of the array to all the comparators at once, that gradually increased from minimum to maximum signal. Each time the ramp signal increased a small step, all the counters incremented. Then, when any given comparator said, okay, the value that's being broadcast right now is now equal to the value I'm seeing in my pixel, it would stop its counter, freezing that value. At the end of a ramp cycle, you could read out all the counters, and have all the brightness values for an entire row of pixels. So now you could have a couple of thousand A to D converters in parallel, depending on the size of the sensor array.
This brought a huge increase in speed. There are obviously huge technical challenges that had to be overcome with the approach, to achieve good digitization accuracy, but they overcame them. That was just one part of the problem, though; once you had those three or four or five thousand pixel values for each row of the sensor, you still have to somehow get all that data off the chip, so that was another challenge.
There are a variety of things to be done about that, but then again, Sony had an advantage with their stacked sensor technology, because it made it possible to have many thousands of connections with this sensor, to some sort of a processor-slash-buffer memory.
Others have obviously closed that gap or even pulled ahead though, as we've witnessed from Olympus with this year's E-M1 II, which can capture full-resolution data at 60 frames per second.
One of the big stories in 2016 is how significantly manufacturers have managed to increase both sensor and processor speeds, and the very real advantages that's bringing for end users.
AA: Are we getting to a point in 2016 where reward-versus-cost is allowing manufacturers to pack in enough processing power to capitalize on these innovations?
DE: Yes, absolutely. But it's not just the level of processing and the cost of it, but also the power budget you have to dedicate to it. The level of integration, the size, the power... in the past you could have had it all if you were willing to put up with a camera that was twice the size and had a fan-cooled heat sink on the back of it, with a belt-mounted battery pack to power it all. What's happened is that the technology has advanced to a point where you can rally at the problem just through a lot of brute force. And the neat thing is that brute force is being thrown at it in a way that's ending up being beneficial to photographers. Not just, 'oh, we've got all this processing power, let's make some cute, useless feature with it.' What we're seeing is better image quality in video, very high frame rates for still photography and much more sophisticated autofocus.
As I just mentioned, if you leave out autofocus, the Olympus E-M1 Mark II can capture 60 full-frame resolution images per second, and save them as both JPEG and RAW. You do give up some things on that: at that point it's an electronic shutter, which has some impact on image quality, and you can also have more rolling shutter compared to a fast focal-plane shutter as a result. Also you only get a single-shot focus at the beginning of the stream of images. But it's still kind of mind-boggling that it's possible to capture very high-quality "still" images at 60 fps.
Autofocus is the other really big consequence of such radically increased processing power. Autofocus has always been a tremendously difficult technology. Especially coming from the SLR side, it's really remarkable just how good autofocus was with the separate AF sensor. Manufacturers didn't have a lot of points to deal with to be able to discriminate what was the target and what wasn't. It's really amazing that Canon and Nikon's autofocus algorithms were sophisticated enough to track subjects quite well, even with what would today be considered to be very few points.
There are some advantages in having a separate AF sensor: you've got less data to deal with, and you can really structure that sensor so that it's easy to get the data off it quickly, as opposed to trying to extract focus pixels from a huge image sensor array. And you can also make the pixels on the AF sensor much bigger, so you get better low-light performance. But the speed with which you can get the data off the main sensor chip these days has really made it possible to not only to equal but to exceed the density of AF points that could be dealt with per focus cycle by separate AF sensors: on-chip phase detection has really matured a lot from the early days, and there were some significant advances in 2016 in particular. The advances in technology and the raw processing power have contributed a lot to conventional SLR designs as well: The AF system that's in the Nikon D5 and D500 is really an incredible leap forward when compared to anything that went before it in the SLR world. There are a huge number of focus points, with a great number of cross points.
When I was interviewing Nikon execs and engineers at last year's CP+, and I asked them what was the limitation - why hadn't we seen denser AF arrays and more cross-points earlier. It really came down to processing limitations. It turns it out that the processing for phase-detect autofocus requires a lot of data crunching and manipulation. You're basically sliding arrays of image data past each other, to see where the point of maximum correlation is. That takes a lot of horsepower. And if you make your regular focus points into cross-points, suddenly you have twice as much data and twice as much processing to deal with. So the image-processing technology has really led to a huge step up in autofocus across the board this year, both in conventional SLRs and mirrorless designs.
AA: So what's the next iteration of this technology?
DE: That's a really good question: we now have lots of AF points to look at, but they're still just passing depth information. The processor can kind of make out objects, and how far they are from the camera, but it's still kind of limited in terms of object recognition. I think that's where we'll see the next round of improvements. We've been moving in this direction, including image information along with the depth information, to better discriminate and track the subject. That's been going on for a while in the SLR world; both Nikon and Canon have exposure sensors that record RGB points across the frame, and that's allowed them to start doing some object identification based on color and position within the frame.
Nikon and I'm sure Canon began using that exposure metering information to also track the objects, and to assist with face recognition. That then translated to being able to track objects, correlating between the depth map information from the autofocus points, and what they were seeing in the RGB exposure sensor, so they could use all that information to identify and track objects across the frame. I think that's the direction where we are going to see a lot more intelligence coming in, in that I think that cameras with enough processing power could get very good at identifying objects by synthesizing image and depth-map information.
We saw something in this direction from Samsung before they exited the market - the NX1 had this really cool mode for catching a baseball coming off of the bat. The camera was set up to look for a small, light-colored object moving across the frame within a range of characteristic speeds. And you would say, 'okay, when the object gets to this point, snap the shutter' (you could move a vertical bar to define the point you wanted). So that was an impressive but somewhat simplistic application of that technology. I think we're going to see much more actual object recognition and tracking in the relatively future.
On the technology curve, we've seemingly, suddenly, hit a level where there's a whole new level of capability. And it's significant for photographers. You come down to basic capabilities: for raw speed, I don't think we need two hundred frames per second in continuous shooting mode. The user interface becomes critical to being able to scroll through all those images and identifying the ones you want. (I think Olympus has done a pretty good job of that, by the way.) A direction Panasonic has taken is to actually just take the best stills from 4K video.
Olympus is processing images through a still-imaging chain as opposed to a video chain, which means a significant difference in image quality: the video compression definitely exacts a price. You can pack a heck of a lot more data on a card or a computer with 4K video, but it won't be the same level of detail or tonal gradation that you'd have by capturing JPEG/RAW.
AA: It comes down to the user needs: a stills photographer versus a videographer.
DE: It's a question of what quality level you need. We're at 4K photo now, I have no doubt that at some point we'll be at 8K photo, meaning still frames extracted from 8K video. It has the distinct advantage that you're using a heck of a lot less storage space. We do have six- and eight-terabyte consumer hard drives now, but it's laborious dealing with 300 RAW images in a five-second burst, compared to a much smaller movie file.
AA: The technology has changed the focus of the problem.
DE: With any kind of technology, you go from 'gosh I wonder if we can do this' to 'yeah we can just barely do this' to 'yeah we can do this pretty well' to 'yeah the technology is so good it doesn't matter any more'. And we're rapidly approaching that point with storage, at least relative to still image photography. Yeah, we're going to come back with fifty, a hundred gigabytes of files, but I've got a ten terabyte hard drive, so it's no problem.
AA: And if there wasn't a problem before with cataloguing and finding images, there may definitely be one now.
DE: Absolutely. It's one thing to have cheap storage space for tens of thousands of image files, but another thing to have to wade through all of them! Some of the approach to that to is find ways to go about logically grouping them. If you've got 300 images from a five-second burst, the camera groups them together in playback. - Although I don't think that software like Lightroom supports cues from the camera to do this quite yet. It probably wouldn't be much of a step to examine the EXIF data and realize the shots were all taken at the same time. There can be intelligence in the cataloguing software, computers can do a bit of content analysis (the same subject is present) to group the photos together.
AA: Any program that's going to be able to speed up the process of cataloguing: analyzing, keywording; is going to be a leader.
DE: I'm thinking about the potential for cloud computing to be able to offer some very sophisticated AI-type functions as part of the photo cataloging. You might have images stored locally but the system would upload lower-resolution copies to the cloud for processing and cataloging. I do think that this is still a bit further out from where we are now, though.
This year was really a perfect storm of technology for high speed, and that spilled over into autofocus quite a bit, just by enabling so many more AF points. These were the huge items in terms of technical advances for the industry, but we didn't have any individual company that had some uniquely different, breakthrough technology, who we could give our Technology of the Year award to. Rather, it seemed like all of the sudden, the technology was available to a number of companies to crank up the speed. The other thing that I think is going to be a big thing over time, but is requiring a change in thinking of photographers, is the area of three-sixty.
AA: An excellent segué. It's literally the next thing in my notes.
DE: Haha, great minds think alike <grins> We've seen things go from the 'yeah we can just barely do this', where they had sixteen cameras bolted together in an aluminum framework the size of a beach ball, down to Ricoh with the Theta and now the Theta-S, being able to give you 360-degree imaging in a very small package. And now Nikon's come out with the KeyMission 360 this year. Nikon's marketing has really been focusing on the adventure category, and I guess it makes sense in that that's the low-hanging fruit - a lot of people have GoPros, and 360 capture is a really cool thing for that segment. And it's a capability that's not really available any other way. I think Ricoh has a waterproof housing for the Theta-S whereas the KeyMission is a very compact, inherently waterproof package. So I guess it makes sense that they'd be focusing on action photography as their initial market foray - but I think the huge potential is for memory-keeping. I remember seeing the KeyMission 360 and thinking 'here, finally, is a reason for everyone to buy another camera'. It's a capability that doesn't exist on your phone, doesn't exist in your camera, and the thing is it's fundamentally different in terms of the experience and what you bring back. The extent of that difference, though, is why it's going to be a long adoption cycle.
Photography is really all about framing and capturing an image. Whereas 360 is completely different. You can certainly construct a playback sequence that guides the user's eye, but the whole point of 360 is that you can also look around elsewhere. It's really a different sort of thinking than anybody who's a still photographer is going to have at the outset.
AA: It's exactly the opposite.
DE: When it comes to recording a memory, I think 'definitely take your still photos', but why on earth wouldn't you want to take this 360 video camera with you? I'm thinking back to family vacations (one in particular) and remember the cameras I had along at the time - a digital SLR, for sure - but also I had a little digicam along. And I recorded a bunch of little video clips on that. And it's amazing the extent to which - not just viewing them, but having viewed them - it's so much more immersive. I have pictures that I took at a little tapas bar in the Basque country in Spain, where the locals were singing along to a jukebox. The photos were nice, but the video of the experience of that is just so enormously richer than the still pictures were. The relatively few video clips I brought back from that vacation are what I remember, what I've gone back and looked at from that trip as opposed to the hundreds of stills I shot.
AA: Of course, the still pictures you can print and hang on your wall, whereas a 360 video clip of that experience, there's a problem with playback.
DE: Exactly. That's very true, but on the other hand, if you look at how people are consuming images now, very few pictures are hung the wall. We're looking at them on computers, we're sharing them on Facebook. But the use case of recording the family vacation, the ability to experience it, to see the people and look around at what's going on, what was going on while you're there, I think that's a hugely compelling proposition.
AA: It speaks to a different method of consuming information: the directed form, where a director is guiding your attention, whereas a 360 degree experience eschews that attention so you have complete freedom. How do you alert the user that something interesting is happening elsewhere in the frame?
DE: Three-sixty has another use case for story tellers: some of the content that Nikon's put together for marketing KeyMission has been a really great example of this. The application for the story tellers and the filmmakers - it's capturing everything that's going on from a stable platform, but then you have this 360 degree bubble you can navigate through, and guide people through it. It's really a kind of post-capture videography. All of this is very different than anything that's come before. The problem for Nikon and everyone else in the space is that any time you're relying on people to think differently in order to be able to sell your product, it's going to be a difficult challenge.
AA: Will Three-Sixty suffer the same fate as the 3-D television?
DE: Yeah, TV manufacturers thought that 3-D would be the compelling reason for people to get a new TV, but it turned out not at all to be the case. From that standpoint the adventure market makes sense as the initial point of entry [for KeyMission]. I think that people who are already recording their kayak trip down the rapids, or their snowboard trip down the mountain, really would enjoy being able to look around them, and would watch the footage multiple times, taking different views. With Nikon coming into the market, I think 360 is becoming more mainstream; Samsung has a product in that space as well. It's kind of peeked its head over the horizon right now, and over the long term I think it's going to be a really big thing. It's a question of whether it will be a big thing quickly enough to support the products the manufacturers are trying to deploy into the space.
AA: As they say: "interesting times." Certainly an interesting conversation, thanks for your time!
DE: Exciting times for photography; my pleasure!