Digital Camera Home > Interview With Jerry Harris

BETTER PERFORMANCE & A MORE INTUITIVE INTERFACE

Adobe's Jerry Harris Rotates
the Photoshop Canvas

By MIKE PASINI
Editor
The Imaging Resource Digital Photography Newsletter

By MIKE PASINI

In a far-ranging interview last week, Adobe Senior Engineer Jerry Harris discussed everything from the Photoshop's new GPU processing to developing an interface that doesn't require you to learn (or remember) quite as much. Photoshop Product Manager Bryan O'Neil Hughes also joined the discussion.

After codeveloping PixelPaint and working at Apple, Harris joined Adobe in 1996 in time to work on Photoshop 5.0 and develop the new paint engine in version 7.0.

A LITTLE HISTORY | Back to Contents

But Harris was just a kid out of college when he attended SIGGRAPH, the Association for Computing Machinery's Special Interest Group on Graphics and Interactive Techniques annual meeting. There he saw the Pixar Image Computer "just flying over these huge satellite images in real time, rotating them, zooming them." That vision never dimmed.

Not until Creative Suite 3 was well underway did Harris find a way to do the same thing in Photoshop. John Brandt had given him a little demo app that showed off a few things you could do by writing code that ran not on the main CPU but on a special chip dedicated to graphics called a Graphics Processing Unit.

The team was so excited by the demo that it tried to get some GPU code into Photoshop CS3. But that turned out to be very difficult. "Photoshop had a lot of very bright minds spending a lot of time trying to make the program draw as little as possible and not draw very often," Harris explained. "And trying to change that to be more of a video game mentality was a huge, huge chore."

But they made progress and discovered "we could do some other cool things." Those things may eventually extend as far as a more intuitive user interface, but the immediate proof of concept was much simpler: rotate the canvas in Photoshop CS4.

And indeed rotating the canvas to a more convenient angle to view or paint is no longer a matter of picking an angle from a menu or guessing at an arbitrary value. Instead, thanks to GPU processing, you just press the R key, grab the image and twirl.

Wait, as someone once said, there's more. The same benefits apply to scaling, too. As Hughes put it, "Just seeing things as they should be is really nice. I know it's always driven me crazy to know that since day one I had to look at my image at 25 percent or 50 percent or 100 percent. The GPU means it looks right no matter how I look at it whether it's video or still."

And there's still more. Harris observed, "MacPaint original had this pixel grid, if you remember, and it was lost over time because essentially we couldn't draw the grid fast enough. So it was nice to be able to bring that back into Photoshop CS4. That's really helpful when you're doing detail work.

"CS4 is really nice. It feels more fluid. You know we've been careful not to say it speeds up a lot of tasks because it's mostly the top layer of things it's speeding up now. You can rotate an image really fast but it's just a screen representation of the image."

Adobe has posted a TechNote describing all of the GPU and OpenGL features in Photoshop CS4 including smooth display at all zoom levels, an Animated Zoom tool, animated transitions with One Stop Zoom, Hand toss image, Birdseye view, Rotate Canvas, smooth display of non-square pixel images, a pixel grid, GPU color matching, Draw Brush tip editing feedback and 3D acceleration. In Bridge, they include Preview panel, full-screen preview, slide show and Review mode.

SPEEDING UP PHOTOSHOP | Back to Contents

As those examples show, tapping into the GPU isn't simply about speeding up Photoshop.

You don't need a GPU just to speed up Photoshop, Harris made clear. The first thing to do, he said, is "put as much memory in that machine as you can afford." And use a 64-bit operating system if you're on Windows, like the 64-bit editions of Vista or Windows 7. Avoid Windows XP, he said.

An operating system like Vista or Mac OS X has "modern window servers that use the GPU themselves to do all this eye candy. And they virtualize the hardware so if you have two things trying to use the GPU at the same time, it makes them play nice."

But you can't expect any more performance from that approach, Harris said. Modern CPUs have hit a technological brick wall. "You can't air cool a chip at over three gigahertz," Harris explained. It isn't about gigahertz any more.

"CPUs are slowly running out of gas," Harris said. "They can't speed up the clock frequency. And they haven't for a while. So they get a newfangled CPU out and they say it's faster but usually things aren't running faster because they're adding cores or something. It's hard to speed up a lot of CPUs so why try?"

And it's the same problem with coding for multicore processors. "It's hard to actually achieve that theoretical goal when running in parallel," he said. "So it's sort of like working with a bunch of mules. You might work with two of them but four or five, forget it. They don't want to behave. Where a GPU is more like a stream of fish you see in the ocean or a flock of birds. They just seem to do better with more of them. More naturally suited."

To get more performance, you have to move code to the GPU.

As Intel Chairman of the Board Craig Barrett said in a recent Gizmodo interview, "You see, everybody's kind of looking at the same thing, which is, 'How do I mix and match a CPU- and a GPU-type core, or six of these and two of those, and how do you have the software solution to go hand-in-hand?'"

And, Harris added, AMD's Fusion "is the same idea. What happens if you can get rid of that bus, if you had unfettered access to the pixel data? It could speed things up dramatically."

The nice thing about moving to GPU code is that you don't have to buy a new system to run the next upgrade. Instead of laying out $1,500 to $3,000 to speed up CS4, you can just buy a $200 video card.

The market for GPUs has been divided between AutoCad users at the high end and gamers at the low end. At NVision 08 last year in San Jose, the first GPU-oriented convention featured a huge darkened ballroom dedicated to gamers who set up their own boxes to compete with each other. At the same time, in a small room to the side, Nvidia CEO Jen-Hsung talked about the data processing capabilities of the GPU that, harnessed together, were already providing supercomputing power to academics in Asia building their own machines.

Nvidia isn't the only GPU in town, though. ATI has been a major player and recently AMD with its Fusion project and Intel with Larabee promise even more options.

Intel, in fact, has recycled its Pentium design as a GPU in its Larabee project, which bundles several Pentium-design chips into a GPU. "They're going to a simpler design but they're going to stick more of them on there and use the same amount of watts."

PICKING A GPU | Back to Contents

So how do you evaluate a GPU?

If you're in that situation now, Harris suggested getting something that's been shipping in the last year. "If you're looking at somewhere between $100 and $400 for a consumer card, I'd try to err on the side of $200 to $300."

"Honestly, getting just any $200 card will do," Harris said. "If you want to future-proof yourself, try to get a card with at least 512MB of RAM. And get one that's more at the high-end of the consumer line. Say the 9000 series Nvidia or higher or a 3000 series or higher on the ATI side.

"These newer cards have floating point that's more dependable. So in the future when we do more on that GPU it will be a more solid foundation than these older ones. Because honestly with the older cards, they just wanted to make your video game look good. They weren't worried about accuracy. It was frame rate the ruled the video game market. That's all that mattered."

In fact early graphics cards (ones that do not support Shader Model 3.0) could not understand simple if-then conditional statements or manage running a loop to do something repeatedly until some condition was met. And some of these cards provided very little room for programs to run anyway.

But Hughes pointed out that any recent machine running a modern operating system probably has a powerful video card. "I run into very few people that bought machines in the last couple years that aren't able to enjoy these benefits and features. It's really more people like myself who have a dual G5 at home that's four or five years old. I went on eBay, found the right card and bought that. But with newer machines I don't see it as being an issue too much."

Harris also recommended looking for PCI Express 2.0 cards. "It's a technical term but a pretty important one. Some of the newer cards do PCI Express 2.0. And all it means is it's twice as fast as the previous one. That's important. It's something I would look for if I were buying a card or a new computer today, but it's usually not a problem with a new computer. I think they're all PCI Express 2.0 now."

PCI Express 2.0 addresses what's an important issue for GPU programming: getting data back from the GPU. That isn't important in game programming where no data is read back from the card.

But we're not yet at the stage where it's worth it to run two GPUs simultaneously (which is possible, as those academic supercomputers demonstrate). "It doesn't really help Photoshop right now, to be honest with you. And it probably won't help us in the short term future. But at some point it will help us.

"I would go for a better single GPU card," Harris added. "I would not invest in an SLI, where you can stick more than one card in the box. Crossfire and SLI. That's not particularly useful for Photoshop. In fact it's not useful at all. So I would go with a card that has more memory like Nvidia 260 and 280 or the ATI 4000 series cards with almost a gig of memory on them. You're pretty future-proof with that."

DATA PROCESSING | Back to Contents

While a new GPU will give you smoother zooms and more accurate image display, it can also help run those calculation intensive operations like filtering. Both 3D painting and the Pixel Bender plug-in use the GPU extensively.

Hughes noted, "We're really careful about how we talk about speeding things up. I will tell you that with Pixel Bender there's a checkbox to enable GPU support. And if you play around with those filters with and without that check it's really a dramatic difference in speed."

In fact, the Pixel Bender team built a toolkit that was used to put things like Rotate Canvas in Photoshop CS4.

BACK TO THE INTERFACE | Back to Contents

If you thought Harris would be content popping that pixel grid back on the image, think different.

"You know how to move something," Harris observed about our basic motor skills. "When you used your first Mac, you had this idea of direct manipulation." Drag and drop. "Unfortunately, that seems to be frozen in time. That's still what direct manipulation is today for the most part. That bitmapped old user experience.

"We can do a lot better today. The iPhone has pointed the way. The Wii has pointed a different way. And if we can tap into those innate things you don't have to think about, then you can learn five or seven things in a short amount of time. You dive into them right away."

One such example is Birdseye view.

Hughes explained what it is, "In Birdseye -- and when we think about the Gigapixel Project -- we've actually slightly downrezed this enormous image where you can zoom all the way into a window at Adobe and see a person in their office." It's Google Earth without the satellites.

Supersized images are not the exception they used to be. With 24-megapixel full-frame sensors and panorama image stitching, they're becoming common.

Harris pointed out, "More people are stitching images together so you're looking at a monster. And once you have all that data, content has to be king. So we've got to find a way to not have so much screen real estate devoted to palettes, but make a more fluid environment where your eyes are focused on what you're interested in and not, if you will, noise or sign pollution."

Birdseye does that, once you learn the trick of holding the H key down and clicking your mouse to zoom out, then reposition the navigator box to zoom right back in. The GPU makes that kind of an interface feasible for navigating a large image at full resolution.

THE ECOSYTEM & TWO THUMBS | Back to Contents

Oddly enough, as we continued our discussion, the hardware and software strains started to merge into one ecosystem.

"If you look at this fundamental problem we have where the free ride is over, we will no longer speed up unless we work hand-in-hand with Intel, Nvidia and AMD," Harris said. "It's going to be hard work but it will pay off. If we don't, Photoshop's not going to speed up and that's one thing everyone can agree on: people will pay for performance. People appreciate that. And you can enable UIs, you just can't do it if you don't have that performance."

Fortunately, he noted, "This is a competitive and, so far, healthy environment. It's just nice to be able to work with all these partners who want to grow the market."

But there's another aspect to this, he continued, that the current user interface is inhibiting. "It's real sad, you know. It's great that tablets that used to cost $3,000 and weigh about 50 lbs. 22 years ago are now down to $200, but we're doing the same thing. You still have the same amount of expressiveness today that you did 20 years ago. Just in a cheaper, smaller form factor."

While many of us may be comfortable at the keyboard and have learned, over the years, how to find a feature buried in menus and dialogs, the younger generation isn't following in our footsteps. Two thumbs are all they ever lay on a keyboard.

"You don't have to explain a Wii do you? They get it. Even an iPhone. Put it in somebody's hands and they're so popular. I mean my kids are telling me about every two days, 'Dad, five more of my friends downloaded the Facebook app for the iPhone.' It's spreading out there like crazy.

"They've grown up with a Nintendo paddle in their hand. That was one issue we had with Birdseye. If people could add another finger to their hand to be more expressive in Photoshop, they probably would pay for the operation.

"We're out of keyboard shortcuts. So we need to find a way for people to be more expressive. So how this ties back into the ecosystem is we need to find a way to enable some use of the gadgets. If at all possible to tap into the skills this young crowd has, the Nintendo paddle skills or the Gameboy skills. They're using that opposable thumb of their non-dominant hand a lot more than we were at their age. They're good at it. You just watch them. They can sit there, they're not even looking at their phone and they can text message. It's unbelievable. We just need a way to tap into those motor skills."

The problem is that while Adobe was building more and more features into Photoshop, the user interface was becoming less and less intuitive. It's called cognitive load theory, Harris told us.

And solutions to the problem like Adobe Configurator require you to stay in what turns out to be a different mode than we inhabit when we are creating.

Harris told us this issue was graphically illustrated when he spent some time with a few artist friends.

One art professor told him about sitting accomplished fourth-year art students in front of a digital painting program and "basically watching them lower their skill level down to that of a high school arts student all of a sudden."

From another of the artists, he learned how "just the fact of having a color palette with a numeric feedback, geeked out these artists. They just started getting all geeked out about trying to use 255 variations of every color. And it just sort of took them out of their moment of serendipity. They couldn't create art very effectively."

And when one art professor demonstrated a few techniques to Harris, he couldn't describe what he was doing while he was doing it. "We started joking about it. It turns out this serendipity, if you watch these people get in their groove in the real world, they don't think a lot about how they use their tools. Everything's a small motor skill or something used at the subconscious level.

"What we have to figure out in the future is what about computer programs introduces cognitive speed bumps or speed bumps in serendipity, if you will."

FINISHING UP | Back to Contents

Hughes observed, "I think there's a good reminder here that there's hardware performance and there's interface performance and the ideal solution is somewhere between where one compliments the other. I know Jerry has definitely shown that in the features he's implemented."

And Jerry replied, "When I was working on PixelPaint, I was the jack of all trades, finisher of none. There were all sorts of features, they were all buggy. It's funny that the ones that intrigued me the most were the ones that no one cared about. And I'm sort of past that these days. I don't want to waste a minute working on something that's not relevant.

"That's a hard goal, finishing a feature. I'll tell you that. You think you've got that thing captured and it's evasive."

It takes a little something more to push beyond the current paradigm, even when you've got a lot of skin in the game, as Harris put it. He cited John Worthington (who "hasn't stopped thinking about Liquify") as a case in point. "The Photoshop team is made up of people like that. That's what makes it strong. Everyone's got a little ..."

Hughes cut in, "a lot of passion."

"Yeah," Harris confirmed, "a lot of passion."

 

---

Jerry Harris is a Senior Computer Scientist on the Adobe Photoshop team. Since joining Adobe in 1996, Harris has developed and tested new drawing technologies with GPUs, OpenGL, layers effects and brush work. Prior to joining Adobe, he was a co-developer of PixelPaint at Apple. For inspiration, Harris pays close attention to innovative technological developments and enjoys reading research articles on digital painting and cognitive science.

Bryan O'Neil Hughes is a Product Manager on the Adobe Photoshop Team and a Product Evangelist for the Lightroom Team. Since 1999, O'Neil Hughes has helped to test, drive and demonstrate Adobe's professional imaging solutions. He speaks worldwide on behalf of Adobe and can often be found leading workshops, seminars and user groups. Beyond Adobe, O'Neil Hughes is a published editor, author and photographer.