Adobe’s new AI-powered Enhance Details feature promises up to 30% more resolution for raw images

 

 

Adobe has announced an update to the entire Adobe photography ecosystem, including Adobe Camera Raw, Lightroom Classic CC and Lightroom CC. The primary focus of the update is the new Sensei-powered Enhance Details feature, which utilizes a brand-new approach to demosaicing raw images. Demosaicing is an important part of the raw conversion process because it works at the pixel level to convert the data captured by the image sensor into something resembling the photo we expect to see.

The new Enhance Details algorithm, powered by the machine learning and computational photography involved in Adobe Sensei, is said to promise an increase in resolution for both Bayer and X-Trans images of up to 30 percent. By applying Enhance Details to your images, you can improve the rendering of fine details, improve the reproduction of fine colors and resolve some issues that users have previously pointed out regarding Fujifilm X-Trans cameras.

Let's dig further into the feature. To help illustrate how the new feature works, it is worth briefly discussing how digital cameras actually capture information and produce an image. Digital camera sensors have a primary photosensor array, which measures the intensity of light for a given pixel, but doesn't capture color data. To produce images with colors, then, there's a second part, the color filter array, which sits on top of the photosensor array. The color filter allows the camera to record the color of any given pixel on the image sensor. A traditional sensor will record one of the three color values (red, green or blue) for any given pixel. This means that every pixel in a raw image actually contains only one of the three colors, and lacks the information for the other two colors. It is here where Bayer and X-Trans sensors differ a bit, as they feature different patterns of color pixels.

 

Bayer sensor versus X-Trans sensor color filter arrays. Image credit: Adobe. 

How then does software take the light intensity and color information from a raw image and recreate an image which resembles what we expected? This is achieved through the process of demosaicing. There are many ways to demosaic a photo and the design choices made at the engineering level will have potentially dramatic effects on the resolution of your images, color fidelity, color accuracy and more. In a basic form, demosaicing averages color values of neighboring pixels. This works fine in images with large areas of constant or similar color, for example, a blue sky. However, when an image is very complicated, a more precise and careful demosaicing process is required for high-quality results.

You could simply advanced demosaicing processes for all images and have better final results, right? The issue here is that advanced demosaicing is not beneficial in every part of every image and it is very computationally expensive. There's traditionally a tradeoff between fidelity and speed. Some fidelity issues which Adobe runs into repeatedly include small-scale details, false colors and zippering. The first is a big issue as we continue to work with cameras featuring higher-resolution sensors. When an image includes small details which near the resolving limit of a camera, it can be turned into ugly moiré artifacts, for example. Demosaicing algorithms can make mistakes, of course, such as when the algorithm interpolates across, rather than along, sharp edges, causing unnatural shifts in color.

 

Comparison showing standard demosaicing (left) and Enhanced Details (right). Of course, a small file on the web is not the ideal way to compare the results. 

Over the years, Adobe has continued to work on its demosaicing algorithms, resulting in better results all the time. However, there are still some problem cases. This is where Adobe Sensei comes in, by leveraging artificial intelligence and machine learning to work with a trained neutral network. By feeding over a billion problematic raw images into the neural network, Adobe Sensei was able to train the new Enhance Details feature to better recognize issues and deal with them appropriately, resulting in images with better resolution and fewer artifacts. Adobe trained two models, one for Bayer sensors and one for X-Trans sensors.

 

Enhance Details versus standard demosaicing. Take particular note of the lights in the image and the colors on the two stacks to the left. You can also see differences in color fidelity on the signage. 

Enhance Details is not the perfect solution for every photographer nor every photograph. Because Enhance Details relies upon Apple's Core ML and Microsoft's Windows ML, it will only work on computers using current operating systems. Further, it's an intensive process and can take time. A faster GPU will result in faster results. It is not recommended for every image, but rather, should be used for images you intend to print or display at very large sizes, images which have artifacts issues present without Enhance Details and images featuring a lot of fine detail.

In addition to the Enhance Details feature, there is also an update for Lightroom CC which adds HDR, Panorama and HDR Panorama merge tools, the Targeted Adjustment Tool and histogram clipping indicators. There is also new camera and lens support, which you can read about here. The latest update for Adobe Camera Raw, Adobe Lightroom Classic CC and Adobe Lightroom CC is available now.