MaticKos Photography
6Apr/160

Sensitive

Liquid shot

Nivea-v2-small

MySpaceFacebookTwitterBlogger PostWordPressDeliciousNetlogTypePad PostTumblrRedditDiggLiveJournalStumbleUponBox.netLinkedInPushaS/B
20Mar/160

My bag

Had some free time this weekend, so did a quick "What's in my (meeting) bag" photo..

MyBag-noise

18Feb/160

Gaia

Gaia // natura portrait //

Gaia-mix1-small

23Jan/160

Petra

Petra // editorial //

Perica-kosmatko-mix1-small

Perica-kosmatko-mix2-small

14Jan/160

Saving disk space – NEF to DNG vs JPG

I'm a Nikon shooter. A Nikon D800 shooter to be exact. I got it after 5-6 year relationship with Nikon D300 and I never looked back. Well, actually I did. For a few things really, from rubber quality, battery life span, vertical grip and so on, but mostly for file sizes.
D800 has 36Mpx and I love every single one of them, except when it comes to storage and backup. Then I hate about half of them! Since I shoot NEF (raw, for those unfamiliar), 14-bit in lossless format, my file sizes are around 41MB each. On an average portrait shoot I do around 500 shots, so that comes to about 20GB per shoot. With weddings the number is around 3500 shots per wedding at about 140GB. Then there are video files, tif exports, working .psd and .psb files and so on and so on, which all comes down to huge amount of new data every year. At the moment I'm at around 12TB.

All this files have to be available for eventual work, stored and backed up. I also never (or try to) delete anything, since you never know when you might need it. Since I have double backup, for every 1TB that I add to my working machine, I have to add 2x 1TB to backup machines.
My workflow is always the same: after the shoot images are developed, processed, retouched, printed/exported, then stored. Once images have been printed, published or sent to the client they are rarely used again. Sometimes a client would need some additional shots, but that happens once or twice a year and usually 1-2 months after a shoot. So 3 months after the shoot, images are *never* touched again. But they still have to be stored. As said before, I won't delete them, you never know when you might need something. We all know Murphys law..

The normal solution of storing is adding more drives and expanding the storage capacity. But that has considerate costs behind, not to mention actual physical space limitations. Another route that I was considering was cloud service, something like Google Drive or Amazon. They now have *unlimited* space for 60€/year which is peanuts when you think about it. I'm spending about 120€/TB/year. First major downsize for me is security. I already encrypt all my data, but the idea of having it somewhere out there, in a cloud.. not something I like. Sure they all claim it's secure, but we've read about numerous account hacks and employees being able to browse your data, etc, etc. No thanks, that's not an option for me. Probably the only provider I would trust is SpiderOak or Tresorit, but their prices are A LOT higher for my amount of data. Another problem is uploading everything to the cloud, which would take years (yes, years)! So no cloud for me.

The only other option with handling ever growing stored photo data was reducing the actual file size. There are two ways of doing this, the classic way would be JPG export. Then there's DNG.
I won't go into details about DNG, you can read that on wiki if interested, just to summarise: DNG is basically a container with jpg, same as NEF/CR2, written by Adobe. What DNG offers is lossless and lossy compression of your existing files. Since most of my backed up NEFs are there just for archive and "just in case" purposes, do they really need to be full size, HQ, full metadata, original NEFs?
There are a lot of articles comparing NEFs and DNGs, examining the advantages and disadvantages of DNG and it seem (as it always is I guess) there are two camps of people: one that totally recommends DNGs and the other that sticks to NEFs. I have to admit I was always in that second camp. I mean we have D800 and it's big & beautiful file for a reason, right? Why would I convert or even compress it? But with all the storage and workflow reasons mentioned above, I wanted to do more research.

My first DNG test

So, a while back I did my first quick DNG test. I had one D800 NEF file, 14 bit, lossless compression, comes to.. 38.4MB
Using Lightroom converter I exported this test file as DNG with Camera Raw 6.6, full size jpg preview, lossless settings and got.. 30.4MB
That's not bad, got round 20% size reduction with that.
Just for fun I though I'll try lossy compression.. so DNG, Camera Raw 6.6, full size jpg, lossy compression I got.. 11.1MB.. That's over 70% size reduction! Yeah, but with that amount of compression there must be artefacts and banding all over the place, clipped highlights and shadows, lots of details lost,.. So I imported both DNG files back into Lightroom to compare them with the original NEF. On surface, they all looked the same. No artefacts, no banding, full color.. what gives? Need more tests.

My second DNG test

Since lossy DNG was so much smaller it had to be clipping some serious shadows and highlights, as well as loosing image data, but how much?
For the test I used two images that I did during summer, when I was playing with HDR. It's the same shot, done through a window that shows inside of an abandoned work station in a factory. There's a window for the highlights and a large shadow area under the desk, so it's perfect for my test. I also had them 2 f-stops over and 2 f-stops underexposed, so I could do more extreme tests.
As before I exported the original NEF file into lossless and lossy DNG format, then imported them back into Lightroom. I then exported them with the same settings into tiffs and in Photoshop put them into one psd file. Reduced the size and saved and jpg 10. So all three files went through the exact same process and it's the same for all the tests/images here.

Here are the original files (2 f-stops underexposed in camera, ISO200, 1/320, f5), exported from LR, no changes done:
(picture description is always under the picture; clicking on picture will open them in 2000px size)

dng-original-fullFull image, no changes

dng-original-thresholdThreshold at 1 bit level
(as expected we can see that lossy has a bit less info then NEF and lossless file, which are almost the same)

dng-original-cropCrop of the above image
(with the naked eye, I can't see any difference)

dng-crop-thresholdThreshold at 1 bit level of the crop

Now I've raised the exposure by 2 f-stops in Lightroom to get the shadows exposed:

dng-plus2-full+2 f-stops, full image

dng-plus2-full-thresholdThreshold at 1 bit level
(same as before, there is some data loss in the lossy format, but not much)

dng-plus2-croplCrop view
(One next to another, we start to see the difference. Lossy has more artefacts and color noise, even a slight color change. Remember, this is +2 f-stops from normal camera exposure)

dng-plus2-100crop100% crop view
(There is a noticeable difference)

Those are the shadows, but what about the highlights?
For those I used the second file, which was already overexposed by 2-stops in camera. So original file is ISO200, 1/80, f5:

dng-highlights-fullFull image, no changes

dng-highlights-full-thresholdThreshold at 255 bit level
(everything looks the same)

Now to see what we can recover and what got clipped by file conversion. Same files as above, I dropped the exposure by - 2.5 f-stops in Lightroom:

dng-highlights-minus2-fullFull image
(everything looks the same)

dng-highlights-minus2-threshold255Threshold at 255 bit level
(I see a couple of tiny dots missing from lossy, but nothing major. All three files were able to recover the same amount of data)

dng-highlights-minus2-threshold250Threshold at 250 bit level preview
(Same as above, can hardly tell the difference)

dng-highlights-minus2-threshold240Threshold at 240 bit level preview
(this is where more highlights come into play and it's the same on all three images.
I only posted threshold shots since you can't tell the difference by the naked eye)

Original NEF and lossless DNG are practical the same, can't spot the difference. I always knew that lossy will clip shadows and highlights, but by these tests it's not that bad. I actually did a lot more tests and exposures, from BW to cross process filters and it's always the same. Some data is lost yes, mostly in the shadow area, very little from the highlights. Quality of highlights remains the same, while shadows begin to show artefacts. Which makes sense, since, as we all know, camera sensors are linear devices and the least amount of data is in the shadows, which becomes visible once you start raising the exposure.

Next thing that I wanted to test was JPG vs DNG lossy. JPG already uses lossy compression, so file sizes should be similar, let's see about the image quality.
So from original 2 f-stops underexposes NEF file as above, I exported full size, 100% quality, AdobeRGB (same as shot) JPG file, then reimported it back into Lightroom. I raised the exposure by 2.5 f-stops, changed the white balance, raised saturation by +25 and reduced the highlights by -50 (just to make the file a bit different and see how some editing effects the image; I also did a test with unedited image, which showed same results). Exported all to tiff, 8-bit, added them to psd files and saved.

dng-jpg-full(full image)
(There is a noticeable difference between JPG and NEF/DNG. It's hard to spot here, but while flipping on/off in photoshop layer it's obvious. JPG is more contrasty, sharper, edges and shadows stronger, highlight are brighter, dynamic range has changed)

dng-jpg-full-thresholdThreshold at 1 bit level
(we can see jpg clips a bit more shadows then NEF or DNG, which are almost the same)

dng-jpg-full-threshold-250Threshold at 250 bit level preview
(just wanted to see the highlights.. not much difference here)

dng-jpg-crop100% crop
(We can clearly see that JPG is a lot better then DNG. There are a lot more artefacts and color noise on DNG then on NEF/JPG)

dng-jpg-crop-thresholdThreshold at 1 bit level preview

Next thing to check are the highlights. I used the second file which was 2 f-stops overexposed in camera:

dnh-jpg-highlights-fullfull image, no changes done
(can't see any difference)

dnh-jpg-highlights-full-threshold255Threshold at 255 bit level preview
(almost no difference)

Now I lowered the exposure by 2.5 f-stops in Lightroom:

dnh-jpg-highlights-2f-full(full image)
(Since the dynamic range for jpg has changed, we get no data once we go below -1 f-stop)

dnh-jpg-highlights-2f-full-255Threshold at 255 bit level
(NEF and DNG are the same)

dnh-jpg-highlights-2f-full-175Threshold at 175 bit level
(this is where JPG data is becoming visible)

With JPG we have to go down about a stop of highlights while with NEF/DNG we are able to get about 2.5 stops of data before no more data was present. So JPG lost some dynamic range there. Here's a 100% crop of the recovered details:

dng-jpg-2-crop(I've added another file from JPG with -1 f-stop exposure. This is where highlights histogram is starting to separate which means no more data is present and we're only turning whites into greys. We can see there's more details and data with NEF/DNG files.
I've also added JPG -2.5 f-stop, since I'm sure someone will ask why don't I just lower the exposure. You can, but no more data will show. Once it's clipped, it's clipped.)

Here's another 100% crop, from the window:

dng-jpg-2-crop-2

dng-jpg-2-crop-2thresThreshold at 255 bit level
(Everything that's white is clipped, no more data can be recovered. It's clear there's more recovered with NEF/DNG then JPG, but not much - see below)

dng-jpg-2-crop-2thres-252Threshold at 251 bit level
(Just 4 bits lower, we start to loose data in NEF/DNG as well)

I did a bunch more tests with different files and found you can recover anywhere from 0.1 - 0.5 f-stops more data with NEF/DNG then with JPG, depending on an image. Which didn't seem a lot to be honest, I was expecting even bigger difference. But then I realised why. All these are Lightroom generated JPGs. Computer has a lot more power then camera, so algorithms must be better. And since I'm already writing a novel here, I did another test just for fun. I wanted to compare camera generated JPG to Lightroom generated JPG.

So, I took another photo that I overexposed by about 4 stops in camera. D800 created NEF and a full size, HQ, JPG. I then converted NEF file into DNG and into JPG in Lightroom. So we now have 4 files: original D800 NEF, original D800 JPG, Lightroom converted JPG and Lightroom converted DNG lossy. Here's how files compare, with nothing done to them:

JPGs-all4(I think it's obvious to all, camera generated JPG is by far the worse. Lots of details are lost, there's blue tint, CA,..)

But we're currently interested in highlights, so let's see how much we can bring back. I lowered the exposure in Lightroom until there was no more data to show. For NEF and DNG that was - 2.3 f-stops, for both JPGs -1 f-stop. This is how that looks:

JPGs-all4-normalOriginal NEF, original JPG from camera, Lightroom converted JPG and DNG lossy
(First thing we see, again, is how bad the camera generated JPG is. Another thing that's different is the dynamic range and shadows compared to NEF/DNG. But we can *fix* that with a curve, which I'll post after the threshold shot)

JPGs-all4-normal-threshThreshold at 255 bit level
(NEF and DNG are practically the same, close third was Lightroom generated JPG, while original camera JPG lost a lot more data)

JPGs-all4-normal-curveDarken curve on both JPGs, to match the shadows

JPG-all4-cropCrop view of the castle

JPG-all4-crop-thresh255Threshold at 255 bit level

JPG-all4-crop-thresh240Threshold at 240 bit level

JPG-all4-crop-thresh220Threshold at 220 bit level

JPG-all4-crop-thresh200Threshold at 200 bit level

JPG-all4-crop-thresh80Threshold at 50 bit level preview - shadows are almost the same (with the darken curve on JPGs, remember)

As we can see, camera generated JPG clips a lot more highlights data then all other files. Which is understandable. So for those shooting JPGs - don't :) Well, if you get the exposure right and don't plan on doing much highlight/shadow postprocess, then I guess it's ok. But I'd still recommend you shoot RAW and then convert to JPG on your computer, you'll get a lot better files with more headroom for eventual editing.
As for file sizes: original NEF is 38.8MB, camera generated JPG is 6.74MB, Lightroom jpg is 9MB and DNG lossy is 9.7MB.

While doing image comparisons between NEF, DNG lossy and LR generated JPG I also discovered that JPG always sharpens the image, even when sharpening is turned off. As an end product image does look better (sharper), but when we have retouching in mind this is not good. Sharpening always comes last in the retouching workflow, not first. With sharpening we get stronger lines and pixels, which are harder to retouch.
Here's a sample:

dnj-jpg-sharpenss100% crop, cross processed image, raised shadows, highlights dropped,... I was making a mess to spot the difference :)
NEF and DNG are practically the same, while JPG has different tones, contrast, dynamic range and especially sharpness.

It still bothered me how badly DNG lossy handled shadows in the first test photo, so I wanted to try another image. The first test above was with a really dark, 2 f-stop underexposed image, so blacks were really crushed and getting clipped, but what about a normal exposure photo?
So I took the image of the castle and tried it out. Here's how the image looks on it's own:

DNG-gradoriginal exposure, no editing

This is how the image compares to the first test image above:

DNG-grad-1bitboth images with threshold at 1 bit

DNG-grad-5bitthreshold at 5 bit
(this is where blacks come in on the castle image)

DNG-grad-10bitthreshold at 10 bit
(we can see that the first test image had a lot more shadows)

So I took this image, converted it to DNG lossy and JPG as before and raised the exposure by 2 f-stops. Here's how it turned out:

DNG2-all

DNG2-all-thresh30bit threshold at 30 bit level

DNG2-all-thresh230bitthreshold at 230 bit level

As before, NEF and DNG have virtually the same shadows and highlights in normal exposure photo, but JPG is not that much different. But we're interested in the shadow quality right now, so let's take a look of the crop:

DNG2-all-crop

Well, while switching layers in Photoshop I can say NEF and DNG are practically the same, while JPG is a bit more contrasty and sharper. But overall the shadow quality is practically the same with all. So with normally exposed image DNG can handle shadows as well as JPG.

Conclusion (finally, right?).. To DNG or not to DNG...? Or to go the JPG way?

First let me say, I'm doing this tests for my own personal needs. I'm not comparing NEF vs JPG shooting or which settings to use, how to shoot, how to export, etc, etc.. I'm simply looking which option is best for me, to archive my "B roll" photos and save some disk space. All this are simple and quick tests, I haven't even tested channel differences, dynamic range and luminosity,..

So.. DNG lossy or JPG.. Well, it depends. Depends mostly on the type of images and their exposure. I always try to shoot ETTR (exposure to the right) and always with the correct exposure, so I *never* adjust the image by more then a stop in Lightroom and even that it's mostly to lower the exposure. With normal exposed image JPG and DNG are very close, with very underexposed images DNG does recover a tiny bit more data, but overall quality of the shadows is a lot better with JPG. DNG also recovers a bit more highlights, while the quality is the same. JPG does change tones, contrast, dynamic range and sharpness, which is my main concern right now.

There's one other thing that's somewhere in the back of my mind and that's compatibility. Adobe is pushing really hard to get DNG to become a recognised standard, but it's not really going their way. They have promised they'll always support the format, but we all know that can change. We're also limiting ourselves to Adobe products (for now at least), if we use DNG. While JPG is universal. We can use any bitmap software out there with jpg.

For my working and "active" files I'll definitely stick to NEFs, but once the project is over and done, converting them to something smaller still looks a really nice idea. If there was no sharpness issue with the JPG, I think I would go that way. But since it's there, I need to do some more tests..

Z vstopom na strani pod domeno "matickos.com" se strinjate z uporabo poškotkov / By entering sites under "matickos.com" domain, you agree to the use of cookies. Vec info / More info

Vse spletne strani pod domeno "matickos.com" uporabljajo piškotke, ki omogocajo boljše in lazje delovanje strani. Nekateri piškotki so za delovanje strani nujni, drugi sluzijo izboljšanju uporabniške izkušnje in statistike obiska. Za nadzor nad piškotki lahko prilagodite svoje nastavitve brskalnika. Ce nadaljujete z deskanjem po spletnih straneh pod domeno matickos.com ali ce kliknete na gumb "Potrdi", se z uporabo poškotkov strinjate.

Uporabljeni piškotki na strani:

FACEBOOK // Uporabljen za spremljanje obiska in objav na socialno omrezje.

GOOGLE // Piškotek za spremljanje gibanja uporabniki po naši strani in kreiranje statistike.

UVC // Piškotek za izdelavo statistike obiska.

PHPSESSID // Piškotek za optimizacijo delovanja spletne strani.

The cookie settings on websites under the domain "matickos.com" are set to "allow cookies" to give you the best browsing experience possible. Some cookies are needed for normal functioning of the page, some are used to enhance user experience and provide statistics. You can adjust your cookies prefferences in your browser.. If you continue to browse websites under the domain "matickos.com" without changing your cookie settings or if you click the "Accept" button below, then you are consenting to this.

Cookies used on this site:

FACEBOOK // Used to track visitors. Features for sharing via Facebook. Does not set a cookie by itself, but if one is present it will read it.

GOOGLE // Used to track visitors. Collect information about how visitors use our site. We use the information to compile reports and to help us improve the site. The cookies collect information in an anonymous form, including the number of visitors to the site, where visitors have come to the site from and the pages they visited.

UVC // Used to collect information about how visitors use out site.

PHPSESSID // Required cookie to improve our site

Zapri / Close