Had some free time this weekend, so did a quick "What's in my (meeting) bag" photo..
I'm a Nikon shooter. A Nikon D800 shooter to be exact. I got it after 5-6 year relationship with Nikon D300 and I never looked back. Well, actually I did. For a few things really, from rubber quality, battery life span, vertical grip and so on, but mostly for file sizes.
D800 has 36Mpx and I love every single one of them, except when it comes to storage and backup. Then I hate about half of them! Since I shoot NEF (raw, for those unfamiliar), 14-bit in lossless format, my file sizes are around 41MB each. On an average portrait shoot I do around 500 shots, so that comes to about 20GB per shoot. With weddings the number is around 3500 shots per wedding at about 140GB. Then there are video files, tif exports, working .psd and .psb files and so on and so on, which all comes down to huge amount of new data every year. At the moment I'm at around 12TB.
All this files have to be available for eventual work, stored and backed up. I also never (or try to) delete anything, since you never know when you might need it. Since I have double backup, for every 1TB that I add to my working machine, I have to add 2x 1TB to backup machines.
My workflow is always the same: after the shoot images are developed, processed, retouched, printed/exported, then stored. Once images have been printed, published or sent to the client they are rarely used again. Sometimes a client would need some additional shots, but that happens once or twice a year and usually 1-2 months after a shoot. So 3 months after the shoot, images are *never* touched again. But they still have to be stored. As said before, I won't delete them, you never know when you might need something. We all know Murphys law..
The normal solution of storing is adding more drives and expanding the storage capacity. But that has considerate costs behind, not to mention actual physical space limitations. Another route that I was considering was cloud service, something like Google Drive or Amazon. They now have *unlimited* space for 60€/year which is peanuts when you think about it. I'm spending about 120€/TB/year. First major downsize for me is security. I already encrypt all my data, but the idea of having it somewhere out there, in a cloud.. not something I like. Sure they all claim it's secure, but we've read about numerous account hacks and employees being able to browse your data, etc, etc. No thanks, that's not an option for me. Probably the only provider I would trust is SpiderOak or Tresorit, but their prices are A LOT higher for my amount of data. Another problem is uploading everything to the cloud, which would take years (yes, years)! So no cloud for me.
The only other option with handling ever growing stored photo data was reducing the actual file size. There are two ways of doing this, the classic way would be JPG export. Then there's DNG.
I won't go into details about DNG, you can read that on wiki if interested, just to summarise: DNG is basically a container with jpg, same as NEF/CR2, written by Adobe. What DNG offers is lossless and lossy compression of your existing files. Since most of my backed up NEFs are there just for archive and "just in case" purposes, do they really need to be full size, HQ, full metadata, original NEFs?
There are a lot of articles comparing NEFs and DNGs, examining the advantages and disadvantages of DNG and it seem (as it always is I guess) there are two camps of people: one that totally recommends DNGs and the other that sticks to NEFs. I have to admit I was always in that second camp. I mean we have D800 and it's big & beautiful file for a reason, right? Why would I convert or even compress it? But with all the storage and workflow reasons mentioned above, I wanted to do more research.
My first DNG test
So, a while back I did my first quick DNG test. I had one D800 NEF file, 14 bit, lossless compression, comes to.. 38.4MB
Using Lightroom converter I exported this test file as DNG with Camera Raw 6.6, full size jpg preview, lossless settings and got.. 30.4MB
That's not bad, got round 20% size reduction with that.
Just for fun I though I'll try lossy compression.. so DNG, Camera Raw 6.6, full size jpg, lossy compression I got.. 11.1MB.. That's over 70% size reduction! Yeah, but with that amount of compression there must be artefacts and banding all over the place, clipped highlights and shadows, lots of details lost,.. So I imported both DNG files back into Lightroom to compare them with the original NEF. On surface, they all looked the same. No artefacts, no banding, full color.. what gives? Need more tests.
My second DNG test
Since lossy DNG was so much smaller it had to be clipping some serious shadows and highlights, as well as loosing image data, but how much?
For the test I used two images that I did during summer, when I was playing with HDR. It's the same shot, done through a window that shows inside of an abandoned work station in a factory. There's a window for the highlights and a large shadow area under the desk, so it's perfect for my test. I also had them 2 f-stops over and 2 f-stops underexposed, so I could do more extreme tests.
As before I exported the original NEF file into lossless and lossy DNG format, then imported them back into Lightroom. I then exported them with the same settings into tiffs and in Photoshop put them into one psd file. Reduced the size and saved and jpg 10. So all three files went through the exact same process and it's the same for all the tests/images here.
Here are the original files (2 f-stops underexposed in camera, ISO200, 1/320, f5), exported from LR, no changes done:
(picture description is always under the picture; clicking on picture will open them in 2000px size)
Now I've raised the exposure by 2 f-stops in Lightroom to get the shadows exposed:
Those are the shadows, but what about the highlights?
For those I used the second file, which was already overexposed by 2-stops in camera. So original file is ISO200, 1/80, f5:
Now to see what we can recover and what got clipped by file conversion. Same files as above, I dropped the exposure by - 2.5 f-stops in Lightroom:
Threshold at 240 bit level preview
(this is where more highlights come into play and it's the same on all three images.
I only posted threshold shots since you can't tell the difference by the naked eye)
Original NEF and lossless DNG are practical the same, can't spot the difference. I always knew that lossy will clip shadows and highlights, but by these tests it's not that bad. I actually did a lot more tests and exposures, from BW to cross process filters and it's always the same. Some data is lost yes, mostly in the shadow area, very little from the highlights. Quality of highlights remains the same, while shadows begin to show artefacts. Which makes sense, since, as we all know, camera sensors are linear devices and the least amount of data is in the shadows, which becomes visible once you start raising the exposure.
Next thing that I wanted to test was JPG vs DNG lossy. JPG already uses lossy compression, so file sizes should be similar, let's see about the image quality.
So from original 2 f-stops underexposes NEF file as above, I exported full size, 100% quality, AdobeRGB (same as shot) JPG file, then reimported it back into Lightroom. I raised the exposure by 2.5 f-stops, changed the white balance, raised saturation by +25 and reduced the highlights by -50 (just to make the file a bit different and see how some editing effects the image; I also did a test with unedited image, which showed same results). Exported all to tiff, 8-bit, added them to psd files and saved.
(There is a noticeable difference between JPG and NEF/DNG. It's hard to spot here, but while flipping on/off in photoshop layer it's obvious. JPG is more contrasty, sharper, edges and shadows stronger, highlight are brighter, dynamic range has changed)
Next thing to check are the highlights. I used the second file which was 2 f-stops overexposed in camera:
Now I lowered the exposure by 2.5 f-stops in Lightroom:
With JPG we have to go down about a stop of highlights while with NEF/DNG we are able to get about 2.5 stops of data before no more data was present. So JPG lost some dynamic range there. Here's a 100% crop of the recovered details:
(I've added another file from JPG with -1 f-stop exposure. This is where highlights histogram is starting to separate which means no more data is present and we're only turning whites into greys. We can see there's more details and data with NEF/DNG files.
I've also added JPG -2.5 f-stop, since I'm sure someone will ask why don't I just lower the exposure. You can, but no more data will show. Once it's clipped, it's clipped.)
Here's another 100% crop, from the window:
I did a bunch more tests with different files and found you can recover anywhere from 0.1 - 0.5 f-stops more data with NEF/DNG then with JPG, depending on an image. Which didn't seem a lot to be honest, I was expecting even bigger difference. But then I realised why. All these are Lightroom generated JPGs. Computer has a lot more power then camera, so algorithms must be better. And since I'm already writing a novel here, I did another test just for fun. I wanted to compare camera generated JPG to Lightroom generated JPG.
So, I took another photo that I overexposed by about 4 stops in camera. D800 created NEF and a full size, HQ, JPG. I then converted NEF file into DNG and into JPG in Lightroom. So we now have 4 files: original D800 NEF, original D800 JPG, Lightroom converted JPG and Lightroom converted DNG lossy. Here's how files compare, with nothing done to them:
But we're currently interested in highlights, so let's see how much we can bring back. I lowered the exposure in Lightroom until there was no more data to show. For NEF and DNG that was - 2.3 f-stops, for both JPGs -1 f-stop. This is how that looks:
Original NEF, original JPG from camera, Lightroom converted JPG and DNG lossy
(First thing we see, again, is how bad the camera generated JPG is. Another thing that's different is the dynamic range and shadows compared to NEF/DNG. But we can *fix* that with a curve, which I'll post after the threshold shot)
As we can see, camera generated JPG clips a lot more highlights data then all other files. Which is understandable. So for those shooting JPGs - don't Well, if you get the exposure right and don't plan on doing much highlight/shadow postprocess, then I guess it's ok. But I'd still recommend you shoot RAW and then convert to JPG on your computer, you'll get a lot better files with more headroom for eventual editing.
As for file sizes: original NEF is 38.8MB, camera generated JPG is 6.74MB, Lightroom jpg is 9MB and DNG lossy is 9.7MB.
While doing image comparisons between NEF, DNG lossy and LR generated JPG I also discovered that JPG always sharpens the image, even when sharpening is turned off. As an end product image does look better (sharper), but when we have retouching in mind this is not good. Sharpening always comes last in the retouching workflow, not first. With sharpening we get stronger lines and pixels, which are harder to retouch.
Here's a sample:
100% crop, cross processed image, raised shadows, highlights dropped,... I was making a mess to spot the difference
NEF and DNG are practically the same, while JPG has different tones, contrast, dynamic range and especially sharpness.
It still bothered me how badly DNG lossy handled shadows in the first test photo, so I wanted to try another image. The first test above was with a really dark, 2 f-stop underexposed image, so blacks were really crushed and getting clipped, but what about a normal exposure photo?
So I took the image of the castle and tried it out. Here's how the image looks on it's own:
This is how the image compares to the first test image above:
So I took this image, converted it to DNG lossy and JPG as before and raised the exposure by 2 f-stops. Here's how it turned out:
As before, NEF and DNG have virtually the same shadows and highlights in normal exposure photo, but JPG is not that much different. But we're interested in the shadow quality right now, so let's take a look of the crop:
Well, while switching layers in Photoshop I can say NEF and DNG are practically the same, while JPG is a bit more contrasty and sharper. But overall the shadow quality is practically the same with all. So with normally exposed image DNG can handle shadows as well as JPG.
Conclusion (finally, right?).. To DNG or not to DNG...? Or to go the JPG way?
First let me say, I'm doing this tests for my own personal needs. I'm not comparing NEF vs JPG shooting or which settings to use, how to shoot, how to export, etc, etc.. I'm simply looking which option is best for me, to archive my "B roll" photos and save some disk space. All this are simple and quick tests, I haven't even tested channel differences, dynamic range and luminosity,..
So.. DNG lossy or JPG.. Well, it depends. Depends mostly on the type of images and their exposure. I always try to shoot ETTR (exposure to the right) and always with the correct exposure, so I *never* adjust the image by more then a stop in Lightroom and even that it's mostly to lower the exposure. With normal exposed image JPG and DNG are very close, with very underexposed images DNG does recover a tiny bit more data, but overall quality of the shadows is a lot better with JPG. DNG also recovers a bit more highlights, while the quality is the same. JPG does change tones, contrast, dynamic range and sharpness, which is my main concern right now.
There's one other thing that's somewhere in the back of my mind and that's compatibility. Adobe is pushing really hard to get DNG to become a recognised standard, but it's not really going their way. They have promised they'll always support the format, but we all know that can change. We're also limiting ourselves to Adobe products (for now at least), if we use DNG. While JPG is universal. We can use any bitmap software out there with jpg.
For my working and "active" files I'll definitely stick to NEFs, but once the project is over and done, converting them to something smaller still looks a really nice idea. If there was no sharpness issue with the JPG, I think I would go that way. But since it's there, I need to do some more tests..