Digital Darkroom Software Review: Darktable

Program: darktable (1.6.3; updated to 1.6.7 late in the review process)
Platforms Available Tested: Linux, Mac, Windows
Installation: normal download dmg & drag app to Applications folder (or wherever you like). Easy.

darktable is probably the closest to Lightroom that you’ll find in the open source space. With the exception of user-definable albums (they’re called Film Rolls in darktable, and they’re a bit strange to use), you can do everything in darktable that you can do in Lightroom, and more, most of it in a few different ways.

Thankfully, I’m not trying for a comprehensive review here… I just want to see if darktable can replace Lightroom for me.

To be honest, I’ve gotten a bit tired of this testing lately. It’s partly that Ramadan has come and I have much lower energy (even for sitting on the sofa and typing), but mostly that I’m tired of massaging the same images over and over again, and really tired of playing/fighting with computer software. I’m also very ready to get back to shooting and I have something of a backlog of images to process and share now. So forgive me if this is a bit briefer than you hoped, or if I don’t cover something important, or didn’t find the easy solution: I’m really just trying to get this over with.

(and especially forgive me for going on and on about the xmp file business…)

General Comments

I like darktable… I’ve reviewed it here before, and so I was somewhat familiar with it when I went in.

darktable is very powerful, and its unconventional methods of tweaking settings make some operations easy and fast. I’ll go into some of these later, In sha’Allah.

There is a comprehensive user manual available, but I find it harder to navigate than RawTherapee’s rockstar manual. I think it’s something about the colors, maybe.

The deployment of the mouse scroll wheel to change sliders and manipulate most any control is a pleasure to use, and when I hopped back into Lightroom to make some comparisons, I kept hovering over sliders and spinning the scroll wheel to no avail.

Adobe could learn something from this…

Importing images into the darktable database is easy, just point it to a folder and it does most of the rest. One thing that distinguishes darktable from its competition is its ability to pull the tags and colors and some settings from Lightroom.

To test this, I applied some random colors and ratings and tags to a couple of images from another project (that I may never finish or share) in Lightroom.
I then imported these into darktable. I expected the info would just be there, given the information in darktable’s user guide, which suggests that tags, color labels, ratings, and GPS data is included on import. However, when these pictures came into darktable, they came in without any of that included.

After looking around, bewildered, for a bit, I noticed the ‘load sidecar file’ in the History stack:

Clicking that button opens a file browser, where you can pick the xmp file associated with your picture, but this did nothing either…

Strange. I wondered if maybe it was due to something in the Mac build, or maybe the presence of Lightroom on the system, so I decided to try on Ubuntu. Sadly, I borked the VirtualBox install (actually, I didn’t, really, I just got fed up with trying to share folders—so very kludgy—and deleted it), so I booted up the little netbook on which I’ve been playing around some lately.

So I loaded some files onto a flash drive, booted up the netbook, moved files over, and went to start up darktable. Just as it finished loading the files into its memory, I heard a clicking and grinding noise from the netbook and the machine froze. One hard reset later, I heard the noise again and the computer booted into the pure BIOS system… no IDE master found, no USB drives connected.

Alhamdulillah and good times. (Off topic: I took the netbook apart—no easy task, that—and extracted the HD with plans to replace it… but a new HD is ~$60-100 and an SSD is about the same. I could replace the machine completely for close to that, so I’m not sure it’s worth it, really. Maybe, but maybe I’ll just try again with VirtualBox.)*

So reading Lightroom tags and settings doesn’t seem work on Mac, and after several frustrating hours spent reinstalling Ubuntu in Virtualbox and getting it all working again, I found find that ratings, colors, etc. and the develop settings are not read from Lightroom’s xmp files in Ubuntu (on VirtualBox) either. I wondered if this might be due to the old version of darktable in the Ubuntu Vivid repository (it gave me 1.4.3), so I added the darktable-release repository and updated to 1.6.7: no joy there either.

So that part of darktable appears to be broken. It’s likely that something was changed in Lightroom’s xmp files since this feature was introduced (back in 1.2, I think), but it’s still broken, and that’s half a day I burned trying to make this work and hunting for a solution.

Ends up, it was my fault all along. Somehow the ‘Automatically write changes into the XMP’ checkbox got unchecked in Lightroom’s Catalog settings, so all the changes were being stored in the Catalog instead of the XMP… maybe I had it set up this way for a reason, but I probably just didn’t know any better, since a quick google led to multiple suggestions to “TURN THIS ON NOW!!!!!”

Pro-Tip: if you don’t have this turned on, turn it on now. Lightroom will write everything you change to an XMP file going forward. And if you want to write all changes to XMP for all your old stuff, that’s possible: select the images that you want to move the changes for, then hit cmd-s (Mac) or ctrl-s (Windows) and it will start chugging through. I did this for all the RAW files from the C5050, LX-7, and D7000. It took awhile… maybe 2 or three minutes for 144 files from the C5050, maybe 20 for ~1300 from the LX7, and a loooong time for the 23K from the D7000, largely because of a stall at some point, just after I shut down for the night. This led to all sorts of problems the next day, and has me looking to pick up some replacements for my media drives. Good times.

Weeks later… my media drives were formatted to NTFS through Paragon in preparation for moving to Linux or Windows (months and months ago). Apparently, this is not a good idea long term. The main drive got corrupted during all the xmp-writing, and I had to wipe it, so I converted both drives to HFS+ (the native Mac file system) and everthing is zippier now. It still took 2-3 days to get all the settings written over, and all this corrupted Lightroom’s preferences, and I had to delete (and lose) all my preferences, but it works again, and more or less fine. SubhanAllah. What a pain.

After all that, I finally got to test out Darktable’s lightroom import stuff…

According to the darktable user manual, it reads the following during import:

• tags and hierarchical tags
• color labels
• ratings
• GPS information

Upon entering the darkroom view, and assuming an XMP sidecar is found, darktable claims to be able to these development settings development from Lightroom-generated XMP files (the corresponding darktable module is in parentheses):

• crop and rotate (crop and rotate)
• black level (exposure)
• exposure (exposure)
• vignette (vignette
• clarity (local contrast)
• tone curve (tone curve)
• HSL (color zones)
• split toning (split toning)
• grain (grain)
• spot removal (spot removal)

So now that (I think) I have the xmp writing thing fixed, let’s test this out. For this, I used Lightroom 5 (which I started this review with and have been using for a couple of years now) and Lightroom CC 2015 (that I’m previewing for another 23 days or so).

First up, the Lighroom 5 xmp.

As far as metadata goes, ratings, colors, tags, hierarchical tags, and the map work fine. In the develop module:

• – crop and rotate – rotation and straightening are correct; the crop is totally hosed, and throws an error on import: “Invalid ratio format. It should be ‘number:number.’ This is easily correctable, but I’d go nuts if I had to re-crop every image in my library.

• – black level – LR5 +93, dt -.0093
• ~ exposure – LR5 +1.45, dt +1.30
• ~ vignette – I think they’re close: lightroom has one slider for something that darktable uses two to define.
• ~ clarity – LR5 +80; dt coarsness +100, contrast +100, detail +.52 – this looks close to my eye, but I can’t really make sense of the numbers
• ~ tone curve – similar, but not quite the same (see screenshots below)
• + HSL – looks identical
• – split toning (split toning) dt completely missed the shadows; highlights show different numbers for the hue, but look close-ish.
• ~ grain (grain) – I can’t really interpret the numbers, but the result looks close
• – spot removal (spot removal) – same number of strokes, but completely different position

So for files from Lightroom 5, darktable sucessfully captures the HSL, does a middling job with exposure, vignette, clarity, the tone curve, and grain; and fails completely with black level, split toning, and spot removal.

Here’s the picture as exported from Lightroom 5:

And here it is as imported into darktable (after fixing the crop):

Close, but I wouldn’t want to subject my whole library to this treatment, nor would I count on this for anything important.

How does darktable fare with newer xmp files from Lightroom CC?

Well, darktable reads the star rating, color, tags, hierarchy tags, and gps perfectly. The develop settings are another story…

• –crop and rotate – darktable throws the same crop error upon loading; this time it’s both more and less off: it completely missed the straightening and read the image as rotated 180° instead of 90°, but looks to have captured the 4:3 ratio more or less correctly.

•  –black level – LRCC +84; dt -.0084
• ~exposure – LRCC +1.10; dt +.99
• ~vignette – looks similar
• ~clarity – LRCC +91, dt coarseness 100, contrast 100, detail .592
• –tone curve – not particularly close
• +HSL – looks identical
• –split toning – way off: dt missed the shadows entirely and the highlight color is way off, like other-side-of-the-color-wheel off
• ~grain – looks similar
• –spot removal – dt doesn’t read it at all, nada.

For files from Lightroom CC (and 6 presumably), darktable again successfully captures the  ratings, colors, tags, hierarchical tags, and GPS coordinates, and is only really successful with the HSL slider conversion. Exposure, vignette, clarity, and grain are close. Black level, tone curve, split toning, and spot removal are all way off.

So here’s the original edit from Lightroom CC:

And here’s darktable’s rendition of those edits:

This isn’t darktable’s fault at all: with every new Lightroom version comes some changes to the xmp that Adobe writes in to force users that share files to upgrade. I expect that borking darktable’s ability to read the xmp files is secondary.

If the darktable authors are interested, I’d be happy to help however I can to reverse-engineer the xmp files. I’m not much of a coder, but I have files and can contribute some time. InshaAllah I’ll get into the forums and offer up some help.

So if I decided to give up Adobe and move to open source today, I’d want to go into Lightroom first and export all of my processed files as 16bit tiffs and revert all images to their import state prior to introducing the files to darktable. Otherwise, I’ll not only lose the edits, but create thousands of hours of work for myself trying to recreate them.

Whew! That was quite the rabbit hole! Let me get on with the review now…

Test 1: Negative Conversion

darktable has three ways to convert a negative: Invert, base curve, and tone curve. They all produce different results.

Invert

With this tool, you select the color of the film base with the (sorta strange implementation of an) eyedropper tool, and darktable  sets that to black, thus effectively rendering the negative inverted.

The bonus to this method is that sliders work as normal. That’s great, for sure, but the film base, in my experience, is usually not true black. The frame numbers are pure white, or close to it, but the film base is usually something other than black, as you can see in any image I’ve ever posted that has the sprocket holes showing. The wall behind the sprocket holes goes black, but the wall is a soft cream color. The film base is orange, and it therefore never goes all the way to black like the wall behind the sprocket holes.

I spent two full evenings playing with this, trying (and failing) to get a usable result. I then found a really helpful video and I used the method presented there to get a bit closer, but still no joy.

The exposure is off, the color never got right, everything was way too muted and any attempt to lighten up the picture area resulted in some ghosting around the edges, banding, and/or blown highlights.

So I just gave up.

Base Curve

The base curve operates similarly to the tone curve that I’ve used in pretty much every other RAW convertor, with one important difference: it doesn’t really work for some reason, because it’s not designed for that purpose at all… [RTFM rears its head yet again.] It’s there to set the output parameters, more or less.

And that’s probably why it was so tricky: first too light, then too dark. As you can see, it ended up almost going monochrome. But all that only took maybe 20 minutes: still too long, but way better than two nights.

This worked much better than the Invert option, but still no good.

Tone Curve

This is probably the method to go for. In fact, there was already an ‘Invert Negatives’ preset here, probably that I made early on in the testing and forgot about, or that I made some months ago when I was thinking about converting negatives in some other software before processing in Lightroom.

Sure, there were way more steps to the history, but this option presented a starting point that made it easier to get the image closer to what I wanted, and took less time than other options.

I’ve spent so much time with this image that I was able to get to this result in maybe 4 minutes: still a long time, but much more acceptable than the other two options, and this is the method I would suggest for converting negatives, since it gives the most open results.

Test 2: ORF Conversion

Flawless. No issues. I could talk some about the process, but if you’ve processed one raw file, you’ve processed them all, more or less.

Test 3: NEF Conversion

Likewise flawless, so for this, I decided to set up a bit of a test between Lightroom and Darktable.

For the Darktable test, I used only the really groovy Equalizer panel; for Lightroom, I used my usual skills. Which one is which, which captures the title (“disco merry-go-round”) better, and which do you prefer? (I missed the aspect ratio a bit… Oh well.)

Final Thoughts

darktable is great. It’s powerful and as easy to use as you make it. The simplicity and variety of the tools in darktable makes processing images simultaneously easier and more complex: it’s easier, because you can often use one or two modules to get a picture to a usable state; it’s harder because of the number of modules and the learning curve.

darktable could use a competent library organizing scheme**—the film roll thing is kludgy as can be—and the controls could be a bit bigger (everything is tiny, by design apparently), but other than that, it could replace Lightroom in virtually any workflow, as long as you exported all of your Lightroom edits as tiff files and reset all pictures to their import state, otherwise you’re likely to lose much of your hard work and cause yourself huge amounts of additional work.

This is not darktable’s fault at all. Again, darktable is a fantastic program, and I could easily see moving to it full time. If I was just getting started in photography, and knew what I know now, I’d use a mixture of darktable and RawTherapee without question, and that’s what I would recommend to anyone just starting out that wanted powerful, nondestructive image editing. Lightroom and other Adobe products are

To be honest, I abandoned this review weeks ago, when the whole xmp fiasco was in full swing, and more or less resigned myself to stick with Lightroom and Macintosh. I’m even in the process of testing Lightroom CC and Lightroom Mobile (so far, the speed of CC/6 makes the $80 upgrade fee completely worth it; the dehaze thing in CC is neato, but probably not by itself worth $9.99/month; Lighroom Mobile is pretty impressive, if a bit limited compared to some editing tools, but its sync bit is much easier than the native mac photo sync, and it’s ability to share directly to VSCOcam and Snapseed make it surely tempting).

More on all that later, maybe.

Since I started this software testing thing, I’ve shot very little and instead spent most of my free, creative time testing applications, making notes, writing reviews and playing/fighting with technology. This has detracted from worshiping Allah to the best of my ability, from time with my wife in the evenings, and made weekends and evenings into work, rather than living time. If I started using darktable/RawTherapee, I would doubtless have a competent workflow down inside a month or two, especially if I started up something like a 30 days project or something else that required me to work with them daily. But to do that, I’d need to spend about a month going through my Lightroom catalogue and exporting 16bit tiff versions of anything I cared about and another bit of time resetting all my images to the import state to make sure darktable didn’t try to hose anything on import (though without a good library update, I probably wouldn’t choose darktable to be the main organizer, so I’d need another week or three to get the library organization aspect of my workflow down), and that would be 3 more months of fighting with technology rather than worshiping, spending time with my family, and looking at light, playing with cameras, processing film, building new rigs to scan negatives, scanning negatives and massaging digital files, etc., all of which I enjoy quite a bit and find mostly relaxing. I enjoy playing with technology too, and don’t much mind researching to find answers to problems and finding solutions to hardware and software issues, but I’m a researcher in my day job and to come home and do more research —especially of the frustrating, broken-tech type—feels like work, except worse, since I’m not even getting paid to be frustrated.

Mac and Lightroom aren’t perfect, far from it, but I’m used to them, I have a workflow on them and it works for me (most of the time, more or less). Sure, they both make some assumptions about my usage and desires, but both are easy enough to work around or subvert, and sure, they’re both far more expensive than the alternatives, but that expense gets you some things that seem to just work, without a bunch of messing around.

If at some future point, I can no longer afford Apple, I’m confident that I could move my workflow to a windoze box and more or less happily rock Lightroom (and iTunes) there with few issues. It wouldn’t be perfect, but it would be doable. And if I couldn’t afford Microsoft either, I could pretty easily move most of my photography workflow and general computing to open source tools. I’d have to give up the iPhone and switch to Android, but that’s largely trivial too: the only thing keeping me with iPhone at present is native syncing with mac, my podcast management scheme, and Hipstamatic, and other than Hipstamatic, I could make due with things available in various open source programs.

Anyway.

tl;dr: darktable is great, but I won’t willingly move from Lightroom to it anytime soon.

Up next: the rest of Ramadan and trying to worship Allah azza wa jall to the best of my ability, spending time with my wife and family, processing the ever-growing backlog of images,*** and maybe a review of Lightroom CC and Lightroom Mobile.

Thanks for reading!

*And I did end up replacing it… found a good deal on an ssd, and the little netbook is now much more pleasant to use… Windows 7 Starter is one of the more painful OSs in existence and it doesn’t seem likely that it will be possible to install 10 on it without paying, so I may end up pulling the ssd out of it and finding some other use for it (maybe install it in the old G3 iMac? that would be fun). But that was a fun project.

**Note: I looked and looked for an open source library organizer that allowed cataloguing and all, and gThumb is the only thing that does what I want. All the others are based solely around actual folders on the hard drive. I could completely change my workflow and start using tags to find everything, but I think I’ll pass on that for now.

***While working through the Lightroom 5 problems, I found that it lost most of the negative conversions I did over the last 3-4 months. Alhamdulillah. There’s nothing like scrolling through your catalog and finding that 50-100 hours of work have evaporated without warning… This alone should be enough to get me off the Adobe teat, but I’m stupid, I guess. At least I’m writing everything to xmp now and backing up to multiple external drives. Barring some catastrophic loss, everything should be more or less stable (though I need a new enclosure or enclosures for my external drives… the WD MyBook Thunderbolt Duo that I’ve been running for nearly 3 years is starting to have some problems with its thunderbolt passthrough, to which I connect my Dell 2408 wfp: the monitor just cuts off at various points (and I’m sure it’s the WD and not the Dell, since I plugged the dell into the mac directly for a few hours and had no issues).