darktable ep 060 - Linear vs Non-linear data

[@charleshacker765]

This was a beautiful episode. I think it is mislabeled as you also go into: 1. Lightroom vs Darktable and 2. Open source. Then finally close with 3. Discussion of organization. 1. Very nice coverage. For a while your honesty was brutal. That was good. It shows the guys on the other side you are aware and not blinded. 2. Open source - The man you were responding to finds the concept of Open Source hard to understand. The fact is Open Source is more secure as you mentioned. Also, you can be sure that any obstacle you face someone else has faced to, such as producing photo books, and someone has made a program to deal with it. That's the power of Open Source. Everyone benefits from working together. 3. On organizing I'm glad you brought up tagging. For me I don't organize by date. I see what you do jumping through various folders as tedious. No joke I just throw it all on one folder and rely on the tags. I also keep this folder on an external drive and copy the pictures locally I plan to edit. (A trick I learned from you.) Then I use a program called Unison that synchronizes that drive to another one. It is unlikely both would fail at the same time. Darktable let's me edit files on the external and local drives just as easily and like you said Darktable has never corrupted anything of mine. Thanks for the good videos!

[@audio2u]

Thanks again. I see you're bingeing! 😃 The more time I spend in Linux, asking questions of those who've been at it longer, the more I come to realize there is less bitchiness than I see amongst windows users. You'll see it occasionally, but nowhere near as often. But this is getting off my point.... I find the Linux community, for the most part, very happy to help you learn, and I love that! I've always loved by the ideal that if I'm not learning something new, then I'm not stagnating, I'm actually going backwards.

[@charleshacker765]

@@audio2u Ha, ha. Yeah. I am bingeing! We all complain about not having enough time, we better make use of it while we can. I picked up the art of photography only recently from an instructional series, so I focused on yours for post-processing. A friend had given me his older camera in March. Until then I never knew any of this stuff beyond point and shoot. On the subject of Windows, their solution is always reinstall. They have no idea what's going on. Say, you know you never talked about tethering. I know it's a radical one, but it is useful for somethings like time lapse photography and also a lot of production work wants to see the immediate results of the shot. These are two useful reasons for it. Anyway, great stuff! You really are an amazing instructor.

[@audio2u]

Yeah, the problem with tethering in darktable is, unless something has changed that I'm unaware of, it only supports a limited range of cameras. And last I looked, Sony wasn't one of them. 😥

[@ma3xiu1]

Just a small point: you said using floating point arithmetic is like multiping the samples by a large value. That in fact would be fixed-point arithmetic, not floating point arithmetic. The key point of using floating point arithmetic is that you break the number up into two parts: a mantissa, which describes the steps within a particular EV range, and an exponent, which says how far along to move the decimal point. So, the value encoded by a floating point number is [mantissa] x 2^[exponent], which looks like the "scientific notiation" we learned in high school physics, except using base 2 instead of base 10 (because computers work in binary). So, the use of an exponent here is a bit like doing a log encoding, where the exponent indicates the EV range, and the mantissa provides the detailed steps within that EV range. This gives us the same benefit as using an integer log encoding in terms of being able to spread out out bits efficiently over many EV ranges, while still allowing us to do math operations on real values that are proportional to the input light intensity. Hope that sort of makes sense.

[@danielspenner3683]

No, it doesn't.😂 However that's not your fault, but my dyscalculous brain!

[@audio2u]

Great explanation, and not something I would attempt to repeat! 😃

[@andrewgreig1197]

Hey Bruce, re Linux security, the first thing to understand is the architecture. Linux is modular and Windows is monolithic. Software in Linux interacts with modules which in turn interact with the kernel , no application can deal directly with the kernel. On the other hand a finance app in windows can write directly to the kernel, and the protection industry is built around ensuring that nothing writes to the kernel if it is malicious. Good luck with keeping up with that. In extreme cases of paranoia a linux OS can run the kernel from a CDROM, impossible to write to. Everything else apart from the kernel can be loaded into the local drive. I ditched Windows in 2000, at which time people were saying that the Linux desktop was not ready for real world use. As a user, not a geek, I happily ran my various businesses on the Linux Desktop from then till now. I have never been hacked, no virus has ever infected my computer and the support from the Open Source community is staggering. With the help of the Linux Community I have an industry strength backup process which is automated and runs as I work. I love this community, I cannot imagine ever straying int the Vendor lockin of Apple or MS.

[@audio2u]

Love it! I've only been a Linux user since 2016, and I'll confess that I'm more a GUI guy than a command line warrior. I had been wanting to try out Linux since around the same time as you (2000) but my first adventure in that direction was disastrous! Due to not understanding file systems, and setting up a dual boot system, I lost an entire partition of data, including the audio files for a mate's album which we were working on at the time. And there was no backup. So, as you can imagine, he's never let me live that down! We laugh about it these days. But when I saw the direction Microsoft was heading with Windoze 10, I decided it really was time to "get the hell out of Dodge". 😃

[@jrnine]

@Andrew Greig Hi together, yes i know, the thread is 2 years old, but i think again and again new users come over and read this stuff. Linux security. It is a long discussion about Linux and windows security. As i understand it correctly, the question also was is open source secure. Also that is a long discussion. For me, i am using linux since 1995, more or less, i want to mention, that also on linux, sometime the security risk is sitting before the monitor. An example: Back in the early 2000, i set up a linux router, standing in the internet via dyn dns, for a friend of mine who has a photo shop in a local city. Why in the inet? We want to simplyfy the service to get the digital stuff from his other shops around. So, for one customer (a local photograph) he wants to set up a special service. So, i set up, for testing purpose, a user with an easy pwd on the linux box. testing around and i forgot later, that i set up this user and do not restrict remote login. So you guess what happened? Hacked over ssh. Afterwards in the log files (the hacker leaves the bash history on the router) i could analyse what he did and i could analyse when i was hacked. I think 2 days after i set up the user, we were hacked. My fault, user was photo and pwd was? ... photo ;-) Ok long story, at the end, the os and the software can be so secure, but it is still important to not forget to use the brain. Get software from known repositories, wait until your favorite software is on this repo and do not download it from other repositories you do not know, use secure passwords ;-) Greetings, Jörn Rink

[@audio2u]

Haha, great story! At least you can be forgiven for not making the password "password"! 😃

[@JillesvanGurp]

One issue with Darktable asset management is the internal database it uses to keep track of what you have imported and where that is on disk. If you import something, and then rename or move the folder on disk, it gets confused. The UX for importing is a bit confusing on this and in my view it is highly annoying it has good way to recover from that. Also if you import a folder and then later add another photo to that folder, Darktable won't see it until you reimport the folder. Most of the information in that database is just there for performance reasons and losing it is no big deal. When you import, things like ratings, keywords, file locations, etc. get copied to it (from the XMP file). So, that makes it easy to move folders between computers. When I'm traveling, I often do a first pass of editing on my laptop and then copy the folder to my desktop for further editing. Just works. It's great. I have no idea how things are in LR, I've not used it in many years and only dabbled with it briefly when I did. I think valid reasons for moving away from LR is simply getting access to a wider feature set. If you are happy with LR and the choices it makes for you creatively, then maybe that is not that interesting. But Darktable gives you a lot more creative options and ways to do things that maybe would not be easy or at all possible in LR. I'm not aware of anything LR does qualitatively that stands out as being clearly better than Darktable. Probably there are a few things. I am aware of an insane amount of stuff Darktable does, that I use, that LR just doesn't have. So, you mostly gain stuff. But as Bruce mentions, the UX definitely has some rough edges and also the whole set of filters and features can be a bit intimidating to new users. For me the best feature is it's free & future proof. I don't have to worry about the product disappearing from the market (like happened to Apple's Aperture), subscription bills or getting stuck on some unsupported version. As long as there's a single developer left to keep on working on it, the product moves forward. My general observation is that it has moved forward a lot since I started using it. Another issue for me is that performance on a Mac sucks due to Apple not supporting OpenCL anymore with updates and the very likely removal of this entirely (in favor of their proprietary Metal) in future versions. I think the Windows version is probably a lot better but it's something to be aware off. I am actually considering buying a PC for running Linux; but that's not for everyone (I'm a developer, I'll be fine).

[@audio2u]

Just on the "working on the laptop" issue, have you tried the "local copies" feature? I did a video on that quite early in this series.

[@dennispeterson1434]

Thanks for another great video Bruce. Your videos never disappoint. I always walk away with some new knowledge. Your efforts are much appreciated.

[@audio2u]

Thanks Dennis!

[@europlatus]

Hi Bruce, I sense some mild frustration or perhaps some confusion in the first half of this video about how much all of this matters, and I totally understand that feeling. I also have read a lot on pixls.us and tried to understand literally everything about image processing, colour, light and why I didn't pay more attention in maths class at school. But at the end of the day, I just feel more confused about how to use the software. I can get some good results, but I'm not really sure why, and maybe that's ok. Not everyone can get their head round these subjects. I'm torn between my natural inclination to understand everything at the fundamental level, and just being a user and admitting that I can't be an expert at everything. For darktable, I feel there needs to be a balance between catering to those who want to dive in deeply and those who appreciate the power of the software but can't or don't want to delve too much underneath the hood. It's arguably getting harder for the latter group recently, but maybe this is because the software is in a transition phase. Maybe after a couple more major releases, it will have moved to full RGB, and a more refined (or obvious) workflow can be put in place.

[@audio2u]

Wow. You totally nailed it. That is EXACTLY where my head is at right now, too. Torn between "I like this approach and the look it gives me, even though it might not be technically correct" vs "I need to understand every little technical detail to be happy". :)

[@europlatus]

@@audio2u Well having watched all your videos, I feel that we're in a similar camp of being technically proficient, probably above average, and naturally curious; but we're not programmers or mathematics wizards. So it's no surprise that we're having the same thoughts about darkroom. On the one hand, over the last year playing with FOSS and being active on pixls.us I've learned so much; but on the other, it's also shown me how much I don't know! Anyway, keep up the great work!

[@stephenward5133]

@@europlatus @Bruce Williams Photography I should have watched this video months ago. Like you both I feel I am in the same place stuck between wanting to apply logic and not necessarily getting all of the way there (like you Bruce I am a sound engineer and Music tech geek so feel I should be able to understand more of this).

[@thingsiplay]

I want to say something about security in open source software too. Generally, it is more secure because it is open source. The problem starts, if there is no compiled version for the newcomer who don't know how to compile the software, especially if its a very complex and hard to do one. So then they rely on pre-compiled versions from third party sites and sources, which can be a huge security factor. That means, even if the software code itself is does not contain malware, the third party distribution COULD introduce malware. Just in theory. But on Linux this seems to be a non issue and I use a Linux OS since 2008 on my main computer.

[@audio2u]

Yeah, and I would imagine darktable falls in the 'not exactly simple to compile' camp!

[@dschneiderch]

just getting into DT and your video series is great! I guess this is potentially a rather basic digital asset management question, but how do you organize your photo files? after shooting, do you just dump dated folders of your images into a "Pictures" folder and then use the keywords/tags in dt to organize from there? or is there a hybrid approach of organizing the photo files into trip-themed or session-themed folders?

[@audio2u]

No, I use Rapid Photo Downloader on Linux for copying from memory card to PC, using YYYY/YYYY-MM-DD as the naming structure,and then just use keywords for EVERYTHING.

[@PeteBarlow]

Bruce, would it be possible to see the whole email. I prefer to read stuff through myself as I find it easier to understand that way as I can go back if there is something I didn't understand the first time through.

[@audio2u]

It was too long to fit on one screen! That was why I broke it down like I did.

[@joaovtaveira]

33:33 wouldn't "LaTeX book template" be a alternative to that?

[@audio2u]

Possibly. I've never used it.

[@JorgeMM]

Thanks for share your point of view. You are really a very good comunicator. Hope more videos becasue I am learning many thing from you. In other hand, I am spanish and you english is perfect to undertand, clear and slow. With your videos I am learning english and darktable. If one day you want to visit Spain, please write me and I will try to help you with your visit. Best regard and thanks again.

[@audio2u]

Jorge, thanks for the kind words! As it happens, my wife and I visited Spain in 2017 and we LOVED IT!! We were in Barcelona (for the Festival of St Joan!), Toledo and Cordoba.

[@JorgeMM]

@@audio2u I am glad you visited my country. Sure you made a lot of photos. Best regards Bruce.

[@danielspenner3683]

Isn't this, what the "old" filmic module does: turn the linear data provided by the camera sensor into a tonecurve? And thus changing the saturation from shadows through the midtones to the highlights.

[@audio2u]

I think the original filmic worked in LAB, but filmic RGB works, as the name would suggest, in the RGB colour space. But what both versions do is allow a user-definable desaturation of the outlying luminosities (the tones near the black point and the white point). Not sure if that answers your question or not.

[@joaovtaveira]

15:46 0.18^(1/2.2)=0.(45)≈0.45, not 0.46.

[@joaovtaveira]

Hi Bruce cheers from Portugal. I have a strong video recomendasion for you, and I'm sure you'll consider it as a "tag lover" you are. Make a video about the new "metadata" tab in "selected image[s]" module in darktable 3.0.1. In my humble opinion, is one the greatest features realised for a few time. I love it but I can´t be sure what "refresh exif" does. Can you help me and have you had tried it already? By the way, I hope SARS-CoV-2/Covid-19 never attacks Oceania like is attacking Europe. Protect yourself and follow the recomendasions. #stayathome

[@joaovtaveira]

You could also include in that video, the other additions to darktable in its 3.0.1 version.

[@audio2u]

Yeah, need to spend some time to dig into that. :)

[@audio2u]

...and yeah, so far, Australia has not been hit as hard as the rest of the world. But I suspect that just means we're 2 weeks behind. So where you guys are at now... that'll be us in a couple of weeks time. :(

[@joaovtaveira]

@@audio2u I think I figured out what is this button is supposed to do. I think it's supposed to reset the image exif and restore the original one. However it behaves in a strange way, I think it doesn't work very well.

[@blisz6781]

Thx Bruce for your explanation. This was very important to know how software works but it was naturally very theoretically. But what does working in linear and non linear practically means ? I read a lot of stuff on pixls. us and other communities. I also watched aureliens videos again and again. But i missed a practical demonstration proceed an image in LAB or RGB that a normal non studied user noticed what this difference practical means ? Perhaps a new video ? en francaises: darktable.fr/2020/01/darktable-3-rgb-ou-lab-quels-modules-au-secours/ Eine deutsche Übersetzung: www.bilddateien.de/blog/2020-01-26-darktable-3-workflow-lab-oder-rgb.html

[@audio2u]

Yeah, I was kind of wondering that after I'd published it. "Did I really answer anything here?" I guess not! Apologies. As for the examples, see that video of Aurelien's where he changes the order of the modules in order to process a jpeg. But as for "what are the practical differences?", I'm really not 100% sure that most of us mere mortals would ever tell.

[@Foche_T._Schitt]

Lightroom's catalog is a clusterfck.

[@audio2u]

Totally!