RFC / WIP - Lens correction

[paolodepetrillo]

I have been experimenting with a lens correction module which uses a LUT to map a radius in the output image to a radius in the input image, rather than any specific distortion model, to correct for distortion and / or TCA.

This first attempt only supports the WarpRectilinear opcode embedded in DNG file metadata - I figured this was a good starting point since it's openly documented rather than being reverse engineered from proprietary tags. It could also easily be extended to support lensfun models, probably without any change to the shader kernel.

I have mostly been testing this with Olympus .orf's converted through Adobe DNG Converter - it reads the proprietary embedded correction and converts to WarpRectilinear.

There is a slight difference from the behavior of the darktable implementation: the correction results in a non-rectangular image with irregular black edges, and darktable automatically finds the largest rectangle inscribed in that image and sets the scale to crop to that rectangle and hide those black edges. This may be needed for some other models like lensfun, but DNG WarpRectilinear typically has the correct scaling implicit in the coefficients already so I don't try to do any auto scaling. The scale slider is provided to override if necessary.

Possible TODOs:

• Support rawloader in addition to rawspeed+exiv2 (Would need to modify rawloader to make the necessary exif tags available)
• Lensfun models
• Correction for Sony and Fuji raws as in darktable and Olympus as in this darktable pr
• Apply tangential coefficients in WarpRectilinear, although I've never seen a file where they were nonzero
• Support other DNG tags WarpRectilinear2 and WarpFisheye
• Option to apply only distortion, only TCA, or both (not possible for DNG but should be otherwise)

[hanatos]

wow thanks for your work on this! it looks really full featured already. will check it out and comment in more detail after that.

yeah the rawloader codepath is kinda important, but it's not hard to grab the metadata from there too.

if you point exiv2 at the .lcp shipped with the adobe dng converter, would it give you the same values? i suppose we could also think about including this kind of data in the camconst repo.

[hanatos]

okay, a few general remarks (will put more detailed ones interleaved in the code):

very cool how you found your way around the code/api even with uploading custom source inside a module without exposing it to the outside!

the idea to store a few precomputed positions in a lut is very much a cpu idea.. i will bet that these handful of multiply-adds in the horner scheme will completely be hidden in the latency to get to the pixels on a gpu. it also simplifies the code and removes the need to upload anything at all.

about the coefficients/parameters: i would like it better if the coefficients were directly passed as parameters to the compute kernel, and maybe even exposed like this in the gui (possible when not using a lut). that way people can at least dial in the right coefficients even if they don't have an adobe raw converter. it may be easier to ask "a friend" with a windows computer to read out the values from the .lcp file (these are plain human readable xmp). it will also enable copy/pasting histories with correction parameters.

long term maybe we can add an option to read .lcp files or text files with the same content, for cameras which don't write the set of required dng tags.

are there other relevant correction models that require a very different set of arguments? i suppose exposing an int to distinguish between model and a float array with sufficient space for the most complex model would always be possible.

[hanatos]

ah, here's my union already. i guess the parameters of the lenswarp module could look the same. not sure about double precision. looking at the coefficients in some random lcp it looks 32-bit float might suffice. doing double arithmetic on gpu is not much fun.

[paolodepetrillo]

I only used a double here because that's how it was encoded in the DNG opcode.

[hanatos]

yeah see general comment. this is not nearly enough to make a gpu sweat per pixel. also since it appears only even powers of r are needed, maybe the square root above (in the hypot) is not actually required to be computed at all, so the special function unit would be idle too.

[hanatos]

you sure this is correct? seems to me r*r is square and r_scale is still linear (because of the sqrt above?).

[paolodepetrillo]

I think that's a typo.

[hanatos]

see general comment: passing the parameters straight can probably do completely without read_source and init and would only use commit_params to copy over stuff from the img_param struct to the committed_param uniforms of the kernel.

[hanatos]

minor convenience thing: you can use -1ul (note the 64 bits) instead of &module->connector[0].roi to skip passing a roi for an input, it will just use the one of the connected counterpart later on.

[hanatos]

yay you even have documentation :D

[hanatos]

for maximum performance in this case, there is a bypass api call like used in the resize module in the link. it will cpu side short-circuit the connectors so the shader will not even be called, avoiding a roundtrip from and to global memory. has the downside that we'll need to check if parameters changed and in this case recreate the pipeline, but shaves off 0.1ms or so..

[paolodepetrillo]

Thanks for the detailed feedback! I had already almost finished Olympus support so I will push that first and then start addressing the comments.

I had not thought about using the lcp files from Adobe DNG Converter. For Micro Four Thirds (and maybe also Fuji and others) there are no lcp files and only the coefficients supplied by the camera body in the exif are used. For Olympus I think the camera/lens has a 2D LUT of parameters indexed by focus distance and focal length, and for each individual shot it interpolates based on the exact settings for that shot.

There are a few different models that could potentially be supported:

• DNG WarpRectilinear - polynomial model with radial and tangential parts. Here I implemented the radial part and ignored the tangential part. Defines the corner of the raw image as radius 1.
• Adobe Camera Model (rectilinear) - used in lcp files, appears to be the same model as WarpRectilinear but the radius is defined differently in terms of focal length independent of sensor size. Lensfun supports it so could follow the example of its implementation.
• Olympus proprietary exif tag - same polynomial as radial part of WarpRectilinear but radius is scaled slightly differently (details here)
• Sony or Fuji proprietary - reverse engineered in darktable, 16 or 9 element LUTs in the exif tag
• Lensfun - poly3, poly5, ptlens models used in the lensfun database
• Latest DNG version also has a new WarpRectilinear2 tag with a different polynomial, more coefficients. There is also a WarpFisheye

It makes sense to have all of the coefficients be module parameters so that they can be set manually by editing the config. Probably the model with the largest number of parameters would be the 16 element Sony LUT. In some cases the coefficients could be modified in commit_params, for example to handle the different radius scaling between WarpRectilinear or ACM.