Improved filter size

Similarly good results as with the 31x31 filters can indeed be obtained with smaller, asymmetric filters.

A darktable fork serving as proof-of-concept can be found here: https://github.com/ILiebhardt/darktable.git.

For trying, just compile, deactivate openCL (only C code thus far), and choose ‚1 pass Markesteijn‘ as demosaicking method (doesn’t work for 3-pass, and wouldn’t really yield advantages, either).

First proof of concept

I have a first working proof of concept ready, and the results look very promising. The source code can be found here.

By now, I also came across publication [5], a paper that could have saved me quite a bit of work.
;-) 

In the end, though, it was fun to do the according adaptations myself.

To recall from the first post: I am using the method of [3], [4] and [5] only for chroma. For luma and for establishing directionality, I use the open source Markesteijn algorithm. I found this to deliver superior results than using either of the two algorithms in its pure form.

This latter algorithm, as many other attempts to tackle this CFA, suffers from severe false colour artifacts in image areas where luminance has a high spatial frequency.

Sample images

Here are some sample images from such difficult image areas for comparison. Image pairs are first Markesteijn, followed by my combined approach using frequency domain chroma. Markesteijn is 3 pass. None of the output has further noise processing or median filtering.

Markesteijn 3 pass

Frequency Domain Chroma

Markesteijn 3 pass

Frequency Domain Chroma

Markesteijn 3 pass

Frequency Domain Chroma

Visual evaluation

The images resulting from demosaicking by the combined method employing frequency domain chroma look notably cleaner. The last image pair, for example, is part of a concrete flower pot, and should be grey, with only traces of greenish brown from moss. This is well achieved by the combined method, whereas the 3 pass Markesteijn algorithm alone struggles.

Next steps

Presently, the filters used are 31x31. This filter size is most certainly overkill. So the next challenge will be to reduce filter size whilst maintaining image quality. [4] suggests rectangular filters supporting the directionality, and I am intending to do the same here.

Ultimately, the aim would be to move from the initial proof of concept towards a more workable prototypic implementation.

References

[5] M. Rafinazari and E. Dubois, “Demosaicking algorithm for the Fujifilm X-TRANS color filter array,” in Proc. International Conference on Image Processing (ICIP2014), Paris, France, October 2014, pp. 660-663.

Starting the challenge...

About a year ago, I took a first shot at demosaicking images taken with a color filter array of the X-Trans type. This first shot was basically a gradient based approach using guided filtering, extending the ideas of [1] and [2] to the X-Trans color filter array.

The code is available at https://github.com/ILiebhardt/xtrans.

However, I was never satisfied with the results in chroma. Although the approach can restore luma information reasonably well, it fails to do so for chroma. The reason is that the X-Trans pattern provides for a more limited spatial resolution of chroma information than the Bayer pattern. Most gradient based approaches make implicit assumptions as to the spatial resolution of chroma information. While this goes wrong for Bayer sensors without AA filter, it goes even seriously wrong for X-Trans sensors.

A more tailored approach

An ideal method would take account of the more limited spatial resolution of chroma information of the X-Trans sensor. It would demosaick chroma 'accepting' this limitation, without the attempt to 'over-estimate' chroma - an approach that can only fail for the X-Trans sensor.

While looking for possible approaches, I came across the work of Eric Dubois et. al., which seems to be more in line with above considerations. Interesting starting points are [3] and [4].

It appears possible to apply this approach to the X-Trans pattern. Playing around with some Matlab sample code, this approach seems to be very good in restoring chroma information. It seems to be weaker in restoring luma information and also a bit weaker in finding directionality.

So an idea seems to be to combine a gradient based approach with such a frequency-domain approach. In such a combination, the gradient based approach would ideally be used for restoring luma and for finding directionality, whereas the frequency-domain approach would be used for restoring chroma.

My idea is to use the open source Markesteijn algorithm, which is based on AHD, as gradient based approach. To the best of my knowledge, the frequency-domain algorithm of Dubois et. al. is not patented.

This blog will serve to keep track of ideas and progress, to show sample images, and to accompany any (potential) progress from this idea, via a first proof of concept, towards (if feasible) a real implementation.

References

[1] Kaiming He, Jian Sun, and Xiaoou Tang: 'Guided Image Filtering' in TPAMI 2013, further info also here: http://research.microsoft.com/en-us/um/people/kahe/eccv10/

[2] Daisuke Kiku, Yusuke Monno, Masayuki Tanaka and Masatoshi Okutomi: 'Residual interpolation for color image demosaicking' Proceedings of IEEE International Conference on Image Processing (ICIP), pp.2304-2308, 2013.

[3] Dubois, E.: 'Frequency-Domain Methods for Demosaicking of Bayer-Sampled Color Images', IEEE Signal Processing Letters, Vol. 12, No. 12, December 2005.

[4] Leung, B., Jeon, G., Dubois E.: 'Least-Squares Luma-Chroma Demultiplexing

Algorithm for Bayer Demosaicking', IEEE Transactions on Image Processing