Starting the challenge...

About a year ago, I took a first shot at demosaicking images taken with a color filter array of the X-Trans type. This first shot was basically a gradient based approach using guided filtering, extending the ideas of [1] and [2] to the X-Trans color filter array.

The code is available at https://github.com/ILiebhardt/xtrans.

However, I was never satisfied with the results in chroma. Although the approach can restore luma information reasonably well, it fails to do so for chroma. The reason is that the X-Trans pattern provides for a more limited spatial resolution of chroma information than the Bayer pattern. Most gradient based approaches make implicit assumptions as to the spatial resolution of chroma information. While this goes wrong for Bayer sensors without AA filter, it goes even seriously wrong for X-Trans sensors.

A more tailored approach

An ideal method would take account of the more limited spatial resolution of chroma information of the X-Trans sensor. It would demosaick chroma 'accepting' this limitation, without the attempt to 'over-estimate' chroma - an approach that can only fail for the X-Trans sensor.

While looking for possible approaches, I came across the work of Eric Dubois et. al., which seems to be more in line with above considerations. Interesting starting points are [3] and [4].

It appears possible to apply this approach to the X-Trans pattern. Playing around with some Matlab sample code, this approach seems to be very good in restoring chroma information. It seems to be weaker in restoring luma information and also a bit weaker in finding directionality.

So an idea seems to be to combine a gradient based approach with such a frequency-domain approach. In such a combination, the gradient based approach would ideally be used for restoring luma and for finding directionality, whereas the frequency-domain approach would be used for restoring chroma.

My idea is to use the open source Markesteijn algorithm, which is based on AHD, as gradient based approach. To the best of my knowledge, the frequency-domain algorithm of Dubois et. al. is not patented.

This blog will serve to keep track of ideas and progress, to show sample images, and to accompany any (potential) progress from this idea, via a first proof of concept, towards (if feasible) a real implementation.

References

[1] Kaiming He, Jian Sun, and Xiaoou Tang: 'Guided Image Filtering' in TPAMI 2013, further info also here: http://research.microsoft.com/en-us/um/people/kahe/eccv10/

[2] Daisuke Kiku, Yusuke Monno, Masayuki Tanaka and Masatoshi Okutomi: 'Residual interpolation for color image demosaicking' Proceedings of IEEE International Conference on Image Processing (ICIP), pp.2304-2308, 2013.

[3] Dubois, E.: 'Frequency-Domain Methods for Demosaicking of Bayer-Sampled Color Images', IEEE Signal Processing Letters, Vol. 12, No. 12, December 2005.

[4] Leung, B., Jeon, G., Dubois E.: 'Least-Squares Luma-Chroma Demultiplexing

Algorithm for Bayer Demosaicking', IEEE Transactions on Image Processing