Cross Spectral Stereo Matching
Flash required for video playback in older browsers.
Issue: Robust scene depth recovery from cross-spectral (i.e. optical / thermal )stereo imagery.
This has major applications in autonomous surveillance where dual optical thermal sensors are
commonly deployed for a range of tasks to provide robust day/night sensing.
Approach: Dense unsigned gradient features in combination with a strong optimization approach.
Prior work either recovers depth from isolated scene objects (LSS features, [Torabi, 2007]), uses simulated cross-spectral imagery (MI, [Fookes, 2004]) or solely addresses radiometric differences in standard stereo image pairs (ZNCC, [Hirschmüller, 2009]). Our approach outperforms this prior work in comparison.
Dense unsigned Histogram of Orientated Gradient (HOG) outperform these feature matching approaches to produce coarse depth images suitable for scene understanding and reasoning.
With simple Winner Takes All (WTA) optimization unsigned HOG features yield unreliable depth results (right). Combined with Scan-line Optimization (SO), Dynamic Programming (DP), Graph Cuts (GC) or Semi-Global Matching (SGM) dense HOG features produce more stable, usable results (below).
Unsigned HOG descriptors are efficiently computed and L2 normalized. Pixel matching is then performed using L1 distance comparison. Strong optimization approaches provide improved depth with DP and SGM providing usable results within reasonable computational bounds.
Full scene depth recovery comparable in quality to that of standard optical stereo is recovered from the same scene. Examples of our results, using a combined HOG and SGM approach, over a range of scenes are shown in the gallery below (click on image to scroll):
Furthermore we illustrate the temporal consistency of our approach in the production of coarse depth maps suitable for use in scene understanding (e.g. object detection priming/targeting/hinting)and navigation (e.g. obstacle avoidance) over the following video examples:
Cross-spectral stereo, outperforming prior work, is demonstrated using dense gradient features. Future work will consider advanced dense descriptors and ground truth evaluation.
| 2012 | ||
| [1] |  | On Cross-Spectral Stereo Matching using Dense Gradient Features , In Proc. British Machine Vision Conference, pp. 526.1-526.12, 2012. [demo]Keywords: stereo vision, thermal, multimodal stereo, thermal stereo, IR stereo, optical thermal stereo. [bib] [pdf] [doi] |
This work was carried out in collaboration with TU Graz.