Visual-LiDAR Fusion for Incremental Automated Infrastructure Inspections

Problem Statement

Routine visual inspections are crucial in gathering structure-level data to inform civil asset maintenance strategies. Enhancing these inspections with computer vision, mobile robotics, and depth sensors can be very useful in improving the repeatability and objectivity of these inspections. Quantifying concrete defects on the surfaces of civil assets is critical to identifying troublesome areas that may require maintenance.

 

Approach

Our approach to automated inspection utilizes simultaneous localization and mapping to estimate the trajectory of a mobile robot around an asset using cameras, an inertial measurement unit, and a LiDAR sensor. A 3D map of the civil asset is then reconstructed using the estimated trajectory and lidar scans. The sizes of said defects are computed by combining the 3D map and automatically detecting concrete defects in the images (using deep neural networks). Due to the sparsity of the 3D reconstruction when using lidar point clouds, a simple depth completion approach is utilized to increase the accuracy of the defect quantifications.

 

Ongoing Work:

To increase the repeatability of these inspections, we are currently working on automated multi-map alignment by utilizing both image and lidar data to align two maps of the same asset gathered at separate times. This alignment then allows for automated defect growth reporting, essential for identifying which defects are more or less significant to provide maintenance.

 

Key Results and Visuals:

Figure 1. Illustration of depth-completion results for defect 3 shown in Figure 4c including (a) the sparse depth map (b) the completed depth map; (c) the semantic segmentation of the defect (red) without depth completion; (d) the semantic segmentation of the defect (red) with depth completion

 

Relevant Publications

  • McLaughlin, J., Thoms, A., Charron, N., and Narasimhan, S. (2020). “Depth Completion From Sparse 3D Maps For Automated Defect Quantification”, ASCE Lifelines Conference, Los Angeles, California, USA, 2022.

 

Students

Jake Frazer McLaughlin

Alexander Thoms

Nicholas Charron