Structured light three-dimensional imaging
Micro-Baseline Structured Light
Papers link:
Summary
We propose a baseline micro structured light ( the MSL), a new three-dimensional imaging method, designed for small-sized devices, such as mobile phones and micro-robots. MSL using a small projector and camera baseline projection hardware cost, and can use a smaller amount of calculation algorithm for scene depth recovery. The main observation is that the smaller the difference in baseline will result in a smaller, thereby achieving a nonlinear first-order approximation of the image formation model SL. This leads to the key results of this theory: the MSL equation A linearized SL image forming method. Since each pixel has two unknowns (albedo and depth), the MSL equation constrained, but can be effectively used to solve partial least squares. We are from a different system parameters such as baseline projection models and to analyze the performance of MSL, and provide guidance for optimal performance. With these insights, we built a prototype to test the theory and practical test.
1. Introduction
We propose a new SL method, known as micro-structured light baseline ( the MSL), which is suitable for such a highly constrained devices, thereby opening deployed on a small, low power and low complexity of the apparatus may be of SL sex. MSL work under the constraints of a small (micro) projector camera baseline, as shown in FIG . 1 ( B), shown below based on the observation: small differences between the projector and the camera baseline results in small pixels.
Our main theoretical insights that minor differences in the structure of an optical image formation model (in the case of unknown (and depth albedo) is non-linear) can be linearized by a first-order approximation. This resulted in a new linear SL constraints derived, i.e. baseline micro structured light ( the MSL) equation, it albedo scene depth and intensity measurements linked.
2. Related Work
Structured light coding techniques:
Generally, SL technology can be divided into single and multi-lens camera method [25]. Multi-lens technology, optical striping [2], a gray scale code [23] and the sine phase shift [3] to the estimated shape of the projection by a plurality of patterns in rapid succession. These techniques can be restored with high precision by calculating the depth of a simple decoding algorithm, but requires a complicated projection apparatus (e.g., the LCD, the DMD), which device can dynamically change the projection mode, such that they are not suitable for dynamic scenes of low complexity and equipment such as mobile phones. Single shot technique a pattern is projected only, dependent on the intensity [32], color [8, 13], or a projector code corresponding to a local neighborhood [9, 20, 14]. Technology is ideal for single mode dynamic scenario; however, these techniques typically use computational complexity decoding algorithm, it is necessary to achieve real-time performance of dedicated hardware. Some single lens having a relatively simple decoding method (e.g., Fourier transform profilometry ( the FTP) [30]), but they made a strong assumptions texture and depth of the scene.
Real- SL System:
Some methods may be used to perform high-speed ( 1000 FPS) SL, or the use of costly high-speed camera can not be ported to the mobile set [12], or a learning-based method recently used, such as ultra-depth [24] and UltraTereo [7]. With enough data, and Kinect [1] and other special hardware, these methods proved to be fast and accurate.
Our target different
Our aim is to develop a simple, analytic, closed form decoding method using conventional differential equations SL equation in a small baseline constraints. An interesting future research direction is to use data-driven techniques to enhance MSL, to potentially further improve the accuracy and speed.
3. Structured Light Preliminaries
We first describe the image forming SL model system to understand the role of structured light projected on the camera baseline system. Image formation model. Consider FIG . 1 ( B) shown in the projector - camera pair. We assume a rectangular configuration projector or a camera, wherein the camera and the projector central horizontal movement B units. We further assume that the projector and the camera have the same spatial resolution and the focal length f. These assumptions are merely for ease of illustration; and analysis techniques for providing a general system configuration and the parameters are valid.
In the next section, we designed a technique requires a projection pattern (but two images are captured), but lower computational cost, can be efficiently implemented in power limited systems. In addition, although the conventional SL baseline system as large as possible, but the proposed techniques are designed for small size of the device, the device allows only a small (micro) between the projector and the camera baseline.
4. Micro-baseline Structured Light
Relations with the differential method
The method of the above analysis and differential recently designed for photometric stereo [5] and the light field based motion estimation [18] are similar. These methods are also linear original nonlinear problem difficult to solve, the resulting solutions analyzed and easy to handle. In the same spirit, MSL can be seen as a differential version of the SL.
Relationship with the light stream
Notably, the MSL matrix like structure tensor LucasKanade tracker [16] in. In the context of stereoscopic disparity similar linear / optical flow and a 2 × formed matrix 2 has been explored [6, 21]. Structure tensor and a matrix is a key difference MSL MSL matrix depends only on the projection mode its derivative. Thus, reversibility MSL matrix depending on the nature of the projection mode can be analyzed, but can not be analyzed according to the scenario.
5. Invertibility of MSL Matrix
This proposition noted that a pattern is not constant or exponential function by projection, in theory, ensure MSL equation has a solution. Next, we discuss the stability of the solution, which is an important consideration in the presence of noise. Thus, when the projection mode is a periodic solution MSL equation is stable. Mode cycle may not be aligned with the analysis window. However, in practice, as we have shown in experiments on small depth estimation bias is robust.
6. Handling Texture Edge
In order to keep the computational simplicity, we assume that the albedo just guiding scaled version of the image. FIG 2 is calculated by a high texture depth description of the object of guiding the standard with respect to MSL advantage of the MSL. The pilot MSL based greatly improved recovery MSL depth accuracy, almost no computational overhead, and thus extends the scope of the proposed method. Since then, all our results are using a boot MSL calculation method.
7. Practical Considerations for MSL
FIG . 3 ( B) illustrates the accuracy of some representative pattern period as a function of the baseline. Obviously, the minimum error corresponding to the baseline period increases as increases.
Small baseline established to ensure that a first order approximation, but suffered triangulation error [31]. On the other hand, large baseline need a big window, so local invariant assumption may not hold. FIG. 4 shows the simulation accuracy as a function of the baseline. For this analysis, given the baseline, we have chosen to achieve the best accuracy of the model cycle baseline. We observed, the MSL in a different set of examples always achieve the highest accuracy between 8-30mm. In practice, the exact choice of parameters depends on several additional factors, such as allowing the resolution, the camera and the projector defocus projector. We found 15mm baseline resulted in the most accurate results, our laboratory prototypes consistent with this baseline (see Figure 6).
Under what kind of equipment constraints, MSL than the existing more appropriate SL technology? The goal is to shape MSL, low complexity and hardware platforms have limited computing resources, and therefore should not be considered an alternative to conventional ranging common hardware. For example, if a system is capable of projecting a plurality of patterns, the phase shift [3] can work accurately even in the narrow base, as shown in Fig. Similarly, if a system has enough computational resources and / or large baseline, the conventional single technique [20, 8, 33, 1, 7, 24] can be achieved than the MSL higher accuracy.
Furthermore, if the system is equipped with two cameras, you can rely on accurate stereo matching technology [17] to a corresponding relationship, despite the high computational requirements. However, when a small volume of the device in question, when the limited hardware and computing power, the MSL promises to provide a light-weight solution. FIG 5 illustrates, the MSL is less than a more accurate matching block 100 mm baseline, while faster. Although the specific number depends on the specific configuration, but the baseline is small and only a single pattern when projected, the MSL is suitable.
8. Experiments
Hardware Settings
Our apparatus includes a 1280 × projectors (DLP 720. AAXA Technologies), F = 8mm and a 2048 × 1536 machine vision camera ( Basler acA2040120uc), F = 12mm. Different focal lengths lead to the projector and the pixel size of the image size in the camera image is 2.5 times. Camera placed above the projector, the baseline level of 15mm, as shown in Fig. The system also has a base in the vertical direction, which is due to unavoidable mechanical constraints. However, since we propose a vertically symmetrical pattern, and thus only consider the difference between the baseline level; vertical baseline level difference does not affect the calculation.
Basic Facts
We used five frameshift frequency phase capturing ground truth depth information corresponding to 1280px, 100px, 50px, 20px and the pattern period of 10px. It is used to expand the low frequency phase, which makes the sub-pixel disparity estimation precision becomes possible.
Comparison of running time on the phone
To assess the real-time, via a projection we ran 800mm dom point mode, the MSL with a block matching algorithm having a micro stereo baseline were compared. The results are shown in Fig. Please note that the projection mode and a decoding strategy is not optimized for narrow baseline; we are here to focus on is more complicated timing rather than accuracy. FIG . 5 ( B) shows the Android devices with conventional (e.g., use of block matching and matching different image resolution based on the operation of the upper googlepixel2xl Semiglobal method (OpenCV [4] implemented comparison between SGBM)). And semi-global matching block matching run time 3MP images are 133ms and 1s. In contrast, MSL at the speed of 27ms at much faster, suggesting that MSL is suitable for mobile platforms.
Video sequences
Light One advantage SL technology is the ability to calculate the depth of the video rate. To verify this, we per second captured sequence of images at 30 frames, the video rate for 3D imaging. Alternate frames are captured, there is no pattern image for guidance. The system output speed 15fps video, and a depth of no video mode, without calculating (without any mode - Scene separation), it is generally used in reality. FIG we show a depth of three representative frame 8. Please note that the depth of change are clearly visible in a variety of gestures. More importantly, the calculation used to estimate the depth of the overhead is very small, it can be output in real time, which makes the MSL become a compelling mobile technology systems.
Experimental evaluation
FIG 9 shows a different texture and geometric complexity of the scene based on several MSL the recovery results 3D. All experiments were captured by a triangle pattern to different periods, it shows the various scenarios which may be used MSL. The first row shows the results with various texture complexity planar object. Mannequin scene demonstrates the limited non-planar texture scene MSL. Note how the three-dimensional model graph forehead and cheeks. Finally, BAS terrain display depth range accuracy but small high spatial complexity. By displaying the graphic 6px period calculated depth map terrain bas obtain a higher spatial resolution. Note that in the 3D model accurately reconstruct the thigh camera. In all cases, the depth error is less than 8 mm.
Failures
Since the MSL is a local windowing estimation technique, the depth of the depth edges thus calculated is smoothed, resulting in adhering to the object boundary (see FIG plane scene 9). High performance texture objects and the complex geometry (e.g., fine structure) will result in violation of the assumption of a constant partial reduced. Second, guidance MSL assumed albedo window is scaled version of the image under ambient light. If the ambient lighting, a projector or lighting or reflecting surface normal spectrum vary widely, thereby resulting in artifacts, this assumption does not hold. Third, the MSL dependent on the intensity of the sub-pixel accuracy - disparity, but is susceptible to indirect illumination, can not work well at each reflection or subsurface scattering (see FIG. 10).
9. Discussion
We propose a new SL technology, it can narrow the baseline, simple, low-cost hardware to run under restrictions and low computing power. By the projection of the camera corresponding to the linear equation, we demonstrate the use of partial least-squares method may be effective depth estimation. It provides a theoretical and practical guidance for the design of the projection pattern. MSL depth calculation can be performed with limited hardware, making it an ideal imaging distance on the phone, UAVs, micro robot and an endoscope.