Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching

Abstract

Vision-based underwater autonomous systems play a significant role in marine exploration. Stereo matching is one of the most popular applications for vision-based underwater autonomous systems, which recovers the geometric information of underwater scenes via stereo disparity estimation. While stereo matching in the air has achieved great progress with the development of neural networks, it generalizes poorly to the underwater scenario due to the challenging underwater degradation. In this paper, we propose a novel Multilevel Inverse Patchmatch Network (MIPNet) to iteratively model pair-wise correlations under underwater degradation and estimate stereo disparity with both local and global refinements. Specifically, we first utilized the inverse Patchmatch module in a novel multilevel pyramid structure to recover the detailed stereo disparity from the input stereo images. Secondly, we introduced a powerful Attentional Feature Fusion module to model pair-wise correlations with global context, ensuring high-quality stereo disparity estimation for both in-air and underwater scenarios. We evaluate the proposed method on the popular real-world ETH3D benchmark, and the highly competitive performance against the popular baselines demonstrates the effectiveness of the proposed method. Moreover, with its superior performance on our real-world underwater dataset, e.g., our method outperforms the popular baseline RAFT-Stereo by 27.1%, we show the good generalization ability of our method to underwater scenarios. We finally discuss the potential challenges for underwater stereo matching via our experiments on the impact of water.

1. Introduction

Stereo matching aims at recovering a scene’s geometric information via stereo disparity estimation, which is a long-standing vision-based task for many underwater autonomous systems [1,2] and applications [3,4,5]. The early rectified stereo matching method focuses on finding the per-pixel disparity along the horizontal baseline in a rectified binocular image pair in a handcrafted manner. This standard pipeline [6] can be roughly divided into four parts: per-pixel matching cost computation, cost aggregation, disparity optimization, and disparity refinement. Despite their high efficiency, these traditional methods are not able to achieve satisfactory results on recent challenging benchmarks.

Recent stereo matching techniques have achieved tremendous progress in the air environment with the rapid development of deep learning. Although the recent advanced network design brings performance gains to some extent, the complex-designed module as well as the larger parameters usually limit their underwater applications due to the high computational and memory costs [7]. To address this limitation, some works propose to treat stereo matching as a special case of optical flow estimation with horizontal line constraints, inspiring the recent popular iterative-based methods that show notable performance with high efficiency. These methods can be roughly classified as local-based methods or global-based methods according to their means of modeling the pair-wise correlation.

The global-based methods, such as RAFT-Stereo [8], which computes all-pair correlations in a global view, utilize a multilevel structure to update the correlations with manageable memory consumption for standard input image sizes. However, it remains a major challenge, especially for underwater scenarios, to process high-resolution images. The local-based methods, such as the CREStereo [9], leverage local adaptive correlation to predict the stereo disparity. Nevertheless, their results are sometimes unsatisfactory under the conditions of occlusion or textureless cases, especially for underwater scenarios [10]. The learning-based underwater stereo matching is rarely studied, and it remains challenging to directly apply the existing in-air stereo matching models to underwater scenarios due to the complex conditions of underwater degradation, occlusions, textureless cases, and low illumination. Moreover, due to the irreconcilable difference between the features in-air and underwater, it is also difficult for the pretrained in-air model to extract underwater features directly, which may result in magnified local defects.

To address these limitations and explore the stereo matching approach for both in-air and underwater purposes, we propose to model accurate pixel correspondences from the perspective of Patchmatch [11]. Generally, a Patchmatch algorithm should contain three basic parts: random initialization, propagation, and random local search. After random initialization, the initialized correlations are passed to their neighbors by propagation according to the law of large numbers. Afterwards, the random search is adopted to refine the correlations. For robust underwater stereo matching with high efficiency, in this paper, we present the novel Multilevel Inverse Patchmatch Network (MIP-Net) that iteratively leverages the inverse propagation strategy [12] in a multilevel pyramid structure to model the pair-wise correlation for stereo matching. With a novel Attentional Feature Fusion module, we iteratively extract deep features with the ResNet [13] encoder to progressively increase the receptive field for underwater feature extraction. The Patchmatch module, which consists of an inverse propagation module and a rectangular local search block, operates on different levels of resolution to update the disparity in a coarse-to-fine manner and output the refined-level prediction, ensuring a robust disparity search to deal with underwater degradation, occlusions, textureless features, and low illumination. The overall structure of our network is illustrated in Figure 1.

Different from most existing methods, our approach has two major advantages. On the one hand, it only performs correlation on nearby pixels iteratively, instead of constructing a global matching cost, which will reduce memory costs significantly. On the other hand, we update the correlation by propagation, and a local random search is performed only for refinement, avoiding the defects caused by using a local search. The highly competitive results on the public benchmark ETH3D [14] and the real-world underwater dataset demonstrate the effectiveness as well as the good generalization ability of our proposed method for underwater stereo matching research.

2. Related Works

2.1. Stereo Matching in Atmosphere

When developing stereo matching methods, measuring per-pixel matching costs and aggregating them is extremely important. Most impactful works make great contributions to the development of accurate and economic volumes. In the era of traditional methods, there have been many inspiring ideas, such as using a support window centered at pixels to compute the matching cost [15,16] and using belief propagation to formulate the cost function [17,18]. Patchmatch was also introduced at this time [19].

After Zbontar and LeCun [20] trained a convolutional neural network to evaluate the matching score, the performance of the algorithm improved rapidly. Mayer et al. proposed the first end-to-end network, DispNet/ DispNetC [21], based on 2D convolutions. Kendall et al. [22] constructed the first 3D architecture, which aims to act as a differentiable approximation to classical filtering algorithms. All these methods achieved state-of-the-art performance at that time, but early end-to-end networks are plagued either by poor generalization ability or high computation costs. To solve these issues, Guo et al. [23] designed group-wise correlation to improve similarity measurement and save memory at the same time. Yang et al. [24] proposed a hierarchical coarse-to-fine architecture based on pyramidal approaches to deal with high-resolution images. Shen et al. [25] constructed a pyramid cost volume to cover multi-scale receptive fields and a 3D warping volume to refine the initial disparity map. Moreover, networks based on optical flow estimation were introduced into stereo matching at this time to deal with the above issues. Tankovich et al. [26] guided the stereo predictions using predicted tiles to leverage the planar geometry of the scene as an inductive prior in the network design. Lipson et al. [8] constructed a 3D correlation volume and introduced a multilevel update operator to provide robust disparity predictions and save memory by avoiding calculation operating at high resolutions. Li et al. [9] introduced an adaptive group correlation layer to be more memory-efficient and overcome imperfect calibrations for real-world stereo cameras. They also constructed a synthetic dataset that focuses on challenging cases in real-world scenes and features various enhancements. Moreover, Cheng et al. [27] introduced the Neural Architecture Search to reduce the human effort in neural network design.

Another line of work focuses on improving the disparity or matching cost. Most traditional methods and early machine learning methods within the stereo pipeline use individual steps such as a consistency check [28] to obtain more accurate disparity maps at the disparity dimension. After the emergence of end-to-end networks, there were many works focusing on optimizing other dimensions. Liu et al. [29] designed a feature fusion module based on the Local Similarity Pattern to provide higher-quality feature maps. Xu et al. [30] used deformable convolution to enhance the standard convolution’s capability of modeling geometric transformations. This strategy is also used in [9,29], where a Cost Self-Reassembling module and a deformable search window are designed, respectively. Patchmatch was also introduced into a 3D architecture by DeepPruner [31] to improve the distinctiveness of the matching cost generated by the backbone. Xu et al. [32] proposed multilevel adaptive patch matching to generate reliable attention weights, which aim to suppress redundant information and enhance matching-related information in the concatenation volume. However, this is only a refinement method so a high memory cost is still unavoidable.

2.2. Stereo Matching under Water

Compared to stereo matching in the atmosphere, underwater stereo matching is much less developed. Xu et al. [7] used zero-based normalized cross-correlation and the Hamming distance to measure similarity and designed the smoothness term based on the color metric to overcome the discontinuity of the disparity map. Zhuang et al. [10] proposed a new method to express underwater images via a direction-information image and position image, and an image optimization method that fuses color correction and dark channels in advance.

In general, most works chose to improve the performance by using extra techniques, such as camera calibration [33], instead of the algorithms themselves. Only a few of them focused on improving the algorithms by using the unique characteristics of images from underwater scenarios, and most of them are traditional methods or early machine learning methods within the stereo pipeline. Therefore, our method based on an end-to-end network is certainly innovative for underwater stereo matching.

3. Methods

The main contribution of our method is that we have customized various components in our network for underwater scenarios.

3.1. Inverse Patchmatch for Stereo Matching

Calculating correlations by Patchmatch is the most crucial step in our method. To better adapt to underwater stereo matching, we first integrated prior experience obtained by observing public datasets while initializing the disparity field. After this, we introduced a process named “Inverse Propagation” [12], which was first developed for optical flow estimation. During this process, we added a vertical constraint to maintain the disparity predictions on the horizontal line. Moreover, due to the unequal states between the vertical and horizontal directions, we applied a rectangular local search window to refine the correlation computed in the inverse propagation stage.

3.1.1. Gamma Initialization

According to the law of large numbers, there are always some good estimates when initializing the disparity field using a large amount of random numbers. In theory, we can use any distribution to generate a random disparity field as long as the appropriate range is covered. However, providing more good estimates will help the network to propagate more correct information during the early training stage, which may allow the network to converge faster. According to the statistics in [34], the proportion of disparity in most public datasets is not uniform. Considering the shape of the distribution, we use gamma distribution to initialize, which is different from [12,31]. Eventually, our final model only needs 6 iterations to obtain very competitive results.

3.1.2. Inverse Propagation for Stereo Matching

The key idea of propagation is to pass the information to its neighbors. In this work, the information that we need to pass is the matching cost under the current disparity prediction. Specifically, we regard the element-wise multiplication of two feature maps as the correlation. Given a disparity prediction d, the target feature map should be wrapped according to the shift in disparity prediction, so the correlation during the propagation process can be formulated by

$$C=F_1·\mathcal{W}(F_2,d+\Delta d)$$

where C is the 4-dimensional correlation cost volume, $F_1$ and $F_2$ are feature maps, $\mathcal{W}$ is the warping operation, and d and $\Delta d$ are the current disparity field and shift disparity. It is noteworthy that we added the extra constraint on the vertical direction, so the disparity prediction d will only fall on the horizontal line.

However, as mentioned in [12], when choosing to receive information from n neighbor seed points and recurrently update it m times, there will be $n\times m$ shifting and warping operations. Therefore, to overcome the computation cost caused by traditional propagation, inverse propagation was proposed. Differing from the original process, inverse propagation directly shifts the feature map with the shift disparity prediction $\Delta d$ before warping the shifted feature map and offsets the shift disparity prediction later. The process can be formulated as

$$C=F_1·\mathcal{S}(\mathcal{W}(\mathcal{S}(F_2,\Delta d),d),-\Delta d)$$

where $\mathcal{S}$ represents the shift operation.

This is reasonable because disparity shift and feature warping are also serial and coupled, which means that inverse propagation should be equivalent to the original propagation when considering obtaining a correlation, but can avoid many operations in the intended design. As a target feature point is scattered to its seed points, the target features can be shifted and stacked in advance. Moreover, since $\Delta d$ is very small in each iteration, if we ignore the backpropagation, the inverse propagation process will become

$$C=F_1·\mathcal{W}(\mathcal{S}(F_2,\Delta d),d)$$

and we can only perform a warping operation once during each iteration. The process of inverse propagation is illustrated in Figure 2.

In our implementation, we followed [12]; we chose 4 static seed neighbors to balance the performance and complexity. In our final model, we only need to shift target features 4 times and warp the shifted target features 5 times to obtain a very accurate correlation.

3.1.3. Rectangular Local Search Window

In order to pursue state-of-the-art amphibious performance, we need to refine the local correlation additionally. We also performed a local neighborhood search with a fixed radius. This radius is much smaller than the size of the images, so that it will be very memory-efficient. Unlike [9,12], we developed a rectangular local search window to emphasize the unequal states between the vertical and horizontal directions. In general, the rectangular local search can be formulated as

$$C=F_1·\mathcal{S}(\mathcal{W}\left(F_2\right),\Delta d)$$

and the rectangular local search block is illustrated in Figure 3.

Thus far, we have introduced the components of inverse Patchmatch for stereo matching. In the following sections, we will introduce the overall architecture of our network, including the feature extraction and multilevel update pyramid.

3.2. Network Architecture

Our network is composed of an encoder and an iterative decoder. In our encoder, we added an iterative Attentional Feature Fusion module to improve the performance in underwater scenarios by providing feature maps with higher quality. As for the decoder, it is composed of a series of inverse Patchmatch modules operating at different resolutions.

3.2.1. Feature Extraction with Global Context

The aim of our encoder (Figure 4) is to map the rectified binocular image pairs to dense feature maps so that our iterative decoder can use them to predict the disparity field. In [8], the authors introduced two types of feature extractors, the feature encoder and context encoder, where the former is used to compute the correlation and the latter is used to initialize the following modules and provide the information of the relationship in context to avoid local noise. As in many previous works [8,9,23], we used residual blocks [13] to extract these two types of features and then construct a feature pyramid, but we did not capture them separately. Instead, we follow [9,12], where the context is split from part of the feature map. This will slim the module and reduce computation and memory costs.

Meanwhile, reflection, scattering and other phenomena can cause much more noise. Inspired by [29], we use more context information to refine the features obtained by the residual block and the solution that we adopt is to increase the receptive field while obtaining features. Most works chose to fuse multi-scale features via interpolation and element-wise addition, but we use element-wise adaptive fusion weight to conceal low-level noise. More specifically, we adopt the two-stage iterative Attentional Feature Fusion proposed by [35], which has proven its ability in classification tasks. The element-wise adaptive fusion weight is generated by a multi-channel attention module. We have global channel context $\mathbf{g}\left(\mathbf{X}\right)$ and local channel context $\mathbf{L}\left(\mathbf{X}\right)$, which are computed by

$$\mathbf{L}\left(\mathbf{X}\right)=\mathcal{B}\left(PWCon{v}_2\left(\delta \left(\mathcal{B}\left(PWCon{v}_1\left(\mathbf{X}\right)\right)\right)\right)\right)$$

$$\mathbf{g}\left(\mathbf{X}\right)=\mathcal{B}\left(W_2\delta \left(\mathcal{B}\left(W_1\left(g\left(\mathbf{X}\right)\right)\right)\right)\right)$$

where X is an input feature, $\mathcal{B}$ represents a bottleneck structure, PWConv1 and PWConv2 are point-wise convolution, $g\left(\mathbf{X}\right)=\frac{1}{H\times W}{\sum }_{i=1}^H{\sum }_{j=1}^W{\mathbf{X}}_{[:,i,j]}$ is the global average pooling, $\delta$ is the Rectified Linear Unit [36], $\mathcal{B}$ represents Batch Normalization [37], and $W_1$ and $W_2$ are two fully connected layers. Then, the feature is refined by

$${\mathbf{X}}^{\prime }=\mathbf{X}\otimes \sigma (\mathbf{L}\left(\mathbf{X}\right)\oplus \mathbf{g}\left(\mathbf{X}\right))$$

where ⊗ represents the element-wise multiplication of the matrix and ⊕ represents the broadcasting addition. Therefore, after the above operations, the element-wise adaptive fusion weight used to refine and fuse features can be obtained.

Thus far, we have provided the weight to refine features while they are fusing. However, ref. [35] points out that it is important to consider how to inject features into the attention module, so an iterative approach is proposed. It is composed of two intertwined multi-channel attention modules so that the weights when injecting into the attention module can be updated iteratively. In general, the iterative attentional fusion can be formulated as

$$\mathbf{Z}=\mathbf{M}(\mathbf{X}\uplus \mathbf{Y})\otimes \mathbf{X}+(1-\mathbf{M}(\mathbf{X}+\mathbf{Y}\left)\right)\otimes \mathbf{Y}$$

where $\mathbf{X}\uplus \mathbf{Y}=\mathbf{M}(\mathbf{X}+\mathbf{Y})\otimes \mathbf{X}+(1-\mathbf{M}(\mathbf{X}+\mathbf{Y}\left)\right)\otimes \mathbf{Y}$, $\mathbf{M}$ is the multi-channel attention module mentioned above.

3.2.2. Multilevel Update Pyramid

Our decoder is a multilevel update pyramid that consists of a series of inverse Patchmatch modules. In each inverse Patchmatch block, the inverse propagation block and rectangular local search block are followed by corresponding cross-connected GRUs [38]:

$$z_t=\sigma \left(Con{v}_{3\times 3}([h_{t-1},x_t],W_z)\right)$$

$$r_t=\sigma \left(Con{v}_{3\times 3}([h_{t-1},x_t],W_r)\right)$$

$$h_t=tanh\left(Con{v}_{3\times 3}([r_t\otimes h_{t-1},x_t],W_h)\right)$$

$$h_t=(1-z_t)\otimes h_{t-1}+z_t\otimes h_t$$

For one inverse Patchmatch block, there is one inverse propagation block and one rectangular local search block with their corresponding cross-connected GRUs concatenated to each other. After the correlation C is obtained by the inverse propagation block or rectangular local search block, it is concatenated with the current disparity field and context feature separated from the output of the encoder and then injected into the GRU to predict the shift disparity. After several iterations of updates, the predicted disparity estimation will be upsampled [38] to a high resolution level and launch another update process. Eventually, after gamma initializing at a 1/16 resolution, the disparity field is updated 6 times and upsampled to a 1/4 resolution for another 6 iterations.

3.3. Loss Function

Following [8,9], we supervise the full sequence of predictions.

$$\mathcal{L}=\sum _{i=1}^N{\gamma }^{N-i}{\parallel d_{gt}-d_i\parallel }_1,\gamma =0.8$$

where $d_{gt}$ represents the ground truth of the disparity, $d_i$ represents the estimated disparity of each iteration, and N is the number of GRU iterations.

4. Experiment

All models mentioned in the following sections were implemented by Pytorch and trained on four RTX2080Ti GPUs with 11 GB memory using the AdamW optimizer and a one-cycle learning rate schedule [39]. We performed a two-stage pretrain process on the synthetic datasets proposed in [9], and the ablation study was implemented on the Sceneflow [21] cleanpass subset. Moreover, we evaluated the method on the ETH3D [14] benchmark and our hand-crafted underwater dataset. All experiments in this section, excluding the results submitted to the benchmark, are described with the mean of three independent realizations.

4.1. Dataset and Metrics

Synthetic dataset. The models used to submit to the public benchmark and compare with other baselines are pretrained on the synthetic datasets proposed in ref. [9]. It is composed of approximately 200K image pairs and each image pair is captured by dual virtual cameras and customarily positioned objects, using Blender to generate the synthetic dataset. This dataset addresses the generalization from synthetic datasets to real-world scenarios, which enables us to pursue better performance on public benchmarks and underwater scenarios. As for the ablation study, we chose another synthetic dataset, Sceneflow [21]. It consists of three subsets: FlyingThings 3D, Monkaa and Driving. This dataset covers a wide range of scenarios, including common objects, animated short films and street scenes. The cleanpass subset that we used consists of around 15K image pairs. The metric that we monitor is the end-point-error (EPE), calculated by $\frac{1}{N}{\sum }_{i=1}|d_i-d_i^{*}|$, and the 1-pixel error (>1 px), as a percentage of the points with EPE larger than 1.

ETH3D. ETH3D [14] is a real-world dataset that covers indoor and outdoor scenes. It is recorded by a high-precision laser scanner and captured by DSLR cameras. The dataset for stereo matching consists of a training set with 27 image pairs and a test set with a size of 20 image pairs. The default metric on the benchmark is also the 1-pixel error, but it also provides other metrics, such as 90% error quantiles (A90) and the root-mean-square disparity error (rms) in pixels.

Underwater dataset. Our underwater dataset consists of 49 high-resolution ($1920\times 1080$) image pairs. The images were captured via a ZED camera and we also used a high-precision laser scanner to generate ground truth disparity. To simulate real-world underwater scenarios, some images were taken under a low-light condition, which is similar to the situation in the deep ocean, while others were taken in more turbid liquid, which simulates nearshore water. We compared our results with other baselines on similar metrics in ETH3D.

Some samples from the above datasets are shown in Figure 5.

4.2. Data Augmentation

Appropriate data augmentation techniques can significantly improve the performance of networks. In this work, we followed RAFT-Stereo [8] and applied techniques such as changing the image saturation, perturbation and stretch before feeding the data into our network. Specifically, we adjusted the image saturation of both left and right images between 0 and 1.4 and the right images were perturbed to simulate imperfect rectification. Additionally, to cover a broader disparity range in the pretraining process, the images and ground truth disparity were stretched by a factor in the range $[2^{-0.2},2^{0.4}]$. One sample after data augmentation is shown in Figure 6.

4.3. Ablation Study

Vertical constraint. As shown in

Table 1. Ablation on the key components of the network.

Experiment	>1 px	>2 px	>4 px	EPE	Parameters
Without AFF	6.734	4.159	2.590	0.589	5.37 M
Without RLS	8.454	5.061	3.150	0.724	3.06 M
Without IP	7.061	4.308	2.655	0.609	3.67 M
Without constraint	18.867	-	-	1.260	5.37 M

AFF—Attentional Feature Fusion, RLS—Rectangular Local Search, IP—Inverse Propagation.

, if we cancel the vertical constraint, which means that we directly handle the stereo matching task as an optical flow estimation task, the result shows that the entire algorithm will fail. This implies that although stereo matching can be regarded as a special case of optical flow estimation, it is still not practical to merge these two tasks intuitively.

Attentional Feature Fusion. We implemented a very lightweight attention mechanism in our feature encoder, which aims to provide feature maps with higher quality. According to the results shown in Table 1, if we replace the attentional fusion with a traditional plus $\mathbf{Z}=\mathbf{X}+\mathbf{Y}$, where $\mathbf{X}$ and $\mathbf{Y}$ are given feature maps, the performance of our network will drop.

Inverse Propagation and Rectangular Local Search. In fact, the inverse propagation module and rectangular local search module can perform stereo matching independently, so we launched two experiments to test their impacts on the performance. The results in Table 1 show that the inverse propagation module has more parameters than the rectangular local search module and makes a greater contribution to the accuracy of the network.

Radius of Rectangular Local Search. To prove the effectiveness of the rectangular local search module, we also explored different searching radii. To balance performance and computation costs, we chose the vertical searching radius to be 1 and horizontal searching radius to be 3, according to the results shown in

Table 2. Ablation on the radius of vertical and horizontal search.

Experiment	>1 px	>2 px	>4 px	EPE
$r_x=1,r_y=1$	6.570	4.062	2.541	0.574
$r_x=2,r_y=1$	6.578	4.145	2.557	0.574
$r_x=3,r_y=1$	6.538	4.008	2.468	0.568
$r_x=4,r_y=1$	6.537	4.007	2.468	0.567
$r_x=2,r_y=2$	7.190	4.472	2.796	0.660

. Moreover, an excessive searching radius in the vertical direction will cause a significant drop in the performance.

Computation Cost. We also tested the impact of different components on the computation cost in our underwater dataset. The results of the fine-tuned model are shown in Figure 7. We can notice that the inverse propagation (IP) will occupy most of the time and memory in the process, but it is still unwise to remove any of the components. Moreover, our full model is able to handle a $1920\times 1080$ image pair in 0.53 s on a single RTX2080Ti GPU and only uses 5.54 GB. This implies that our method is very memory- and time-efficient. Additionally, to prove the efficiency of our method, we also tested our full model on an Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz CPU and the runtime was 95.36 s, which means that our method can even be applied to edge devices where resources are not sufficient.

4.4. Performance in Amphibious Scenarios

The results shown in the following sections were obtained based on our pretrained model. We launched a two-stage pretraining process on the 200K synthetic dataset in [9]. In the first stage, the network was trained with the maximum learning rate being $6\times {10}^{-4}$ and a batch size of 8. The random crop size was set to be $512\times 384$. The first stage lasted for 100K iterations. As for the second stage, the network was trained for another 100K iterations with a maximum learning rate $2\times {10}^{-4}$ and a batch size of 4. The random crop size was set to be $768\times 384$ at this stage.

We monitored the metrics in the pretraining process. As seen in Figure 8 and Figure 9, the end-point error was calculated on the training batch for each step and the valid EPE was calculated on the validation set.

We can notice that while implementing a two-stage pretraining process, the model can achieve very competitive performance at stage 1 and can still obtain additional improvements after stage 2. This phenomenon fits the design of our network.

ETH3D Benchmark

As with many other real-world datasets, ETH3D is too small to train directly on its training set, so we applied the fine-tune strategy. We fine-tuned it for 100 iterations, with the maximum learning rate being $2\times {10}^{-4}$ and a batch size of 8. The random crop size was set to be $512\times 384$. During our experiments on the validation set that we used, we noticed that although we could improve the performance using extra inverse Patchmatch iterations, it is still not worthwhile considering the time cost.

Observing the samples in Figure 10, we noticed that our method could handle the overexposed and underexposed areas extraordinarily well, which will especially benefit underwater stereo matching. Eventually, our results were compared to other baselines, as shown in

Table 3. Results compared to other baselines on ETH3D benchmark [40].

Method	>0.5 px	>1 px	>2 px	>4 px	A90	A95	A99	EPE
CRE-Stereo [9]	3.58	0.98	0.22	0.10	0.28	0.39	0.92	0.13
GM-Stereo [41]	5.94	1.83	0.25	0.08	0.39	0.55	1.01	0.19
EAI-Stereo [42]	5.21	2.31	1.14	0.70	0.34	0.65	2.35	0.21
RAFT-Stereo [8]	7.04	2.44	0.44	0.15	0.39	0.57	1.20	0.18
HITNet [26]	7.83	2.79	0.80	0.19	0.41	0.65	2.07	0.20
Ada-Stereo [43]	10.22	3.09	0.65	0.20	0.50	0.70	1.45	0.24
DeepPruner [31]	10.77	3.52	0.86	0.23	0.53	0.74	2.05	0.26
NLCA-Net [44]	12.05	3.84	1.04	0.35	0.53	0.79	2.54	0.27
HSM [24]	11.33	4.00	1.36	0.63	0.51	1.19	2.35	0.28
Ours	6.75	2.06	0.64	0.34	0.37	0.56	2.33	0.21

>1 px is the default metric.

.

4.5. Underwater Scenarios

In this section, we describe fine-tuning on the training set for 7000 steps with a batch size of 4 and the maximum learning rate set to $4\times {10}^{-4}$. The random crop size was $640\times 400$.

Different from the scenarios in the atmosphere, the underwater environment will significantly increase the difficulty for stereo matching, despite the fact that our images were captured under a high resolution. From the results shown in

Table 4. Results compared with other baselines on the test set of the underwater dataset.

Method	>3 px	>5 px	>7 px	EPE	Time
GWC-Net [23]	53.89	47.36	40.20	7.14	0.55
DSM [45]	51.13	40.20	33.01	5.67	4.81
CRE-Stereo * [9]	51.02	39.68	1.25	5.32	7.52
RAFT-Stereo [8]	51.02	39.68	1.25	5.32	7.52
Ours	50.08	27.27	17.66	3.88	0.53

* Non-official code repository.

, our method outperforms others by 27.1%, but the accuracy compared to other results on ETH3D drops significantly. It is noteworthy that all methods shown in Table 4 were fine-tuned on the training set and the inference process of all methods was operated on a single RTX2080Ti GPU.

In order to better visualize the output of our network, we transform our disparity maps into depth maps by

$$depth=\frac{b·f}{d}$$

where b is the length of the baseline, f is the focal length and d is the disparity. The final depth maps and their ground truths are also shown in Figure 11.

Moreover, to further explore the impact of water on underwater stereo matching, we launched a series of experiments with different numbers of GRU iterations. Figure 12 shows the experiments on the turbid group in our test set. In the first image pair (first row), where there are many larger objects, the performance of our network seems to be more affected by the turbid water, which leads to significant bias in the first iteration. However, for the second image pair, where there are more small objects, the bias caused by turbid water is not as significant as in the first pair but requires more GRU iterations to be erased. Meanwhile, smaller objects are more difficult to perceive, especially for underwater environments. This is why we implement the Attentional Feature Fusion module in our feature encoder.

For the low-light group in the test set shown in Figure 13, we adopted a similar strategy. In the first image pair, the objects appear very vague against the low-light environment but our network fails to distinguish them from the background. This implies the oversmoothing problem, which is very common in convolution networks. Although our Attentional Feature Fusion module can alleviate the phenomenon, it is still a great challenge for underwater stereo matching. As for the second and third image pairs, the main challenge is to recover the details of the objects. Our network shows a strong ability to deal with this issue.

Furthermore, comparing Figure 12 and Figure 13, we can notice that in different environments, the network can achieve satisfactory results with different numbers of iterations. This implies that we can adjust it in real-world practice to pursue real-time stereo matching.

5. Conclusions

In this paper, we have proposed a novel Multilevel Inverse Patchmatch Network with local and global refinement to perform underwater stereo matching. Based on the similarity between the disparity and optical flow, we utilize inverse propagation, which was originally designed to calculate the correlation for optical flow estimation. Specifically, we maintain the displacement falling on the horizontal line to satisfy the stereo matching task. Moreover, to improve the performance of our network in underwater scenarios, we customize other components, such as adding an attention mechanism for feature fusion and resizing the local search window.

Eventually, after pretraining our networks on a large synthetic dataset, we fine-tuned the pretrained models on the ETH3D dataset and our hand-crafted underwater dataset. According to the results shown on the benchmark’s official website and our dataset, our methods achieved competitive performance. Additionally, we launched a series of ablation studies to show the impact of each component on the performance of our network, as well as a series of experiments to explore the impact of water.

When analyzing the results in underwater scenarios, we discovered that although our method had already outperformed other novel methods, it still suffered from noise in the underwater environment. In the future, we plan to construct an underwater synthetic dataset where we can emphasize the impact of noise. We believe that this can significantly improve the performance. Moreover, our network also suffers from the oversmoothing problem. Inspired by [29], we plan to add cross-entropy loss to our loss function because it has been proven to be useful to overcome the oversmoothing problem [46].