Monocular 3D Object Localization Using 2D Estimates for Industrial Robot Vision System

In determining the precise position of the object, we initiate the process by establishing real-world coordinates through the capture of a checkerboard image, enabling the identification of its corners. Subsequently, for each object present in the image, we leverage an advanced object detection method to discern their respective image coordinates. In the context of detecting checkerboard corners, we establish the correlation between the image coordinates and their corresponding real-world coordinates. Notably, the relationship between distances in images and their counterparts in the real world is not always linear due to distortion. Figure 4 delineates the sequential steps employed to ascertain the initial object position:

• Detect Checkerboard Edges and Corners: Utilizing the Hough transformation, we identify checkerboard edges and the coordinates of their intersections, namely, checkerboard corners, as depicted in Figure 4a.

• Select Fixed Real-World Coordinate: Choose a fixed real-world coordinate in pixel image space, denoted as O_r = [x₀ y₀]_T.

• Estimate Object Center Position: Employing object detection, estimate the center position in pixel image space, represented as P = [x y]^T.

• Calculate Object Position in Real-World Coordinates: Compute the object’s position in real-world coordinates using the formula P_r = (P – O_r)r signifies the average ratio between the length in pixels and the length in the real world of each cell in the checkerboard pattern.

Figure 4.

The progress of the calculation of the object position in the real-world coordinate

While utilizing the checkerboard calibration method to estimate the location of objects proves to be straightforward and readily applicable in industrial settings, its efficacy diminishes significantly when dealing with 3D-shaped objects. In instances involving such objects, the bounding box generated through deep learning methods may not align accurately with the true object location.

Specifically, the coordinates of the bounding box’s center are unlikely to correspond to the actual center of the object within the resultant image. It’s crucial to note that the bounding box’s center can accurately represent the object’s center only if the object is precisely positioned at the center of the camera’s projection onto the floor. To address this limitation, we employ a convolutional neural network (CNN)based object segmentation approach to divide objects, detected in Section 3.1, into two distinct planes. This segmentation involves using the upper plane S_u to establish the initial center of the object, followed by noise reduction and edge extraction from the lower plane.

The initial step involves applying a Bilateral filtering operation, which can be expressed mathematically as: 1I′(p)=1Wp∑q∈ΩI(q)·Gσs(‖p−q‖)·Gσr(‖I(p)−I(q)‖){I^\prime }(p) = {1 \over {{W_p}}}\sum\limits_{q \in \Omega } I (q)\cdot{G_{{\sigma _s}}}\left( {p - q} \right)\cdot{G_{{\sigma _r}}}\left( {I(p) - I(q)} \right)

where:

• -

  I'(p) is the filtered intensity at pixel p.

• -

  W_p is the normalization term.

• -

  I(q) is the intensity at pixel q.

• -

  G_σs is the spatial Gaussian kernel with standard deviation σ_s.

• -

  G_σr is the range Gaussian kernel with standard deviation σ_r.

• -

  Ω is the spatial neighborhood of the pixel p.

Here, k is the size of the kernel, and σ is the standard deviation of the Gaussian distribution.

After that, the edge detector includes gradient computation, non-maximum suppression, and edge tracking by hysteresis. The magnitude of the gradient (G) and the gradient direction (θ) are calculated as follows: 2G(i,j)=Gi(i,j)2+Gj(i,j)2G(i,j) = \sqrt {{G_i}{{(i,j)}^2} + {G_j}{{(i,j)}^2}} 3θ(i,j)=arctan(Gj(i,j)Gi(i,j))\theta (i,j) = \arctan \left( {{{{G_j}(i,j)} \over {{G_i}(i,j)}}} \right) where G_i and G_j are the partial derivatives of the image. After obtaining the gradient magnitude, nonmaximum suppression is applied to thin the edges. Finally, edge tracking by hysteresis involves setting two thresholds, T_high and T_low. Any edge pixel with a gradient magnitude above T_high is considered a strong edge, and pixels connected to strong edges and with a magnitude above T_low are considered weak edges.

Then we divide the edge into two main border lines: the upper line (E_u) and the lower line (E_l). The upper line is the contact line between the two planes S_u and S_l, and the lower line is the boundary of the lower plane S_l. Figure 5 illustrates the progress of object segmentation and edge extraction.

Figure 5.

The progress of object segmentation and edge extraction

We introduce a calibration technique designed for object position calibration. Upon completion of the object segmentation and edge extraction processes, we obtain crucial components: the initial position of the object (center of the upper plane S_u), the upper line E_u, and the lower line E_l. The translation vector V_ul is then calculated using Algorithm 1 to predict the final position of the object by shifting the initial point according to V_ul. In this process, each point on the upper line E_u calculates its distance to the lower line S_l, determining the smallest vector length. The visualization results are depicted in Figure 6a. Subsequently, the translation vector V_ul is defined as the vector with the maximum length within the set of distance vectors for each point, as illustrated in Figure 6b.

Figure 6.

Illustration of the estimated translation vector

This calibration technique facilitates accurate object positioning by accounting for the spatial relationships between the upper and lower components. However, the efficacy of Algorithm 1 can be compromised by suboptimal segmentations. To mitigate this issue, we introduce Algorithm 2 as a solution. In essence, if the vector V_ul derived from Algorithm 1 is accurate, then upon projecting the upper line E_u along the translation vector, the distance between the resultant line Ŝ_u and the lower line S_l should be approximately, or equal, to 0.

Leveraging this concept, we construct the set of neighboring vectors 𝕍 by maintaining the initial point of the vector V_ul unchanged and selecting k neighboring points for the terminal point. Subsequently, we project the upper line following each translation vector in 𝕍 and choose the vector that results in the smallest distance.

This novel algorithm serves to enhance the robustness of the calibration process against suboptimal segmentations, ensuring more reliable results.

To evaluate the proposed method for estimating the location of objects, we conducted experiments using a vision system with a camera positioned above, parallel to the floor, and at a distance of 40cm. The real-world coordinates were defined using a checkerboard pattern, where each square was measured 3 × 3cm². Further details about the experimental setup, including hardware specifications, are provided in Table 1. In our experiments, we used a specific type of object to evaluate the performance of our localization algorithm. The object selected for this study is a standard electrical charger, commonly found in households and industrial settings. This object was chosen due to its well-defined shape and easily recognizable features, which facilitate accurate detection and segmentation. The dimensions of this object are 4.5cm × 3.0cm × 3.5cm. The dimensions, shape, and sharpness of the localized object significantly impact the performance of the localization algorithm. The object’s well-defined edges and distinct features enable the deep learning models to accurately detect and segment it from the background. The size of the object ensures that it is neither too small to be overlooked by the detection model.

The rectangular shape with flat surfaces and sharp edges facilitates precise boundary detection during the segmentation process. The clear and distinct contours of the object enhance the segmentation accuracy, leading to better calibration results.

The evaluation metrics employed for a comprehensive assessment are Intersection over Union (IOU) for object segmentation, Mean Average Precision (mAP) for object detection, and the Euclidean metric for geometric accuracy.

Table 2 presents the results of various detection models evaluated in terms of mean average precision – mAP, model size – MS (MB), precision – Pr, and recall – Rc. The Yolov3 [33] model achieves a mAP of 90.0% with a model size of 8.7 MB, accompanied by precision and recall scores of 85.9% and 84.6%, respectively. In contrast, the Fast R-CNN [34] model demonstrates superior performance with a mAP of 97.0% despite a larger model size of 12.9 MB, achieving precision and recall scores of 93.4% and 42.1%, respectively. The MobileNet [35] model offers a mAP of 94.8% with a relatively compact model size of 4.6 MB, achieving precision and recall scores of 93.8% and 93.4%, respectively. The Yolov4 [36] and RTMDet [37] models exhibit competitive mAP scores of 96.8% and 96.9%, respectively, with larger model sizes of 60.0 MB and 52.3 MB. The Yolov7 [38] and Yolov8 [39] models can achieve superior accuracy with a mAP of 97.1% and 97.8%, respectively, with a precision of 95.7% and 95.5%, and a recall of 93.1% and 94.4%, respectively. Our proposed model outperforms the others with an mAP of 98.7%, precision of 98.6%, and recall of 97.0%.

Additionally, our model exhibits a relatively compact size of 7.0 MB compared to other models, indicating its efficiency in terms of memory usage. These results underscore the effectiveness and efficiency of our proposed object detection model for accurately detecting objects in various scenarios, making it well-suited for practical deployment in real-world applications.

A comparative analysis of various object segmentation models is presented in 4. Among the models evaluated, Yolov5 [40] emerges as a strong contender, showcasing a notable mAP@0.5 score of 98.7%, a precision rate of 97.1%, and a recall rate of 96.2%. These metrics indicate its robust ability to accurately identify objects within images while maintaining a relatively compact model size of 7.4 MB, making it an efficient choice for resource-constrained environments. In addition, Yolov7 [38] and Yolov8 [39] demonstrate a commendable performance with high mAP scores of 99.0% and 99.2%, respectively, along with impressive precision and recall values. However, what sets our proposed model apart is its exceptional performance across all metrics. With an outstanding mAP@0.5 score of 99.8%, precision rate of 99.1%, and recall rate of 97.9%, our model surpasses all others in terms of segmentation accuracy while maintaining a moderate model size of 28.9 MB. These results underscore the efficacy of our segmentation model in accurately delineating objects within images.

Figure 7 visually illustrates the experimental results of our proposed method, including object detection, object segmentation, and object calibration. These visualizations provide a comprehensive insight into the efficacy and accuracy of each phase of our methodology. Object detection showcases the ability of our model to accurately identify and localize objects within the scene, laying the foundation for subsequent processing steps. Object segmentation highlights the precision with which our algorithm delineates the boundaries of detected objects, ensuring accurate localization and analysis. Finally, object calibration visually demonstrates the refinement and optimization of object positions based on real-world coordinates, validating the effectiveness of our calibration approach in enhancing spatial accuracy.

Figure 7.

Visualized examples of experimental results: figure (b): the orange point is the Yolo center, figure (d): dark red is the upper part center. The vector created by the blue points is a translation vector; the light blue point is the correction center

The experimental results in Table 3 offer quantitative insights into the performance comparison between our proposed method, the traditional approach, and the Regression-based method [30], our previous method [30] (Regression-based method). Across multiple folds and samples, our method consistently demonstrates superior performance in terms of position error metrics.

For instance, in Fold 1, Sample 1, the traditional method yields position errors of Δx = 9.69 mm and Δy = 5.51 mm, while our proposed method achieves significantly lower errors of Δx = 0.38 mm and Δy = 1.14 mm before correction, and Δx = 0.95 mm and Δy = 0.76 mm after correction.

The average position errors across all folds and samples further highlight the effectiveness of our proposed method. On average, our method achieves position errors of Δx = 1.12 mm and Δy = 0.94 mm after correction, compared to Δx = 8.30 mm and Δy = 8.96 mm for the traditional method, which reduces the position error by 87.64%. Similarly, the regressionbased method yields average errors of Δx = 1.43 mm and Δy = 1.86 mm, indicating a noticeable improvement over the traditional approach but still inferior to our proposed method.

These quantitative results underscore the significant reduction in position errors achieved by our proposed method compared to both traditional and regression-based approaches. The superior accuracy and precision offered by our method are particularly advantageous in applications where precise object localization is paramount, such as robotic manipulation, augmented reality, and autonomous navigation systems.

The processing time for each phase of our proposed method is summarized in Table 5. In the object detection phase, our algorithm takes approximately 15 ± 2 milliseconds to detect objects within the camera’s field of view. Subsequently, during the object segmentation phase, which involves segmenting the detected objects into upper and lower planes, the algorithm also requires around 40±5 milliseconds. Finally, in the object calibration phase, where the precise position of the objects is determined based on the segmented data, the processing time remains consistent at approximately 300 ± 10 milliseconds. This efficient processing time across all phases underscores the real-time applicability and practical feasibility of our proposed method for object localization in industrial vision systems.