Abstract:Most commercially available optical see-through head-mounted displays (OST-HMDs) utilize optical combiners to simultaneously visualize the physical background and virtual objects. The displayed images perceived by users are a blend of rendered pixels and background colors. Enabling high fidelity color perception in mixed reality (MR) scenarios using OST-HMDs is an important but challenging task. We propose a real-time rendering scheme to enhance the color contrast between virtual objects and the surrounding background for OST-HMDs. Inspired by the discovery of color perception in psychophysics, we first formulate the color contrast enhancement as a constrained optimization problem. We then design an end-to-end algorithm to search the optimal complementary shift in both chromaticity and luminance of the displayed color. This aims at enhancing the contrast between virtual objects and the real background as well as keeping the consistency with the original color. We assess the performance of our approach using a simulated OST-HMD environment and an off-the-shelf OST-HMD. Experimental results from objective evaluations and subjective user studies demonstrate that the proposed approach makes rendered virtual objects more distinguishable from the surrounding background, thereby bringing a better visual experience.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the lack of visual clarity caused by color mixing between virtual objects and the real background in optical see - through head - mounted displays (OST - HMDs). Specifically, since OST - HMDs transmit the physical background and virtual objects simultaneously through an optical combiner, the image perceived by the user is actually a mixture of rendered pixels and background colors, which makes it a challenge to achieve high - fidelity color perception in mixed - reality (MR) scenes. Therefore, the paper proposes a real - time rendering scheme, aiming to enhance the color contrast between virtual objects and the surrounding background, thereby improving the identifiability of virtual content and enhancing the user's visual experience.
### Key Points of the Paper
1. **Problem Definition**:
- **Color Mixing Problem**: Due to the optical properties of OST - HMDs, the colors of virtual objects will mix with the background colors, causing virtual content to be difficult to observe clearly in low - contrast situations.
- **Limitations of Existing Solutions**: Hardware solutions usually involve complex optical components, and software solutions such as subtractive compensation may reduce the brightness of virtual content and decrease its contrast with the background.
2. **Method Overview**:
- **Color Contrast Enhancement**: The paper proposes a method based on constrained optimization to enhance the color contrast between virtual objects and the background by adjusting the chromaticity and brightness of the display colors.
- **Constraints**:
- **Color Difference Constraint**: Ensure that the difference between the optimized color and the original color is within an acceptable range.
- **Chromaticity Constraint**: Prevent the reduction of chromaticity during the optimization process.
- **Brightness Constraint**: Avoid the optimized color from being too low or too high in brightness, which may affect the visibility and surface contrast of virtual objects.
- **Just - Noticeable - Difference Constraint**: Ensure that the difference between the optimized color and the background color is sufficient to be perceived by the human visual system (HVS).
3. **Algorithm Steps**:
- **Pre - processing**: Perform Gaussian blurring and field - of - view calibration on the background video to extract spatial color information and reduce the flickering caused by high - frequency details.
- **Color Space Conversion**: Convert the display colors and background colors from the RGB space to the LAB space, taking advantage of the perceptual uniformity of the LAB space and the independence of chromaticity and brightness.
- **Optimization**: Find the optimal display color under the satisfaction of the above four constraint conditions.
### Formula Analysis
- **Color Difference**:
\[
\Delta E^*_{ab}(x, y)=\sqrt{(L^*_x - L^*_y)^2+(a^*_x - a^*_y)^2+(b^*_x - b^*_y)^2}
\]
where \(L^*\), \(a^*\), \(b^*\) represent the brightness and chromaticity components in the LAB color space respectively.
- **Color Optimization**:
- **Ideal Optimal Color**:
\[
I =-\frac{B}{\text{dist}(B, O)}=-\text{norm}(\overrightarrow{OB})
\]
where \(B\) is the coordinate of the blurred background color, and \(\text{dist}(B, O)\) is the distance from \(B\) to the center \(O\) of the unit sphere.
- **Color Difference Constraint**:
\[
\overrightarrow{DE}=\min(\text{dist}(D, I), \lambda'_E)\cdot\text{norm}(\overrightarrow{DI})
\]
where \(\overrightarrow{DI}\) is the change vector from the original display color \(D\) to the ideal optimal color \(I\), and \(\lambda'_E\) is the user - specified scaled color difference threshold.
- **Chromaticity Constraint**:
\[
\overrightarrow{DC}=t_{ch}\cdot\overrighta