Abstract:Due to the difficulty of replicating the real conditions during training, supervised algorithms for spacecraft pose estimation experience a drop in performance when trained on synthetic data and applied to real operational data. To address this issue, we propose a test-time adaptation approach that leverages the temporal redundancy between images acquired during close proximity operations. Our approach involves extracting features from sequential spacecraft images, estimating their poses, and then using this information to synthesise a reconstructed view. We establish a self-supervised learning objective by comparing the synthesised view with the actual one. During training, we supervise both pose estimation and image synthesis, while at test-time, we optimise the self-supervised objective. Additionally, we introduce a regularisation loss to prevent solutions that are not consistent with the keypoint structure of the spacecraft. Our code is available at: <a class="link-external link-https" href="https://github.com/JotaBravo/spacecraft-tta" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of performance degradation in spacecraft attitude estimation due to the domain gap between training data and test data. Specifically, the performance of supervised learning algorithms will significantly decline when they are trained with synthetic data and then applied to actual operational data. This is because it is very difficult to completely replicate the actual operational conditions during the training process. To solve this problem, the authors propose a test - time adaptation (TTA) method based on predicted - view synthesis. This method takes advantage of the temporal redundancy between consecutive images acquired during close - range operations, and achieves adaptation by extracting features, estimating attitudes, and synthesizing reconstructed views. By comparing the photometric differences between the synthesized view and the actual view, a self - supervised learning objective is established, and a regularization loss is introduced to ensure the consistency of the key - point structure. ### Method overview 1. **Attitude estimation**: - Use a key - point - based method to estimate the attitude of the spacecraft. Given an image \(I\) of the target spacecraft, first obtain the feature representation \(F\) through a feature encoder, and then regress the heatmap \(\hat{h}\) through a heatmap decoder. - Each channel in the heatmap represents the probability distribution of a key - point in 2D image coordinates. Estimate the attitude \(\hat{T}\) from the 2D - 3D correspondence through a PnP solver. 2. **Test - time adaptation**: - Achieved by jointly optimizing the supervised task and the self - supervised task. In the training phase, the two tasks are simultaneously optimized to share the feature representation; in the test phase, the model is updated through the self - supervised task to align the features of the target test domain. - The self - supervised task is based on novel - view synthesis, that is, estimating the relative attitude change \(\hat{T}_{t' \to t}\) from the images at two adjacent time points \(t'\) and \(t\), and then predicting a new view \(\hat{I}_t\) according to \(\hat{T}_{t' \to t}\). - Optimize the self - supervised objective by comparing the photometric difference \(\ell_I\) between the predicted view \(\hat{I}_t\) and the actual view \(I_t\). 3. **Heatmap regularization**: - To prevent the heatmap response from deviating from the true key - point projection, two loss terms are introduced: - PnP loss \(\ell_{pnp}\): Ensure that the position of the maximum value of the heatmap is consistent with the 2D key - point coordinates estimated by PnP. - Heatmap loss \(\ell_{\dot{h}}\): Ensure the consistency of the heatmap structure by generating a pseudo - ground - truth heatmap \(\dot{h}\) and comparing it with the predicted heatmap \(\hat{h}\). - The total regularization loss is \(\ell_{reg}=\ell_{pnp}+\ell_{\dot{h}}\). ### Main contributions 1. Proposed a self - supervised test - time adaptation framework based on novel - view synthesis, which explicitly associates the attitude estimation with the image reconstruction process. 2. Introduced the heatmap regularization loss, which avoids the degenerate solutions in the key - point estimation process and ensures the consistency of the key - point structure. Through this method, the paper aims to improve the accuracy of spacecraft attitude estimation in new test domains, especially in the presence of domain gaps.

Test-Time Adaptation for Keypoint-Based Spacecraft Pose Estimation Based on Predicted-View Synthesis

Pose Estimation and Neural Implicit Reconstruction Towards Non-Cooperative Spacecraft Without Offline Prior Information

TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation

Test-time adaptation for 6D pose tracking

Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation

Pose Estimation for Cross-Domain Non-Cooperative Spacecraft Based on Spatial-Aware Keypoints Regression

Pose Estimation for Non-Cooperative Spacecraft Rendezvous Using Convolutional Neural Networks

SPIN: Spacecraft Imagery for Navigation

Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management

Towards Bridging the Space Domain Gap for Satellite Pose Estimation using Event Sensing

MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

Pose Estimation for Non-Cooperative Rendezvous Using Neural Networks

Robust Multi-Task Learning and Online Refinement for Spacecraft Pose Estimation across Domain Gap

Single Image Test-Time Adaptation for Segmentation

From Question to Exploration: Test-Time Adaptation in Semantic Segmentation?

Test-Time Adaptation for Optical Flow Estimation Using Motion Vectors

Open-World Pose Transfer via Sequential Test-Time Adaption

Online Supervised Training of Spaceborne Vision during Proximity Operations using Adaptive Kalman Filtering

MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation

Joint Spatial-Temporal Calibration for Camera and Global Pose Sensor

Meta-Auxiliary Learning for Adaptive Human Pose Prediction