Development and Adaptation of Robotic Vision in the Real-World: the Challenge of Door Detection

Abstract

Mobile service robots are increasingly prevalent in human-centric, real-world domains, operating autonomously in unconstrained indoor environments. In such a context, robotic vision plays a central role in enabling service robots to perceive high-level environmental features from visual observations. Despite data-driven approaches based on deep learning pushing the boundaries of vision systems, applying these techniques to real-world robotic scenarios presents unique methodological challenges. Traditional models fail to represent the challenging perception constraints typical of service robots and must be adapted for the specific environment where robots finally operate. We propose a method leveraging photorealistic simulations that balances data quality and acquisition costs for synthesizing visual datasets from the robot’s perspective used to train deep architectures. We then show the benefits in qualifying a general detector for the target domain, showing also the trade-off between the effort for obtaining new examples and the performance gain. In our extensive experimental campaign, we focus on the door detection task (recognizing the presence and traversability of doorways), validated in a real-world robot deployment.

Scenario

State-of-the-art object detectors are subject to critical domain shifts when used in robotic applications, as standard training datasets largely neglect the noisy, constrained, and challenging operational conditions a robot faces on the field. In this work, we propose a methodological approach for properly and efficiently adapting these models for service robots. We validate our approach with an extensive on-field experimental campaign with our Giraff-X robot, considering a particularly significant perception task: door detection.

Method

The method is structured around the two principal phases that define the lifecycle of a mobile service robot: the development phase and the deployment phase. During development we create a General Detector (GD), designed using a pose sampling strategy in simulation to develop a photorealistic visual dataset representing typical visual perceptions of a robot. During deployment, the robot is introduced for autonomous operation in a target environment, usually for an extended period; this often involves a domain shift. Given the long-term nature of this phase, there is an opportunity to incrementally fine-tune the GD with data collected in the target environment, obtaining a Qualified Detector (QD). Our method demonstrates a trade-off between the required annotation effort and the resulting performance improvements.

Experiments

We evaluate our door detectors with an extensive on-field campaign in which we teleoperate a Giraff-X robot to map 4 real environments (3 university facilities and an apartment), acquiring images at 1 Hz with a low-resolution Orbbec Astra Camera. These data compose our \(\mathcal{D}_{\texttt{real}}\) dataset. The full workflow is applied to three prominent object detectors: DETR, YOLOv5, and Faster R-CNN.

For each detector, we synthesize four versions of a general detector using different dataset configurations: \(\mathcal{D}_{\texttt{iG}}\) (images from a synthetic simulator), \(\mathcal{D}_{\texttt{DD2}}\) (real images from a public dataset), \(\mathcal{D}_{\texttt{G}}\) (our photorealistic dataset from the robot perspective), and \(\mathcal{D}_{\texttt{DD2+G}}\) (their mixture).

Evaluated on real robotic data, detectors trained on \(\mathcal{D}_{\texttt{iG}}\) perform poorly due to a lack of photorealism. Counter-intuitively, our \(\mathcal{D}_{\texttt{G}}\) outperforms the real-world \(\mathcal{D}_{\texttt{DD2}}\) in certain configurations, and \(\mathcal{D}_{\texttt{DD2+G}}\) obtains the best performance across all reference detectors — demonstrating that the robot perspective is important for generalization in robotic applications.

We then assess the qualification procedure with 4 rounds of increasing data per environment, obtaining \(QD_{e}^{15}\), \(QD_{e}^{25}\), \(QD_{e}^{50}\), \(QD_{e}^{75}\). Qualification boosts performance for the target environment, with large gains in the first rounds and diminishing returns afterwards — suggesting that even a coarse visual inspection of the target environment can yield an environment-specific detector significantly better than the general one.

BibTeX

@INPROCEEDINGS{antonazzi2023doordetection,
      author={Antonazzi, Michele and Luperto, Matteo and Basilico, Nicola and Borghese, N. Alberto},
      booktitle={2023 European Conference on Mobile Robots (ECMR)},
      title={Enhancing Door-Status Detection for Autonomous Mobile Robots During Environment-Specific Operational Use},
      year={2023}, pages={1-8}, doi={10.1109/ECMR59166.2023.10256289}
}
@misc{antonazzi2024roboticvision,
      title={Development and Adaptation of Robotic Vision in the Real-World: the Challenge of Door Detection},
      author={Michele Antonazzi and Matteo Luperto and N. Alberto Borghese and Nicola Basilico},
      year={2024}, eprint={2401.17996}, archivePrefix={arXiv}, primaryClass={cs.RO},
      url={https://arxiv.org/abs/2401.17996}
}

Research