Multi-Sensor 3D Data Visualization in Virtual Reality for Planetary Science and Mission Operations

The MIT Resource Exploration and Science of our Cosmic Environment (RESOURCE) project aims to bridge the operations gap between current uses of VR as a training tool and future uses as an immersive interface for mission support operations and science. We describe the development of "digital twin" field sites within a VR environment from several low cost and commercially available sensors for photography, LiDAR, combination LiDAR-RGB, and photospectrometry. Three-dimensional point-cloud datasets of rocky outcrops were taken from Marblehead, Massachusetts and Svalbard, Norway. These sites were selected for the presence of features across multiple scales and fidelity as analog sites for a terrestrial space environment. They contain large landscape features like hills, depressions or permafrost features (>>1m scale), boulders (~1m scale), small rocks (~10cm), and small scale geologic features such as veining or fracturing (<1cm). These data were collected using a Velodyne LiDAR, Sony digital photography camera, DJI aerial drone, and Hamamatsu mini-spectrometer. Depth models were reconstructed with Open3D and AliceVision/Meshroom. These models were aligned and exported to a VR-capable file format. Geologic data was manually checked for key feature presence, and the point clouds were compared for distance accuracy and feature matching with the photogrammetric model in CloudCompare. Finally, the aligned models and point clouds were rendered in the Unreal 5 VR game engine. As part of this work, we discuss the data collection techniques for dense LiDAR point clouds, as well as handheld and aerial drone photogrammetry. We also discuss considerations for data resolution and rendering capability for a VR interface that can be used for either mission planning or for real science analysis. The low cost and flexibility of the acquisition hardware allows for high extensibility both on Earth for remote sensing, but also in deep space for science targets and mission analytical tasks.