The DIPR challenge is now closed but we have kept the Evaluation and FAQ sections of this page available for informational purposes. Please email us at [email protected] for access to the dataset but please do not send any contest submissions. The DIPr focused on improving short-term IMU fallback tracking for VR systems. The first phase of this challenge used a synthetic dataset created by Arcturus Industries and we released sample code.
This contest will be evaluated by running your code on a private dataset after submission. We plan to update a public leaderboard on a regular basis.
For this challenge we will evaluate submissions using multiple tracking loss simulations on test sequences. This simulation and associated evaluation is performed as follows:
We selected several intervals on each test dataset, where the IMU fallback will be initialized and used. The start points and tracking loss duration are fixed (saved in hdf5 files). They may be chosen arbitrarily or based on real-life tracking loss.
You IMU fallback prediction on each segment will be aggregated to compute the Velocity Mean Absolute Error (VMAE) per dataset
$$ VMAE =\frac{1}{N} \sum_{i=1}^N \rVert v^{pred}i-v^{gt}i \lVert{L1}~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\ = \frac{1}{N} \sum{i=1}^N \lvert v^{pred}{i,x} - v^{gt}{i,x} \rvert+\lvert v^{pred}{i,y} - v^{gt}{i,y} \rvert+\lvert v^{pred}{i,z} - v^{gt}{i,z} \rvert $$
We will then average your VMAE among all intervals and all datasets
We use velocity based metrics to reflect what’s important in SLAM for VR. Indeed, a good velocity estimate is crucial to prevent motion sickness or flyaway. Moreover, the L1 norm is less sensitive to large errors that sometimes occur.
We recommend using the provided code as a start, and we require you to send your submissions in form of code modifications to a third party. Using the same method as us to fuse deep learning with IMU data (EKF) is not mandatory. You may also decide to change the CNN prediction type. You may propose your own methods or a novel methods (even non deep learning).