This repository provides a robust and automated framework for running, benchmarking, and evaluating containerized SLAM (Simultaneous Localization and Mapping) systems. It is designed for streamlined use in robotics competitions and research, leveraging Docker and Docker Compose for portability and reproducibility.

• Automated Pipeline: Execute the entire workflow—running SLAM, playing data, and evaluating results—with a single command.
• Containerized & Reproducible: Runs any Docker-based SLAM system, ensuring consistent environments.
• Flexible Configuration: Easily configure dataset paths, playback parameters, and the target SLAM image via a central .env file.
• Modular Structure: Cleanly organized source code, entrypoints, and configuration make the framework easy to understand and extend.

This framework is the official evaluation tool for the following robotics competitions:

• SLAM Challenge Series @ CTU in Prague: comrob-ds.fel.cvut.cz:555

• SLAM Pipeline and Evaluation Framework
  • Features
  • 🏆 Used In
  • Table of Contents
  • 📦 Prerequisites
  • 🚀 Quick Setup
  • 🎯 Usage Scenarios
    • 👨‍💻 For Competition Participants
      • Step-by-Step Example: Building and Testing a Local SLAM Image
      • Preparing Your Own SLAM System
    • 큐 For Dataset Curators
  • 📁 Example Dataset
  • ⚙️ Advanced Usage & Details
    • Manual Control
    • Visualization with RViz
    • Development Workflow (for this Repository)
  • 🛠 Troubleshooting
  • 📜 License

Ensure you have the following software installed on your system:

• Docker Engine
• Docker Compose

This guide covers the initial setup of the framework. For specific use cases, please see the Usage Scenarios below.

1. Clone the Repository:

   git clone https://github.com/comrob/slam-bench.git slam_competition
   cd slam_competition

2. Create Environment File: Copy the example environment file. You will configure this file based on your chosen scenario.

   cp .env.example .env

This framework is designed for two primary user groups. Please follow the guide that matches your goal.

Goal: To containerize your SLAM system and test it locally, simulating the exact process used by the official evaluation platform.

Recommendation: This guide walks you through building a provided example (VINS-Mono-crl) from source and running it with this framework. Following these steps is the best way to understand how to prepare your own SLAM system for the competition.

1. Clone the Example SLAM System: First, get the source code for VINS-Mono-crl, which serves as a template for a dockerized SLAM solution.

   git clone https://github.com/comrob/VINS-Mono-crl.git

2. Build the SLAM Docker Image Locally: Navigate into the cloned repository and build the Docker image. This process simulates you building your own SLAM system's image.

   cd VINS-Mono-crl/docker
   make # vins-mono-crl:latest will be build
   cd ..

   You now have a local Docker image named vins-mono-crl:latest ready for testing.

3. Set up the Evaluation Framework & Dataset: If you haven't already, complete the Quick Setup for this slam-bench repository and download the shellby-0225-train-lab dataset from the Example Dataset link.

4. Configure the .env File: In the slam_competition directory, open your .env file and point it to your locally built image and the downloaded dataset.

   # Absolute path to the dataset directory
   BAGFILES_PATH_HOST=/home/user/datasets/shellby-0225-train-lab
   
   # Name of your locally built Docker image
   SLAM_IMAGE=vins-mono-crl:latest

   Note: Replace /home/user/ with the actual path on your system.

5. Run the Full Pipeline:

   ./run_pipeline.sh

   This will use your locally built vins-mono-crl:latest image to run the evaluation. This mirrors the exact process you'll follow for your own algorithm.

After running the example, you are ready to adapt the process for your own solution. Keep the following in mind:

• ROS1 is Straightforward: If your solution already runs in ROS1, the containerization process is very direct. You primarily need to create a Dockerfile that builds your workspace and runs your launch file.
• Dockerfile Best Practices: Your Dockerfile should be clean and minimal. We highly recommend studying the VINS-Mono-crl Dockerfile as a template. It should:
  1. Install only the necessary dependencies.
  2. Build a minimal ROS workspace.
  3. Use CMD or ENTRYPOINT to execute the roslaunch command that starts your SLAM node. Ensure the topics in your launch file match the topics provided in the competition datasets.

Goal: To prepare and validate a new dataset for use in a competition.

Recommendation: To ensure your dataset is compatible with the evaluation platform, you should test it with this framework using a known, working SLAM algorithm.

1. Structure Your Dataset: Organize your dataset files into the required directory structure. See the Example Dataset section below for the correct layout (sensors, reference, calibration, etc.).

2. Test Locally: Configure the .env file to point BAGFILES_PATH_HOST to your new dataset's location. Use a reliable, locally built SLAM image (like vins-mono-crl:latest from the example above) for SLAM_IMAGE to run the pipeline.

   ./run_pipeline.sh

   If the pipeline completes successfully and generates a sensible evaluation report, your dataset is correctly structured.

3. Contact Organizers: Once you have validated your dataset, contact the competition organizers to arrange for it to be uploaded to the official evaluation server.

We provide a sample collection of datasets to help you get started.

To test the framework with your own custom dataset, it must follow the same directory structure, including providing sensor calibration files:

<your_dataset_root>/      # e.g., /home/user/datasets/my_awesome_dataset
├── sensors/
│   └── *.bag             # One or more .bag files containing sensory data
├── reference/
│   └── reference.txt     # The ground truth trajectory file
└── calibration/
    ├── intrinsics.yaml   # Intrinsic parameters for cameras, etc.
    └── extrinsics.yaml   # Extrinsic transformations between sensor frames

This project is licensed under the MIT License. See the LICENSE file for details.