SigmaCollab is a dataset that enables research on human-AI physically situated collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world, plus an additional 8 sessions in which the system guided an expert in performing the same tasks.

The dataset, described in detail in this arxiv paper, was collected with an open-source mixed-reality AI application called Sigma (itself described in this arxiv paper and in an IEEE VR extended abstract). The application-driven and interactive nature of the SigmaCollab dataset bring to the fore novel research challenges for human-AI collaboration, and provide more realistic testing grounds for various AI models operating in this space. Additionally, the Sigma system is open-source, and you can download and run it yourself to collect your own / additional data.

SigmaCollab includes a set of rich, multimodal data streams: the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. The raw set of data streams included is summarized in the table below:

SigmaCollab also includes manual segmentation and transcripts for the user utterances, word-level timings for both user and system utterances, task success annotations, as well as post-processed gaze information (e.g., projection of gaze point in the various camera images). For more details regarding the dataset contents, structure, and data formats, please see the Dataset Structure documentation page.

The data is hosted on HuggingFace and available under a CDLA-Permissive-2.0 license. You can download the dataset by first cloning this repo:

git clone https://github.com/microsoft/SigmaCollab
cd SigmaCollab

And then use the following wget command to download the entire dataset (~112GB):

wget -i download/all_sessions

You can also download only portions of the dataset, corresponding to each of the modalities / directories described in Dataset Structure, e.g.:

wget -i download/all_sessions.image           # downloads images (depth, color, leftfrontgrayscale, rightfrontgrayscale) for all sessions
wget -i download/participant_sessions.image   # downloads images (depth, color, leftfrontgrayscale, rightfrontgrayscale) for the participant sessions
wget -i download/all_sessions.image.color     # downloads the color images for all sessions
wget -i download/expert_sessions.audio        # downloads the audio for the 8 expert demonstration sessions

In the commands above, you can use the all_sessions, participant_sessions, and expert_sessions aliases to specify which sessions to download, followed by the modality names as described in Dataset Structure. The commands will download a set of corresponding .tar.gz files, which you can decompress into a data subdirectory as follows:

mkdir data
for f in *.tar.gz; do tar -xzf "$f" -C data; done

If you use this dataset in your research, please consider giving a star to the repo and please cite the following paper:

@misc{bohus2025sigmacollabapplicationdrivendatasetphysically,
      title={SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration}, 
      author={Dan Bohus and Sean Andrist and Ann Paradiso and Nick Saw and Tim Schoonbeek and Maia Stiber},
      year={2025},
      eprint={2511.02560},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={https://arxiv.org/abs/2511.02560}, 
}

The data for the SigmaCollab dataset is made available under the CDLA-Permissive-2.0 license. The files in this GitHub site are made available under the MIT License.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit Contributor License Agreements.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.