Experiments¶

An experiment is a folder containing an experiment.yaml file and, after a run, an outputs/ directory.

Folder Layout¶

my-experiment/
├─ experiment.yaml
└─ outputs/
   ├─ camera-1/
   │  ├─ results.parquet
   │  ├─ body_embeddings.parquet
   │  ├─ face_embeddings.parquet
   │  ├─ transcript_segments.parquet
   │  └─ transcript_words.parquet
   ├─ p1-mic/
   │  ├─ transcript_segments.parquet
   │  └─ transcript_words.parquet
   └─ speech_turns.parquet

Configuration Format¶

The current experiment format is versioned. Input paths may be absolute or relative to the experiment folder. An experiment is identified by the folder it lives in, and so carries no name of its own.

version: 1
glasses_videos:
  - id: p1-glasses
    path: videos/p1.mp4
    gaze_path: videos/p1-gaze.tsv
    timeline:
      offset: 0.0
      shifts: []
fixed_videos:
  - id: camera-1
    path: videos/camera-1.mp4
    timeline:
      offset: -1.25
      shifts:
        - at: 512.4
          seconds: 0.18
audio:
  - id: p1-mic
    path: audio/p1.wav
    glasses_video: p1-glasses
    timeline:
      offset: 0.0
      shifts: []
pipeline:
  glasses_video:
    object_tracking:
      detector: yolo26m
      reid: osnet_x1_0_msmt17
      tracker: botsort
      object_classes:
        - 0
      embeddings_per_track: 32
    face_detection:
      model_name: antelopev2
      det_size: 640
      det_thresh: 0.5
      embeddings_per_track: 32
    body_pose:
      model_name: yolo26m-pose.pt
      conf: 0.25
  fixed_video:
    object_tracking:
      detector: yolo26m
    face_detection: null
    body_pose: null
  speech:
    transcription:
      model_name: primeline/whisper-large-v3-turbo-german
      language: de
      beam_size: 5
      vad_filter: false

Inputs¶

Inputs are grouped by type, each in its own list. Every input needs an id that is unique across all the lists, since it names that input's output directory; for the same reason it has to be usable as a filename, so it cannot be empty or contain a path separator.

Each input has a timeline describing where it sits on the experiment's shared clock. The defaults describe a device that started with the experiment and kept time:

offset: seconds added to this input's own clock to reach experiment time, since every device was switched on at its own moment. Defaults to 0.0.
shifts: content the recording lost partway through, if any. Each entry has an at, where the loss falls on the recording's own clock, and seconds, how much is missing there, which is added to the offset from that point on. Defaults to none.

This file is the on-disk form. At runtime each GlassesVideo, FixedVideo or Audio owns a Timeline object alongside its results, reached through Experiment.glasses_videos, .fixed_videos and .audio. Inputs are added, removed and renamed through Experiment so their ids stay unique.

glasses_videos: video recorded by a participant's glasses-mounted camera. This is the input that carries eye tracking, so it needs a gaze_path as well as a path: the gaze samples the same device recorded, as a TSV file. They share the video's clock, and so its timeline. Video formats with audio, such as MP4, include that audio during video playback. for when glasses video and audio becomes out of sync; amongst other things because that does not happen at just one point in time and can be a very small offset differences.
fixed_videos: video from a camera at a fixed position in the room.
audio: audio recorded on another device, such as a directional microphone aimed at one participant, or a single microphone recording the whole group. If the audio is from a specific participant, the optional glasses_video field identifies the id of the glasses worn by the participant. Embedded video audio is handled by video playback, not as an audio input.

Every list may be empty: an experiment with no inputs at all is valid, which is what a new one starts as before any files have been added to it.

Pipeline¶

Each type of video input has its own block under pipeline, so e.g. a room camera can be tracked with a different detector than the glasses cameras, or skip a stage they run. Omit a block to use its defaults.

For the video blocks — glasses_video and fixed_video — object_tracking is required, while face_detection and body_pose are optional; omit either key or set it to null to skip that stage.

The speech block runs transcription over every input that carries audio — the audio inputs, and any video whose camera recorded a sound track.

Object Tracking¶

Object tracking combines object detection, re-identification, and a tracker:

detector: object detector model reference.
reid: re-identification model used to keep track IDs stable.
tracker: BoxMOT tracking algorithm.
object_classes: COCO class IDs to detect.
embeddings_per_track: number of best body-appearance embeddings to keep per tracklet.

Face Detection¶

Face detection runs InsightFace inside each tracked person box and keeps the best face above the configured threshold:

model_name: InsightFace model pack.
det_size: square detector input size.
det_thresh: minimum face detection confidence.
embeddings_per_track: number of best face embeddings to keep per tracklet.

Body Pose¶

Body-pose detection runs an Ultralytics YOLO pose model inside each tracked person box:

model_name: YOLO pose weights.
conf: minimum pose confidence.

Transcription¶

Transcription runs faster-whisper over the whole recording, keeping the segments it produces and the timing of every word within them.

model_name: Whisper model. The default primeline/whisper-large-v3-turbo-german is accuracy-tuned for German; primeline/whisper-large-v3-german is its full-size alternative, and large-v3 is the accuracy-first multilingual model. Standard Transformers checkpoints are converted to FP16 CTranslate2 weights on first use and then loaded from the shared model cache.
language: ISO 639-1 code, defaulting to de for the German model. Change it for another language, or leave it unset to detect from the first 30 seconds.
beam_size: decoding beam width.
vad_filter: skip silent stretches.