Publication

CLOT: Closed Loop Optimal Transport for unsupervised action segmentation

Conference Article

Abstract

Unsupervised action segmentation has recently pushed its limits with ASOT, an optimal transport (OT)-based method that simultaneously learns action representations and performs clustering using pseudo-labels. Unlike other OT-based approaches, ASOT makes no assumptions about action ordering and can decode a temporally consistent segmentation from a noisy cost matrix between video frames and action labels. However, the resulting segmentation lacks segment-level supervision, limiting the effectiveness of feedback between frames and action representations. To address this limitation, we propose Closed Loop Optimal Transport (CLOT), a novel OT-based framework with a multi-level cyclic feature learning mechanism. Leveraging its encoder-decoder architecture, CLOT learns pseudo-labels alongside frame and segment embeddings by solving two separate OT problems. It then refines both frame embeddings and pseudo-labels through cross-attention between the learned frame and segment embeddings, by integrating a third OT problem. Experimental results on four benchmark datasets demonstrate the benefits of cyclical learning for unsupervised action segmentation.

Categories

computer vision, pattern recognition.

Author keywords

Temporal Action Segmentation; Unsupervised Learning; Optimal Transport Theory

Scientific reference

E.B. Bueno Benito and M. Dimiccoli. CLOT: Closed Loop Optimal Transport for unsupervised action segmentation, 2025 International Conference on Computer Vision, 2025, Honolulu, Hawai'i, pp. 10719-10729.