Motion and geometry-aware methods for automatic sign language processing
Loading...
Date
2026-02-27
Authors
Σαρτίνας, Ευάγγελος
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Automatic Sign Language Processing aims to interpret signed communication from visual input, enabling accessible interaction between Deaf and hearing communities. Despite substantial progress driven by deep learning and large-scale datasets, robust sign language processing remains challenging due to the fine-grained, highly structured, and multimodal nature of signed articulation. In particular, many contemporary approaches rely predominantly on data-driven temporal modeling of pose sequences, often overlooking the underlying geometric and kinematic structure of human motion that governs sign production.
This dissertation investigates geometry and motion aware methods for automatic sign language processing, with a focus on pose-based representations derived from skeletal keypoints. The proposed approaches make use of geometric information extracted from hand and wrist motion and hand skeletal pose for sign language video summarization and recognition.
First, the thesis introduces training-free motion-based summarization techniques for continuous sign language videos. Using wrist motion trajectories, a keyframe selection method based on time-parameterized curvature is proposed to extract lexically salient frames while significantly reducing temporal redundancy. This approach is extended to three dimensions, incorporating both curvature and torsion to capture bending and twisting motion. Extensive quantitative and human-based evaluations demonstrate that the proposed summarization methods preserve semantic content more effectively than existing baselines, while remaining lightweight and interpretable.
Second, the dissertation addresses geometry-aware modeling of handshape and skeletal motion for sign recognition. Complementary pose-level distance measures are utilized, including point cloud-based handshape distances and an articulated pose distance that respects hierarchical kinematic structure. Building on these formulations, a self-supervised distance-preserving embedding framework is introduced to transfer explicit geometric similarity into compact latent representations. These embeddings are integrated into Transformer-based recognition models, where they act as strong inductive biases and yield consistent improvements in recognition accuracy on large-scale benchmarks.
Overall, this work demonstrates that explicitly incorporating geometric and kinematic priors into sign language processing pipelines enhances both motion summarization and recognition performance. The findings highlight the value of geometry-informed representations as a principled bridge between human articulatory structure and modern learning-based models, contributing toward more efficient, robust, and interpretable sign language technologies.
Description
Keywords
Automatic sign language processing, Sign language recognition, Sign language summarization, Sign language summarization, Self-supervised learning, Frenet–Serret geometry