Neural networks : deep learning strategies for problems with limited data

Thumbnail Image




Τσουρούνης, Δημήτριος

Journal Title

Journal ISSN

Volume Title



Small sample size learning (SSSL) problem arises when the available training data are limited, making it challenging for machine learning models to capture meaningful patterns and provide accurate predictions. In computer vision applications, constraints on training data are common due to data collection difficulties or high annotation costs. This PhD thesis focuses on exploring deep learning strategies tailored for addressing the SSSL problem, with a specific emphasis on developing efficient training methods for convolutional neural networks (CNNs) when only a limited amount of data are available. Different approaches exist based on the space being considered: data augmentation techniques in the input space, approximating target functions with regularization and pretraining in the model space and encoding relationships between data points within a latent feature space. In this dissertation we propose methods that attack SSSL in one or multiple spaces simultaneously. The applications studied in this thesis include biometric verification in the offline signature verification (OffSV) problem, which currently lacks a large available offline signature dataset, and the biomedical problem of human epithelial type-2 (Hep-2) cell classification through indirect immunofluorescence (IIF) microscopy images, involving a challenging annotation process. Initially, shallow representation learning approaches, utilizing traditional computer vision techniques, are studied as a baseline scenario of approaching SSSL. This enabled us to gain valuable insights into the intrinsic characteristics of the studied problems and enhances the interpretability of the results. Subsequently, a hybrid scheme combining hand-crafted descriptors with a CNN model is proposed. Hand crafted features can create representations with desired invariance characteristics, hence when used as input to a CNN, can provide a more effective starting point for training with limited samples size. A different path to address the SSSL problem studied in this dissertation involves utilizing external data from a similar domain with data abundance. These data can serve as information carriers within a sophisticated training procedure, aimed at enhancing performance in the target problem that suffers data limitations. Such methods were developed in the context of OffSV, where auxiliary handwritten text data were utilized during the training of CNNs in the writer identification task, managing to learn effective encodings of signature images by employing domain adaptation techniques, achieving comparable performance or even surpassing models trained on thousands of signature images. The first such approach proposed in this thesis is explicit domain adaptation, which encompasses metric learning using an additional transformation layer trained via contrastive loss, used to transform the outputs of a pretrained CNN model. The second proposed technique is implicit domain adaptation, implemented through teacher supervision in the Feature-based Knowledge Distillation (FKD) scheme. This method leverages both local and global information from intermediate representations of the teacher to facilitate efficient knowledge transfer. Results demonstrate that the proposed approaches effectively address the SSSL problem in the OffSV domain, operating in either the feature space or the model space, by utilizing auxiliary data in the input space to overcome the challenges posed by the data limitations.



Deep learning, Neural networks, Convolutional neural networks, Small sample size learning