This course introduces modern NLP using transformers and the Hugging Face ecosystem, covering datasets, tokenization, and fine-tuning of pre-trained models for downstream tasks. It includes advanced topics such as continual pre-training, instruction tuning, and efficient training techniques. The course also extends to multimodal applications, including speech processing (speaker identification, speech-to-text, wake word detection) and computer vision tasks such as image classification, object detection, depth estimation, and image super-resolution, with hands-on experience on real-world datasets and challenges.

Pre-requisite: BSCS3004 : Deep Learning

Lectures-Tutorials-Practicals-Credits: 4-0-0-4

Week 1: Modern NLP with transformers; Hugging Face ecosystem (datasets, tokenizers, models, trainers); Dataset loading, processing, and streaming.

Week 2: Tokenization techniques; Tokenizer pipeline (normalization, pre-tokenization, encoding/decoding); Training custom tokenizers.

Week 3: Fine-tuning transformer models; Task-specific adaptation; Parameter-efficient vs full fine-tuning; Evaluation metrics.

Week 4: Pre-training and instruction tuning; CLM/MLM objectives; Efficient training (mixed precision, checkpointing); PEFT methods (LoRA, QLoRA).

Week 5: Spoken language identification; Audio-based classification tasks.

Week 6: Speaker diarization; Identifying speakers and segmentation in conversations.

Week 7: Speech processing; Speech-to-text (ASR) and text-to-speech (TTS) systems.

Week 8: Wake-word detection; Personalized keyword spotting systems.

Week 9: Image classification; CNN architectures (AlexNet, VGG, ResNet, DenseNet, ViT); Dataset challenges (variation, imbalance).

Week 10: Object detection; Models (YOLO, R-CNN); Handling real-world datasets and class imbalance.

Week 11: Depth estimation from images; Models (U-Net, Pix2Pix); Challenges like low-light conditions.

Week 12: Image super-resolution; GAN-based methods (SRGAN); Applications to medical imaging.