This course covers the Transformer architecture, pretraining and fine-tuning of language models, and different tokenizer types such as BPE, WordPiece, and SentencePiece. It also examines LLM architectures, common datasets, and challenges in handling long-context attention, and various fine-tuning techniques for large language models. This course is also offered as an elective for PG-diploma students.

Pre-requisite: BSCS3004 : Deep Learning

Lectures-Tutorials-Practicals-Credits: 4-0-0-4

Week 1: Transformer architecture foundations; Self-attention mechanisms; Multi-head attention.

Week 2: Decoder mechanisms (teacher forcing, masked attention); Positional encoding (including sinusoidal); Normalisation techniques (batch and layer normalisation).

Week 3: Language modelling fundamentals; Transformers for language modelling; Causal language modelling; Generative pre-trained transformers (GPT); Pre-training and fine-tuning.

Week 4: Decoding strategies (greedy, beam search, sampling-top-k, top-p); BERT and masked language modelling; Next sentence prediction; Pre-training and adaptation to downstream tasks.

Week 5: Tokenization challenges; Subword tokenization methods; Byte Pair Encoding (BPE); WordPiece and SentencePiece tokenizers.

Week 6: Coding Assignment.

Week 7: Modern transformer models (BART, GPT-2); Prompting techniques; Model design choices and performance factors; Architectural and objective variations; Pre-training data scale; Fine-tuning strategies; Multi-task learning.

Week 8: Scaling in language models; Data sourcing and cleaning; Preprocessing pipelines; Pre-training datasets and data considerations.

Week 9: Coding Assignment.

Week 10: Coding Assignment.

Week 11: Efficient attention mechanisms; Time and space complexity of attention; Local and sparse attention; Low-rank approximations; Fast inference techniques.

Week 12: Positional encoding techniques; Relative position encoding; Rotary embeddings (RoPE); ALiBi and alternative positional methods.