Training of Large Language Models | 26 Feb 2026

Source: TH

Why in News? 

At the India-AI Impact Summit 2026, Sarvam AI unveiled two indigenous Large Language Models (LLMs), marking a significant step in India’s AI ecosystem. 

• The models are designed to be less power- and compute-intensive while improving performance in Indian languages, highlighting India’s growing capability in training cost-efficient, locally relevant LLMs.

How are Large Language Models (LLMs) Trained?

• LLM: A Large Language Model (LLM) is an artificial intelligence system built using transformer-based neural networks that can understand, interpret, and generate human language. 
• Trained on massive datasets, LLMs use deep learning to identify patterns in text and learn how words and sentences relate to each other. 
• Their performance depends on data quality, and they are further improved through fine-tuning for specific tasks such as translation, summarization, and question answering.

Training

• Data Collection & Pre-processing: LLM training begins with collecting a massive corpus of text from sources like the internet, books, Wikipedia, and code repositories. 
  • The text is converted into tokens (small units of words or subwords) so machines can process it. 
  • The data is then cleaned to remove spam, duplicates, bias, and harmful content, ensuring the model learns accurate and high-quality language patterns.
• Pre-training (Self-Supervised Learning): In the pre-training stage, the model learns language patterns using next-token prediction, where it predicts the next word in a sentence. 
  • Most modern LLMs use the Transformer architecture with self-attention, enabling them to understand relationships between distant words. 
  • This stage produces a base model that understands grammar, facts, and reasoning but cannot yet follow instructions effectively.
• Supervised Fine-Tuning (Instruction Tuning): During supervised fine-tuning, the model is trained on curated prompt–response pairs created by human experts. 
  • This teaches the model how to respond to instructions, answer questions, and perform tasks like summarization. 
  • As a result, the model learns the format of conversations and becomes more useful for real-world applications.
• Alignment using RLHF:  In the alignment stage, developers use RLHF (Reinforcement Learning from Human Feedback) to ensure the model produces safe, unbiased, and helpful responses. 
  • The model generates multiple answers, which humans rank based on quality and safety. 
  • A reward model learns these preferences, and the LLM is optimized to produce responses aligned with human values and ethical standards.
• Limitations of Training LLMs in India: Indian AI models face several bottlenecks, including the scarcity of high-quality datasets in Indian languages, which affects accuracy and inclusivity. 
  • Many models consume more tokens by translating inputs into English to improve performance, increasing computational costs. 
  • Additionally, training LLMs in India remains challenging due to limited capital availability and the lack of immediate commercial use cases, which constrains large-scale investment in domestic AI development.