Indic Pretraining Scales to 2.8 Trillion Tokens with Open Data Pipeline

Indic, a collaborative project involving AI4Bharat and NVIDIA, has achieved a major milestone in AI research by scaling pretraining to 2.8 trillion tokens. This effort, led by Mitesh Khapra and supported by institutions like IIT Madras, marks a significant advancement in multilingual AI development. The open data generation pipeline enables researchers and developers worldwide to access and utilize this vast dataset, fostering innovation in natural language processing and machine learning.

The project builds on previous work, including the Translation • 8B • Updated • model, which was a key step in expanding multilingual capabilities. The team has leveraged advancements in data curation and computational resources to scale pretraining efforts. This includes contributions from NVIDIA researchers such as Niket Agarwal and Raviraj Joshi, who have played a pivotal role in optimizing the training process and ensuring the quality of the generated data.

The dataset encompasses 12 Indian languages, including Hindi, Bengali, and Tamil, as well as other regional languages like Manipuri and Santali. This extensive coverage is made possible by the collaboration between AI4Bharat and NVIDIA, which has enabled the creation of a robust and diverse training corpus. The initiative also includes contributions from researchers such as Mohammed Safi Ur Rahman Khan and Anoop Kunchukuttan, who have focused on improving translation accuracy and model performance across multiple languages.

In India, the release of this dataset is expected to have a significant impact on the local AI ecosystem. Indian builders and organizations can now leverage this open data to develop more accurate and context-aware language models tailored to the needs of Indian users. The involvement of Indian regulators such as TRAI may lead to new guidelines for data usage and model deployment, ensuring that AI development aligns with local requirements and standards.

The open data generation pipeline not only benefits Indian developers but also has global implications. By making this extensive dataset available to the broader research community, the project encourages collaboration and innovation in multilingual AI. This initiative underscores the importance of open-source contributions in advancing AI research and ensuring that language models are inclusive and representative of diverse linguistic communities worldwide.