Hugging Face released a 2.8 trillion token Indic pretraining corpus and an open data generation pipeline. The pipeline enables scalable, reproducible data creation for low-resource language model training.
WHY IT MATTERS
This provides a foundation for training more capable Indic language models, addressing data scarcity in AI development for South Asian languages.
Hugging Face Blog