A Gold Standard Bengali Raw Text Corpus

42,37,440 Words | 1,460 Tittles | XML format | 3 domains

Bengali is the official language of West Bengal and Tripura. It belongs to the Indo-Aryan language family. Bengali is influenced by Sanskrit. Greater use of Bengali has contributed to the growth of the language in terms of vocabulary and the number of styles and registers. Bengali is spoken over the whole of West Bengal, Tripura and Bangladesh and in some parts of Bihar, Orissa, and Assam. Bengali refugees, who have settled in Andaman after 1950, have also carried the language there. LDC-IL Bengali Text Corpus developed according to various factors such as quality of the text, representativeness, retrievable format,  size of corpus, authenticity, etc. For collecting text corpus LDC-IL adopts a standard category list of various domains and a prior set of criteria. The corpus of Bengali text can be broadly classified as literary and non- literary texts. A huge amount of literary texts are available in Bengali but scientific texts are less thus LDC-IL attempts to develop balanced text corpora of Bengali. Data has been collected from books, magazines, and newspapers and it is verified true to the original text.

Bengali Text Corpus encoded in a machine-readable form and stored in a standard format. The major encoding being used is Unicode and stored in XML format. The data is embedded with metadata information. The corpus has been created from the contemporary text in a typed method. 

The available Text Corpus details:

• Ramamoorthy, L., Narayan Choudhary, Sonali Sutradhar, Arundhati Sengupta, Sankarshan Dutta, Priyanka Das & Saswati Karmakar. 2019. A Gold Standard Bengali Raw Text Corpus. Central Institute of Indian Languages, Mysore.
• Choudhary, Narayan & L. Ramamoorthy. 2019. "LDC-IL Raw Text Corpora: An Overview"  in  Linguistic Resources for AI/NLP in Indian Languages, Central Institute of Indian Languages, Mysore. pp. 1-10.