COMP 472/6721   Mini-Project 3 Solved

Description

Automatic Language Identification

In this project you will build and experiment with a probabilistic language identification system to identify the language of a sentence.

Your Task

You will build a character-based n-gram model for 3 languages: English, French and a language of your choice that uses the same character set.  As a basic setup, you must:

• Train a unigram and bigram character-based language model for each language.
• Use your language models to identify the most probable language of a sentence given as input.

Training Set: As training set, you will start with the corpora available on Moodle (2 books in English, and 2 books in French, where diacritics have been removed). For the 3^rd language, you must find your own training corpora.  The Web is a great place to find electronic texts.  Look at the Web page of Project Gutenberg^[1] or Internet Archive^[2] for good starting points.

Character Set: Make sure that all 3 languages use the same character set.  In particular, you should not take diacritics (accents, cedillas…) into account.  The French corpora on Moodle have been cleaned of diacritics.  For the basic setup, use only the 26 letters of the alphabet (i.e. ignore punctuations and other characters) and reduce all letters to their lower case versions.

Log Space: In order to avoid arithmetic underflow, remember to work in log space.  Any logarithmic base would work; but for the basic setup, use base 10.

Smoothing: For the basic setup, both your unigram and your bigram models must be smoothed using add-delta, with delta=0.5.

Programming Environment:

You can use Java, C, C++ or Python.  If you wish to use another language, please check with me first.

Input:

Your program will take as input:

• 3 file names (txt, trainEN.txt and trainOT.txt) containing the training texts for each language, and
• a file containing the sentences to be classified.

Output:  Your program will output:

1. a dump of the language models in the following files: txt, bigramFR.txt,  unigramEN.txt, bigramEN.txt, unigramOT.txt, bigramOT.txt

Each file should contain a dump of the language model in list format.  For example,

for the unigram models:   for the bigrams models:

P(c … P(

2. For each input sentence in the input file:
   1. on the console, the most probable language of the sentence
   2. a dump of the trace of that sentence in a file name out#.txt where # is the number of the sentence in the input file.

Each output file must contain the sentence itself, a trace and the result of its classification, following the format below.

For example, if the input file contains 30 sentences to test, then you should generate 30 output files named out1.txt to out30.txt.  If the first sentence is What will the Japanese economy be like next year? then out1.txt will contain:

3. Submit the output of your program with 30 input sentences. Your 30 sentences must include: a. the following 10 sentences
   • What will the Japanese economy be like next year? (EN)
   • She asked him if he was a student at this school. (EN)
   • I’m OK. (EN)
   • Birds build nests. (EN)
   • I hate AI. (EN)
   • L’oiseau vole. (FR)
   • Woody Allen parle. (FR)
   • Est-ce que l’arbitre est la? (FR)
   • Cette phrase est en anglais. (FR)
   • J’aime l’IA. (FR)
4. 10 sentences that your system classifies correctly
5. 10 sentences that your system gets wrong

Experiments:

As with the previous mini-projects, you are expected to perform additional experimentations with your program beyond the basic setup specified above.

Examples of experiments include: experimenting with a variety of values for n, for delta or for the character set, a variety of languages (similar or different), etc.  For each experiment, report and analyse your results in your report.

Deliverables:

The submission of the project will consist of 3 deliverables:

1. The source code and an executable that runs in the lab machines
2. The 30 output files when your language model is trained with the corpora given
3. A Report
4. A Demo

The Code:

Submit all files necessary to run your code in addition to a README.txt  which will contain specific instructions how to run your program on the desktops in the computer labs.