Analyse textuelle


Objectif

This course introduces the main statistical tools for natural language processing (oNLP). 
Natural language processing covers any application based on extracting information from textual data, and has been a core component of modern search engine, spam detectors or automatic translation systems.
After a short introduction of the standard classical statistical tools, we will dive into modern approaches based on Deep Learning that are currently used by the industry.
This course requires knowledge of coding and statistics, and will be validate through an experimental project.

Plan

Part I Traditional tools for NLP — Edouard Grave
— Introduction classical problems in NLP
— Count based statistical methods
— text classification
— Language modeling and n-grams models
— Phrase based machine translation

Part II Deep learning for NLP — Armand Joulin
— Word embeddings
— Recurrent networks
— Attention mechanisms and transformer networks
— Neural language modeling
— Neural Machine Translation

Références

– Brown, P.F., Pietra, V.J.D., Pietra, S.A.D. and Mercer, R.L.,  “The mathematics of statistical machine translation: Parameter estimation”. Computational linguistics. 1993.
– Hochreiter, S. and Schmidhuber, J., Long short-term memory. Neural computation. 1997. 
– Goodman, J. T. "A bit of progress in language modeling." Computer Speech & Language. 2001.
– Och, F. J., and Ney, N. "A systematic comparison of various statistical alignment models." Computational linguistics. 2003.
– Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. “A neural probabilistic language model”. Journal of machine learning research. 2003.
– Mikolov, T., Karafiát, M., Burget, L., ?ernocký, J. and Khudanpur, S. "Recurrent neural network based language model." In Eleventh Annual Conference of the International Speech Communication Association. 2010.
– Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. “Distributed representations of words and phrases and their compositionality”. In Advances in neural information processing systems.  2013.
– Bahdanau, D., Cho, C. and Bengio, Y.. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473. 2014.
– Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.. "Attention is all you need." In Advances in Neural Information Processing Systems. 2017.
– Bojanowski, P., Grave, E., Joulin, A. and Mikolov, T. “Enriching Word Vectors with Subword Information”. Transactions of the Association for Computational Linguistics. , 2017.
– Joulin, A., Grave, E., Bojanowski, P. and Mikolov, T., “Bag of Tricks for Efficient Text Classification”. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: 2017.