Robust Word Predictions

Abstract

Natural Language Processing (NLP) is a sub-field of Artificial Intelligence (AI) that allows machines to process and comprehend human languages in order to bring machines nearer to language understanding. In older days, statistical methods were predominant where the rules were written / calculated manually. Recent advances in Machine Learning and Deep Learning have led to many breakthroughs in various sub-fields of NLP which include language modeling, machine translation, speech recognition etc. Language Model (LM) forms a building block of all the NLP applications where the task is to predict the next word given previous words. The main drawback of the language model is that it is limited to predicting next word and the training requires significant amount of vocabulary. This research aims for the development of generalized language model in which the prediction is not just limited to next word but to any word in the future text. The research tried to expand the horizons of language modeling problem and make it more generalized in terms of understanding context as well as making prediction. This work proposes a Neural generalized language modeling technique and tested on two kinds of databases i.e., BBC News articles as well as Wikipedia exploring the use of word embeddings, attention mechanism to understand the context and making context based word predictions. The main focus lies on predicting non-stop words and meaningful words. Also, this work explored the memory capabilities and Noise tolerance of the developed model.