Hello World! It’s Siraj and let’s make our own language translator using TensorFlow Today there are about 6,800 different languages spoken across the world and in an increasingly globalised world nearly every culture has interactions with every other culture in some way that means there are an incalculable number of translation requirements every second of every day across the world Translating is no easy task A language isn’t just a collection of words and of rules of grammar and syntax it’s also a vast inter-connecting system of cultural references and connotations and this reflects a centuries old problem of two cultures wanting to communicate but are blocked by a language barrier our translation systems are fast improving though, so whether it be an idea or a story or a quest each new advancement means one less message will be lost in translation. During the Second World war, the British government was hard at work trying to decrypt the morse coded radio communications that Nazi Germany used to send messages sucurely, known as Enigma. They decided to hire a man named Alan Turing to help in their effort and when the American government learnt of their translation effort they were inspired to try it themselves, post war. Specifically because they needed a way to keep up with Russian scientific publications. The first public demo of a machine translation system, translated 250 words between Russian and English in 1954. It was dictionary based so it would attempt to match the source language to the target language word for word The results were poor since it didn’t capture syntactic structure. The second generation of systems used Interlingua that means they changed a source language to a special intermediary language with specific rules encoded into it then from that generated the target language. This proved to be more efficient, but this approach was soon overshadowed by the rise of statistical translation in the early 90s Primarily from engineers at IBM Innovation at IBM Watch this A popular approach was to break the source text down into segments then compare them to an aligned bi-lingual corpus using statistical evidence and probabilities to choose the most likely translation. Nowadays the most used statistical translation system in the world is Google Translate and with good reason. Google uses deep learning to translate from a given language to another with state of the art results So how do they do this? Let’s recreate their results in TensorFlow to find out The dataset we’ll be using to train our language translation model is a corpus of transcribed TED talks It’s got both the English version and the French version and our goal will be to create a model that can translate from one to the other after training. We’ll be using TensorFlow’s built in data_utils class to help us pre-process our data set and we’ll start by defining our vocab size which is the number of words we want to train on from our dataset. We’ll set it to 40k for each which is a small portion of the data then we’ll use the data_utils class to read the data from the data directory. Giving it our desired vocab size and it will return the formatted and tokenised words in both languages We’ll then initialise TensorFlow placeholders for our encoder and decoder inputs Both will be integer tensors that represent discrete values they will be embedded into a dense representation later We’ll feed our vocabulary word to the encoder and the encoded representation that’s learnt to the decoder. Now we can build our model Google published a paper more recently discussing a system they integrated into their translation service called Neural Machine Translation. It’s an encoder decoder model inspired by similar work from other papers on topics like text summarisation. So whereas as before Google Translate would translate from language A to English to language B with this new NMT architecture, it can translate directly from one language to the other It doesn’t memorise phrase to phrase translations instead it encodes the semantics of the sentence. This encoding is generalised so it can even translate between a language pair like Japanese and Korean that it hasn’t explicitly seen before. So I guess we can use a LSTM recurrent network to encode a sentence in language A the RNN spits out a hidden state ‘s’ which represents the vectorised contents of the sentence. We can then feed ‘s’ to the decoder which will generate the translated sentence in language B, word by word. Sounds easy enough right? WRONG! There is a drawback to this architecture, it has limited memory that hidden state ‘s’ of the LSTM is where we’re trying to cram the whole sentence we want to translate ‘s’ is usually only a few hundred floating point numbers long The more we try to force our sentence into this fixed dimensionality vector the more lossy our neural net is forced to be. We could increase the hidden size of the LSTM after all they’re supposed to remember long term dependencies but what happens is as we increase the hidden size ‘h’ of the LSTM the training time increases exponentially. So to solve this we’re going to bring ‘Attention’ into the mix If I was translating a long sentence, I’d probably glance back at the source sentence a couple times to make sure I was capturing all the details. I’d iteratively pay attention to the relevant parts of the source sentence We can let neural nets do the same by letting them store and refer to previous outputs of the LSTM This increases the storage of our model without changing the functionality of the LSTM So the idea is once we have LSTM outputs from the encoder stored we can query each output asking how relevant they are to the computation happening in the encoder Each encoder output gets a relevancy score which we can convert to a probability score by applying a softmax activation to it. Then we extract a context vector which is a weighted summation to the encoder outputs depending on how relevant they are. Memory ain’t enough, pay attention Memory ain’t enough, pay attention (In Hindi) Memory ain’t enough, pay attention (In German) Memory ain’t enough, pay attention (In Spanish) Memory ain’t enough, pay attention We build our model using TensorFlow’s built in embedding attention sequence to sequence function giving it our encoder and decoder inputs as well as a few hyper parameters we define like the number of layers. It builds a model that is just like the one we discussed TensorFlow has several built in models like this that we can drop into our code easily So normally this alone would be fine and we could run this and the results would be decent but they added another improvement to their model that requires MORE CODE A 100 GPUs and a WEEK OF TRAINING Seriously that’s what it took we won’t implement it all programatically but let’s dive into it conceptually If the outputs don’t have sufficient context then they won’t be able to give a good answer we need to include info about future words, so that the encoder output is determined by the words on the left and the right. We humans would definitely use this kind of full context to determine the meaning of a word we see in a sentence. The way they did this is tho use a bi-directional encoder so it’s two RNNs. One that goes forward over the sentence and the other goes backwards. So for each word it concatenates the vector outputs which produces a vector with context from both sides. and they added a lot of layers to their model. The encoder has one bi-directional RNN layer and seven uni-direciotnal RNN layers The decoder has eight uni-directional RNN layers The more layers the longer the training times so that’s why we use a single bi-directional layer if all the layers were bi-directional the whole layer would have to finish before layer dependencies could start computing But by using uni-directional layers, computation is going to be more parallel. We’ll initialise our TensorFlow section, then our model inside of it Let’s see some results after training. First I’ll give it this phrase Looks good and now another phrase DOPE! While it’s not perfect and we still have a way to go we’re definitely getting closer to having a universal translation model. Breaking it down Encoder-Decoder architectures are for state-of-the-art performance in machine translation by storing the previous outputs of the LSTM cells we can judge the relevancy of each to decide which to use via an attention mechanism. And by using a bi-directional RNN, the context of both past and future words is used to create an accurate encoder output vector. The coding challenge winner from last week is Ryan Lee This was very impressive, he created a recipe summariser by scraping a 125,000 recipes from the web and documented it all beautifully with installation steps so you can reproduce the results yourself. WIZARD OF THE WEEK! and the runner up is Sarah Collins her code converts scientific papers to text and prioritises them by topic. This weeks coding challenge is to create a simple translation system using an encoder-decoder model. All the details are in the readme, post your github link in the comments and I’ll announce the winner next week. Please subscribe for more programming videos Check out this related video and for now I’ve got to get a better GPU So, thanks for watching!