TensorFlow Tutorial #21 Machine Translation

How to translate between human languages using a Recurrent Neural Network (LSTM / GRU) with an encoder / decoder architecture (sequence-to-sequence model) in TensorFlow and Keras. Demonstrated on the large Europarl data-set from the European Union.
github.com/Hva...
This tutorial has been updated to work with TensorFlow 2.1 and possibly later versions.

[@hvasslabs]

This tutorial has received a high ratio of dislikes. It is a pity that people don't explain why they dislike it. I suspect it is my "political rant" about Europe that some people disagree with. If you have more technical reasons to dislike this video then please explain that in a comment.

[@leemobaia]

There is nothing to dislike. Machine Translation is very big task. Some people even did phd on Machine Translation. You did a great job by explaining it in single video tutorial. But you have to understand that it is daunting for many people. it is great work. you are awesome.

[@mertsefatarhan]

It's g****e developers and they don't like explaining in comments as you know :))

[@owen1226]

I feel sad that Canada is doing what European countries did. Somebody did horrible things, but people can NOT say negative comments to their groups, instead we are forced to learn their culture and understand their hate.

[@raheelsiddiqi8285]

You have not properly shown the training of the model. When I trained it on my computer, it is giving me val_loss of NaN even after few epochs.....

[@raheelsiddiqi8285]

Can you please show how the training progressed? What was your training and validation losses after the first epoch? I am getting NaN for both after the first epoch......

[@reacionario2026]

5 stars for technical content. 5+ for political content. You have courage and love! Thanks for both content!

[@sharma_learn]

Hats off to your content(I have watched #20 and #21 of this series and both of them are just phenomenal). Thank you.

[@djcoolio8096]

Magnus this is yet another gem from you! Great political rant...the world is a fucked up place in general :)

[@yannickpezeu3419]

ROFL I wasn't ready for the political talk ! It's so good to hear engineers who actually have a brain !

[@hvasslabs]

Are you from Eastern Europe? They're the only ones in Europe who take these problems seriously, because they know what a totalitarian ideology looks like. Check the video I'm uploading right now on Long-Term Stock Forecasting, it has a 20 minute introduction with a political discussion about the year 2020.

[@stevecoxiscool]

HAHA !!!! I actually started grocking RNN's with this video. One of the hardest parts of this whole AI/Deep learning stuff (for me) is the data transformations from words/sounds/images into inputs for Tensorflow/Keras what ever. Great rant !!!! There is a HUGE pusification process going on in the USA as well. Really sad. Anyways ..... back to AI and Deep nets.Thanks for all the help.

[@hvasslabs]

I actually have no use for NLP myself, I was just curious how it worked and indeed how you transform the data and work on it inside a Neural Network. Regarding the politics, it is a tragedy what is happening in the Western World! It is as if we have made a cultural suicide pact. In Western Europe you will get branded a racist or a nazi for any slight criticism of mass immigration. The problems in Europe have grown completely out of control because criminals and fanatics haven't been deported, so the rest of us could live in peaceful harmony. But our courts refuse to deport even the worst criminals who do not even have European citizenship, because of a very vague convention about "human rights" in the European Union. European natives don't have any human rights anymore, only criminals do. Now thousands of terrorists from ISIS are coming back to Europe from Syria so we can expect a lot of problems in the coming decade.

[@Mars_Network]

I report your video because if your hateful speech

[@victorsung3582]

Wow, more tutorials are coming?! HYPE!!!

[@tanmaygupta638]

I hope I can meet you in person in future!!!Keep up the good work!!

[@romanzakharov6344]

Very informative tutorial(the best I think in this field), thank you very much) I cant find similar tutorial), what do you think about use pretrained embeddings for example for englesh language?(this will reduce the time of training)

[@hvasslabs]

Thanks! If you improve this tutorial with your ideas, just put it on your github and post a link here.

[@victorsung3582]

Holy sh*t this is gonna be epic!

[@hvasslabs]

Thank you! :-)

[@1234rediff]

You are doing an amazing job.. Keep it up. Learning so much from your videos...

[@ashikkapadiya]

Sir, you're great. Thank you for your tutorials.

[@KshitizRimal]

I like your videos.. they are honest and impactful. Please keep on posting more such cool videos in future as well. And I enjoy your political rant as well.. i didn't know the state of sweden right now. I am really sad to hear that

[@hvasslabs]

Thanks very much! The political situation in Western Europe is tragic and frustrating. It is hard to see how it can be resolved peacefully. Just the other day there was yet another Salafist terrorist attack in France.

[@NepaliFolk]

"I enjoy your political rants" रे | कत्ति हास्नु |

[@yoursaviour7613]

Best video ever

[@mashurshalehin4972]

This is really a great video man. Clears a lot of queries. But I still have a question. Will this work for like direct translation from French to Danish or something like that where we don't use English?

[@yannickpezeu3419]

thanks for the great tutorial ! And I must say I agree with you on a lot of the politics you talked about

[@RandomShowerThoughts]

lol the rant was great

[@longfellowrose1013]

thanks for your tutorials, really helpful and meaningful!

[@ShadArfMohammed]

Thanks so much for putting so much effort into this tutorial.keep it up.

[@bkovnkk6105]

thanks for the video haha i think we all live in shit hole

[@stevenbrown5668]

Thank you so much for posting this video. Your teaching and explanation style is outstanding.

[@hvasslabs]

Thanks!

[@VictorGallagherCarvings]

I am glad I found your video. I am working on an english to english translator and was hoping to pick up some ideas on network design. As for your rant, I don't blame you one bit. You sound very frustrated.

[@hvasslabs]

English to English translation? Please explain :-)

[@VictorGallagherCarvings]

Sorry for being vague, what I mean is I am trying to make a semantic simplifier . I found a dataset of 167,000 english sentence pairs with the target set being a simplified version of the source set. With some preprocessing and some minor changes to your model I started training a few days ago. Unfortunately this model will not train on two gpus. For some reason Keras does not split the batches correctly. So testing various models has gone slowly. Also I have had problems with the validation loss hanging somewhere in the range of .95 and 1.05, regardless of what I try. Your videos are some of the best I have seen and your code documentation is the absolute best. Regards, Victor

[@beizhou2488]

Many thanks to you. Thanks for this tutorial, I finally coded and trained my own simply machine translation model translating English to Chinese. Although the model is heavily overfitting, I am very happy. I am going to dive into the image captioning tutorial tomorrow. Hopefully, I can understand it in depth. Thank you again, you are so brilliant.

[@hvasslabs]

Thanks again for the kind words. I'm glad you are learning from these tutorials!

[@sandeerads]

Can you please tell me that how to save the trained model and the way how it can be loaded back again for reuse ?

[@BoneCrushGaming]

Hi I need some help. I have tried to get it working with the current code, but the model_train.compile() won't work. The specific error message is...ValueError: Rank mismatch: Rank of labels (received 2) should equal rank of logits minus 1 (received 2). I have tried just copying and pasting lots of the code from the Jupiter Notebook into what I have and it still doesn't work. It seems like it doesn't like the sparse_cross_entropy function we created. Any thoughts? Thanks so much for any help!

[@torpuotmajiok4980]

i really like your tutorial ,i,m doing projects to develop my native language on google . so can you plz share ur experience with me on how to do this ??

[@thobekile3802]

torpuot majiok I'm working on a similar project

[@torpuotmajiok4980]

@@thobekile3802 that great to hear z bro.. so can we plz share our experience

[@hannaseyoum9724]

I'm trying to do the same. How is your project coming along?

[@youssefel-shabasy833]

9:00 : 11:50 WTF 😂😂🤣🤣

[@shumshersubash]

u r awesome man.. loved ur videos.

[@JoseCastillo-fl8jn]

Omg you sound angry about that countries...

[@hvasslabs]

Do a search for grooming gangs in the UK. Doesn't that enrage you?

[@suryagaur7440]

Hey Dude great video as always. I am having one question, I am trying to pass 2d vector (like yours) instead of one-hot encoded 3d vector and use softmax as an activation function instead of 'Linear' one but I am getting an error that my decoder need 3d vector and I am passing 2d vector. Sorry for my naivety !!!

[@TerenceChill286]

Thanks a lot for the excellent tutorial. I got one question though. Your custom loss function expects y_true, y_pred values. When you compile the model the way you did, where does the model know from what to feed to your loss function. I suppose, that keras automatically feeds the final model output and what has been specified as the target_tensor to the loss function. But how does it know which one is y_true and y_pred since the argument names are arbitrary

[@mrvishwjeetkumar]

Hi Thanks for awesome tutorial. but i have some doubt in architecture of encoder-decoder, like how error back propagate . and how weight of encoder optimize. it will be very helpful if you explain.

[@gurpreetchawla2066]

Watched your videos today and all i could say that i got Best content till now on these projects with the practical implementations of the project whether of movie sentiment or text conversion from one language to another .Hats off bro !! ..Eager to watch some more Exciting projects on Tensorflow+NLP

[@thomasgordonclausen]

Hej Hvass, Tak for en stribe super klare videoer! Jeg sætter stor pris på dem. Thomas Google Translated: Hi, Hi, Thanks for a series of super clear videos! I appreciate them. Thomas

[@hvasslabs]

Velbekomme! Jeg går ud fra det her er din hobby? I bruger vel ikke TensorFlow i DSB?

[@thomasgordonclausen]

Det er hobby for min part, men som firma følger vi med i AI og eksperimenterer med forskellige løsninger :-)

[@saurabhkulkarni2177]

Can you please cover chatbots as well? also text summarization might be nice

[@23232323rdurian]

Yeah but please tell me where I am confused: I've been using the SAME dataset for 2+ years to develop my Synthetic Text Generator: Europerl Parallel Corpus\\English europarl-v7.es-en right? Well in practice it's a lengthy list of sentence-pairs that were HAND-PAIRED BY HUMANs back in the past, and CAREFULLY MANICURED TO MATCH AS EXACTLY AS POSSIBLE.... and probably with an eye to Machine Translation cuz it AVOIDs notorious problems such as: (quotation complications, indirect speech, full-stop '.' Tokenizing Hell, Named Entities, etc....) n one of very few quality long texts of sentence-pairs available for free on Internet (please inform me of others) ==>It's not virgin new heretofor non-existent input data you are trying to translate, it's rather one of the 2 million PRE-EXISTING Danish sentences, that ALREADY HAVE a PRE-EXISTING 'best fit' English, that a Human, not a machine or Machine Learning, judged optimal and paired together Isn't it? If not, please clear me up.... -->Now there are MANY ways to match em up, but they were ALREADY HUMAN HAND-MATCHED to start with... so the humans might as well have just assigned each of 2 million Danish sentences an integer as its Lemma, then the SAME integer as the Lemma for the equivalent English 'translation' in a Python Dictionary for example.... -->U dont need Tensorflow or Keras to do that....(tho they can and do), in fact it's ALREADY BEEN DONE BY HUMANs, otherwise none of this 'translation' would work at all.... -->And *ANYHOW*, after you get Tensorflow to match the sentence pair the HARD WAY (by TensorFlow), then why not just iterate thru the whole database of pairs n correlate em? For example, the Danish INs could be the Lemma to a Python Dictionary while the English OUT would be the Gloss n it'd probably match FASTER than by TensorFlow/NeuralNetwork/GPU everytime except for the first time... ===> Please I am trying to learn something, I'm not trying to annoy, so please clear me up... I use similar techniques, even the SAME English database for my Synthetic Text Generator.... ====> I find the bottleneck is inability to process all 54 million Tokens of data in one session by Python3.6.1 (doesnt fit, takes forever) so TensorFlow on giant GPUs is probably faster...but I dont own the hardware.... and *THANK YOU* for your hard work to produce this useful video.......the few people who truly understand contemporary Machine Translation are keeping it pretty secret, and most of em probably work in VERY FEW select places, such as Google Translate..... It's not so much a 'trade secret' as it is just VERY COMPLICATED, HARD TO VERBALIZE, and translation quality is improving so fast there's not much point writing it down, cuz it'll be OBSOLETE by the time it get documented.....

[@hvasslabs]

I didn't really understand your question. As I recall, the Europarl data-set consists of human translations between different European languages - I actually know some of the people who made the Danish translation. Our model is learning mappings of sequences of words in the source language to the target language. Ideally the model should also learn sub-sequences instead of just the entire sequences of words, so it can handle new variations of the words it has already seen. But of course, if you present the model with something it has never seen before, it will get confused and output something wrong. I think I try with some Danish slang which it cannot handle, because the Europarl data-set uses a more formal language.

[@SunilSharma-sp2vk]

Cool :)

[@abdulazizalhefdhi9133]

We would like to know the training/validation accuracy/loss

[@mohammedraihanhussain1438]

Sometimes bringing up politics is not a bad thing. I loved listening to something different in the middle of the tutorial. Please continue this practice. Not only politics but any other interesting topic in the middle of a tutorial increases my focus. lol. Thanks :)

[@hvasslabs]

Thanks! My political remarks are very controversial in Western Europe - which is of course why we have the big problems we now do, because you aren't allowed to talk about these issues.

[@TheWorpler]

Excellent tutorial sir - and you are correct about the state of politicians in much of Western Europe. Tusind tak!

[@hvasslabs]

The past 30 years have seen left-wing politicians in Europe pissing on the working and lower classes of society. Finally the Social Democratic Party of Denmark has started to wake up and realize they don't want to live in a new Saudi Arabia or Afghanistan after all. The irony is that two of the main people driving this new agenda of the party are Lars Aslan (Danish-Turkish I believe) and Mathias Tesfaye (Danish-African). (Velbekomme :-)

[@MiKenning]

Imagine entering a lecture-hall and your mechanics lecturer starts ranting about the price of bread.

[@hvasslabs]

Your comment is incredibly rude and arrogant. I don't see any advanced lectures / tutorials on your KZbin channel, but I do see that you like Jeremy Corbyn and that you have put up a video criticizing the Catholic Church, which of course explains your attitude here. In this video I am showing a list of European languages related to the data I am working on. I then decide to make a very angry political statement because it is entirely appropriate the way things are going in Western Europe - thanks to politicians like Jeremy Corbyn. The following Salafist attack happened a 5-minutes walk from where I used to live in Copenhagen and where my father's family has lived for almost 100 years. I crossed this exact spot several times every day when I lived there. At the funeral procession for the terrorist were some 1500 muslims, some of them aggressively shouting a victorious ALLAHU AKHBAR in the streets of Copenhagen after their friend murdered two people with a machine gun! And you think this is like "discussing the price of bread"? What is the matter with you?! en.wikipedia.org/wiki/2015_Copenhagen_shootings

[@zakikhalid3170]

Hi, i am from Malaysia, i am working on my translation machine project. i find your video is funny. i hope i can meet you in person in future. i would like to learn from you regarding natural language processing.

[@hvasslabs]

I was in Kuala Lumpur maybe 5 years ago. It was hot! :-)

[@zakikhalid3170]

@@hvasslabs hahaha, maybe it was dry season. We do have wet season starting october to december but its crazy, everywhere is flooding. Come here one more time, i will show you around. :-D

[@MrA_TheFIST]

Hey, i trained this model on cornell dialogue corpus but it is giving same result at each inference. Please respond its kinda important i have to show project in college.

[@hvasslabs]

I also experienced this problem when I was working on the tutorial. I think it can happen when the model hasn't trained long enough, or when the loss-function does not calculate the mean over the entire batch and sequences as I did in the function sparse_cross_entropy() using tf.reduce_mean().

[@MyDanish100]

Bhaiya , I have my final year project presentation on neural machine translation. So can you share your project work and with me which you did on nmt. I will be thankful of you Please bhaiya I am in trouble because i am not understanding anything and I have to submit the project in upcoming 2 months. My mail id- [e-mail] Phone number- 7992211249

[@MyDanish100]

Akshay dadwal bhaiya please

[@karFLY1]

Recently I've tried PyTorch. And it is so cute! What do you think about it?

[@hvasslabs]

Haven't tried it.

[@saurabhkulkarni2177]

whenever you say "So called" you get me

[@erikbrorson9529]

Håller med om att Sverige är coolare än Danmark ;) I was brought here from a comment you left on a keras github issue, regarding your implementation of the sparse cross entropy loss. I tried to implement it this way (telling keras to have a 2 dimensional output) in one of my own projects (also seq-2-seq in keras) but found some strange behaviour related training. In essence, if I train the model for some amount of steps/epochs, the loss will decrease more when using the non-sparse loss function that the sparse version. I managed to reproduce the issue using toy data, I posted the notebook here erikbrorson.github.io/2018/03/24/Figure-out-problem-with-sparse-cross-softmax/) Do you have any idea what might cause this?

[@hvasslabs]

Why aren't you furious about what is happening to your previously so beautiful country Sweden? Do you think it's going to get better? I can tell you that it's not - it's going to get exponentially worse! Sooner or later Swedish law and culture will be completely gone. What you are seeing now is just the very small beginning. Just a few weeks ago I saw in the news that a Swedish court of law had made a verdict that was straight out of Sharia law. If the Swedish people had bothered to read the Quran you would have known this was exactly what happened. The courts are going to overturn that verdict, but in 50 years you will no longer have the political or judicial power to do that, because there will be so many Salafists that they will effectively be ruling your country. You can see how well that has worked for the Muslim world. They are a population of more than 1.5 billion people and yet they have only 2 (TWO!) Nobel prizes in science. One of those was a man who worked in England, the other was working in USA. Why is that? Because under Sharia law you will get KILLED if you criticize Islam. How can you make any scientific progress if you cannot criticize a 1400 year old book that is scientifically wrong?! That is the tragic future of Sweden!

[@hvasslabs]

Regarding your technical question, I don't know what the problem is. I started another thread on GitHub about this but it doesn't seem to have very high priority for the developers: github.com/tensorflow/tensorflow/issues/17150

[@erikbrorson9529]

Yes, I saw that issue. My problem was really that it trained extremely slow, as compared to the ordinary cross entropy implementation. I ran a language model last night, with categorical-crossentropy it converges nicely but with longer run times, the same model implemented with the tf.sparse... loss function, using your trick with the placeholder. The cost isn't decreasing at all in the first two epochs. I was curious if you had witnessed similar behavior, or if the issue lies somewhere else.

[@JoseCastillo-fl8jn]

English is Danish with noise and viceversa.. :p

[@hvasslabs]

Hehehe! The languages are indeed closely related! Some words are almost exactly the same in English and Danish - however, an English person will not understand Danish and vice versa without schooling. Danish is closer to Swedish and Norwegian and sometimes I understand them but if they speak in a local dialect then I understand almost nothing of what they say.

[@vito135c]

Thank you

[@shahargiv]

thanks!