Training BERT #4 - Train With Next Sentence Prediction (NSP)

Next sentence prediction (NSP) is one-half of the training process behind the BERT model (the other being masked-language modeling - MLM).
Although NSP (and MLM) are used to pre-train BERT models, we can use these exact methods to further pre-train our models to better understand the specific style of language in our own use cases.
So, in this video, we'll cover exactly how we take an unstructured body of text, and use it to pre-train a BERT model using NSP.
Meditations data:
github.com/jam...
Jupyter Notebook
github.com/jam...
🤖 70% Discount on the NLP With Transformers in Python course:
bit.ly/3DFvvY5
📙 Medium article:
towardsdatasci...
🎉 Sign-up For New Articles Every Week on Medium!
/ membership
📖 If membership is too expensive - here's a free link:
towardsdatasci...
🕹️ Free AI-Powered Code Refactoring with Sourcery:
sourcery.ai/?u...

[@sinabd1396]

Even after 3 years I haven't found such an amazing, easy to follow tutorial series on this topic. I really loved it James. Thank you!❤

[@goelnikhils]

Amazing Video. Amazing Explanation with so much ease.

[@niteshmethani9884]

Amazing video, James. It's just what I was looking for. Bought your course on udemy just to learn from you.

[@jamesbriggs]

that's awesome, happy to have you here!

[@anshoom]

@James Briggs superb job with the explanation, your voice is very soothing as well, kept me focused+awake while watching, ha ha! Curious if it matters to do MLM and NSP training in any particular chronology? In a separate context, should you have insight, I am trying to pre-train on my corpus for sentence embeddings, does both stages of training are still required or just MLM is enough ? Thank you!

[@basharmohammad5353]

Thanks for the nice video. Really helpful!

[@mohamadrezabidgoli8102]

Hi James, Thanks again for the high quality content you are making. Can you also make a video for using (and fine-tuning) BERT for binary classification problems (like say sentiment analysis or machine-generated determination)?

[@jamesbriggs]

hey just saw your comments - will definitely add it to the list, I think you already found one on Bert for classifications, there's also this one: kzbin.info/www/bejne/ppvXn555fKqfmac Which includes the saving/loading of your model at the end too I haven't heard of machine-generated determination before, do you have any links to articles/videos on it? Could be interesting to look into...

[@sxmirzaei]

Hi James, thanks for the video! How do you save the model, load and use it for prediction? none of the videos in the playlist cover it. Thanks!

[@Wigglebus]

Hi, great video. Could you also explain or show how to make predictions with this model?

[@d3v487]

Amazing Guide Love it💗. Very intuitive Explanation🧠. I find out that there are not that many good resources about Text Summarization and fine-tuning it. Please Upload a Video of Fine-tuning Text Summarization model on a dataset using Huggingface Transformers. It'll be very helpful for every NLP Practitioner. Love From India❤️🙏🏻.

[@jamesbriggs]

That would be awesome, will add it to the list!

[@sinamon6296]

Very helpful. Thank you ! subscribed :)

[@jamesbriggs]

Awesome, thankyou!

[@intoeleven]

Thanks for providing this wonderful course! Wonder where is this jupyter notebook?

[@jamesbriggs]

Added to video description :)

[@sabrinabani7973]

If I have several paragraphs and want to see if they are consecutive or not, how do I write label =0 or label =1?

[@vijayendrasdm]

Hi James Great video again.!! Please clarify We are creating just a pair of data/record i.e sent_a, sent_b, label from one paragraph . Suppose a paragraph has 10 sentences , shouldnt we come up with more records ( , ) from same paragraph

[@jamesbriggs]

ideally yes if you have plenty of sentences, we can also do paragraph 1 followed by paragraph 2, etc (doesn't need to be 'sentences' specifically, just as long as there is some sort of topic continuity between the two), the code here is written to be as simple as possible though :)

[@vijayendrasdm]

@@jamesbriggs thanks James

[@TO-il3vc]

wouldnt it be important to make sure the random sentence for sentence B doesnt end up being the TRUE next sentence? I implemented a simple while loop to check if that is the case, before appending to sentence_b list.

[@jamesbriggs]

Yeah you can add that in. I don’t here is to keep the example as simple as possible, and with a large dataset the actual impact on performance would be minor due to the (probable) low number of times this happens

[@abdallahmahmoud8642]

Hey James, thanks a lot for the information. I have a question when you have time, I currently have a pre-trained model using BertForMaskedLM, my goal is to fine-tune it with BertForNextSentencePrediction. How should one go about it? Also I might be able to transfer the weights to BertForPretraining and fine tune it there. Is that correct? Thanks!

[@jamesbriggs]

hey Tito! you should be able to save the MLM model with save_pretrained('path/to/model ') and then you should: ``` from transformers import BertForNextSentencePrediction model = BertForNextSentencePrediction.from_pretrained('path/to/model') ```

[@abdallahmahmoud8642]

Thanks a alot, this was really useful