Rules Based NER in Python (Named Entity Recognition for Digital Humanities 02)

GitHub repo for this series: github.com/wjb...
In this video, we explore rules-based NER on Harry Potter, book 1. The purpose of this video is to show the limitations of rules-based NER. In the next video, we will see how these limitations can be overcome with machine learning-based methods in the spaCy library.
Harry Potter text: www.pauladaunt....
Geeks4Geeks source referenced: www.geeksforge...
If you enjoy this video, please subscribe. I provide all my content at no cost. If you want to support my channel, please donate via
PayPal: www.paypal.com...
Patreon: / wjbmattingly (its my www.themedievalworld.com account as well).
If there's a specific video you would like to see or a tutorial series, let me know in the comments and I will try and make it.
If you liked this video, check out www.PythonHumanities.com, where I have Coding Exercises, Lessons, on-site Python shells where you can experiment with code, and a text version of the material discussed here.
You can follow me at:
/ wjb_mattingly

[@python-programming]

Check out the Textbook for this series: ner.pythonhumanities.com/intro.html

[@python-programming]

GitHub Repo for the files for this series: github.com/wjbmattingly/ner_youtube . Please note, we will explore rules-based methods in more detail in video 4 when we learn more about spaCy.

[@anjumrohra9778]

Amazing video Sir. I am facing an issue while loading a json file using "json.load()". It results in an error as "JSONDecodeError: Extra data: line 2 column 1 (char 4144)". Any suggestions would be of great help. Thanks :)

[@python-programming]

Thanks! You need to specify the file as an object as an argument. So load(f).

[@anjumrohra9778]

@@python-programming Thanks for replying! I did it, but still, I am facing the same error.

[@python-programming]

@@anjumrohra9778 can you take a screenshot of your code and the JSON file? It may be as simple as a missing comma in the JSON.

[@anjumrohra9778]

@@python-programming sure I will. And where should I send the screenshot?

[@python-programming]

@@anjumrohra9778 Sorry about that. I probably shouldn't reply before I've had coffee =). You can send them to me as a DM on Twitter (wjb_mattingly).

[@shry92]

Thanks for the really detailed instruction, William I could follow along easily. With the known limitations of the rules based approach, could you please provide an example of where this approach might be used or even preferred over the ML approach?

[@python-programming]

You will actually see videos coming out over the next few weeks that talk about how spaCy 3.5 has expanded their rules-based pipelines to allow fuzzy searching which makes them even more powerful. A classic example for an EntityRuler would be something that has a finite list. I am on a project right now and the leader wanted to extract Continents as a distinct type of Location. I could include CONTINENT into an ML model, but that would be unnecessary since there are only 7 continents in the world and they appear in only a few variant forms in English (such as Oceania). This problem is much better solved with an EntityRuler.

[@zyxwvutsrqponmlkh]

Do what you want cause a pirate is free, you are a pirate! Yarr har fiddle dee dee Being a pirate is alright to be Do what you want cause a pirate is free You are a pirate! You are a pirate! Yeah! We got us a map (A nao!) To lead us to a hidden box Thats all locked up with locks (with locks!) And buried deep away Well dig up the box (The box!) We know its full of precious booty Burst open the locks And then we say hooray! Yarr har fiddle dee dee If you love to sail the sea Weigh anchor! Yarr har fiddle dee dee Being a pirate is alright to be Do what you want cause a pirate is free You are a pirate! Arr yarr ahoy and avast Dig a deep and youre digging too fast Hang the black flag at the end of the mast! You are a pirate! Yay! Were sailing away (Set sail!) Adventure waits on every shore We set sail and explore (Yaar har!) And run and jump all day (Yeah!) We float in our boat (The Boat!) Until its time to drop the anchor Then hang up our coats (Aye, aye!) Until we sail again! Yarr har fiddle dee dee If you love to sail the sea Land ho! Yarr har fiddle dee dee Being a pirate is alright to be Do what you want cause a pirate is free You are a pirate! Yarr har! Wind at your back lads wherever you go! Blue sky above and blue ocean below You are a pirate! You are a pirate!