Is web scraping legal? 🫢😳

🔗 Follow me on LinkedIn 👉 / luke-b
🆇 OR on X/Twitter 👉 / lukebarousse
Courses for Data Nerds
==================================
📜 Google Data Analytics Certificate (START HERE) 👉🏼 lukeb.co/GoogleCert
💿 SQL for Data Science 👉🏼 lukeb.co/SQLdataScience
🧾 Excel Skills for Business 👉🏼 lukeb.co/ExcelBusinessAnalyst
🐍 Python for Everybody 👉🏼 lukeb.co/PythonForEverybody
📊 Data Visualization with Tableau 👉🏼 lukeb.co/Tableau_UCDavis
🏴‍☠️ Data Science: Foundations using R 👉🏼 lukeb.co/RforDataScienceJH
➕ Coursera Plus Subscription (7-day free trial) 👉🏼 lukeb.co/CourseraPlus
👨🏼‍🏫 All courses 👉🏼 kit.co/lukebarousse/data-anal...
Build a Portfolio
==================================
👩🏻‍💻Build portfolio here 👉🏼 hostinger.com/luke
Rebate Code: "LUKE"
My Portfolio 👉🏼 lukebarousse.tech/
Books for Data Nerds
==================================
📚 Books I’ve read 👉🏼 kit.co/lukebarousse/book-reco...
📗 Data Analyst Must Read 👉🏼 geni.us/StorytellingWithData
📙 Tableau 👉🏼 geni.us/tableau
📘 Power BI👉🏼 geni.us/powerbi
📕 Python 👉🏼 geni.us/pythontricks
Tech for Data Nerds
==================================
⚙️ Tech I use 👉🏼 kit.co/lukebarousse/computer-...
🪟Windows on a Mac (Parallels VM) 👉🏼 lukeb.co/ParallelsFreeTrial
👨🏼‍💻 M1 Macbook Air (Mac of choice) 👉🏼 geni.us/M1macAir8GB
💻 Dell XPS 13 (PC of choice) 👉🏼 geni.us/DellNewXPS13
💻 Asus Vivo Book (Lowest Cost PC) 👉🏼 geni.us/AsusVivoBook15
💻Lenovo IdeaPad (Best Value PC)👉🏼 geni.us/LenovoIdeaPad15
Social Media / Contact Me
======================
🙋🏼‍♂️Newsletter: www.lukebarousse.com/
🌄 Instagram: / lukebarousse
⏰ TikTok: / lukebarousse
📘 Facebook: / datavizbyluke
📥 Business Inquiries: [e-mail]
As a member of the Amazon, Coursera, Hostinger, and Parallels Affiliate Programs, I earn a commission from qualifying purchases on the links above. It costs you nothing but helps me with content creation.
#dataanalyst #datascience

[@carlosalba9690]

Alternative Title: “Dude discovers TOS” lmao

[@gregthwuen]

If you never registered an account on LinkedIn and never accepted the TOS, you can't violate the TOS. Of course your country's laws still apply, which may prohibit sth like web scraping.

[@carlosalba9690]

@@gregthwuen it’s not illegal to scrape web data generally speaking. But the LinkedIn EULA applies to any person or entity that uses LinkedIn. If you don’t agree you’re expected to not use the software and delete it. Any person or entity that uses LinkedIn is also subject to the LinkedIn User Agreement, Privacy Policy and Cookie Policy. On the second bullet point of section 8.2 of LinkedIns user agreement they explicitly state that you will not “Develop, support or use software, devices, scripts, robots or any other means or processes (including crawlers, browser plugins and add-ons or any other technology) to scrape the Services or otherwise copy profiles and other data from the Services;” Users of a website do not need to be registered in order to be considered users. LinkedIn differentiates between “Members” and “Visitors” in their paperwork. LinkedIns policy is not the law of land at least in the US but they can send cease and desist , ban you and even sue you for violating their terms. This also applies to folks in the EU as far as I remember.

[@quebono100]

I thought the same. xD wtf.

[@joseluislopes3956]

​@@carlosalba9690 but LinkedIn does not give you access to 99% of the website without creating an account?

[@immortalsun]

It’s an informative video.

[@NicEeEe843]

So companies won’t let us scrape their info but they’ll happily sell ours?

[@LukeBarousse]

🙌🏼

[@eeHMFIC]

Correct. Your data is the commodity.

[@kakterius]

That is also why they don't want you scraping it xD

[@dabbopabblo]

So you have an issue with that but happily agree to their tos to benefit from their free services?

[@LukeBarousse]

@@dabbopabblo very good point, it's probably why I don't read TOS's very well...🤣 but I would argue that it's not necessarily free, they're getting my data

[@kardz1848]

Alternative title: "data scientist tries to find job by collecting data(gone wrong)."

[@LukeBarousse]

🤣

[@JenOween]

Imagine if LinkedIn took phishing job posts and scam posts as seriously as they take scraping.

[@VoidplayLP]

Data is what they sell so scraping hurts the bottom line lol

[@RidingWithGerdas]

Next time when you scrape, add some randomness to your process to look less like a bot

[@LukeBarousse]

This is a good point! Actually did some time variation randomness, but that wasn't enough

[@RidingWithGerdas]

@@LukeBarousse can imitate random clicks back and forth with Selenium

[@LukeBarousse]

@@RidingWithGerdas Yeah, I think the main problem was I was using the same IP address... think a proxy would be better

[@StrokeMahEgo]

@@LukeBarousse how would that matter? People log on to social medias including LinkedIn from the same ips all the time. (Home, work, etc) very routine.

[@BenRangel]

@@StrokeMahEgo Yeah but most bot detectors are still quite simple and look look for abnormal request per minute from certain the same ip, userAgent, etc. A more advanced detection could look at stuff like time spent. if 100 visit is never more than 1 seconds each - it's a bot. (Allthough most bot detectors are usually quite basic )

[@lachee3055]

In Australia, if it is publically available it's fair game as long as it's not a detriment to the service and other users.

[@tjdjultima]

I’ve done similar tasks professionally. Rotate your IPs, purchased leases to residential IPs work well, and you can set request headers to better imitate a “real” browser instead of whatever webdriver you’re using. A lot of times you can isolate the data call without having to render a bunch of images and just fire that as it’s own request through postman or whatever and then only get the json for every listing. LinkedIn is pretty notoriously tough to do thoroughly though.

[@EpicNESMetal]

How is that helping if you have to log in with your account? Isn't it much more obvious if the same account is beeing used by many different IP adresses?

[@beastly_neon]

@@EpicNESMetal multiple accounts are created using different ips

[@buddysteve5543]

As I like to say, if there is the will there is a way! That pretty much applies to everything except death and taxes! LoL!

[@MmeHyraelle]

And thats why i need an account to view linkedin now... Thanks.

[@UlrichTonmoy]

MS be like only we are allowed to scrape public data and steal private one but not the other way around

[@Pod-Z]

Scraping actual useful stuff is prob my second favorite programming activity, forget the law do it anyway and if they want to come for you barricade yourself in a log cabin and let the k go

[@LukeBarousse]

NGL, I can agree, it is pretty fun to scrape data

[@adio1679]

What’s your first favorite?

[@Pod-Z]

@@adio1679 I havent done it in a few years but Making Runescape bots in Java , they usually have great library's, alot of support and you see instant results even after just a few lines of code. its pretty satisfying

[@EllaNut]

I believe it is illegal to scrape certain sites such as government sites, also if you cause a DOS that is illegal.

[@vijayragav1865]

what does "let the k go" mean? Could you please explain. I am confused

[@volterkeg]

It's not illegal, but it can to lead to some extremely overwhelming situations for the site if left unregulated. Whether or not a website is ok with it, you should time your bots. Don't run your bots with uncapped speed. Some websites even require you to follow some guidelines like one page per sec. The benefit of a bot should be automated consistency not speed.

[@eliasb6244]

3 things: - proxy pools - rotate IP addresses - randomize sleeps between requests

[@jithendra.k.sfirst_yr_b.sc9574]

I'm into this... Did some illegal stuff, by being ignorant....😅

[@LukeBarousse]

🤣

[@forbiddensouls]

I myself built a scraper called "Linked In Booster" All it does is, it searches people with ur search string that can be anything, and start sending connection requests to people to boost ur network..... I didn't know that it was legal, altho i didn't get banned but stopped doing it. Also there is a plugin that comes with puppeteer, that tricks any of the AI metrics system that it is a human that's operating the app. I tried it on KZbin and it worked.

[@wanderingronin305]

Not illegal just against their use policy. Company policies aren't laws

[@jithendra.k.sfirst_yr_b.sc9574]

@@wanderingronin305 i know, it's just "I" words🥲😶

[@Jajajaja1231]

@@wanderingronin305 Then how did a whole legall case was taking place by this¿

[@gorillaz9694]

When i built my first web scraper, i already noticed that it probably illegal becuase i need to bypass the "I'm not a robot" chapta.

[@blenderowl6495]

You know that breaking ToS, while bans you from the service, doesnt mean what you did was illegal. When you sign up to use a service, lets say for in this case first person online shooter, they usually ask you to click "I agree to the terms of service" in order to continue. This document dictates what you can and cannot do with the video game. Any form of cheating is against ToS, selling your personal account is against ToS, sharing your account with another player (pressumably to boost your rank) is against ToS. If you get caught breaking these rules the service has the right to ban you from that service, i repeat ban and not arrest.

[@gorillaz9694]

@@blenderowl6495 I see, thank you for the insight.

[@ssherwood7245]

So when you scrape schedule the read to occur at a random time and with day spread. Also if you occasionally use the account to comment it will confuse their system

[@kexec.]

for the sake of your time, linkedin lost the battle since it was public data

[@harshitsati]

Arrest me officer 😳 ⛓️ I'm a criminal

[@LukeBarousse]

😜

[@sauce6534]

You should have made or bought dummy linked in accounts, used those as scrapers as well

[@vishnudixit7754]

I tried doing something similar on Instagram, but scrape the like count of a page using selenium autoscrapper, but immediately got banned. I freaked out and deleted the account and the email associated with the account, I'm glad I'm not the only one this happened to 😂

[@Benexdrake]

I have my own Web Scraper, for Crunchyroll, Imdb, Pokémon, Pokémon Tcg, Magic Tcg and Honda Parts in C#, this project makes much fun. I use Selenium and Httpagility for it.

[@chedisLoL]

Imagine that. You web scrape a Python job. Use the bot to apply to the job and state that the submission was automated and done via a bot. You get hired and simultaneously banned from linked in…

[@LukeBarousse]

🤣

[@kizhissery]

No huge website allow scraping data , last thing to do is settimeout between each mouse movement but then scraping would take ages. If I would scrape I might directly fetch backend REST api , providing headers and dynamically updating cookie every 12hrs, also huge apps like fb uses gql, so may not feasible or learn gql endpoint which provide entire data.(only happen if you know all the queries for gql)

[@thanhquachable]

i am just curious, if you directly fetch backend API, they have even more reasons to sue/charge you because the backend API is not publicly available for us to make calls to without their explicit consenst 😂? If we simply render the whole page , at least "this is what I and everyone sees publicly", i am just smart enough to extract data I need to quickly lol. But yeah, getting a nicely formatted json file with all data you need is very tempting hahahha

[@peterbauer1494]

It shouldn’t be illegal, public information should be public information. But like... I get why LinkedIn doesnt want bots running rampant on their website

[@markpolop5171]

You need to rotate ip’s and user agents to reduce chances of being caught and flagged as a bot

[@Michael-ty2uo]

This sums up my experience with scraping Facebook marketplace

[@SportsIncorporated]

A few years ago I scraped data that was in the public domain, from websites around the world. I never had a problem with accessing the web pages. The problem was that the webpages changed. You had to constantly rewrite the scraping code, or change inputs to scraping tools. It might have cost less and reduced a lot of stress. Just by hiring low cost labor to manually input the data.

[@drowsy4400]

Or.. you sign up to get an email when a job of your interest opens up

[@junkoscarlet6586]

Scrape so fast, the backend crashes

[@ericadacunhaferreira9611]

This was actually a project idea that I had for quite some time, to see job distribution in different states/countries, cross relate to salary by company from GlassDoor and all that, while researching, I discovered that there is an informal LinkedIn API, so you don’t actually need to scrape all the data, quite helpful There are a bunch of articles on Medium about it too

[@nasimicin]

Linkedin: not permit crawling Google, Bing: Do crawling anyway Is this some kind of bot discrimination?

[@LukeBarousse]

Yeah I think so 🤷🏼‍♂️

[@peasantlord135]

I imagine it's king knocking your door to do you a favor vs a beggar knocking your door for money 😂

[@birdpump]

It's called rate limiting, it can be bypassed with multiple proxies.

[@Michael-Martell]

I hope the dev teams that received the ticket to block you watches this.

[@thrashassault1]

When modal screen didnt answered and your script keep diggin in the backgroınd they catch you

[@HaseebHeaven]

I already knew that thats why never tried with LinkedIn. There are Github projects for that as well but doesn’t come with warranty.

[@motoshan]

Another video where the title question never gets answered. Brilliant.

[@chinchan9]

How do I stop getting banned while scraping websites?

[@ysdhnm]

All actions on my scrapers pass though a randomizer. Button hit coordinates, time between clicks, list processing (avoid sequential link following) and splitting up processing of payloads. Humans take breaks and so should scrapers, create multiple accounts with a generated user agent and proxy working in shifts leveraging timezones.

[@skeletonboxers7336]

I’ve scraped linked in and indeed before and all you need to do is add some scrolling in between or buffer it with some time so it isnt instantly making http requests at impossible for human speeds. I consider it a way to automate the menial part of scrolling and glancing when i could just have it to the side while I work, eat, etc, still not legal sure, but in a way I’m still confining it to a relatively quick reader instead.

[@LukeBarousse]

This is good to know!

[@zaskens8083]

What if we try to make a fast way to scrap manually data?

[@jalilsharafi]

who said you're not allowed to do something only because they wrote it somewhere, did you sign it? if not I don't see how that can be used in any court against web scraping

[@jalilsharafi]

@Jhon Doe yes then you’ve signed something but I can go on any realestate website and search whatever without making an account, I may as well web scrape their data by sending queries and create my own database … I can’t see how’s that any violation…

[@jalilsharafi]

@Jhon Doe further even if you’ve signed some terms and conditions even then you should be allowed to use the publicly available information

[@jalilsharafi]

@Jhon Doe ban yes, sue in court no

[@IArkProject]

"Are you one of us?" Haha perfect clip

[@RadenHZ26]

Because of that ToS, now i scraping data manually for my client, and it was pain in the arse. Lmao

[@nirvansiga5575]

I had a similar issue, adding a small delay using 'sleep' helped get around the bot checker. edit: forgot to mention that it was another site not linkedin that i was scraping so results may vary.

[@ArikShalito]

If you find a way to scrape without creating an account and missing the small letters you agreed on, scrape on, brave warrior, the law is on your side.

[@antipainK]

Yeah, if it's performed commercially it would light up my "grey area" indicator, but for personal non-profit projects, I think it's perfectly fine.

[@satishrkulkarni114]

Can TOR be used ? Guess thats even more illegal

[@naikiran9624]

Shit, I just got this error yesterday as no jobs found. Yeah should have read that first.

[@TinaHuang1]

it's not illegal if you don't get caught right :x

[@LukeBarousse]

Exactly!! 🚔😳

[@mateocortes9546]

same thing happened to me, luckily was able to solve it by using a vpn 😂

[@LukeBarousse]

I want to try this as well at some point! Thanks for sharing this!

[@audr]

How did you build your scraper? RPA? something else?

[@nemodot]

Used to work for Avature, a SaaS company that was for talent search. We had scrappers for every effing database, some provided an API, most of the time it was pure webscrapping. For linkedin we had to do some type of chrome extension to manage to manually extract canidate resumees.

[@racvets1]

From what I have heard, since you logged in, any data accessed is bound by their TOS, aka your screwed. Now, if the data is publicly accessible without a login, that is different. That is like putting a no photography sign in front of an outdoor place, not really legally enforceable. (Not a lawyer)

[@devilliersduplessis7904]

Willing to share a dataset with a fellow Data scientist?

[@LukeBarousse]

Yeah! So the jobs I scraped is now pretty outdated... but if you go to my "How I use Python" video I have a new dataset that is publicly available via Kaggle in the description... also the video has more info on the dataset

[@BrianGivensYtube]

But if you went through manually, it would be fine. But because you can do it quickly, it’s banned.

[@titodenino]

what the purpose of scraping and how could someone use it and what is it?

[@dbanga5]

Did you use proxies?

[@DendrocnideMoroides]

but why does it not like web scraping?? it is anyways publicly available data

[@lilmrmagoo]

because someone can then go and make another website that copies them.

[@acedigibits9079]

your bot might have been rate limited or soft banned. Secondly if you are scraping publicly available data for personal usage then there is nothing illegal in it, you are simply saving time instead of visiting those manually.

[@itznukeey]

You wanted to say you had a low delay on your web scraper

[@Adomas_B]

So they can collect our data anytime anywhere but we can't do the same?

[@vincentjanse]

What frameworks did you use? I'm trying to figure out how to scrape tiktok and KZbin for the most popular videos.

[@LukeBarousse]

selenium

[@iamTMBTM]

Super novice move… most sites have had anti scraping clauses in their terms for well over a decade.

[@ericadacunhaferreira9611]

Yeah, I was actually surprised that he didn’t know that

[@southredmondtoxik1885]

I make a weather API. But now it give me an error like you have been blocked because we have registered an unusual ammount of traffic from your IP address. So I can't finish my project because of this. How can I solve this issue

[@rouisaek]

IDK if the bot you program have some sort of rate limiting or like a delay of 1sec between each request!!

[@voidpointer398]

Did you used selenium? And how did you automate the bot to work after regular intervals?

[@LukeBarousse]

Yeah selenium! just ran it daily myself and built the script to request data at random intervals

[@voidpointer398]

@@LukeBarousse oh, thanks for replying. I also studied about it and found an automated way of doing it by using windows task scheduler. You can either use the pre installed gui or can use pywin32 for python.

[@oguz-qb5rl]

Tutorial on building a web-scraper from scratch?

[@LukeBarousse]

Let me see what I cando on this, I appreciate the recommendation! 🙌🏼

[@Karmasu_L]

But the website is allowed to use cookies and other tool to pull whatever data from user that they can?

[@christianherrera4729]

Alt tite: Dude doesn't know what robots.txt is

[@SandraGonzalezUslar]

Just LinkedIn or other platforms too??

[@rorschacht8478]

Try to access without accepting TOS. If you manage to, then you'll be completely in the clear as there are no laws against bots or scraping. The only reason you could be charged for anything is if you break TOS, which can't happen if you never accept them.

[@MattIn3rtia]

"Is web scraping legal" Google has left the chat

[@xasser]

Multi accounts and residential or mobile proxies with unique user agents. Will work depend on how much you think this data is worth.

[@nohedsheikh3764]

it's ban because that's how you don't spend your useful time on their website and don't watch ads .

[@cbjueueiwyru7472]

Terms of service doesn't mean it's illegal. It just means it's the terms you agree to when using their service

[@saurabhrawat3878]

Did you have course for web scrapping

[@LukeBarousse]

I don't... I need to look into this more

[@LunaticEdit]

Honestly this is true for 99% of all websites with data worth scraping. If you want to scrap you're going to have to work in some mitigation logic, and _always_ scrape through a proxy - not to hide your tracks so much as to not lock yourself out if you actually use their site legit.

[@yosbel12]

Use proxies

[@cameronord7750]

They have anti scraping measures now too. I mean the site basically useless if you dont scrape it because the search is literally dogwater and i found it was the only way to actually filter the results to get actually relevant jobs

[@bosshaug5672]

Lmao I did the same thing on indeed and got banned for like a month haha

[@LukeBarousse]

🤣 Dangit Indeed!!!

[@da_ta]

thanks for this

[@dylanakent]

Data viewed by the public on the internet via a privately owned corporate site does not necessarily equal public data.

[@theshuman100]

why we cant have nice things. some company decides to just download and reupload a website as their own

[@WolfSingh]

Why didn't you just use proxies ?

[@Schlohmotion]

Look closely. The TOS says "third party software". If I was a lawyer I would argue, that you wrote the scraper yourself. Meaning no software of a third party was involved; Just yours - Software made by one of the two parties involved.

[@LukeBarousse]

I didn't catch this! This is good! 😈

[@mjt1517]

Third party means any software not directly made or authorized by LinkedIn/Microsoft. Any software made by a user would be third party software.

[@Schlohmotion]

@@mjt1517 I don't know how your country defines "third party" legally.... But in my country, the third party is called third party, becaus it is literally the third party (the first and second party are the parties that set up a contract and accept said contract).

[@brockobama257]

Web scraping should be legal and information should be free and available to everyone

[@shahraanhussain7465]

Then how would linkedin earn, Somehow they are also selling the data in the market with different name.

[@NeroCat9999vr]

You didn’t need to read anything. It’s your computer, with your code, scraping fully public info. If anything, you should work on your code more and try to scrape more. There’s nothing illegal about code development on your own PC

[@mjt1517]

I don't care about the legality of scraping, but it's not just his computer. He's using his computer to interact with THEIR computer network. So there's more involved in this than just what you've stated. But again, I dgaf about what they want. I'll scrape whatever I damned well please. TOS or no TOS.

[@kevinfultz07]

But what did you do with all that “dadda”?

[@demonetiz3d]

Next time you scrape data, dont post the whole confession online

[@ab5441]

I would assume no. It is not illegal to write down or screen shot that information then share it. So why would it be illegal to automate the task?

[@parkuuu]

I made the same using Python Selenium and BS4, and it still works. The omly trick is not to log in. Voila.

[@LukeBarousse]

I like this approach of not logging in; I should have done this from the beginning

[@parkuuu]

@@LukeBarousse It doesn't show results based on your profile tho. I tried searching the same parameters when logged in and not, both show different results, and SOMETIMES it gives me the slider captcha which can be avoided by setting longer sleep periods

[@scottcampbell2707]

The TOS in the video bans third-party software. If you write it yourself, it is not third-party (if it os considered third-party, who would the third-party be?)

[@voxelfusion9894]

The company is first party. The user is 3rd party. The tos are accurate.

[@akam9919]

@@voxelfusion9894 ...wouldn't you be the second party...since you are the one agreeing (or "agreeing") to the TOS?

[@AlsamaZip]

Or, you could use their free Jobs API to grab that data instead. Removing the hassle of having to log int and scrape html.

[@MainDoodler]

Use proxy + different headers

[@PS3PCDJ]

Go through a public dataset manually LinkedIn: 😄 Go through a public dataset with a bot LinkedIn: 😠

[@fevicoI]

Chad web scraper says everything is legal

[@benjamintaylor2757]

Arent there multiple companies that base the whole business model on scraping data from LinkedIn and selling it as leads ?

[@LukeBarousse]

Yep, quite a few actually!

[@stillready6405]

It it not possible to scrape data, and not get detected as a bot?

[@FiReLScar]

Ah yes, what next are they gonna ban google for crawling it?

[@oldnerdsteve]

Yeah, it's always good to read the terms and conditions before you do something out of the ordinary.

[@jazzyfriends4197]

Proxies ?