Is Web Scraping Legal? My Take on Automation and Fair Use

[@adoolaard]

Great video! I only have one question: Me, as a software developer want to create a web application in which we can make a list of LinkedIn sales navigator leads (activated on my account). Then with the account of me and my colleagues, we would send a sales pitch to the people in this list. So we are using it to make profit, and the list will be provided by my account. Is this then illegal what I’m doing? Thanks in advance!

[@PythonSimplified]

hmmmm I would think that the profit part makes it a bit problematic... definitely worth to speak to a lawyer or legal consultant before processing with this venture 🤔 There are legal pathways there for sure, as systems that generate and contact leads exist on the market - just make sure you are aware and following the correct guidelines... to find our those guidelines - I would only trust the advice of professional legal consultants, otherwise it's a guessing game with too much liability 🙃

[@PythonSimplified]

Oh and I forgot to mention - I actually have some really nice LinkedIn scrapers/bots publicly available! It may save you some time if you choose to move on with the project: ⭐ LinkedIn Connecting Bot: kzbin.info/www/bejne/bZKsk2mfhr-eetk ⭐ LinkedIn Messaging Bot: kzbin.info/www/bejne/jpWphqN8iqmnbpo And here's the Github Repository of those projects just in case: github.com/MariyaSha/WebScraping_Linkedin Good luck and I hope everything works out! 😊

[@adoolaard]

@@PythonSimplified Thanks for your answer!! I think that i will definitely have to talk to a lawyer then. Quite expensive unfortunately 😬 I think that i can make a lot of profit and learn really much by making this tool. So i will definitely try to continue with it! Thanks you for the great explanation of how to make these scrapers! Keep up the great work! 💪❤️

[@OGIMxGaMeR]

So if I am scraping public information created by other websites and publish it, I am going against copyright? Even if the data is publicly available?

[@ishid4592]

3:27 "When I say dummy accounts, I don't mean stupid accounts" haha don't know why this made me laugh so much ! =D

[@benjaminauer788]

nice video! But I have another question regarding the copyright topic: If you webscrape for your yt-purposes and show this data in a video.... is it still legal?

[@HugoM946]

When it comes to web scraping I believe the grey area lays not on the method you get the information or the amount but in the speed you are able to pull it. Automation can be REALLY fast and hammering a server with requests will get you throttled in no time if the server is minimally prepared for this kind of operations. In my opinion it's more of an ethical issue than a legal one, like ignoring robots.txt lol.

[@littlebrit]

Some websites like Amazon are blocking me time to time. When I do scraping from the cloud, I have to wait few hours. Of course not from my home IP, I have not been blocked yet. Their T&C says that I should not scrap.

[@primosoma]

A site that don’t want to be scraped can measure time between requests or analyse the behaviour of the site usage and eventually block your IP address. This can be more annoying then a blocked account.

[@PythonSimplified]

I absolutely agree primosoma! 😀 There's ways around it though - since Selenium runs from an automated browser windows - it's very hard to block you! This problem, however, is very popular among headless browsers (let's say Scrapy, or even the headless Selenium browser). A good way to avoid detection in terms of time measuring is to randomize the wait between selecting elements with: time.sleep(random.randint(0,10)) This would generate a new random wait instance anywhere between 0 seconds and 9 seconds - which will make it even harder to track you 😉

[@bashinho]

What most people do not understand: The internet is based on copy. If you watch an website - it is copied to your RAM, so you have a working copy on your computer. You do not watch an webseite on the server, you watch a copy. If you watch a stream the whole movie is copied to your computer RAM. and so on.

[@redaoutarid6465]

Thank you Mariya for your very nice topics. I would love what you do. I am a web developer and have been very interested in web automation lately. I update my knowledge in Python to start well in this field. What advice can you give me to be a professional in web automation?

[@memeghost9716]

How can I make Desktop Web scraping Application...?

[@PythonSimplified]

Simply use a GUI Library to create the interface of the application, and use my webscraping tutorials for the functionality. I have a bunch of these, but for example - checkout my Kivy interface tutorial: kzbin.info/www/bejne/j3XTaGaNn7N8icU And checkout my Web Scraping LinkedIn tutorial: kzbin.info/www/bejne/bZKsk2mfhr-eetk If you combine the two - you can make a very powerful scraper desktop app! Good luck! 😊

[@reprovedcandy]

Question: Web scraping by mimicking network requests and extracting the results as JSON. Does anyone know the legality difference in that case?

[@sandeepyadav1478]

Hey, can i ask for some help? Need little help with youtube api with python. Can you look into that! I will be so much thankful.

[@WahlVids]

Don't some sites restrict the number of requests? I learned there's a way to get around this by varying the times to fool the sites bot from catching your scraper or preventing DDoS attacks, please, correct me if I'm wrong. In the end I completely agree with your assessment. After all, we're getting data mined every day by numerous companies! They actually sell our info, but this is usually buried deep within terms of service and privacy notices, lol.

[@siamahmed8287]

Hey can we make api from scraping data from website?

[@healthwithbhavesh6030]

Can you please start the Selenium python series for the beginners?

[@rohitw5518]

Very neatly explained about web scraping 😀

[@PythonSimplified]

Thank you Rohit, I'm glad you liked my explanation! 😀

[@raynercadrelo8206]

Hi, I would like you to talk about creating a GUI app with Kivy. Thanks for this video.

[@RS-tx4bu]

Thank you for the blessing! Best, most logical advice yet. Keep up the good work. 👍

[@alanwest546]

You have a wealth of knowledge. Keep teaching

[@manu-singh]

So what topics would you cover in the ai vidoes

[@ronin2963]

Follow the rule of 2s and 3s. Never associate 2 or more identifying pieces of information. Plus I think you nailed Artificial Intelligence

[@furkancetin3195]

İf I scrape an Instagram account's followers and send them automated DM would it be legal? I know you're not lawyer just curious about your opinion :)

[@kesavguru415]

waiting eagerly😁

[@PythonSimplified]

Thank you so much Kesav! I'm heading to the live chat as we speak! 😁

[@reprovedcandy]

Agreed! If you can do it manually, automating it should be legal in every instance. The problem is, US law does not see it that way. Ridiculous!

[@TheTirailleur]

Could I feed hundred of photos for ai with-out having troubles ?

[@joseluisbeltramone599]

In full agreement with your explanation, Mariya. Thank you!

[@nikluz3807]

I’ve always sort of had this question. Thank you for the breakdown.

[@AquiHeme]

Am I the only one who noticed that that she turned her face to the right when she said that she is a reliable person?

[@Shan224]

I think you have a great take here. Another thing to keep in mind that wasn’t touched on, is using bots for denial of service attacks. Or even inadvertent DOS via heavy scraping, which can be a serious issue for smaller sites run a by a few individuals.

[@SinanAkkoyun]

1:45 ask game hackers lol xD Great video btw!

[@andrivr4920]

Thanks to you i discover about colaboratory! Thank you! I really like it you're a knowledge!

[@nathanielmarner6134]

Thanks a lot for your videos. I learn a lot from you👌

[@panth5501]

Great video, glad you clarified your thoughts :)

[@PythonSimplified]

Thank you so much Priyank! 😀 I was writing the same reply on so many comments that I eventually decided to have mercy on my fingers and film a vlog instead hahahah 🤣 I assumed that if I see dozens of comments asking me the same question - it means that there are hundreds of others that were wondering about the exact same thing 😉

[@kesavguru415]

@@PythonSimplified I got a idea because of tired of replying...😅 Make a video on auto reply to KZbin comments using Selenium 😅

[@PythonSimplified]

@@kesavguru415 hahahaha sounds like a great plan! I think the last lesson I'll be teaching in AI is "how to replicate Mariya's brain - step by step tutorial" 🤣🤣🤣 And then I retire and more to Florida! 🤪

[@emreyavuz4706]

Very clear explanation 👍

[@chiranjeebroychowdhury7759]

Nice video! Perfect clarifications.

[@PythonSimplified]

Thank you so much Chiranjeeb! 😀 I saw so many confused comments in the past few days - I couldn't get away with a proper review of the subject (well, I probably could, but it would have involved so much typing that it's much better just to film it 😁)

[@rafales1525]

Wow thank You for this vid Beauty 😉😊 Very very very valuable info 😊

[@LuisSolis]

Thank you for your opinion, an amazing video again.

[@PythonSimplified]

Thank you so much Luis, it was a pleasure chatting with you today! so many lovely folks were asking me about this - I absolutely had to film a video on the subject! 😀

[@signorgatto5893]

Very good explaination, better than my teacher 😂 I know you may just thought about it but: what about making a group with your subscribers?

[@PythonSimplified]

Hi Signor, I don't think I'll be able to keep up with all the comments in a group chat... you saw how challenging it was for me to reply to everybody during the premiere 🙁 I think the best solution is going live and replying to comments while streaming... typing is probably not gonna work when trying to communicate with 30,000 lovely folks 😀😀😀

[@signorgatto5893]

@@PythonSimplified Yea sure, that would be hard to have a group with over 30.000 🤣 anyway, good on your side!

[@PythonSimplified]

​@@signorgatto5893 thank you! I'm sure that eventually we'll have a discord server, but it would be more of a communication between viewers who love Python rather than me chatting with everybody 😀 For now, I prefer to keep everyone on one platform so I don't loose my mind hahahaha I'm eventually hoping to have enough regular viewers so I can hire somebody to help me out and then I'll have much more time to improve communication 😊

[@yashvander-bamel]

@@PythonSimplified WOW!! about 14.8k new people in just 2 months (one of them being me :p). You're getting famous haha

[@wispcondanoi]

the best line of the video: [esto de ser bilingüe toca hacer windows + space] "Nobody cares about the 30 photos of cats you download to your computer" - No mas hago mención por que en donde estan las comillas también está la ñ

[@jigarpatel2264]

Scraping and automation testing is looking like same but works differently 😄😉

[@PythonSimplified]

Aren't we using the same tools for it though? I've seen Selenium implemented both with automation testing and web scraping 🤔 Why splitting it into 2 categories when the legality of one would reflect the legality of the other? 😉

[@jigarpatel2264]

@@PythonSimplified Can you send me link of your video (selenium with python ) if you have in your channel!! It will be great for me to see and understand better. Thanks🙌

[@PythonSimplified]

Absolutely @@jigarpatel2264 ! I have a bunch of these videos and I actually included the links in the description of this video - but here's a copy as well 😁: WEB SCRAPING INSTAGRAM: kzbin.info/www/bejne/n3uqp4x4fZ6gmbs WEB SCRAPING LINKEDIN: kzbin.info/www/bejne/bZKsk2mfhr-eetk WEB SCRAPING FACEBOOK: kzbin.info/www/bejne/iaS7lKylmtuejJI WEB SCRAPING GOOGLE: kzbin.info/www/bejne/mqOnlZVngqd1nKs Have fun scraping! 😉

[@gwh0]

Yes. A browser is "code". The purpose of a website is to allow a browser to "scrape" it.

[@PythonSimplified]

I agree! You would also need to allow search engines to scrape your site if you want to get any type of traffic. Some people are so paranoid though, you'd be surprised to see how many folks are mad at me for posting web scraping tutorials that even a child can learn on his own just by looking at the documentation 😵😵😵

[@z2.060]

nice vid, make a tutorial on how to login to google account using python pls ?

[@mohamedpedrou2898]

I totally agree with you

[@jayprajapati2208]

Thanks to remember me!! 😅😄

[@PythonSimplified]

Hahahaha I always remember you Jay!!! 😁😁😁

[@ZurioSi]

Amazing video! Thanks for the info and I wish you a wonderful day !! :). Can't wait to watch AI videos in the future if you make them.

[@PythonSimplified]

Thank you so much Dark Wolf, I'm gonna post the first video in the series in the beginning of next week - it's an unofficial chat about Machine Learning Paradigms (Supervised Learning vs. Self-Supervised Learning vs Reinforcement Learning) and I will most give real life examples of each to get the message through easier 😀😀😀 (mostly trying to avoid fancy word while properly explaining the subject... it's not easy! hahaha but I think I was able to do that 😉 I'll know better when I start editing this on Monday)

[@shaikafrid2204]

WE NEED KIVY FRAMEWORK TUTORIALS

[@wtfdoiputhere]

Im not a lawyer but i never seen anyone being arrested for using bs4 so were good boys

[@PythonSimplified]

I agree layZ! I was actually surprised to see how many people were on the fence about web scraping! so many people are scared of their own shadow these days, it's getting hard to tell what is true and what is plain paranoia 😅

[@wtfdoiputhere]

@@PythonSimplified if it hasn't been b4 then its not real and thats y i keep web scraping lol

[@mushinart]

Well said , Maria

[@smalirizvi8026]

Keep it up Mariya!! Apart from being beautiful, you are super talented and guiding and helping others with clarity is something you are gifted with. Keep it up. I (along with every boy 😂) are with you 😇

[@PythonSimplified]

Thank you so much Ali! I'm so happy to read your wonderful feedback! I sure will keep it up! 😁😁😁

[@smalirizvi8026]

@@PythonSimplified you r most welcome! We have every reason to appreciate and support you!

[@fassoyangce1075]

4:34 OMG!!

[@frankhabermann9083]

Your first argument requires that everything I can do with keyboard and mouse is legal. Pretty sure that this is not the case so this argument is invalid. Your second argument basically takes uses cases you assume to be legal and generalize that therefore all uses cases are legal. This is non sequitur. There are also a lot of hacking tools around, check out Kali Linux for example. It automatizes lots of things you could do with keyboard and mouse and you can use it to check your own Network and Servers for security flaws (which is legally totally fine), but you can also use it to get access on other peoples Networks and Servers without their permission, which is not so legal. However, I could make the very same arguments for Kali Linux. Your reasoning is flawed.

[@PythonSimplified]

I don't think you understood me correctly Frank 😊 My first argument is not claiming that everything you can do with a mouse and keyboard is legal - I did not say such a thing! I did claim that there's nothing you can do with automation that can't be done manually with a mouse and keyboard. Which is very different from what you understood. Automation is a daily practice for many professional developers, that's what I was claiming in my second argument. Not sure how you understood from it that if some cases are legal than all cases are legal - because I did not say that as well! 🤔 I understand that you're distorting my arguments to fit your own perspective, but it has nothing to do with what I actually said. In fact, in the very end of the video, we discussed the illegalities so either you missed that part, either you're trying very hard to find flaws in my logic 😅 (which is ok! I really don't mind arguing! but let's stick to the facts, no? especially since both of us have access to this video and to the exact wording I used) And just because something can be used for hacking - doesn't make it an illegal practice. I can use Python for hacking - but Python is not illegal. It's always a case by case situation, and that's exactly why I'm encouraging everybody to look up information on their end which will be relevant for their particular experience. I hope I cleared up the confusion as you completely misrepresented everything I said in this video 😉

[@frankhabermann9083]

@@PythonSimplified First of all, thank you for answering. So we agree, that it is not about the tools, it is about what you do with them. First you talked about webscraping and then you went ahead with "there is absolutely nothing you can do with selenium,....that you can not do manually with your mouse and keyboard..." - so your argument was specifically about the tools - I hope you see why I may have understood you wrong, it is not about distorting your arguments, at least not on purpose. Then you said "many professional developers are using webscraping on a daily basis..." - ok, here you are talking about the task, not the tool, but you do use examples you assume to be legal. You are correct in that sense that this would imply that this task is not generally illegal, but it would be incorrect to conclude that it would be legal. You did not explicitly make the second claim and I got you wrong there, so my mistake, sorry. However, I want to add "people are doing it on a daily basis" and "why would those tools exist" are not convincing arguments, from a legal perspective: there are lots people doing illegal stuff frequently and outlawed things exist. From a logical perspective: "people are doing it and therefore it is legal to do" would imply something is legal as soon as some people do something. I think we can easily agree that this implication has nothing to do with reality and hence premise is false. I'd like to point out that I agree with the points you are trying to make in the video, it is just that I still have a problem with the structure of your reasoning. Anyway, I like your videos, they are a useful introduction into things you can do with Python. Keep up the good work.

[@PythonSimplified]

No worries @@frankhabermann9083, I didn't really film this video to convince anybody, I filmed it because I was getting lots of questions asking what I think about the topic 😊 The mouse and keyboard argument is an answer to people who think that there's a level of "hacking" or computer magic involved with web scraping. My argument explains it's not the case, as we're simply automating tasks we can already do with our existing peripheral devices 😀 "if you can do it with your mouse and keyboard - there's no reason why you can't do it with automation" - this is my reasoning. It's very different from what you understood from it 😉 Whether I'm talking about the tool or the task - having legitimate businesses and highly qualified developers engaged in such operations - should give people some confidence about using these tools and performing the tasks. Businesses have much more to lose than individuals when engaging in illegal practices, especially when they publicly admit doing it. It's far from "if people are doing it - than it's legal". No such thing was said in this video. It's your take on my words - but it's not what I said or even remotely meant to say. If it doesn't give you enough confidence - it's ok! I'm encouraging you to look for external sources of information and do your own research, which will probably give you more confidence to web scrape. If you're looking for somebody to confirm legality beyond any doubt - please consult with a lawyer. He'll give you all the perfect arguments because it's his job and he was trained to do so. KZbin won't be the platform to look for such answers and this video is representing my take on the subject. I'm only sharing my thoughts here and it's ok if you don't like the way I think - I just don't really like when people distort my words and I would really appreciate if you didn't... Also, please have a look at the rest of the comments here, and see what professional developers are saying in response. You might be surprised how far off you went from the actual intentions of this video 😉 And sorry about the long rant 😅 hahahah I appreciate you taking the time to clarify your comment above, I felt like I owe you a detailed response as well 😀

[@artabra1019]

i got this kind of project in upwork my client have 10000 instagram usernames i create bot that will automatically message in this 10,000 account.

[@m.b786]

what is public is public ... end

[@PERSISTENTxMF]

👏

[@kvelez]

Interesting.

[@freezoulou]

ho yeah IA is next :)

[@PythonSimplified]

Yey! I'm so happy to see you're excited about the topic! I wasn't sure how everybody will react as AI is not necessarily a Python-themed subject... but it sure can be easily implemented with Python! 😉

[@PiezasdeAjedrez]

Missing the old world wild west web times :v

[@PythonSimplified]

Me too! 😫😫😫 But hey, they're working on blockchain based internet as we speak! It's just a matter of time until the old school internet is back on! 😉 It's the only way to save what's left of our online freedom and we're heading there quite fast 😁

[@b213videoz]

1:01 😂

[@VidyaSagar-hc9nu]

💖😍

[@anjanbora7943]

😂😀

[@at4095]

You rock

[@othaibounheng4637]

Come to Laos after covid19, mary with me and do phython things to benefit the world.

[@PythonSimplified]

hahaha Thank you Othai, Laos sounds fantastic! 😁 I'm nowhere near being single though 😅

[@othaibounheng4637]

@@PythonSimplified i love coding but I dont have the basic knowledge of coding, especially phython. I have the best idea to share with u and create it together to benefit the world for the traditional finance system + modern blockchain integration idea using phython if you come to Laos after covid19, marry with me and teach me how to code and work with it. This idea is to let intenet user crossborder spending money like sending a message between fiat and blockchain. You are my style of love and you look a little bit asian like me. I like your content and you na. You will benefit for the cost of living compare to the same cost of living in your country, if you come to live in Laos, cause you are expert in phython, you can live anywhere in the world and get same salary. In Laos if you get salary equal to US$ 3000-5000, you can live like a king or queen and you can support your own big 5-6 people in the family if you have.

[@geraldofrancisco5206]

Dont know, but I dont care

[@jasenmichael]

Talk nerdy to me