Cleaning up 1000 Scraped Products with Polars

[@JohnWatsonRooney]

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/JohnWatsonRooney/ . You’ll also get 20% off an annual premium subscription.

[@crissydogg]

Brilliant as always

[@fredde7356]

Hey John, can you please continue the scraping livestream with your test site? Would love to see how to handle the drop-down menus, Java script and how to handle stricter cloudflare rules Would be happy to hear about some news, if you plan to continue

[@JohnSmith-y8o]

I'm parsing my scraped data into Pydantic models. Never looked into dataframes pandas etc. should I? :D

[@JohnWatsonRooney]

definitely, it's great for analysis and transforming data

[@Rice0987]

I've got next message in VS: Missing required CPU features. Install the `polars-lts-cpu` package instead of `polars` to run Polars with better compatibility. AND it starts working!

[@Rice0987]

This is additional option if you have such issue. So, if you have it you need to install that second package after installing polars.

[@historiadeunahora6383]

this also helped me, thanks!!

[@prashantbhosale6745]

hi, can you show us how to extract all the data related to the title/field from the pdf file.

[@Septumsempra8818]

Has anyone built an engine to mimic guman behavior? How does mone move the mouse and scroll etc to mimic human behavior. I have a site that requires a from to be filled in to change location. But it triggers captcha without fail. Does anyone have tips on mimicking human behavior or how to type something into a captch form

[@pypypy4228]

What are the advantages over Pandas?

[@JohnWatsonRooney]

It’s going to be faster but for what I do there aren’t any really - I just wanted to try it out

[@gokulyc]

Code / repo link?

[@einekleineente1]

0:20 you promised to link to the other video wehre you got the data... now I am sad that the link is not there .. and I never would get the idea to look at your channel to see which videos you posted in the last days.. ;-)

[@DM-py7pj]

might be video titled "Website to Dataset in an instant" based on quick scan of field names.

[@mishmohd]

Cleaning data is so passe, we up here dry cleaning data

[@DM-py7pj]

Looks v similar to PySpark.

[@JohnWatsonRooney]

i've not used that but looks good

[@bakasenpaidesu]

.

[@JohnWatsonRooney]

..

[@abdulrahmanharoon3165]

Is it faster than pandas?

[@bakasenpaidesu]

10x

[@JohnWatsonRooney]

It’s written in rust and yes it is generally faster than pandas

[@abdulrahmanharoon3165]

Thanks @@JohnWatsonRooney

[@Septumsempra8818]

Kinda the same speed. It shines when the data gets big. Once your file is 1Gb+ it makes pandas look old.

[@abdulrahmanharoon3165]

Thanks @@Septumsempra8818