Scrape Home Depot Product, Pricing & Inventory Data

👉 Scrape Home Depot Products & Data: stevesie.com/apps/home-depot-api
MITM Proxy: mitmproxy.org/
Jupyter Notebook: github.com/Stevesie/jupyter-n...
How do you collect public data from The Home Depot? Companies have been doing it for years, so what's their secret? In this video, we'll do our best to find out using network traffic interception to uncover a hidden API.

[@johanneszwilling]

Thx for that approach with the proxy!

[@StevesieData]

🙌 Of course! More apps on the way...

[@johncarroll6910]

Thank you... Very helpful.

[@StevesieData]

Glad it was helpful!

[@hammadhassankhan]

This is definitely good shit man. Love you boy.

[@StevesieData]

Thanks!

[@danielmathew6634]

sorry to be so offtopic but does anybody know of a method to get back into an Instagram account?? I was dumb forgot my account password. I would love any assistance you can give me!

[@alanmcrae8594]

Excellent technical work! Isn't it a shame that in our current economic system the suppliers of construction materials put legal & technical obstacles in the way of their customers easily getting up to date pricing info for the list of products they will need to purchase for their current & future construction projects. (Yeah, let's make life as difficult for each other as we can.) The amount of valuable work time that is wasted by skilled craftsmen just to get materials pricing & availability must be staggering. It's especially wasteful because the actual data is already stored on computer servers that are accessible over the internet, and most builders use computers to prepare their construction estimates. (Nah, we can't allow that. Gotta control our markets as much as possible. Of course, if you're willing to pay for it we could maybe help you out.)

[@StevesieData]

Thanks for the feedback Alan! Yes, but more companies are starting to wisen up and offer public APIs (even free up to say X requests per month before you have to pay) like Google, KZbin, Twitter, Reddit, Spotify, etc... even Best Buy & Kroger when it comes to retail and I hope more companies follow in these footsteps. But unfortunately you're right about construction. I don't know of a retailer like Home Depot with an official public API. However, I wouldn't assume Home Depot is trying to purposely limit you. They may simply not have the technical resources nor incentive to maintain a public API. For example, Netflix tried maintaining one around 2013 but had to shut it down because it cost them too much to maintain and no one was using it. So Home Depot may be in a similar boat and they simply don't mind if people access their data in unofficial ways as described in the video - as opposed to investing the technical resources to maintain one (may cost over $200k/year to hire an engineer or two to maintain it).

[@completesolutionscontracti9422]

@@StevesieData Thank you for the great information. I am developing an excel based estimating sheet and would like to add an inventory of commonly used items that updates pricing automatically. Is there a practical way to do this or are you saying it would need to be constantly maintained by an engineer?

[@jadewhite8464]

@@StevesieData.

[@kingchuwo]

Can it find an item/items at a certain percentage off?

[@donhemmerling2645]

Good stuff, but I am looking to get the cost of products based on the materials needed for a job. In the estimating app you select the materials need then add the quantity per unit, gallons, square ft. or linear ft. and a total cost will be produced for each item. My challenge is finding a way to get the updated prices on all of HDP or Lows products on any given day.

[@StevesieData]

Hey Don, great app and idea! Unfortunately you'll never be able to get ALL the prices on any given day since they'll also vary by store. I would suggest building up a list of the relevant SKUs / Item IDs that you need, and you can look them up each day to build a daily tracker... as I know at least a few people are actively doing this for similar projects ;)

[@donhemmerling2645]

@Stevesie I read your disclosure and will not accept it to protect myself form any legal action. I am looking for something that can be used along with my app when published. For that reason your disclosure prevents me from using your service. I also do not want to know any more of the service you created so I can't be charged with copy write. I will continue looking for a solution for my app. If by any chance you are interested in combining our efforts. My email address: [e-mail]

[@christophermccall9228]

@@StevesieData You say you know at least a few poeple who are actively doing this.........care to share the exact key word search ? I'd love to see even a small sampling of this being done for the purpose of quoting jobs (estimating).

[@wow1521]

hey does this still work? i am trying but the mobile app does not load when im intercepting

[@marcodeconto1838]

Does this approach expose in-store pricing? With a SKU and store id, is it possible to query local prices and inventory?

[@StevesieData]

Great question Marco! Yes, you can provide a Store ID to this endpoint. This article: stevesie.com/docs/pages/scrape-upcs-from-home-depot-skus has a bit more info, you can see how we're able to match SKUs with a Store ID on that endpoint to get back the price & inventory.

[@donhemmerling2645]

@Marco DeConto Thanks for the question.

[@SideOfHustle]

The length I will go for a 12' skeleton decoration.

[@hollymaguire4464]

I am here too to make a bot to get the damn skeleton. I just want a giant skeleton

[@SideOfHustle]

@@hollymaguire4464 you don’t need this. You can just write a selenium bot. If you just follow their twitter you’ll get one. The bots weren’t even buying them this year. The skeleton didn’t sell out for hours

[@jerrynewland5231]

Wouldn't it be Easier to set a macro with Google sheets and start one single column at 10000000/200000000 i can ahow you what i mean