Automate Google trends analysis with Python

[@TypicallyThomas]

I feel like you primarily explain if statements throughout this video but the first 15 minutes are quite helpful

[@Pythonenthusiast]

The goal of my videos on this channel is to show that cool stuff can be built with basic knowledge. I understand it can be boring if your knowledge is more advanced. Thank you for the comment!

[@TypicallyThomas]

@@Pythonenthusiast Oh yeah that's great. Really got the sense this is great for beginners, but over the course of this series the videos got significantly shorter which is good, because you're still explaining the basics but also not staying on the simple stuff so much it takes up most of the video

[@Pythonenthusiast]

Thanks for the feedback, it is much appreciated!

[@roguegalaxi1]

Thank you so much, now i completely understand the pytrends

[@Pythonenthusiast]

I am happy this helped!

[@brenogoulart8091]

Hi, the pytrends is returning me empty values sometimes when the search term has a space in it like 'fidget spinner'. And when i check it in the website there is data available. How can i solve this issue? Thanks

[@mohammadrachman1344]

Hi! Thank you for sharing, this is gold mine. Greeting from indonesia

[@Pythonenthusiast]

Thanks Mohammad, glad you enjoyed it!

[@fbaldimguerra]

Thanks for that tutorial. Could you, please, help me with this problem? When I run [pytrends =TrendReq(hl='en-US')] I got the error message ssl: certificate_Verify_failed How can I fix it?

[@sanjayusha9536]

Can we like get the output of the trend data in Excel format...Don't know if I sound lame. Just asking as I believe programmers can do anything.

[@saniiiboy]

This video is SO helpful, thanks for making it 👍🙏

[@Pythonenthusiast]

Thank you for the kind words :)

[@immac4236]

Thank you so much. This video is very helpful. I wonder is it possible to extract "daily" data longer than one year.

[@kaziarmanahmed7537]

When will the next video publish?

[@Pythonenthusiast]

I am working on the next one at the moment. If everything goes as planned, it will be uploaded within 24 hours from now. The entire series should be out in less than 1 week.

[@adilchergui6357]

Many thanks for the explanation, I have a question Is there a way that I can search on one keyword but in multiple categories? " cat = ['0', '3', '74'] "

[@Pythonenthusiast]

Hey there Adil, that is a good question. You can only search for 1 category at a time. However, if you have a list of categories (in your example a list of 3 categories), you can run it through a loop and get results for each category. Hope this helps!

[@adilchergui6357]

@@Pythonenthusiast broo thank you so much for helping

[@adilchergui6357]

@@Pythonenthusiast I am very sorry for the inconvenience I tried, but I couldn't I am a beginner in fact this is the first program I make Could you please give me a resource that I can do this matter I can send you the code I'm trying to make Thank you so much

[@Pythonenthusiast]

​@@adilchergui6357 Create a list categories = ['0', '3', '74'] Then, try the following: for category in categories: pytrends.build_payload(keyword, category, timeframes[0], geo, gprop) data = pytrends.interest_over_time() After this, you need to decide what you will do with this data. As you have 3 categories, with the for loop, it will run 3 times, one for each category. I hope this is clear.

[@adilchergui6357]

@@Pythonenthusiast I am so sorry I have tried a lot with him and there is no result I don't know what I'm doing wrong This is the code : shrib.com/#Leonardo8e4Ob6N I will be very happy if you help me Either way I'm glad I reached this stage thanks to you

[@sunny80one]

Does anyone know of a function or line that will pull data from a specific date range. For example. I'm working with pytrends and I'm looking at several movies from various time frames, I need data pulled from a week prior from release of each movie and a week after each release of the movie. Does that make sense to anyone?

[@MsFleishers]

Thank you so much for this - super clear and helpful! I was wondering if you have a suggestion to get the daily trends x countries? Would you loop thru and feed the geo input in the over time? Any ideas would help - I'm pretty novice but some cookie crumbs will help!

[@Pythonenthusiast]

Hey there, thanks for the kind words. Your logic is on point and good job for understanding the way the library works. What you've explained is indeed what I would do if I want to accomplish such a task. Step 1: Storing all the country codes into a list "countries" Step 2: Create a for loop that would run through this list and it would change geo to be equal to the element of the list. Step 3: Extract the trends Good luck!

[@MsFleishers]

@@Pythonenthusiast thanks! Well I tried. But keep getting errors. This is what I have, would you give me some clues where i might be doing the countries loop wrong? The keyword loop part works on its own. Really really appreciate any help!! from pytrends.request import TrendReq import matplotlib.pyplot as plt import numpy as np import matplotlib.dates as mdates import pandas as pd import time startTime = time.time() pytrends = TrendReq(hl='en-US') colnames = ["keywords"] df = pd.read_csv("keywords.csv", names=colnames) df2 = df["keywords"].values.tolist() df2.remove("keywords") dataset = [] countries = ['india', 'united_states', 'united_kingdom', 'netherlands', 'brazil', 'china'] for c in range(0,len(country)): count = [countries[x]] for x in range(0,len(df2)): keywords = [df2[x]] pytrends.build_payload( kw_list=keywords, geo=count, timeframe='2021-01-01 2021-02-09') data = pytrends.interest_over_time() if not data.empty: data = data.drop(labels=['isPartial'],axis='columns') dataset.append(data) result = pd.concat(dataset, axis=1) result.to_csv('search_trends_test.csv') executionTime = (time.time() - startTime) print('Execution time in sec.: ' + str(executionTime))

[@Pythonenthusiast]

@@MsFleishers Hey there, thanks for the comment. It is difficult to read the code this way, I do see 2 for loops, I am not sure if the 2nd one is added into the 1st one or not. You can reach out to me via email and I'll help you out. However, maybe one tip is to think of the countries. At the moment, I see that interest_over_time is used, which requires the geo-code in its 2-letter abbreviation. If you want to access trending_searches, then you need to use the full name.

[@sumayyakamal8857]

Thanks a lot for the awesome work! But the mean shows us the mean of the word search FREQUENCY (how often it's been searched) , right?

[@Pythonenthusiast]

Hey there, indeed, it is related to the # of searches and it is all relative frequency. We do not have information related to the number of searches but we do know how that changes over time compared to the highest point (interest of 100 is the point where the keyword has been searched for the most). Hope this helps!

[@studyeurope3589]

Amazing Tutorial. keep it up :)

[@alayetmanel6986]

i have an error 429 everytime how can i solve that ?

[@Serbian_Travel_Junkie]

Can you tell us how to do this analysis for other countries such as France, Switzerland, Germany, Serbia, Croatia, Slovenia...?

[@Pythonenthusiast]

Hi Aleksandar, absolutely! Selecting the country is mentioned in this tutorial. The variable "geo" should be changed to the country. If you go to the google trends website and select the country France, what you will notice in the URL is that geo=FR. In the Python script, you need to set geo = 'FR'. If you select Serbia, you'll notice that geo=RS. Every country has its code and you can filter based on that. Hope this helps!

[@spag5296]

This video is super helpful. Exactly what I needed. Thank you 🤩

[@Pythonenthusiast]

Glad you enjoyed the video!

[@spag5296]

@@Pythonenthusiast One little question, do you think there is a limit to how many queries you can make with this API? I imagine that Google would block your IP address

[@Pythonenthusiast]

@@spag5296 That is a very good question, indeed, there is a limit. I have not been able to find info on the official amount as there are limits on requests per minute, but also per day. What I have read is that if you have a script running 24 hours, it is advisable to have 1 request per minute (almost 1,500 requests per day). I have not yet tested this to the point of exceeding the limit, but if you do, please share the result.

[@DarkTobias7]

Hey how can I change the timezone or hl to my country?

[@kriptosahamusa1216]

How do you install the pytrends and do the setup before start?

[@Pythonenthusiast]

Hi there, it is installed the same way as any other library.

[@ChrisLuyckx]

Great Video! How would you download the output to excel?

[@Pythonenthusiast]

Sorry for the late reply, there are plenty of ways to do that, probably the easiest is to use Pandas and save the output to excel.

[@dw-px1ev]

hey man how would i calculate the [-52:] and [:52] number for the other timeframes?

[@Pythonenthusiast]

Hey there, I'll try to explain my reasoning behind this and why it would not work with other timeframes. When I use the 5y timeframe, it returns weekly data and if I want to get the data for the first year, then I'll need the first 52 weeks. Now, 5 years is a pretty long period to spot if there is a shift in certain trends. If you are to use a different timeline, they are not long enough to spot a trend, with maybe the exception of 1 year in some cases. However, if you do want to recreate this, you have to start by thinking of the following: 1. How long is my timeframe? (Is it 1month, 3months, 12months, or 5 years?) 2. Assuming you can identify the interest at any period, what kind of comparison would give you insights? In my case, I measured the interest 5 years ago vs. the interest last year as there's a quite large gap. 3. Then, you need to proceed by understanding the data that you have and ask the question, is my data daily or weekly? 4. Once you have an idea of what you need to compare (from step 2) and you know the data (step 3), you can figure it out if you need to take the first 52 data points that represent the first year or maybe the first 30 if it is daily data and you would like to extract a month. I hope this helps!

[@franciscocastillosaker]

Hey, thanks for the video. Is it possible to set a non stop live scraping system? I am trying to set up an alert that notifies me automatically. Something like "when topicX search volume increases 50%, send me an email". Any ideas on how to do it?

[@Pythonenthusiast]

Hey Francisco. It is absolutely possible! There are a few things you need to ask yourself first: How would you measure the 50% increase? Is it 50% compared to the previous day, week, or maybe month? A simple "While True" look would run forever, so that is an easy approach. As for the emails. I plan to create a tutorial on that sometime, probably using smtplib, so you can take a look into that as well. I hope this was helpful!

[@dicasdoaprendiz]

Hey, friend! Excellent tutorial. One of the best on the subject, I would say. I have a question: is there a way for me to print only keywords that have more than, for example, more than 50 average interest in a given period? I ask this because I have more than 600 specific keywords in my database and instead of analyzing the result one by one, I would like to print only those that have a satisfactory result for me. It's possible? Thanks.

[@Pythonenthusiast]

Of course, you already laid down the logic. Your script should run the analysis starting with the average interest in a given period. If it's above 50, it should continue with whatever analysis you'd like to see. if it's below 50, it can move to the next keyword.

[@lukeschreuder3697]

Dear Kosta, Your videos are incredibly useful as I am doing some research related to Google Trends activity with no previous Python experience. For this research I need to obtain daily search volumes over a period of around ten years for a little under a 1,000 keywords. I have found a script that should be able to extract the monthly data and convert it to daily frequencies, but the problem is I can not get it to work properly as I am not sure which parameters in the code I actually need to change and how to obtain the output. I am assuming I need to include a result.to_csv() command at the end, but I would not know where the results are actually stored. Please let me know if you have encountered similar problems and if you know any resources I could use to gain more insights into solving this issue. Kind regards, Luke

[@jamielee6787]

Hi Luke, I am working on a similar project. Have you found a way to do it?

[@thegingehshow]

Hey would it be possible to have the user Input the keyword(S) instead of having it inside the code itself? Beginner at python. Thanks for your tutorials and help

[@lodostic1015]

Try converting the all keyword list to an input variable

[@emoryolsoff96]

Thank you so much.

[@dw-px1ev]

and is there any way to get the lowest and highest values of searches?

[@Pythonenthusiast]

Absolutely. The highest value would always be 100. You can find the lowest as well and you can move a step further and find when (during which week), the keyword has been searched the least/most.

[@fredesforslag7511]

@@Pythonenthusiast Hey! Great video. Im looking to find peak months (or week) of a given keyword. How would I do that?

[@JonhyyyPistoleiro134]

Possible to share the code from github?

[@Pythonenthusiast]

I will share the code and add a link in the description at the end of the series. There are a few more parts of the library that I think are valuable and will cover in the following videos.

[@Pythonenthusiast]

The code is shared and the link is in the description.

[@JonhyyyPistoleiro134]

@@Pythonenthusiast Thanks!

[@TheStanka89]

really interesting thank you very much. a litte doubt: how can i avoid to obtain "partial data" and get all the weekly data in the last 5 years?

[@Pythonenthusiast]

What do you mean "partial data"?

[@TheStanka89]

@@Pythonenthusiast at 22:34 you get an output with 10 rows of data, instead of a row for every week from 2015 to 2020. The label above data is "Fidget spinner isPartial". I would like to get all the historical data of the last 5 years. (Sorry for my weird english)

[@Pythonenthusiast]

@@TheStanka89 Oh, now I understand yoru question. Well, actually, what you have mentioned above is not fully correct. If you take a look at the data printed, you'll see that it is 261 rows x 2 columns, which is roughly 5 years x 52 weeks. When you print a large data set, only the first 5 and the last 5 rows are being printed in the Shell (You can see that by the dates displayed), but that doesn't mean that there are only 10 rows. Regarding the isPartial, that is a column that provides you with some additional information as to whether the data for that given week is complete or not. As you can see, isPartial was False for all weeks, except for the last row as the data for the last week has not been fully processed. I hope this is clear. If not, please let me know!

[@TheStanka89]

@@Pythonenthusiast ahhhh, ok, now it's clear. Exporting in CSV I am able to collect all the data. I totally misunderstood that "IsPartial". Thank you very much

[@Pythonenthusiast]

@@TheStanka89 Great! My apologies if I wasn't clear enough in the video.

[@matteomuendler3417]

I'm a teen and my dad is asking me to use pytrends and to help him as his "RA" (Research Assistant). This video was super helpful. I'll watch the rest of the series too and hope I can help him with it. If I have any questions what would be the fastest way for me to ask?

[@Pythonenthusiast]

I am happy to hear that the video was helpful to you. I try to be responsive here as much as I can and also via email.

[@sadijemehmeti2584]

Thank you for your videos. They are very helpful! 🤩 Is it possible to get data over time for each region

[@Pythonenthusiast]

Hi Sadije, thanks for the question. That should be possible if the region is specified as geo.

[@JonhyyyPistoleiro134]

Amazing! Thanks.

[@Pythonenthusiast]

Glad you liked it!

[@alayetmanel6986]

The request failed: Google returned a response with code 429

[@ramoses1]

Your knowledge about the topic is clear, but please don't make the video so long, you are putting some information that is actually not important for the tutorial, it makes pretty painful to watch despite the good content.