Wordpress Robots.txt Best Practices - How to Optimize WordPress Robots.txt [Still true in 2023!]

[@wplearninglab]

If you enjoyed this video don't forget to the like it and subscribe! Then check out this playlist about speeding up WordPress: kzbin.info/www/bejne/mXOnimB5jbygf5I

[@franshescomviran5741]

Bloody brilliant mate! Cheers from Australia! 🎉

[@17aig]

As always thank you so much for this tutorial

[@wplearninglab]

You're welcome Nissim, thanks for watching! Let me know if you have any questions :)

[@WilliamFernandesvideos]

Congrats and thanx !

[@wplearninglab]

Thanks and thanks for watching!

[@weldingempire]

Thank you so much. I was having a lot of trouble trying to have ahrefs crawl my site and adsense. wonderful and clear tutorial.

[@personaldronerepair6141]

You Rock !! Thank you for your efforts and time in!!

[@wplearninglab]

Thanks, you rock too!

[@mahmoudgh2025]

Thanks

[@Lettucelu]

Thank you!!!!

[@worldtraveler303]

Thanks so much!!!! 🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿

[@TheBlades2012]

Hi, When I put my sitemap on just like you did it comes up with the red X on the left?

[@MH-jp1fv]

awesome thank you!

[@Electronzap]

Good stuff :)

[@wplearninglab]

Thanks Paul and thanks for watching!

[@17aig]

Could you please add links to the Robert text tester and the blackhole Thanks in advance

[@wplearninglab]

For sure, here you go: Robots.txt Google Search Console link: support.google.com/webmasters/answer/6062598?hl=en Black Hole for Bad Bots tutorial: kzbin.info/www/bejne/Znykq3p6fdZgZ68

[@victoragburum1858]

Every time I crawl my site with semrush, it keeps detecting errors that my some pages could not be crawled, pls does anyone have an idea how to fix that issue

[@rgkh1503]

Hi there, thanks for the video, but this seems not to be the best practice anymore (cf usage of meta tags)..any update of this video planned?

[@shelbythomas7744]

What is your favorite contact form and spam blocker for that form?

[@wplearninglab]

Happy forms is pretty awesome and the free version is quite powerful. For free forms I also like Contact Form 7. I know it's old and a bit clunky, but you can build out lots of simple form types and there are plenty of addons for more functionality. If the form needs to have calculations done on data entered in the fields I like Gravity Forms. I like Recaptcha v3 and/or honeypot fields to reduce spam. ..I hope that helps, let me know if you have any further questions. Thanks for watching!

[@tevinkiprop1883]

Hey so I have a large website and when I click on to coverage I have the discovered- but not indexed yet message also when I try to use the URL inspection it displays noindex. kindly help since only a few of my pages are being indexed

[@frankstone]

Thanks to your previous tutorial, I have installed IThemes Security and activated the "hide backend" feature. Will the robots.txt file provide additional site security that is not already installed?

[@wplearninglab]

Hi Frank, If you use the "hide backend" feature of iThemes Security you wouldn't need the user agent rules in the robots.txt file. The only thing that may benefit you is having the link to the sitemap in the robots.txt file. But you can also add your sitemap directly to the webmaster tools in major search engines instead. I hope that helps :) Let me know if you have any further questions. Thanks for watching!

[@GBFragrances]

Hi! How do I edit the robot txt file that's blocking Google to crawl my pages?

[@Cognitoman]

Should be in the public part of the website

[@siddhant2002]

Do you have any video to how to make money as wordpress developer

[@wplearninglab]

Hi Sidhant, I don't at the moment. From my experience, you'll need to (1) have the development skills, which it sounds like you have and then (2) you have to go out and find clients. Part 2 is the hardest for most people. You have to go where your ideal clients meet up (conferences, meet ups, etc.) or you have to go door to door with an offer they can't refuse.

[@mustafatankus9343]

hello, only the homepage of my site comes up in google searches. None of the articles on my site appear in google searches. Do you have any idea about the reason? I registered the property on google search console, submitted the sitemap to google. I checked the robots.txt file but I can't seem to solve this problem in any way.

[@budsgrower9970]

I have the same problem. I hope someone will answer

[@nachogalvezantunez1915]

Great Bjorn! Really thankfull! My site got broken with the "500 Internal Server Error" after adding the "Really Simple SSL" Plugin to my Wordpress and Google Search Console is telling me that's because my server (at Google Cloud Platform) bloqued Google search engines (Googlebot) so they can not find my domain. Google Search Console though the robots.txt taster tool is prompting me to go back to the last usefull "robots.txt" downloading it to my computer and uploading it to my server. I can´t find any "robots.txt" file at my "html" directory; acording to your video I just have to upload the one downloaded from "robots.txt taster" tool, is this right? I really apreciate very much in advanced and hope you can help me. Best regards.

[@pahtashow]

How to make a sitemap in 2022 wordpress ?

[@guadadis]

Hi! Do you know what could be happening with my file robots.txt? Every so often it automatically switches to user-agent: * Disallow:

[@mauwiks]

Mine too

[@Cognitoman]

Prolly something with your server from your hosting company

[@starsproduction9676]

It's the robot.txt make your blog post show in search results