Welp, I take it back—I think I'm just an idiot.
This is what I get for panic sharing to social media rather than taking 5 more minutes to figure something out. 🤦
I believe I've found the issue will wait for validation of the fix. Apolgogies.
Almost immediately after I updated my robots.txt, I started seeing pages reported as blocked by robot.txt in Google Search Console.
Curious as to why this might be happening.
Do these UAs have more to do with Search than stated by the docs? Any thoughts from @johnmu.com (or anyone else)?
I tried blocking AI bots from crawling my website using the list at github.com/ai-robots-txt/
The list includes several Google UAs:
Google-CloudVertexBot
Google-Extended
GoogleOther[-Image|-Video]
None of these should affect Google Search according to developers.google.com/search/docs/...
Lead generation?
Upper text "When you ask an SEO consultant what they actually do for work" Lower text "The work is mysterious and important"
The work is indeed mysterious and important
