Support #26006
openhard to find on Google
0%
Description
Levi Waldron reports
The site is hard to find on Google. A site:epibios.commons.gc.cuny.edu search returns only about 17 of the 76 URLs in our sitemap, all dated from last spring. None of our Student Resources pages or this semester's news are indexed. The sitemap (/wp-sitemap.xml) and even robots.txt are blocked to non-browsers, which could explain the poor search engine performance.
Any advice?
Updated by Boone Gorges 8 days ago
Could you share a URL?
I have done some initial testing with non-browser access to /sitemap.xml (not wp-sitemap.xml - I don't think this is a correct URL?) and robots.txt and I was able to fetch them, but this could definitely be specific to other aspects of the requesting entity (user agent, ip range, etc). Having a specific URL will help our hosting provider search through firewall logs.
Updated by Marilyn Weber 8 days ago
Apologies - https://epibios.commons.gc.cuny.edu/
Updated by Marilyn Weber 4 days ago
Please let me know if you have an update. But there's no rush from Levi.
Updated by Boone Gorges 4 days ago
I asked the host about it, and they said that requests to sitemap.xml etc are going through just fine. The firewall doesn't show widespread blocking of sitemap requests, or more general requests from Google and other search engine indexers.
I suspect that Levi noticed some issues with programmatic access #26007 and suspected that a firewall-level block sitemap.xml is the cause. But I don't see any direct evidence of that. If Google indexing for the site in question is not as good as the user wishes, it could have to do with some content-level SEO issues. I've copied Colin and Scott here in case they have questions or thoughts about that aspect of things.