Support #17010
openrobots.txt
0%
Description
Wole Oyekoya writes
"I have been wondering why my publications (preprints) are not showing on Google Scholar and I just realized that the default robots.txt disallows pdf files. See https://virtualself.commons.gc.cuny.edu/robots.txt. I’d appreciate if you let me know how to edit the robots.txt. Thanks!"
Updated by Boone Gorges almost 2 years ago
Our robots.txt file is shared across all sites on the Commons.
I'm not sure why we block pdf, doc, and docx files. This must be something that we added ourselves. Unfortunately, we don't track robots.txt in our repo for some reason. The last-touched date of the file is Oct 10 2018, but looking back at my email, it looks like this is when I added the Crawl-delay directive, which doesn't help explain why we exclude the other document types. I've paged back through many years of emails with the word 'robots' and was able to find that the robots.txt file did not have these directives in 2014, but this is not super helpful :)
Ray, do you have any recollection of why we did this? Is there danger if we remove the directives? My initial thought was privacy but it seems to me that our various privacy settings (eg automated htaccess files for non-public sites; serving bp-group-documents via PHP; etc) should prevent problems like this.
Updated by Raymond Hoh almost 2 years ago
Ray, do you have any recollection of why we did this?
Unfortunately, I do not.
Is there danger if we remove the directives?
I don't think so. I think we should move our custom robots.txt customizations to the 'robots_txt' filter or the 'do_robots' action at a higher priority and remove the /robots.txt
file from the production server. That way, other plugins that use these hooks could potentially make modifications to robots.txt
such as Better WordPress Google XML Sitemaps
, Yoast SEO
and Jetpack
.
If we decided to use a static robots.txt file due to performance reasons, then we can just remove the file type directives from the robots.txt file.
Updated by Boone Gorges almost 2 years ago
- Status changed from New to Resolved
- Target version set to 2.0.10
Thanks, Ray. I agree with your assessment.
In https://github.com/cuny-academic-commons/cac/commit/088ae4270d56dafdaf4ff7291c8aafd4db8c99aa I made the following changes:
- Moved our robots.txt generation to the robots_txt filter
- Removed the file-type blocks discussed here (pdf, doc, docx)
I deployed this as hotfix and moved the existing robots.txt file to robots.txt.bak, which we can reference in the upcoming weeks if we notice any issues.
Updated by Marilyn Weber almost 2 years ago
Thanks. Does this mean Wole can now edit? He is asking how to do that.
Updated by Boone Gorges almost 2 years ago
No, users cannot edit their robots.txt. However, the .pdf restriction has been lifted sitewide, so Google should now be able to crawl these files.
Updated by Raymond Hoh almost 2 years ago
Boone, I noticed that the generated wp-sitemap.xml file for the main site includes a lot of post types and taxonomies that may not be applicable such as:
- https://commons.gc.cuny.edu/wp-sitemap-posts-cacsp_paper-1.xml - Could keep.
- https://commons.gc.cuny.edu/wp-sitemap-posts-cac_course-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-posts-cac_course-2.xml
- https://commons.gc.cuny.edu/wp-sitemap-posts-tapor_tool-1.xml - Could keep.
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_college-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_department-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_title-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-group-documents-category-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-group-documents-category-2.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_associated_item-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_access-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_comment_access-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_folder_in_group-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_tag-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-tapor_category-1.xml - Could keep.
- https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacsp_paper_tag-1.xml
For post types, we can use the 'wp_sitemaps_post_types'
filter to omit certain post types from wp-sitemap.xml
:
add_filter( 'wp_sitemaps_post_types', function( $retval ) { unset( $retval['cac_course'] ); return $retval; } );
For taxonomies, some of the taxonomies can be addressed by setting the 'publicly_queryable'
property to false
. If the taxonomies need to be publicly queryable, we can use the 'wp_sitemaps_taxonomies'
filter similar to the 'wp_sitemaps_post_types'
filter above.
- https://commons.gc.cuny.edu/wp-sitemap-users-1.xml
- https://commons.gc.cuny.edu/wp-sitemap-users-2.xml
- https://commons.gc.cuny.edu/wp-sitemap-users-3.xml
Or we could keep it if we change the permalink from /author/USERNAME/ to /members/USERNAME/ so it references the BuddyPress profile URL instead with the following:
// Replace author profile URL with BP profile URL. add_filter( 'wp_sitemaps_users_entry', function( $retval ) { if ( is_main_site() ) { $retval = str_replace( '/author/', '/members/', $retval ); } return $retval; } );
Updated by Boone Gorges almost 2 years ago
- Status changed from Resolved to Assigned
- Assignee set to Raymond Hoh
These are all good catches, Ray.
For posterity, here's some reasoning:
- Hide https://commons.gc.cuny.edu/wp-sitemap-posts-cacsp_paper-1.xml because we no longer publicize the existence of Papers
- Hide the cac_course post types because they link to individual course URLs that don't represent anything (courses themselves are Sites, or Groups, or both)
- Hide https://commons.gc.cuny.edu/wp-sitemap-posts-tapor_tool-1.xml because we're not advertising TaPOR tools.
- Hide the taxonomies because they're needed for internal use, and are not meant to be publicly browseable in this way.
Looking over these taxonomies, I don't think any of them need to be publicly queryable. (IIRC the technical meaning of publicly_queryable
is: can you access a taxonomy archive at a public URL - is that all it does?)
Or we could keep it if we change the permalink from /author/USERNAME/ to /members/USERNAME/ so it references the BuddyPress profile URL instead with the following:
I really like this idea (rather than hiding the directory).
Ray, could you go ahead and implement these changes?
Updated by Raymond Hoh almost 2 years ago
Ray, could you go ahead and implement these changes?
Done in https://github.com/cuny-academic-commons/cac/compare/710bf1d...eaef35d984.
Looking over these taxonomies, I don't think any of them need to be publicly queryable. (IIRC the technical meaning of publicly_queryable is: can you access a taxonomy archive at a public URL - is that all it does?)
Yeah, I believe you're right. After reading the PHPDoc, I went one step further and switched from 'publicly_queryable' => false
to 'public' => false
since 'public'
defaults to true
if not set: https://github.com/WordPress/WordPress/blob/a921157f3a1fff5cd403612d7b5201043917860e/wp-includes/class-wp-taxonomy.php#L343. And the 'public'
value gets passed down to 'publicly_queryable'
, 'show_ui'
and 'show_in_nav_menus'
if not set.
For CACAP-CAC, I've set those taxonomies to 'public' => false
in https://github.com/cuny-academic-commons/cac/commit/15de62be8528332fdf9cf43b76b930ee5e56c57f.
For BuddyPress Docs, I've opened a PR: https://github.com/boonebgorges/buddypress-docs/pull/717. For CAC Group Library, I've also opened a PR: https://github.com/cuny-academic-commons/cac-group-library/pull/15. Until those PRs are merged, I've removed them from the sitemap temporarily in https://github.com/cuny-academic-commons/cac/compare/cb3539c...eaef35d984.
Updated by Boone Gorges almost 2 years ago
- Target version changed from 2.0.10 to 2.0.11
Updated by Boone Gorges almost 2 years ago
- Target version changed from 2.0.11 to 2.0.12
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.0.12 to 2.0.13
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.0.13 to 2.0.14
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.0.14 to 2.0.15
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.0.15 to 2.1.1
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.1 to 2.1.2
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.2 to 2.1.3
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.3 to 2.1.4
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.4 to 2.1.5
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.5 to 2.1.6
Updated by Boone Gorges over 1 year ago
- Target version changed from 2.1.6 to 2.1.7
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.7 to 2.1.8
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.8 to 2.1.9
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.9 to 2.1.10
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.10 to 2.1.11
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.11 to 2.1.12
Updated by Boone Gorges about 1 year ago
- Target version changed from 2.1.12 to 2.1.13
Updated by Boone Gorges 12 months ago
- Target version changed from 2.1.13 to 2.1.14
Updated by Boone Gorges 11 months ago
- Target version changed from 2.1.14 to 2.1.15
Updated by Boone Gorges 11 months ago
- Target version changed from 2.1.15 to 2.1.16
Updated by Boone Gorges 10 months ago
- Target version changed from 2.1.16 to 2.2.1
Updated by Boone Gorges 10 months ago
- Target version changed from 2.2.1 to 2.2.2
Updated by Boone Gorges 9 months ago
- Target version changed from 2.2.2 to 2.2.3
Updated by Boone Gorges 9 months ago
- Target version changed from 2.2.3 to 2.2.4
Updated by Boone Gorges 8 months ago
- Target version changed from 2.2.4 to 2.2.5
Updated by Boone Gorges 8 months ago
- Target version changed from 2.2.5 to 2.2.6
Updated by Boone Gorges 7 months ago
- Target version changed from 2.2.6 to 2.3.1
Updated by Boone Gorges 7 months ago
- Target version changed from 2.3.1 to 2.3.2
Updated by Boone Gorges 6 months ago
- Target version changed from 2.3.2 to 2.3.3
Updated by Boone Gorges 6 months ago
- Target version changed from 2.3.3 to 2.3.4
Updated by Boone Gorges 5 months ago
- Target version changed from 2.3.4 to 2.3.5
Updated by Boone Gorges 5 months ago
- Target version changed from 2.3.5 to 2.3.6
Updated by Boone Gorges 4 months ago
- Target version changed from 2.3.6 to 2.3.7
Updated by Boone Gorges 3 months ago
- Target version changed from 2.3.7 to 2.3.8
Updated by Boone Gorges about 2 months ago
- Target version changed from 2.4.1 to 2.4.2
Updated by Boone Gorges about 1 month ago
- Target version changed from 2.4.2 to 2.4.3