CUNY Academic Commons

Our robots.txt file is shared across all sites on the Commons.

I'm not sure why we block pdf, doc, and docx files. This must be something that we added ourselves. Unfortunately, we don't track robots.txt in our repo for some reason. The last-touched date of the file is Oct 10 2018, but looking back at my email, it looks like this is when I added the Crawl-delay directive, which doesn't help explain why we exclude the other document types. I've paged back through many years of emails with the word 'robots' and was able to find that the robots.txt file did not have these directives in 2014, but this is not super helpful :)

Ray, do you have any recollection of why we did this? Is there danger if we remove the directives? My initial thought was privacy but it seems to me that our various privacy settings (eg automated htaccess files for non-public sites; serving bp-group-documents via PHP; etc) should prevent problems like this.

Ray, do you have any recollection of why we did this?

Unfortunately, I do not.

Is there danger if we remove the directives?

I don't think so. I think we should move our custom robots.txt customizations to the 'robots_txt' filter or the 'do_robots' action at a higher priority and remove the /robots.txt file from the production server. That way, other plugins that use these hooks could potentially make modifications to robots.txt such as Better WordPress Google XML Sitemaps, Yoast SEO and Jetpack.

If we decided to use a static robots.txt file due to performance reasons, then we can just remove the file type directives from the robots.txt file.

Thanks. Does this mean Wole can now edit? He is asking how to do that.

No, users cannot edit their robots.txt. However, the .pdf restriction has been lifted sitewide, so Google should now be able to crawl these files.

Thanks, I let him know

Boone, I noticed that the generated wp-sitemap.xml file for the main site includes a lot of post types and taxonomies that may not be applicable such as:

• https://commons.gc.cuny.edu/wp-sitemap-posts-cacsp_paper-1.xml - Could keep.
• https://commons.gc.cuny.edu/wp-sitemap-posts-cac_course-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-posts-cac_course-2.xml
• https://commons.gc.cuny.edu/wp-sitemap-posts-tapor_tool-1.xml - Could keep.
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_college-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_department-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacap_position_title-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-group-documents-category-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-group-documents-category-2.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_associated_item-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_access-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_comment_access-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_folder_in_group-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-bp_docs_tag-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-tapor_category-1.xml - Could keep.
• https://commons.gc.cuny.edu/wp-sitemap-taxonomies-cacsp_paper_tag-1.xml

For post types, we can use the 'wp_sitemaps_post_types' filter to omit certain post types from wp-sitemap.xml:

add_filter( 'wp_sitemaps_post_types', function( $retval ) {
    unset( $retval['cac_course'] );
    return $retval;
} );

For taxonomies, some of the taxonomies can be addressed by setting the 'publicly_queryable' property to false. If the taxonomies need to be publicly queryable, we can use the 'wp_sitemaps_taxonomies' filter similar to the 'wp_sitemaps_post_types' filter above.

• https://commons.gc.cuny.edu/wp-sitemap-users-1.xml
• https://commons.gc.cuny.edu/wp-sitemap-users-2.xml
• https://commons.gc.cuny.edu/wp-sitemap-users-3.xml

Or we could keep it if we change the permalink from /author/USERNAME/ to /members/USERNAME/ so it references the BuddyPress profile URL instead with the following:

// Replace author profile URL with BP profile URL.
add_filter( 'wp_sitemaps_users_entry', function( $retval ) {
    if ( is_main_site() ) {
        $retval = str_replace( '/author/', '/members/', $retval );
    }

    return $retval;
} );

Ray, could you go ahead and implement these changes?

Done in https://github.com/cuny-academic-commons/cac/compare/710bf1d...eaef35d984.

Looking over these taxonomies, I don't think any of them need to be publicly queryable. (IIRC the technical meaning of publicly_queryable is: can you access a taxonomy archive at a public URL - is that all it does?)

Yeah, I believe you're right. After reading the PHPDoc, I went one step further and switched from 'publicly_queryable' => false to 'public' => false since 'public' defaults to true if not set: https://github.com/WordPress/WordPress/blob/a921157f3a1fff5cd403612d7b5201043917860e/wp-includes/class-wp-taxonomy.php#L343. And the 'public' value gets passed down to 'publicly_queryable', 'show_ui' and 'show_in_nav_menus' if not set.

For CACAP-CAC, I've set those taxonomies to 'public' => false in https://github.com/cuny-academic-commons/cac/commit/15de62be8528332fdf9cf43b76b930ee5e56c57f.

For BuddyPress Docs, I've opened a PR: https://github.com/boonebgorges/buddypress-docs/pull/717. For CAC Group Library, I've also opened a PR: https://github.com/cuny-academic-commons/cac-group-library/pull/15. Until those PRs are merged, I've removed them from the sitemap temporarily in https://github.com/cuny-academic-commons/cac/compare/cb3539c...eaef35d984.