CUNY Academic Commons

I'm seeing the same thing with other Jetpack-connected sites. I've sent a support request to Automattic for further diagnosis, and I'll follow up here when I have info.

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

Boone Gorges wrote:

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

I feel that this is a separate issue though. I was able to connect via JetPack just fine up until the last week. The problem with the "down" site has been ongoing for much longer.

Can you please explain in more detail? You're suggesting that there's more than one thing going on here, but the only problem you specified in the original report was the "site cannot be accessed" notice on wordpress.com. If you are unable to use Jetpack in some way, can you please specify exactly what you're seeing and what you expect to be seeing? If there are two separate issues - one of which is new and one of which is ongoing - could you please spell out this difference in more detail?

Thanks for the screenshots. You mentioned earlier that "the problem with the 'down' site has been ongoing for much longer". Can you explain what that means? Does this mean that the message "this site cannot be accessed" message has been there for a long time, but the "missing" content on wordpress.com is new?

Boone Gorges wrote:

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

Do we have any reference points of the sites using HTTP for the problematic Jetpack sites?

If so, can we check the Jetpack debugger results page for such a site at /wp-admin/admin.php?page=jetpack-debugger? There's a section that notes the "Sync IDC URLs". If these URLs are HTTP and these sites are the ones experiencing these symptoms, then let's make the change.

That being said, I checked the Jetpack debugger page for Raffi's site and the "Sync IDC URLs" are using HTTPS, not HTTP. The SELF test fails, but everything else passes.

Whereas on another Commons site using Jetpack (openatcuny), the SELF test passes.

Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Boone Gorges wrote:

Thanks for the screenshots. You mentioned earlier that "the problem with the 'down' site has been ongoing for much longer". Can you explain what that means?

Ah, I thought you were speaking of https://redmine.gc.cuny.edu/issues/8571. This is where JetPack periodically sends emails saying so and so site is down. It eventually goes up. Sorry if I was mistaken.

Does this mean that the message "this site cannot be accessed" message has been there for a long time, but the "missing" content on wordpress.com is new?

Ah, no. They both seemed to happen at the same time. Again, sorry for the misunderstanding.

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

Raffi Khatchadourian wrote:

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

Tried it. I disconnected and tried to reconnect but now I am getting an error:

https://jetpack.wordpress.com/jetpack.authorize/1/?response_type=code&client_id=145852292&redirect_uri=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-admin%2Fadmin.php%3Fpage%3Djetpack%26action%3Dauthorize%26_wpnonce%3D0c364a99ec%26redirect%3Dhttps%253A%252F%252Fcybercamp.commons.gc.cuny.edu%252Fwp-admin%252Fadmin.php%253Fpage%253Djetpack&state=6303&scope=administrator%3A112beb4f56855ae0fdc0758223e40ad6&user_email=raffi.khatchadourian%40hunter.cuny.edu&user_login=khatchad&is_active=1&jp_version=6.3.2&auth_type=calypso&secret=QnxWcaIQ4iV3DBiw5gT8YfNYGEmODF6o&locale=en&blogname=Hunter+College+Cyber+Security+Summer+Camp+2018&site_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&home_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&site_icon=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-content%2Fblogs.dir%2F3473%2Ffiles%2F2018%2F05%2Fcropped-cyber-security-1915628_960_720.png&site_lang=en_US&_ui=75815771&_ut=wpcom%3Auser_id&_as=list&from=landing-page-top

Invalid request, please go back and try again.
Error Code: invalid_client
Error Message: Unknown client_id.

Raffi Khatchadourian wrote:

Raffi Khatchadourian wrote:

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

Tried it. I disconnected and tried to reconnect but now I am getting an error:

https://jetpack.wordpress.com/jetpack.authorize/1/?response_type=code&client_id=145852292&redirect_uri=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-admin%2Fadmin.php%3Fpage%3Djetpack%26action%3Dauthorize%26_wpnonce%3D0c364a99ec%26redirect%3Dhttps%253A%252F%252Fcybercamp.commons.gc.cuny.edu%252Fwp-admin%252Fadmin.php%253Fpage%253Djetpack&state=6303&scope=administrator%3A112beb4f56855ae0fdc0758223e40ad6&user_email=raffi.khatchadourian%40hunter.cuny.edu&user_login=khatchad&is_active=1&jp_version=6.3.2&auth_type=calypso&secret=QnxWcaIQ4iV3DBiw5gT8YfNYGEmODF6o&locale=en&blogname=Hunter+College+Cyber+Security+Summer+Camp+2018&site_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&home_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&site_icon=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-content%2Fblogs.dir%2F3473%2Ffiles%2F2018%2F05%2Fcropped-cyber-security-1915628_960_720.png&site_lang=en_US&_ui=75815771&_ut=wpcom%3Auser_id&_as=list&from=landing-page-top

Invalid request, please go back and try again.
Error Code: invalid_client
Error Message: Unknown client_id.

And now my site isn't working at all :(. http://cybercamp.commons.gc.cuny.edu

I'm looking into it.

Regarding cybercamp.commons being unavailable, there's a bug in Jetpack that triggers a fatal error for the "Simple Payment" widget if the Simple Payments module is not activated. I assume this has something to do with being disconnected from wordpress.com. I'll try to reproduce it so that I can submit a report to Automattic, but in the meantime, I've patched the plugin with a workaround.

Regarding the connection issues, Ray said:

If so, can we check the Jetpack debugger results page for such a site at /wp-admin/admin.php?page=jetpack-debugger? There's a section that notes the "Sync IDC URLs". If these URLs are HTTP and these sites are the ones experiencing these symptoms, then let's make the change.

I think something more complex is happening than what the jetpack-debugger can report. In all cases, it appears that HOME_URL and SITE_URL on the debugger are HTTPS. But I think that this is because of the complex filtering+unfiltering dance that Jetpack does with these options. See Jetpack_Sync_Functions::get_raw_or_filtered_url() and try to follow the logic :) Briefly, I think that the UI is reporting that Jetpack believes that the site URL is https://, but wordpress.com gets an unfiltered value (see cac-functions.php and cac_set_url_scheme()) and thinks that it's http://.

I verified this on a separate site by switching home and siteurl to https://. This alone did not fix the problem: I had to disconnect and reconnect Jetpack, which - according to my hypothesis - triggered a resend of the 'home' and 'siteurl' values to wordpress.com, this time with the proper https:// protocol. After that change, wordpress.com was able to reconnect correctly.

Raffi, it would be helpful if you could verify this. I've changed the URLs of cybercamp.commons to HTTPS. Please go through the disconnect-and-reconnect process again, and see if wordpress.com reports that things are working. If so, I think we will have narrowed the issue down.

Unfortunately, if this is the answer, I'm unsure how we can fix all of the URLs in an automated fashion. It might be necessary to reach out to Automattic with a list of sites and ask them to change them.

I reconnected. It seems to be working now.

While investigating, I realized that WP-CLI believed that the URL for cybercamp.commons was still http://. It turns out that this is due to a filter we put in place long ago to handle our migration to SSL-only. The logic works by forcing 'home' and 'siteurl' to the current URL scheme, using WP's `set_url_scheme()`. The problem is that CLI requests are not interpreted as being SSL. My theory is that that the intersection of this behavior with the fact that scheduled tasks are run via CLI means that Jetpack was reindexing the site as being non-HTTPS at the next cron job.

I've put a hotfix in place so that cac_set_url_scheme() uses 'https' always (for *.commons.gc.cuny.edu domains) rather than by detecting the scheme of the current request. I waited a few minutes, and it appears that Jetpack resynced itself and is now communicating again. Raffi, can you have a look?

The above suggests that we might just be able to change the URLs and have Jetpack automatically fix its stored URLs on the next sync.

http://cybercamp.commons.gc.cuny.edu looks good but http://khatchad.commons.gc.cuny.edu does not (as well as another on OpenLab). Reconnecting.

Raffi Khatchadourian wrote:

http://cybercamp.commons.gc.cuny.edu looks good but http://khatchad.commons.gc.cuny.edu does not (as well as another on OpenLab). Reconnecting.

Ok, khatchad.commons.gc.cuny.edu seems to be working (for now) after the reconnect to JetPack.

Thanks, Raffi. I've just run a script that changes all 'home' and 'siteurl' values to https, as well as the networkwide siteurl. Let's leave this open to see what sticks.