Project

General

Profile

Bug #10125

JetPack issues

Added by Raffi Khatchadourian about 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority name:
High
Assignee:
Category name:
WordPress Plugins
Target version:
Start date:
2018-08-08
Due date:
% Done:

0%

Estimated time:

Description

JetPack seems to have connection issues. I am getting a "site cannot be accessed" error on wordpress.com.

Screenshot from 2018-08-13 19-23-31.png (475 KB) Screenshot from 2018-08-13 19-23-31.png Raffi Khatchadourian, 2018-08-13 07:24 PM
Screenshot from 2018-08-13 19-23-16.png (580 KB) Screenshot from 2018-08-13 19-23-16.png Raffi Khatchadourian, 2018-08-13 07:24 PM
Screenshot from 2018-08-14 14-01-04.png (614 KB) Screenshot from 2018-08-14 14-01-04.png Raffi Khatchadourian, 2018-08-14 02:01 PM

History

#1 Updated by Boone Gorges about 1 year ago

I'm seeing the same thing with other Jetpack-connected sites. I've sent a support request to Automattic for further diagnosis, and I'll follow up here when I have info.

#2 Updated by Boone Gorges about 1 year ago

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

#3 Updated by Raffi Khatchadourian about 1 year ago

Boone Gorges wrote:

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

I feel that this is a separate issue though. I was able to connect via JetPack just fine up until the last week. The problem with the "down" site has been ongoing for much longer.

#4 Updated by Boone Gorges about 1 year ago

Can you please explain in more detail? You're suggesting that there's more than one thing going on here, but the only problem you specified in the original report was the "site cannot be accessed" notice on wordpress.com. If you are unable to use Jetpack in some way, can you please specify exactly what you're seeing and what you expect to be seeing? If there are two separate issues - one of which is new and one of which is ongoing - could you please spell out this difference in more detail?

#5 Updated by Raffi Khatchadourian about 1 year ago

Boone Gorges wrote:

Can you please explain in more detail? You're suggesting that there's more than one thing going on here, but the only problem you specified in the original report was the "site cannot be accessed" notice on wordpress.com. If you are unable to use Jetpack in some way, can you please specify exactly what you're seeing and what you expect to be seeing? If there are two separate issues - one of which is new and one of which is ongoing - could you please spell out this difference in more detail?

See the attached screenshots and notice the error message on the top left "This site cannot be accessed." Also, note that it says that there are no pages but indeed there are many pages on http://khatchad.commons.gc.cuny.edu. This is also the case for my other sites on the commons.

#6 Updated by Boone Gorges about 1 year ago

Thanks for the screenshots. You mentioned earlier that "the problem with the 'down' site has been ongoing for much longer". Can you explain what that means? Does this mean that the message "this site cannot be accessed" message has been there for a long time, but the "missing" content on wordpress.com is new?

#7 Updated by Raymond Hoh about 1 year ago

Boone Gorges wrote:

I happened to be at an event on Friday with a technician from the Jetpack team, and he helped me track down the issue to HTTPS redirects. Basically: the canonical URL for most Commons sites is non-SSL - http://... - but we force traffic to HTTPS where possible using Apache redirect rules. Jetpack's health-check script does not always play nicely with these redirects. So, periodically, Jetpack reports that Commons sites are unavailable.

I'd like to run scripts that do two things:
- For existing sites with *.commons.gc.cuny.edu addresses, change the canonical URL to https:// ('home' and 'siteurl')
- Change the network 'siteurl' option to https://, so that future sites are configured correctly

I'm fairly certain that this is safe to do, but I would like Ray to confirm this logic before I do it :)

Do we have any reference points of the sites using HTTP for the problematic Jetpack sites?

If so, can we check the Jetpack debugger results page for such a site at /wp-admin/admin.php?page=jetpack-debugger? There's a section that notes the "Sync IDC URLs". If these URLs are HTTP and these sites are the ones experiencing these symptoms, then let's make the change.

That being said, I checked the Jetpack debugger page for Raffi's site and the "Sync IDC URLs" are using HTTPS, not HTTP. The SELF test fails, but everything else passes.

Whereas on another Commons site using Jetpack (openatcuny), the SELF test passes.


Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

#8 Updated by Raffi Khatchadourian about 1 year ago

Boone Gorges wrote:

Thanks for the screenshots. You mentioned earlier that "the problem with the 'down' site has been ongoing for much longer". Can you explain what that means?

Ah, I thought you were speaking of https://redmine.gc.cuny.edu/issues/8571. This is where JetPack periodically sends emails saying so and so site is down. It eventually goes up. Sorry if I was mistaken.

Does this mean that the message "this site cannot be accessed" message has been there for a long time, but the "missing" content on wordpress.com is new?

Ah, no. They both seemed to happen at the same time. Again, sorry for the misunderstanding.

#9 Updated by Raffi Khatchadourian about 1 year ago

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

#10 Updated by Raffi Khatchadourian about 1 year ago

Raffi Khatchadourian wrote:

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

Tried it. I disconnected and tried to reconnect but now I am getting an error:

https://jetpack.wordpress.com/jetpack.authorize/1/?response_type=code&client_id=145852292&redirect_uri=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-admin%2Fadmin.php%3Fpage%3Djetpack%26action%3Dauthorize%26_wpnonce%3D0c364a99ec%26redirect%3Dhttps%253A%252F%252Fcybercamp.commons.gc.cuny.edu%252Fwp-admin%252Fadmin.php%253Fpage%253Djetpack&state=6303&scope=administrator%3A112beb4f56855ae0fdc0758223e40ad6&user_email=raffi.khatchadourian%40hunter.cuny.edu&user_login=khatchad&is_active=1&jp_version=6.3.2&auth_type=calypso&secret=QnxWcaIQ4iV3DBiw5gT8YfNYGEmODF6o&locale=en&blogname=Hunter+College+Cyber+Security+Summer+Camp+2018&site_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&home_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&site_icon=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-content%2Fblogs.dir%2F3473%2Ffiles%2F2018%2F05%2Fcropped-cyber-security-1915628_960_720.png&site_lang=en_US&_ui=75815771&_ut=wpcom%3Auser_id&_as=list&from=landing-page-top

Invalid request, please go back and try again.
Error Code: invalid_client
Error Message: Unknown client_id.

#11 Updated by Raffi Khatchadourian about 1 year ago

Raffi Khatchadourian wrote:

Raffi Khatchadourian wrote:

Raymond Hoh wrote:

Boone Gorges wrote:
Raffi, I read Jetpack's troubleshooting page and they say if a site is experencing connection problems that disconnecting Jetpack and reconnecting it could fix things:
https://jetpack.com/support/getting-started-with-jetpack/troubleshooting-tips/ (point 7)

Can you give that a try?

Ug, I was afraid of that. It's happened before. It's annoying because I have many sites connected to JetPack. I'll give it another shot. Thanks!

Tried it. I disconnected and tried to reconnect but now I am getting an error:

https://jetpack.wordpress.com/jetpack.authorize/1/?response_type=code&client_id=145852292&redirect_uri=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-admin%2Fadmin.php%3Fpage%3Djetpack%26action%3Dauthorize%26_wpnonce%3D0c364a99ec%26redirect%3Dhttps%253A%252F%252Fcybercamp.commons.gc.cuny.edu%252Fwp-admin%252Fadmin.php%253Fpage%253Djetpack&state=6303&scope=administrator%3A112beb4f56855ae0fdc0758223e40ad6&user_email=raffi.khatchadourian%40hunter.cuny.edu&user_login=khatchad&is_active=1&jp_version=6.3.2&auth_type=calypso&secret=QnxWcaIQ4iV3DBiw5gT8YfNYGEmODF6o&locale=en&blogname=Hunter+College+Cyber+Security+Summer+Camp+2018&site_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&home_url=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu&site_icon=https%3A%2F%2Fcybercamp.commons.gc.cuny.edu%2Fwp-content%2Fblogs.dir%2F3473%2Ffiles%2F2018%2F05%2Fcropped-cyber-security-1915628_960_720.png&site_lang=en_US&_ui=75815771&_ut=wpcom%3Auser_id&_as=list&from=landing-page-top

Invalid request, please go back and try again.
Error Code: invalid_client
Error Message: Unknown client_id.

And now my site isn't working at all :(. http://cybercamp.commons.gc.cuny.edu

#12 Updated by Matt Gold about 1 year ago

  • Category name set to WordPress Plugins
  • Status changed from New to Assigned
  • Assignee set to Boone Gorges
  • Priority name changed from Normal to High

#13 Updated by Boone Gorges about 1 year ago

I'm looking into it.

#14 Updated by Boone Gorges about 1 year ago

Regarding cybercamp.commons being unavailable, there's a bug in Jetpack that triggers a fatal error for the "Simple Payment" widget if the Simple Payments module is not activated. I assume this has something to do with being disconnected from wordpress.com. I'll try to reproduce it so that I can submit a report to Automattic, but in the meantime, I've patched the plugin with a workaround.

Regarding the connection issues, Ray said:

If so, can we check the Jetpack debugger results page for such a site at /wp-admin/admin.php?page=jetpack-debugger? There's a section that notes the "Sync IDC URLs". If these URLs are HTTP and these sites are the ones experiencing these symptoms, then let's make the change.

I think something more complex is happening than what the jetpack-debugger can report. In all cases, it appears that HOME_URL and SITE_URL on the debugger are HTTPS. But I think that this is because of the complex filtering+unfiltering dance that Jetpack does with these options. See Jetpack_Sync_Functions::get_raw_or_filtered_url() and try to follow the logic :) Briefly, I think that the UI is reporting that Jetpack believes that the site URL is https://, but wordpress.com gets an unfiltered value (see cac-functions.php and cac_set_url_scheme()) and thinks that it's http://.

I verified this on a separate site by switching home and siteurl to https://. This alone did not fix the problem: I had to disconnect and reconnect Jetpack, which - according to my hypothesis - triggered a resend of the 'home' and 'siteurl' values to wordpress.com, this time with the proper https:// protocol. After that change, wordpress.com was able to reconnect correctly.

Raffi, it would be helpful if you could verify this. I've changed the URLs of cybercamp.commons to HTTPS. Please go through the disconnect-and-reconnect process again, and see if wordpress.com reports that things are working. If so, I think we will have narrowed the issue down.

Unfortunately, if this is the answer, I'm unsure how we can fix all of the URLs in an automated fashion. It might be necessary to reach out to Automattic with a list of sites and ask them to change them.

#15 Updated by Boone Gorges about 1 year ago

  • Target version set to Not tracked

#16 Updated by Raffi Khatchadourian about 1 year ago

I reconnected. It seems to be working now.

#18 Updated by Boone Gorges about 1 year ago

While investigating, I realized that WP-CLI believed that the URL for cybercamp.commons was still http://. It turns out that this is due to a filter we put in place long ago to handle our migration to SSL-only. The logic works by forcing 'home' and 'siteurl' to the current URL scheme, using WP's `set_url_scheme()`. The problem is that CLI requests are not interpreted as being SSL. My theory is that that the intersection of this behavior with the fact that scheduled tasks are run via CLI means that Jetpack was reindexing the site as being non-HTTPS at the next cron job.

I've put a hotfix in place so that cac_set_url_scheme() uses 'https' always (for *.commons.gc.cuny.edu domains) rather than by detecting the scheme of the current request. I waited a few minutes, and it appears that Jetpack resynced itself and is now communicating again. Raffi, can you have a look?

The above suggests that we might just be able to change the URLs and have Jetpack automatically fix its stored URLs on the next sync.

#19 Updated by Raffi Khatchadourian about 1 year ago

http://cybercamp.commons.gc.cuny.edu looks good but http://khatchad.commons.gc.cuny.edu does not (as well as another on OpenLab). Reconnecting.

#20 Updated by Raffi Khatchadourian about 1 year ago

Raffi Khatchadourian wrote:

http://cybercamp.commons.gc.cuny.edu looks good but http://khatchad.commons.gc.cuny.edu does not (as well as another on OpenLab). Reconnecting.

Ok, khatchad.commons.gc.cuny.edu seems to be working (for now) after the reconnect to JetPack.

#21 Updated by Boone Gorges about 1 year ago

Thanks, Raffi. I've just run a script that changes all 'home' and 'siteurl' values to https, as well as the networkwide siteurl. Let's leave this open to see what sticks.

#22 Updated by Boone Gorges about 1 year ago

  • Target version changed from Not tracked to 1.13.8

https://github.com/cuny-academic-commons/cac/commit/b45a65d176e0c1b7104b258b2c6aeea45331724b is the changeset where SSL is forced for 'home' and 'siteurl'.

Moving this into the next milestone. If we don't see a recurrence before then, we can close out.

#23 Updated by Boone Gorges 12 months ago

  • Status changed from Assigned to Resolved

Haven't heard more about this, so I'm going to close. Thanks for your patience, Raffi.

Also available in: Atom PDF