CUNY Academic Commons

Added a few watchers.

Logins are not working because WP is unable to update the list of user sessions in the usermeta table.

I've traced this back to a MySQL problem. It's not currently possible to run any delete or update operations on the mysql database for cdev. The process times out. There are sometimes errors to the effect of "ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction" but even this is not consistent - sometimes it just hangs.

I suspect that this may perhaps be linked to disk space. My understanding is that we recently had an issue with disk space on the production Commons site. The problem was exhibited differently because of the large amount of traffic that the production site receives. In the current case, it could be that there's not enough space on the disk to build the new indexes after an update, or something like that.

We're nearing the end of the debugging that it's possible for our team to do, and we'll need to ask IT for help - at the very least, they can restart the mysql daemon, but they may also need to look into disk space issues. Let's talk about this during today's call.

Some text for an email to IT:

==

The CUNY Academic Commons team recently discovered some odd behavior on our test site, which can be visited by pointing commons.gc.cuny.edu to 146.96.128.252 (LDV2) in your hosts file.

We've traced the issue down to a problem with MySQL. On LDV2, we use a database 'commons_main', accessed by user 'commons_admin'. Currently, we are able to run SELECT statements, but not UPDATE or DELETE operations. These operations either freeze up the CLI, or they timeout with an error message "ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction"

To reproduce, connect to the mysql database on LDV2 and try the following:

USE commons_wp;
UPDATE wp_blogs SET public = 1 WHERE blog_id = 5;

A few guesses:
1. An existing mysql process is preventing a table lock from being cleared. A restart of the mysql daemon might help to clear it.
2. The database is running out of disk space and so cannot perform write operations.

Could your team please help to investigate?

Hi Colin - Yes, I get the same problem.

Seems to be working for me now after the "mysql service on LDV2" restart that Angus Dickson just let a few of us know about via email.