Out-of-control queries on production server
This is a master ticket to track potential causes of recent performance problems on the production server.
Symptom: Webserver RAM occasionally spikes, with no corresponding traffic increase. Spikes correspond to highly increased traffic between webserver and database server, suggesting a query loop that's out of control.
I've been collecting diagnostic data over the past 24 hours about the following:
- requests that trigger large numbers of database queries
- requests that take a very long time to complete
and I have some preliminary findings, which I'll spell out in specific tickets.