We will be sending an email blast using Mail Chimp in the near future to all Commons users and we want to measure the impact of the email through analytics.
How many users clicked on the email to visit the Commons?
How many of those users were dormant or inactive accounts?
How many of those users visit the Commons a second time within a week?
Perhaps there are inactive users who reactivate their use of the Commons who do so without clicking the email link, but use some other method, like a favorite in their browser. How do we measure them?
So, we will need to define some parameters like what does it mean to be an inactive account?
Then we will have to capture the metrics. Can we use Mail Chimp or Google Analytics or do we have to run some custom queries?
#1 Updated by Raymond Hoh almost 6 years ago
Mailchimp should have built-in analytics to change the URLs we want to track into URLs that Mailchimp will track for analytics purposes.
As for tracking users from Mailchimp, we could do something with Google Analytics like this:
#2 Updated by Matt Gold almost 6 years ago
That's interesting, Ray. I agree that Mailchimp will provide us with some interesting data related to the mailing. What we are hoping to see is whether this mailing helps improve usage of the Commons over time -- and particularly to see whether it helps activate members who have recently been inactive. The idea (which came from Lisa Rhody, our new Deputy Director of Digital Initiatives of the GC) is to take the pulse of the number of active users who had been active in the last (day/week/month/year) one week before the mailing, and then do run those same tests/measures a week, two weeks, a month later and see whether there had been any effect.
That would provide a pretty rough indication of users potentially newly made active by the mailing (and of course, correlation is not causation etc -- no way to know whether these changes were really provoked by the email etc)
Of course, if we could be more specific -- to compare a list of users who hadn't been active on the CAC in a year before the email, and then get the same list and compare afterwards -- would be great and might give us more specific data on this, as would the direct MailChimp/Google Analytics connection you linked to.
#3 Updated by Boone Gorges almost 6 years ago
As long as they log into the site, it will be pretty easy for us to query for a list of users who (a) have not been active in x weeks/months, and (b) are active in y days after the email goes out. (last_active in wp_bp_activity) It will be hard (and obtrusive) to test whether the login originates from the mailchimp email, which is the only way we could determine whether it was the email that caused them to log in.
#4 Updated by Matt Gold almost 6 years ago
That's great, Boone. It would be fine with me to "query for a list of users who (a) have not been active in x weeks/months, and (b) are active in y days after the email goes out. (last_active in wp_bp_activity)"
Are you able to set up these queries on a chron job of some sort so that we could get daily reports from, say, today through Dec. 20?
If not, can you please take a reading now/soon and then do another a week or so after the release/email message?
#5 Updated by Boone Gorges almost 6 years ago
- Target version set to 1.9
I'm not sure I'll have time to set up anything automated by the time we launch 1.9. Multiple pre-release data points would be of limited utility in any case - testing to see who's been active in the last 30 days on Monday, and then again on Tuesday, will only show us how many idle users became active between those days, which doesn't give us much useful info if we're releasing on Wednesday.
For the moment, I will plan to run two manual queries:
1. Immediately before the 1.9 release
2. Immediately before the email blast (will need coordination from whoever is pulling that trigger)
Then I can run manual queries at specified intervals after the blast.
We need to decide on the idle interval we are interested in. 14 days? 30? 3 months? I can do multiples, but let's not go overboard. I suggest maybe 14 days and 3 months, to get a short-term and long-term spread. Internally, I will keep a list of user IDs matching each of our chosen idle intervals, and will use those IDs to do lookups post-blast.
#7 Updated by Boone Gorges almost 6 years ago
Can we do 14 days, 3 months, and 1 year as the idle intervals?
This sounds great.
I am sending the email blast and will coordinate with you. How long before hand do you need notice?
Not long. It's not critical that the query be run within seconds of the blast - it's highly unlikely that a longtime-idle user will happen to return in the few moments before you send the blast. Just drop a note in this ticket when you have a scheduled time, and I'll try to run the query within the hour before that.
#8 Updated by Boone Gorges almost 6 years ago
Note to self that I now have a script to generate the data. Usage (from wp root):
$ wp eval-file ~/wp-cli-scripts/query-inactive-users.php
Log files are stored at /home/commons/inactive-logs/.
I've made a note in the release checklist to generate the log just before release. Matt, please let me know when you'll be sending the blast, and I'll record another set of logs.
I'll need modify the script when we want to get results post-release (adding an IN clause for the pre-recorded users).
Matt, we'll need to decide how to manage post-release logs. Do you want to run once daily for a week or two? Maybe that's too much data?
#9 Updated by Boone Gorges almost 6 years ago
- Assignee changed from Boone Gorges to Matt Gold
- Target version changed from 1.9 to Not tracked
I've generated the initial set of inactive logs, immediately after the 1.9 release. We should run another set immediately before the email blast goes out. I'm reassigning this ticket to Matt so that he remembers to coordinate this with me.