CUNY Academic Commons

Thanks for starting this ticket, Boone. It would be great if we could leave a foundation here for more complex logic down the road. But for starters, I think for all three of People/Groups/Sites we've been coalescing around:

-From the user's campus
-Most public privacy level only
-Most recently active

For groups I think we could potentially get more complicated because we have the added measurement of total members, and perhaps some groups will dominate the recent activity metric (or it's just too noisy to seem like a real recommendation). I don't think we just want to show the biggest groups, as that could have little variance over time also. It would be nice if new groups worked their way in sometimes, whatever we choose. Is it possible to go with highest growth rate over a time period, like two weeks, rate being new members in that span divided by total members? Or open to other thoughts.

It would be nice if new groups worked their way in sometimes, whatever we choose.

"Creation" counts as "activity", so they'll end up in the most-recently-active list.

Is it possible to go with highest growth rate over a time period, like two weeks, rate being new members in that span divided by total members? Or open to other thoughts.

We don't have any mechanism for tracking growth over time, so I recommend that we table this suggestion. I think that activity is likely to be a good proxy for this anyway.

In fact, activity is likely to be a good proxy for lots of measures of "interestingness" that I can think of, so I agree that it's a good place to start with all three content types. As you note, the one danger is that extremely busy groups could create noise. But I think we should wait to address this "problem" until we've seen it in action. So, I think that your suggested criteria are good for now: show the most recently active items that are public and from the user's campus(es).

Sounds good to me. I'll keep an eye on this ticket if you hit any snags with that logic or want to tweak it.

Based on the following criteria:

So, I think that your suggested criteria are good for now: show the most recently active items that are public and from the user's campus(es).

Here's a brief tech overview:

Users:

• User's campus is stored in BP xprofile field data under the field "College".
• A BuddyPress user does not have an option for privacy, so we can ignore this.
• User's recently active timestamp is stored in the BuddyPress activity component.
• BP members loop (bp_has_members()) cannot be filtered by date_query. Can be an upstream improvement to BuddyPress.

Groups:

• Group's campus is stored in group meta under the "cac_campus" meta key. (See cac_get_group_campuses().)
• A BuddyPress group can be filtered by public status, so we will only query for public groups.
• Group's recently active timestamp is stored in group meta under the "last_activity" meta key.
• BP groups loop (bp_has_groups()) cannot be filtered by date_query. Can be an upstream improvement to BuddyPress.

Sites:

• Site's campus is stored in BP site meta under the "cac_campus" meta key. (See cac_get_site_campuses().)
• Only public sites are recorded into the BP blogs table, so we can just do a regular BP site loop query.
• Site's recently active timestamp is stored in BP site meta under the "last_activity" meta key.
• BP sites loop (bp_has_blogs()) cannot be filtered by date_query. Can be an upstream improvement to BuddyPress.

Implementation notes / questions:

• Will use a custom cac/suggestions REST API endpoint for this. Will probably extend the existing BuddyPress REST controller classes, but will override the register_routes() and get_items() methods to add our custom campus filter.
• The date_query improvements would allow us to do fine-grained recently active lookups. For example, recently active groups from X campus over the last three months. However, this is not a requirement.
• What should we display if there are no suggestions or not enough suggestions? A message to visit the respective directories?
• How often should a suggested item be seen? A simple random order is mentioned in the initial post. However, if we wanted to do something more complicated, do we want to record how many times a certain item is shown with a transient and to omit those items from the suggestions query for X number of days? Perhaps the recording should only take place for the first paginated set of suggestions. Feedback appreciated.

Regarding Colin's question:

Is it possible to go with highest growth rate over a time period, like two weeks, rate being new members in that span divided by total members?

This might be possible by:

1. Asking BuddyPress to pull up all the "X joined the group" (joined_group) activity items over the last two weeks.
2. From the results, we parse out the group for each joined_group activity item and tally the count. Then we can calculate the "new members / total members" rate that way.

This can only be possible for groups since we do not record an activity item when someone joins a site.

What should we display if there are no suggestions or not enough suggestions? A message to visit the respective directories?

Will this ever actually happen? If we're filtering only by shared Campus, it seems like there will always be results, since each campus is fairly well represented in our site/group/user directories. But, in case it ever does happen, perhaps we could simply lift the Campus requirement and randomly pull from the entire Commons community. (Or, if we are operating using a restriction like "active in the last month", we could bump that number up to two or three months in order to fill in the missing slots.)

How often should a suggested item be seen? A simple random order is mentioned in the initial post. However, if we wanted to do something more complicated, do we want to record how many times a certain item is shown with a transient and to omit those items from the suggestions query for X number of days? Perhaps the recording should only take place for the first paginated set of suggestions. Feedback appreciated.

This feels like it's bound to get complicated. If the pool of available suggestions is large enough, then I propose that random suggestions will be "good enough". In other words, if there are four slots for suggested groups, and they're being pulled from a pool of 50 qualifying groups, then on average, you'll only see a duplicate every couple of page refreshes. So I guess my suggestion is to make the suggestion criteria loose enough so that the pool of suggestions is fairly large, rather than trying to force the appearance of randomness.

At some point down the road, if our suggestions are based on more sophisticated logic, it could turn out that they're "too" specific. In that case, we might consider a dismissal mechanism for individual suggestions. (Twitter had this at one time - not sure if it still exists.)

Is it possible to go with highest growth rate over a time period, like two weeks, rate being new members in that span divided by total members?

This sounds really complex to me, and I suggest that we set it aside for a future iteration. Keep the suggestion logic simple for now, and we can complicate it down the road.

A question for Ray and the group: Will the suggestion query exclude items that you are already connected to? In other words, don't suggest groups that a user is a member of, members that the user is friends with, sites that the user is a member of?

Will this ever actually happen?

(Or, if we are operating using a restriction like "active in the last month", we could bump that number up to two or three months in order to fill in the missing slots.)

Good call. Maybe this will occur for groups when we go with some more complicated logic as discussed above. We can increase the time period if we decide to go that route as you mentioned.

If the pool of available suggestions is large enough, then I propose that random suggestions will be "good enough". In other words, if there are four slots for suggested groups, and they're being pulled from a pool of 50 qualifying groups, then on average, you'll only see a duplicate every couple of page refreshes.

Now that you mention it, random makes more sense and is easier!

Is it possible to go with highest growth rate over a time period, like two weeks, rate being new members in that span divided by total members?

I actually think what I suggested is not that complicated. The BP activity component already has the ability to filter by date. But as I mentioned above, this would only work for groups due to the joined_group activity item. Update - the one thing is filtering out private groups after the initial activity query though, so we might get less results afterwards.

Will the suggestion query exclude items that you are already connected to?

I would probably recommend excluding since if you're already connected, you probably wouldn't be as interested in viewing the suggested item.

I've got a first pass of this ready for testing on cdev.

Once you are logged in, navigate to the homepage and view the widgets in the third column under "Recent Posts". I've added a suggestion widget for each component -- Members, Groups and Sites. The widgets are lazy-loaded, meaning the suggestions only load when they appear on screen and not before.

Right now, the algorithm is as suggested: show the most recently active items that are public and from the user's campus(es). I primarily did this so we can see results on cdev :)

I didn't add a date query to limit the results to say the most recently active items from X months with a random sort, but the functionality is already coded and ready to go if we decide to go in that direction. I've also added a patch upstream for BuddyPress to limit results by date for members, groups and sites here.

To test suggestions with a different college, you will need to go to https://commons.gc.cuny.edu/wp-admin/users.php?page=bp-profile-edit to select a different one for your user account. I tested with CUNY Graduate Center.

• While testing on cdev, I found that the College data for older users was saved as serialized data before moving to a string format. So I had to account for that in the member xprofile_query here. Question for Boone, were users able to select multiple Colleges on the registration page in the past? If so, I'll have to rework the suggestions queries to handle this.
• Unlike the member's College data, the group and site's College meta is saved with the college key, not the college full name. So I created a helper function called cac_get_cuny_campus_key_by_field() to fetch the college key from the fullname (I haven't committed this function yet).
• I decided to utilize the existing BuddyPress REST endpoints rather than implement a new REST namespace and used a custom type=suggestions in the query loops to modify the query.

• I haven't included the "Add Friend" button to the members suggestions widget as outlined in the new homepage wireframes yet.
• I haven't added pagination support or the ability to refresh the suggestions with a new set yet.
• Styling adjustments once the new homepage is at a suitable stage in development.

Awesome! Thanks for sharing your progress, Ray. I've looked through the cac-suggestions repo and it's looking awesome.

While testing on cdev, I found that the College data for older users was saved as serialized data before moving to a string format. So I had to account for that in the member xprofile_query here. Question for Boone, were users able to select multiple Colleges on the registration page in the past? If so, I'll have to rework the suggestions queries to handle this.

I don't recall what the registration screen used to look like. But I just did a query on the production site and it looks like 3064 out of 5444 values of 'College' are serialized arrays, so we have to do something about it. I'd like to move toward a unified treatment of College/Positions data, but this will take a good amount of work across the codebase. So, for now, a LIKE query like the one you've linked to is probably best.

Unlike the member's College data, the group and site's College meta is saved with the college key, not the college full name. So I created a helper function called cac_get_cuny_campus_key_by_field() to fetch the college key from the fullname (I haven't committed this function yet).

Blergh. See above. I intentionally moved to storing the key rather than the name, because the display name occasionally changes. If you're going to go with a single source of truth, I'd make it the key (cac_get_cuny_campuses()) and then build a translation map to the name(s) of each college. Sounds like this might be the opposite of what you're suggesting, but there's more than one way to skin this cat.

I think we can hold off on additional interface mods until after we have the first round of visual designs.

But I just did a query on the production site and it looks like 3064 out of 5444 values of 'College' are serialized arrays, so we have to do something about it.

Were you able to quickly see if any of those serialized arrays contained multiple colleges?

I'd like to move toward a unified treatment of College/Positions data, but this will take a good amount of work across the codebase.

Ahh, I just noticed the Positions field from the CAC Advanced Profiles plugin and on the Groups Directory filter interface. Did you want me to follow similar logic for the filtering?

If you're going to go with a single source of truth, I'd make it the key (cac_get_cuny_campuses()) and then build a translation map to the name(s) of each college.

I chose the opposite approach because the value from a user's xprofile data is the full college name, not the key.

This is the code for cac_get_cuny_campus_key_by_field:

/**
 * Fetch the campus key by campus data field.
 *
 * @param  string $value Value to search for.
 * @param  string $field Field to search from. Default: 'full_name'
 * @return string
 */
function cac_get_cuny_campus_key_by_field( $value = '', $field = 'full_name' ) {
    foreach( cac_get_cuny_campuses() as $key => $data ) {
        if ( $data[$field] === $value ) {
            return $key;
        }
    }
    return '';
}

I'll commit this function once I get your sign-off :)

Were you able to quickly see if any of those serialized arrays contained multiple colleges?

mysql> select count(*) from wp_bp_xprofile_data where field_id = 2 and value like 'a:1:%';
+----------+
| count(*) |
+----------+
|     2676 |
+----------+
1 row in set (0.01 sec)

So it's about 400 that have more than one. We could simply convert these to separate entries in the data table, but I'm unsure if this would break a lot of other stuff.

Ahh, I just noticed the Positions field from the CAC Advanced Profiles plugin and on the Groups Directory filter interface. Did you want me to follow similar logic for the filtering?

In theory, data from Positions should be synced to College. See https://github.com/cuny-academic-commons/cac/blob/1.18.x/wp-content/plugins/cacap-cac/includes/widgets/positions.php#L80, https://github.com/cuny-academic-commons/cac/blob/1.18.x/wp-content/plugins/cacap-cac/cacap-cac.php#L13

In fact, it could be that this is where the serialized entries for College are coming from.

In any case, maybe you could do a couple of queries to see if Positions is, in fact, mirrored in College. (Like, maybe compare a list of user IDs in each place - there should be none in Positions that aren't also reflected in College) If so, there's probably no need to check Positions.

All this is demonstrating how the separation between the two is really problematic, so if you have bright ideas about how to tackle the larger problem, I'd be glad to hear them.

I chose the opposite approach because the value from a user's xprofile data is the full college name, not the key.

As long as the name of the college doesn't change, this should work OK. We went through a process of changing a couple college names in the last year or two, but at that time I ran a script that did the search-replace in the bp_xprofile_data table as well. So they should match up.

Ray, can you take the reins to integrate this feature into the new homepage? The main sections (https://www.figma.com/file/3DMoVZ5ZG79AcwZOmYH7nn/CUNY-Homepage-design---Development-Handoff?node-id=820%3A1486, see "Homepage (logged-in)") are:
- Campus Activity https://github.com/cuny-academic-commons/cac/blob/f3158f150819ff9c7ae901c07dc4ba4b215ac0e6/wp-content/themes/bp-nelo/parts/home/sites-logged-in.php#L20
- Groups from your campus (I already implemented a quick-and-dirty version of this at https://github.com/cuny-academic-commons/cac/blob/f3158f150819ff9c7ae901c07dc4ba4b215ac0e6/wp-content/themes/bp-nelo/parts/home/groups-logged-in.php#L2, but feel free to modify)
- Members from your campus https://github.com/cuny-academic-commons/cac/blob/f3158f150819ff9c7ae901c07dc4ba4b215ac0e6/wp-content/themes/bp-nelo/parts/home/members.php#L46

You'll notice that, in the case of Groups, I took the results of the 'Campus' query, and excluded them from the more general 'Commons-wide' query. This is a small nicety that it would be cool to carry over to the others, but probably not critical.

I was fearful that the Members queries would be very difficult to handle, but now I recall that you already worked out something that is viable. If you think it's good enough, let's go with it. Otherwise, this might be the time to consider a central way of recording object-campus affiliations, maybe a table:

id | object_id | object_type | campus

where an object might have multiple campus affiliations, and 'campus' would be the immutable campus slug (see cac_get_cuny_campuses()) rather than the name, which can change over time. This kind of logic might help to simplify the queries across the board, and would also make it easier to do queries across object types in the future. On the other hand, this kind of change is a decent amount of work, between building the CRUD tools, writing a migration script, and then implementing the sync mechanisms between XProfile and the new table, etc. So I leave it up to your judgment whether you think it's worth tackling at all, and if so, whether it can/should be done now vs later.

See also #14184.

In theory, data from Positions should be synced to College.

In fact, it could be that [the Positions widget] is where the serialized entries for College are coming from.

I can confirm that this is the case.

Whenever the Positions widget is saved, it mirrors the college data back over to the "College" XProfile field. So the "College" XProfile field can solely be used and we can omit the lookup for the college data in the Positions widget.

I was fearful that the Members queries would be very difficult to handle, but now I recall that you already worked out something that is viable.

With that in mind, this will work for member queries: https://github.com/cuny-academic-commons/cac-suggestions/blob/67de19f6cd22cb08c6a2c81775eb63501c25838a/src/Get.php#L18-L41

It's using a LIKE query and the value is querying the College name, rather than the campus key due to how the College XProfile field is saved. As you mentioned, this has to potential to be out-of-sync down the line if the college name changes. If we're concerned about this, let's talk things over in #14184.

The "Groups from your campus" and "Members from your campus" will be covered by the Suggestions plugin.

I originally coded Suggestions as a widget, so I'm going to either create new widget sidebars for each component (members, groups) or call the widget directly into the template.

The hardest part is actually coding the Campus Activity portion. This would probably require a DB query that roughly resembles:

SELECT * FROM wp_bp_activity WHERE
( item_id IN ( cuny_get_user_campus_group_ids() ) AND component = 'groups' ) OR
( item_id IN ( cuny_get_user_campus_site_ids() ) AND component = 'blogs' )


When the cuny_get_user_campus_group_ids() and cuny_get_user_campus_site_ids() functions are created, they'll need to be cached. What do you think, Boone?

The object-campus affiliations custom DB route could help with the group and site campus relationships, but I'm going to try to do this without the custom DB at this time.

The main sections (https://www.figma.com/file/3DMoVZ5ZG79AcwZOmYH7nn/CUNY-Homepage-design---Development-Handoff?node-id=820%3A1486

The Figma link doesn't work for me. Is there an updated link?

The Figma link doesn't work for me. Is there an updated link?

I've pinged Sara to add you.

When the cuny_get_user_campus_group_ids() and cuny_get_user_campus_site_ids() functions are created, they'll need to be cached. What do you think, Boone?

Yes, I think that's as good as we can do. This is one of those queries that we should carefully monitor - the slow query log, Query Monitor, etc. Joining against a separate 'campus-affiliation' table is likely to be much faster, but I'm fine to hold off on this change for now.

The object-campus affiliations custom DB route could help with the group and site campus relationships, but I'm going to try to do this without the custom DB at this time.

This sounds like a good approach for now. If we notice issues down the road, we can change out the wiring.

Latest updates on cdev include the "Campus Activity" and "X For Your Campus" suggestions blocks for the logged-in homepage.

Some notes:

• The "Campus Activity" block currently does not have any restrictions for activity type. Should we limit this block to only blog posts and blog comments?
• The "Browse Activity Across CUNY" and "More Site Activity" links do not work yet. I'm assuming we want to link to the activity directory page (a.k.a. the "News" page) and filter the items that way?

Boone, the suggestion blocks require the cac-suggestions plugin. I've added it as a submodule here: https://github.com/cuny-academic-commons/cac/commit/f50a2775a085087a2eeccc690548825491278c6f

You'll notice that, in the case of Groups, I took the results of the 'Campus' query, and excluded them from the more general 'Commons-wide' query. This is a small nicety that it would be cool to carry over to the others, but probably not critical.

I've included the ability to manually exclude items from suggestions. See https://github.com/cuny-academic-commons/cac-suggestions/commit/84dfa18ced36afa54548ee05097ad6aa60fe4aa4 and the implementation here: https://github.com/cuny-academic-commons/cac/commit/794d06b51caf67fbd35dc67e5a81167d103c087e.

Let me know if anyone encounters any issues.

One thing I noticed is that, since the Featured items are loaded asynchronously and not rendered as part of the synchronously rendered "global" stuff, it can result in some grid mismatches between columns.

This isn't really a problem due to being loaded asynchronously. We're splitting the "Members Across CUNY" and "Members From Your Campus" blocks into separate columns, so they use separate grids. I also don't think this is an issue. Could also mitigate the problem by truncating member names, pronouns and campus so they do not take up more than one line, but that isn't very good for UX. Although if we decide to truncate, we could perhaps use a mouseover effect so the parentheses are expanded to view the full line.

The "Campus Activity" block currently does not have any restrictions for activity type. Should we limit this block to only blog posts and blog comments?

This is done. See https://github.com/cuny-academic-commons/cac/commit/cc4ad6f1d0504e547b12ace339a9f548049eed33.

One other thing I forgot to mention is the "Campus Activity" block for the logged-in homepage will not show up if your campus does not have any blog posts. When this happens, the "Recent Site Activity" block will span the entire width of the section.

Sorry to jump in late here, but a couple points on the Campus Activity specs:

- That feed isn't actually in the Sites blade, right? It's right below the Shortcuts blade, before the Groups/Sites/Members blades. As such, we may not be limited to only Site activity on a campus.

- We talked through this a bit many moons ago here: https://redmine.gc.cuny.edu/issues/13999#note-18
And settled on these items for that feed, as long as they're public:
- New group created
- New site created
- New site posts
- New group topic

I'd be up for adding comments in as well - better to be too lively than not lively enough - and we could always scale it back.

Just following up from the call, where we decided that the two "activity" columns in question would be, from left to right, "CUNY-Wide Activity" and "Activity from Your Campus(es)" to match the lower columns that also go from wide-to-campus left-to-right.

Then these two columns are going to show the same set of (public) items, changing only on the right to limit by your campus:

- Group created
- Site created
- Site posts and comments
- Group forum topic and posts

We acknowledged this will be a firehose, but let's see in practice how it shapes up. This will also allow each column to have the same formatting, as opposed to before when we had post-specific formatting on one side.

I think we can also nix the "View the latest..." "Discover the latest..." tags underneath each header, as it's pretty self-explanatory now.

On cdev, I've updated the activity section for the logged-in homepage as per the dev call and what Colin wrote above. (Commit - https://github.com/cuny-academic-commons/cac/commit/b5d1ef2cff0d3ff6d1073547d4a6c51ea4d28377.)

I'll leave Ray to look into these issues, but I wonder if some of the homepage stuff might be related to overly aggressive caching.

But why in the screenshot is it missing the "new test topic colin" from "dec new group colin" that I see in the CUNY Wide stream?

Ah, the "new test topic colin" group is private. Previously, only public group activity from the user's campus was displayed.

I've just added a commit so the logged-in user's campus groups -- regardless of group status -- is also displayed in the campus activity block: https://github.com/cuny-academic-commons/cac/commit/e66f4bf2a6f4b5d853646d8af2da3798f979a50f.

Can you test again, Colin?

I also made a post on this site, marked for Brooklyn College, that doesn't seem to be appearing in CUNY Wide or Campus columns but is in the internal full activity streams.

By default, BuddyPress does not record activity items for private sites, so that's probably why. This is a longstanding BuddyPress issue: https://buddypress.trac.wordpress.org/ticket/4831.

Hi Ray, you were right about that group being private, but yes the forum post is showing up now despite that. Makes sense to me that you'd see your own private groups/sites in the streams if logged in, but I wonder if it's confusing to show them for groups and not sites (since we can't get around that BuddyPress issue with sites). Maybe it's not a big deal.

That site I mentioned is actually public, though, so the BP issue shouldn't apply:

https://teachtest.commons.gc.cuny.edu/

I just did another post there and I see it in my Campus Activity and on the internal streams, but I don't see it in CUNY-Wide Activity yet. Maybe that's just a lag issue (only did the post a few minutes ago) but thought I'd mention it.

Attaching a screenshot here, because now I see the site post in question in CUNY Wide but not in Campus Activity, and all of my activity items there are 3 days old.

I forgot to mention that in the Campus Activity block, we exclude items that are already visible in the CUNY Wide block so you will not see duplicates in both blocks.

That site I mentioned is actually public, though, so the BP issue shouldn't apply:

https://teachtest.commons.gc.cuny.edu/

I see the problem now. The issue is due to the caching of the user's site campuses. I'm working on rewriting this logic to deal with when a new site is added with the user's specific campus(es). Will update this ticket when I'm done.

Ok Ray, thanks for both of your points here and the caching work. That sounds right about the logic to avoid duplication between the two columns.

I see the problem now. The issue is due to the caching of the user's site campuses. I'm working on rewriting this logic to deal with when a new site is added with the user's specific campus(es). Will update this ticket when I'm done.

This is done and is available for testing on cdev. See https://github.com/cuny-academic-commons/cac/compare/52bd665...c9116fd.