Project Browser Initiative Meeting for [February 2, 2022] [#3264404]

Good day all! I am still wrapping up my daily standup but start giving a wave here!

shaal	Hello :blob_wave:
fjgarlin	:wave: hi there
mradcliffe	:wave: Hi
leslieg	Hi everyone
mandclu	:wave: from :canada:
tim	:wave:
AmyJune (volkswagenchick)	AMyJune in California!! Hi everyone
thejimbirch	:wave:
Gábor Hojtsy (he/him)	Hey
jjcs227	:wave::skin-tone-3:
dww	:wave:
bircher	:wave:

1️⃣ We are trying to generate a list of the "top 100 Drupal 9 modules" so we can target the right list of ones to attack. We're trying to identify "which are the first modules someone will see on the 'recommended' tab?"

chrisfromredfin	The sort/filter criteria we're recommending is here and below: #3240314: Decide on algorithm for "recommended" modules
chrisfromredfin	Option (1) Can we just identify the Top 100 by doing a filter on drupal.org? Unknown: when sorting by "most installed" is it most installed by release or most installed overall - we want by release.
chrisfromredfin	Option (2) Use Drupal's API to pull and aggregate this information. We need someone to write a "scraper" of sorts to pull this info and identify the Top 100. I have something started but it's not perfect. Would need to be enhanced to get us what we actually want.
chrisfromredfin	Option (3) We use the drupal.org database dump to query this information right from the DB. This requires someone who's writing these queries to have signed the NDA with the DA.
chrisfromredfin	In anyone willing to investigate this? This is critical information needed to be generated by someone technical so the SBS can get to work on improving the project module data (descriptions, logos, etc) on d.o.
mandclu	If there's no feasible way to do something like query usage for 9-compatible releases, this is something I could help do manually
mradcliffe	Could a maintainer of module that might be excluded from the recommended list because it isn't site-builder focused, but still popular by download (e.g. devel), be able to change categories to get it included as recommended?
chrisfromredfin	The API does provide release usage broken down by at least major (d9, ex.g.) but would still need to be aggregated. Similarly, I believe this data would be in the database if we went this route, broken down by release which would then need to be summed up across "all releases compatible with major"
mandclu	I think that given the intended audience, developer-focused modules make sense to exclude, just not sure how easy it is to make that call without it being subjective
mandclu	ctools is a good example of a module a site builder likely wouldn't get value from on its own, but would likely get pulled in by other modules that require it
mradcliffe	Is there a top 100 based on "stars", which in my opinion is on the extreme-side of subjectivity given GitHub stars.
mradcliffe	Or maybe Top 1000 downloaded, but weighted by stars?
chrisfromredfin	@mradcliffe we've had a lot of discussion about stars and they're basically useless to us because they're too subjective.
chrisfromredfin	a lot of people have devel starred, ex.g.
chrisfromredfin	someone new to drupal may still want devel, btw
chrisfromredfin	@hestenet (he/him) or @drumm may be able to answer how the sort for option (1) works on d.o now.
tim	I would like to help with this too. I am not an expert, but willing to learn what is needed. Will look at getting NDA if that option is best. (which need to look at closer to determine)
chrisfromredfin	@tim how technical are you? Are you able write complex SQL queries and/or write PHP to ingest an API?
drumm	Stars are relatively new too, they don’t have a lot of data.
tim	Well I like complex. I am currently doing queries in google sheets now which is probably basic compared ... I have created 2 applications in the past with MS acccess that did moderate queries.I can do PHP but that is probably the biggest weakness honestly. But wanting to develop skills in that area, just not an expert yet!
drumm	Download stats are useless, they measure how often CI/CD systems download things. We don’t do the log processing for them anymore.
chrisfromredfin	the "usage" you mean, @drumm?
chrisfromredfin	because this is a bombshell if so; we're basically assuming the usage stat is like our primary mechanism to sort popularity
tim	@drumm When did you stop processing them?
drumm	Was responding toOr maybe Top 1000 downloaded
mradcliffe	I think update module reporting is different from download statistics.
drumm	Right, we us usage stats
drumm	*use
drumm	To answer the original question - yes we sort by usage within a core compatibility when filtering https://git.drupalcode.org/project/drupalorg/-/blob/7.x-3.x/drupalorg_se... (edited)
chrisfromredfin	!!! ok that's awesome^^
chrisfromredfin	This means @leslieg like we were discussing perhaps we filter on d.o to identify top 100
chrisfromredfin	As long as we're able to match up all the criteria from the linked ticket.
chrisfromredfin	OK this is the closest search I could come up with: https://www.drupal.org/project/project_module?f%5B0%5D=im_vid_44%3A13028...…]vered&text=&solrsort=iss_project_release_usage+desc&op=Search
chrisfromredfin	@drumm do you know if I tweak the query string could I filter by two statuses (Actively Maintained OR Seeking CO-Maintainers) /
drumm	Not that I know of.
chrisfromredfin	ok so @leslieg that's something to consider, we can't QUITE use D.o to match exactly the query we'd run for recommended.
mradcliffe	So option 2 or option 3?
chrisfromredfin	right. I'm not sure. part of me thinks option 1 gets us close enough? But 2 or 3 gets us there perfectly...
Gábor Hojtsy (he/him)	there is this job https://dispatcher.drupalci.org/job/project_analysis_d10/ that has the dump of most used releases of drupal 9 compatible projects at https://www.drupal.org/files/project_analysis/projects_d10.tsv?75
Gábor Hojtsy (he/him)	this powers the Drupal 10 readiness dashboard on dev.acquia.com so the ordering of usage there is sourced from this data
mandclu	That data looks promising, but it seems like it shows some projects as a dev release even if there is a stable release in that branch
mandclu	Also, when I sort by usage, I see a number of cases where projects appear on consecutive rows, which suggests that the usage number is still per project instead of per release
mandclu	For example
leslieg	I’m also not clear on where the usage numbers are coming from. For token it looks like it may be looking at usage for the entire 8.x-1.x release, even though I believe only releases greater than 8.x-1.7 are Drupal 9 compatible. This could be fine as the most popular modules still would include Token. Trying to get a list of most popular for the project browser based on the version the site is currently running on seems difficult to figure out. Could be just me not understanding though.
mandclu	An approach we might try would be (as an example) just grabbing a list of the 200 most popular modules overall, and then manually go through and pull the numbers for D9-compatible releases. We could probably even divide and conquer if a few people were willing to help
chrisfromredfin	That is totally a simple approach, not sure if that's a ton of work or not, given that it could be queried…?
drumm	Compared to the important work, suggesting improved descriptions and icons, that’s a tiny amount of work.
Gábor Hojtsy (he/him)	@mandclu for this dump that they repeat does not mean they are for the project :smile: this is the query for generating the tsv: https://git.drupalcode.org/project/infrastructure/-/blob/main/stats/proj...
Gábor Hojtsy (he/him)	this is the usage part:(SELECT sum(count) FROM project_usage_week_release puwr INNER JOIN field_data_field_release_project fdf_rp ON fdf_rp.entity_id = puwr.nid INNER JOIN field_data_field_release_category fdf_rc ON fdf_rc.entity_id = fdf_rp.entity_id AND fdf_rc.field_release_category_value = 'current' WHERE fdf_rp.field_release_project_target_id = prsv.nid AND puwr.timestamp = (SELECT max(timestamp) FROM project_usage_week_release)),
Gábor Hojtsy (he/him)	I guess the answer to whether this is about the project depends on what’s in project_usage_week_release since that is where the last timestamp is taken. Based on the structure of that this may be summarizing all current (Drupal 8/9) project releases for the project or just the last one.
Gábor Hojtsy (he/him)	Hm, yeah based on https://git.drupalcode.org/project/project/-/blob/7.x-2.x/usage/project_... and the above query not limiting to specific release of a project it is indeed for all current releases of the project and not just for the latest release in general.
Gábor Hojtsy (he/him)	I think the reason we did this is that if you only look for the usage data for the latest release, the uptake will be small, so the ordering will be very different based on when the last release was made.
mandclu	I think it may be more complicated than that because for some projects, only the last release branch is D9 ready, but others may have multiple D9-ready branches.One other question to consider: If a project has multiple D9-ready branches but only one is stable, for the sake of measuring D9 popularity, is the sum of usage across those branches the right number, or only the stable one?
Gábor Hojtsy (he/him)	I think all D9 compatible branches is a better indication, otherwise when a module becomes D9 compatible, it will start at the end of the list (0 usage for that stable release).
chrisfromredfin	yes, my assumption was: "Query for all the releases compatible with version of Drupal, and sum their usage"
mandclu	But at least one of those branches must have a stable release
chrisfromredfin	ugh, yes. :slightly_smiling_face: THO I think that that information is available from the project page, but not positive.
mandclu	Yes, exactly. I'm confident this could be compiled manually from the project pages. If it could be queried directly, that would obviously be even better
chrisfromredfin	or perhaps it's some combination of writing the best d.o filter we can, and then manually pulling the info we can't quite get
chrisfromredfin	that probably makes the most sense.
chrisfromredfin	OK so @leslieg and I just had a call about this. We're going to grab the first 150 or so from d.o filtering by "any" development or maintenance status, and she'll open tickets for all this work. I think the Discover Drupal students can then help up by going to each module and determining if its maint and dev status jive with our filter, and if not will eliminate from the list.
chrisfromredfin	We have to use the API because maint/dev status is not shown on the project detail page, but I showed her how to get that info.
chrisfromredfin	https://www.drupal.org/project/project_module?f%5B0%5D=&f%5B1%5D=&f%5B2%...…]vered&text=&solrsort=iss_project_release_usage+desc&op=Search
chrisfromredfin	woo
tim	So we are going down the path of option 1?Option 1 as stated at the top of this thread.... (edited)
bircher	@chrisfromredfin yea that is basically the query I posted about in the other meeting that sparked this thread... https://drupal.slack.com/archives/C01UHB4QG12/p1643790700019799?thread_t...

2️⃣ To recap our synchronous meeting from last week, we decided to switch out the current mechanism of syncing all the data from d.o into a "database fixture" is being replaced instead by stubbing in an "API fixture" - that is a fixture that returns some semi-fixed JSON (could be initially populated with real D.O data?). This will help us define the API we want while still rapidly iterating on a prototype, and lets us then tell the DA what API we expect d.o to provide (as well as defining an API that other folks might want to provide privately, since we'll have a pluggable backend system).

chrisfromredfin	FIRST, we need an issue for this if there isn't one already. SECOND, we need someone to work on it. :wink:
mradcliffe	ping @prodigyjoe, @Devon Ek, @jjcs227 @AmyJune (volkswagenchick) .
mradcliffe	Are there notes for the synchronous meeting? I tried searching the project for the Meeting component but I couldn't find any. It would help to have the full notes in being able to find or create an issue about what the problem/motivation, proposed resolution, and remaining tasks are.
mradcliffe	This might be related: #3224709: [API] Include API translation layer with mock/fixture implementation until d.o API is ready ?
chrisfromredfin	^ Yes, OK that's already the issue.
chrisfromredfin	We have not done notes for a while, because we've been too busy. Last week we had a thread to ask if the Disc Drupal students might be able to do our SLack notes! Easy way to contribute :slightly_smiling_face:
Gábor Hojtsy (he/him)	I think @timplunkett (he/him) was maybe taking notes last week? But yeah the only topic was how the data is sourced for the module and the above interim solution was the best we could come up with.
dww	Sounds like a prudent move. I guess that means we don’t need the release migration immediately...
chrisfromredfin	correctamundo ^

3️⃣ Members of the Discover Drupal program have joined this channel and will be helping us out with the Project Browser Initiative. If you are from that group, please say hi and let us know where your interests are. Back end development, design, testing, site building or anything else (including I have no idea at this point). All are welcome to help and no experience is needed

tim	Also are you still looking for an Admin?
AmyJune (volkswagenchick)	Hi there! I am one of the mentors for the graduates.
mradcliffe	Hi, I'm involved as a Drupal mentor. I've been looking at the spreadsheet. Yesterday @jjcs227 was looking for a place to get started so I suggested reviewing some of the notes in the spreadsheet to start.

4️⃣ Still looking for a volunteer to help record meeting notes as issues. :slightly_smiling_face:

AmyJune (volkswagenchick)

Leslie is creating an issue with instructions, and hopefully the Discover Drupal students can help with this

Participants:

shaal, fjgarlin, mradcliffe, leslieg, mandclu, tim, AmyJune (volkswagenchick), thejimbirch, Gábor Hojtsy, jjcs227, dww, bircher, chrisfromredfin, drumm