Project Browser Initiative - Site Builders Subcommittee meeting Jan 18, 2022 [#3263717]

1️⃣ I had a discussion with @nod_ about the use of Svelte as a front-end framework. There are two schools of thought here. (1) This could be the project that pushes Svelte into Drupal core, but we would definitely have to fight for it. Advantages are that (a) there's no JS library that's dependent -- ie we don't need to get "Svelte in core" in order to use Svelte, since ultimately what's compiled is vanilla JS. (2) This is a new thing, and it therefore reduces our time-to-market. We have lots of Drupal knowledge about render arrays and using the Drupal.ajax framework. With that knowledge, we may be able to get more people onboarded to work on the project, and we relegate "advanced project browser" UI to a contrib module, even while Project Browser is in core.

chrisfromredfin	We might as well have this discussion now, earlier, since we're not too far deep into the work. Personally, I'm at "well this is already done in Svelte" - however, hardly anyone is showing up to do work and I can't help but wonder if the Svelte thing is scaring people off. I would rather get some kind of MVP to market fast. I just don't know WHICH accelerates the project: ripping out work already done in favor of something people are more comfortable with or (b) using the head start we already have and just continuing. I'm curious to hear what EVERYONE thinks about this, but especially those folks who are very close to core. If there's a resounding "there's no way this is gonna get in like this" then we likely have no choice. ?
chrisfromredfin	Obviously one major factor in this switch would be if the work were already done. If there were an open MR that pushed Svelte out of the project and replaced it with core Drupal.ajax and render arrays... well, I'd certainly have an easier time leaning one way or another.
chrisfromredfin	@Ron Northcutt I know you have opinions here, and don't want them to go unheard.
Gábor Hojtsy (he/him)	I don’t think Svelte is scaring people away, there are much higher level meta things that people are not showing up here, that could be its own thread, happy to discuss and not derail this thread with that.
Gábor Hojtsy (he/him)	On the concrete topic of using Svelte or not, whatever code is generated by Svelte is based on libraries that would need to be vetted by core committers, including security practices and backwards compatibility practices. Eg. if they like to release new breaking changes all the time and not backport security fixes to older branches (or not even maintaining older branches), that is a problem.
Gábor Hojtsy (he/him)	(Assuming project browser offers some kind of API and is not entirely internal that nobody should build anything on top of).
Gábor Hojtsy (he/him)	Drupal would need to sign up for backporting fixes from the newer branches, coordinate with their release cycles, etc. This is not in itself a problem but we’ve been seeing how heavy it has been with backbone, jquery UI, and various other libraries already used by core.
chrisfromredfin	I partially think of this as like "does core use sass" - except yes, I guess there are also security consideration since the JS it compiles may have vulnerabilities. This is probably the do-or-die for "which JS framework gets into core...?" if any... :slightly_smiling_face:
shaal	Vue!:stuck_out_tongue_winking_eye:cc: @bnjmnm
Gábor Hojtsy (he/him)	Yeah the JS it compiles is still based on 3rd party JS which has releases and potential vulnerabilities, etc.
Aaron McHale	There's also huge advantages to building this using Views, not just because it's already in Core and so doesn't introduce additional dependencies, but also because it reduces the barrier to entry for someone contributing (they don't need to learn a whole framework just to contribute to PB), it's easy for a site/distro to customise the views to suit their needs, any work done here to improve Views for PB is a win for everyone else (say a new component or way of rendering is needed, adding that to Views benefits the whole of Core and the community).This also has to be maintainable in the long-run, sure it might be nice to pick the shiny tool to do the job, but I've seen that problem play out way too many times, where then what looks shiny and cool now turns into a maintenance burden further down the road. Add to that everything else that's in core which needs tracked and maintained, adding a framework just to Core that's only used by PB is not sustainable.There's also the UX/UI/A11y considerations, basically it has to look like and fit into the Admin UI. We can't introduce something that behaves differently here, we have to use existing components and improve those components where needed. Claro, and the various components in Core, has been through extensive accessibility and usability testing/tweaking. So PB project saves a lot of time and effort by just using what's already there.
Gábor Hojtsy (he/him)	How many plusses I can add to @Aaron McHale? ;)
xjm	If the initiative team is actually interested in using Svelte, you should start a "[Policy, no patch]" issue in the core queue to propose using that as a dependency. It would need signoff from release managers, backend and frontend framework managers, and JavaScript committers.There is a strong chance that the answer would be "No, we're not willing to add that as a core dependency."Even if the answer were "yes", it would take a lot of time and work to get to that point of allowing it as a core dependency.Dunno if that helps put things in context, but from my perspective and a few others, Svelte was useful as a way to build a prototype for research. Research is done now and it's time to start architecting the actual planned MVP. :slightly_smiling_face:
irinaz	@xjm this is very important point about signoff from release managers, backend and frontend framework managers, and JavaScript committers on additional framework that is used on project targeted for core. Can you give more details on this topic?
xjm	See the core dependency policy and core governance documentation for some background concepts :slightly_smiling_face:
Ron Northcutt	Late to the party, but some thoughts:Svelte was chosen specifically because it has a low learning curve and because is compiles to pure JS. That makes it easier for non-JS developers to use/extend, and it doesn’t carry any specific dependencies with it. Contrast this with jQuery, React, Vue, etc.Using views is tempting from a Drupal perspective, and @mglaman has a cool contrib module that aims to solve this problem. However, this would be yet another major hurdle to get into core on a short timeline. In any event, to do this right would require something like that module in core which would be amazing, but seems unlikely. Pure JS app is much more reasonable.The concept that PB (which is doing something new) can’t or shouldn’t introduce new concepts/patterns becuase it is the only thing to use it… is a circular argument. We can’t do new things in core because new things aren’t done?Accessibility and design are very important topics. My suggestion has been to provide minimal CSS for layout, but otherwise expect to inherit the CSS from the admin theme. As for HTML structure related accessibility, that work will need to be done either way.Security risk - this should be pretty minimal. We don’t need auth for the client app, and unlike other approaches that rely directly on a whole chain of library dependencies, this is generated code. If anything, we could define the limitations of including external libraries. The alternative being discussed is vanilla JS created by hand, which should carry a similar risk profile.Security #2 - because this is generated code without a need for external libraries, then this drastically reduces the risk. Once the PB component is created, tested, and launched… it could in theory never need to be changed. So, in the event that the Svelte community fails in some major way (which doesn’t appear to be likely), then this code would remain unaffected. This is another reason why it is such an attractive option - stability is not tied to external dependencies.JS PR - another very interesting advantage is that by using a very modern (but reliable) JS tool that is currently quite popular (and gaining in popularity), it telegraphs to the rest of the JS community that Drupal core is modernizing in interesting ways. On the contrary, if we use custom vanilla JS, AJAX, jQuery, etc… that does little to help Drupal’s reputation in JS circles.
Ron Northcutt	So, I see some great ideas and very legitimate concerns, but so far I see nothing that indicates using Svelte generated JS for the UI is a show stopper.
nod_	We could view data remote it (would be faster to vet for core than svelte) or we could hardcode it doesn't need to be a generic solution for consuming remote data with views. If you're not seeing how svelte can be a show stopper I suggest you look at #3259323: [meta] Project list rendering & path to core commit I'd be happy to have answers to the few questions that will come up. Especially around extensibility and translation
nod_	Most of these wouldn't be a show stopper for a contrib module though
nod_	And I'm more and more convinced that who we need to attract in drupal are css devs not Js devs to increase market share
nod_	Also who will maintain the sveltejs code? I personally don't look forward to it, so it would need someone willing to be core js maintainer to take care of it, with all it implies.
Gábor Hojtsy (he/him)	@Ron Northcutt Either I am missing something or there are some misunderstandings here. That there is no active runtime dependency does not mean that code generated from 3rd party libraries is not based on dependencies. Those could still have issues that would need to be solved in collaboration with the 3rd party. Drupal has been swinging to the side of using a lot of 3rd party components on the backend and it’s been painful when they made new major versions available and did not fix security issues in old ones, or they got abandoned (eg. doctrine annotations is causing us a lot of headaches there) or they released security fixes without notice or used public security reporting mechanisms. That is why for whatever new 3rd party dependencies, a dependency evaluation is made to explore their release cadence, security practices, etc to see how well it matches Drupal’s. That there is no authentication code in svelte per say does not make it inherently secure, this UI is targeted at a level of user that can install new extensions on the site after all, which is as dangerous as it gets.
Aaron McHale	If we also consider it from the perspective of someone wanting to contribute to Drupal and get involved in the community, they already know some PHP, maybe some CSS and JS, they start learning Drupal and contributing in the issue queue, maybe they fix a few issues in Core. Now they want to contribute to Claro, and so discover they need to learn a CSS and JS libraries, as well as the toolset, so they do that. Then they want to contribute to a part of core which still uses jQuery, now their knowledge of jQuery is limited because they didn't consider it to be a "modern" JS approach, so now have to learn some jQuery. Then they discover Drupal's own JavaScript libraries, so they learn those, and, then they discover the decoupled menu initiative so they learn the tooling for that. Finally, they realise there is a improvement they want to contribute to PB, but in order to contribute, they have to learn another JS framework. Maybe it's at that point, or maybe it's earlier, that they decide that it's not worth the effort learning yet another framework/library so don't bother contributing.For me personally, I can relate to that feeling of being overwhelmed by the JS ecosystem. As someone who is traditionally a backend developer, the amount of different JS frameworks out there makes learning modern JS feel like a mountainous task, and if Drupal ends up with a bunch of different front-end frameworks, for me, that just makes me less likely to contribute to the front-end. If PB is using Views, that makes it a lot easier for someone like me to contribute, because I already know enough Drupal and Views to contribute. Even if PB didn't use Views, but it used an existing Core JS framework, we already have a pool of people skilled in those frameworks, so speeding up contribution and improvements, making the road to stabilising and including PB a much easier road.Now I'm not saying we shouldn't innovate, or introduce new tools, we absolutely need to be open to doing that, but with the amount of different tools and libraries that Drupal already has, there must be a strong case for introducing a new one, and so far I don't see a compelling reason that the PB can't just be built in Views and reuse existing components. The effort involved now to get a few enhancements into Views would be very worthwhile, and the overall effort required to build PB in Views and maintain it long-term, would be significantly reduced. Additionally, the entire Drupal community benifits when we improve and enhance Views. As illustrated above, the health of the community is in part dependent on Drupal being maintainable, and not having a code-base that diverges along too many different technologies. If Drupal diverges too much then the skills in our community are fragmented and contribution slows. Look at the core Aggregator module, it has a dependency on a PHP library that is only used by the Aggregator module, and now that's part of the reason it's being removed from Core, because that library has become a blocker to us upgrading Drupal. Now I'm not saying that would happen to PB, Aggregator has other problems these days, but if PB is doing it's own thing, there is a risk that the same could very well happen one day if it's off doing it's own thing.From the perspective of a site builder, as I mentioned earlier, if they want to change the way that PB is displayed or is rendering something, or they want to use that data in a different way. Having all of that in Views, significantly reduces the effort involved in doing they, they just have to modify a View in the UI or create a new one. Maybe they want to change which categories are displayed, or limit which extensions are available. That could be done easily in Views. If the PB interface is built entirely using a JS framework, we'd be asking potentially non-technical site builders to learn a JavaScript framework and how to compile that framework before they can make any changes to the PB interface. At that point we'll inevitably end up with a lot more support requests and feature requests being generated, and I guarantee some of those will be "how do I do this in Views?".
Ron Northcutt	We do have an open (and stale) issue to look at using plugins to manage the backend piece. This means that different organization (or people) can create their own integration that uses the same interface to surface different sets of projects (or even non-Drupal ones): #3249984: Plugins discussion - user stories
Ron Northcutt	Using views doesn’t solve this problem, it actually makes it worse. Because now you have to create a whole new views plugin (and lots of custom code) to do what you want. Views would let you customize the PB interface to a degree - but not much. Keep in mind that we are not dealing with Drupal entities here, but external data from an API. So, unless we have something like views_remote_data, there is no real advantage to using views in the long run except that it is already in core.
Ron Northcutt	@nod_ That issue is very helpful, but while a few things (like translations) would be easier, it would simply be moving the challenge to a different part of the stack, and would be creating blockers in different areas. The fact that no one has changed the original implementation since @grasmash created it is actually a good thing. It proves that once the UI component is in a good spot, it shouldn’t need to be updated all that often. once.js hasn’t had a meaningful change in many months, and probably not much before then as well. Is that a problem? I don’t think so - its an indication that it is stable and there isn’t a need to change things.
Ron Northcutt	To be clear, I’m not attached to using svelte here. I personally think it makes a ton of sense and should be easier to manage than vanilla JS in the long run. But, the basic architectural pattern of having a UI component that uses a defined API call/route as the interface (which is where the plugin would provide flexibility) is a solid pattern.
Ron Northcutt	If we decide to use views, then we need to ensure we have a reliable and reusable way of easily integrating remote data and using views to query an API. Thats an ENORMOUS benefit, but also a large task. Otherwise, using views with a custom/clunky plugin is just creating a different type of “singleton” in core.
Ron Northcutt	If the goal here is to use this as an opportunity to extend the utility of views and open the door for other types of integrations, then I think it makes sense to pivot. However, creating a single, one off and very specific views plugin in core that only serves this one purpose feels like a lot more technical debt than supporting a JS component UI that can be reused to browse various inputs. Whether that JS component is handcoded JS or generated JS is a relatively minor detail in the grand scheme.
Aaron McHale	For plugins, there's probably a lot of overlap there with auto updates and the underlying package manager that's being developed, so definitely worth being aligned in that respect.
Aaron McHale	I think the difference between creating a Views plugin and building the whole thing on a JS framework is that a Views plugin is within an existing ecosystem, there's no other dependencies or third-party requirements there, we use existing components, yes there's a element of future maintenance, but that level off commitment is minimal in comparison to introducing a whole new JS framework, integrating that and keeping up to date with it
Ron Northcutt	JS Framework is a bit of a red herring. This could be rewritten in vanilla JS instead of svelte if the preprocessing framework was a blocker.
Gábor Hojtsy (he/him)	@Ron Northcutt currently d.o does not have an API we can query to the extent we need that I am aware, so PB replicates the needed data locally and thus can use whatever is available locally (views is best that Drupal has for this, people can create custom “app stores”, “suggested apps” blocks, etc with Views/Layout builder). We’ll have a meeting later this week to discuss the “remote API” bit if that is at all a possibility.
Ron Northcutt	@Gábor Hojtsy (he/him) do you know how much data the site would have to replicate? Its a massive amount.
Gábor Hojtsy (he/him)	@Ron Northcutt currently 9MB https://git.drupalcode.org/project/project_browser/-/blob/1.0.x/fixtures...
Ron Northcutt	I went ahead and updated the issue, but my suggested TLDR is here:We use a JS UI for the frontend and a Drupal plugin based backend that does the query of the API and transforms the results appropriatelyWe invest in something like views_remote_data so that Views can properly be used to query and display API data
Ron Northcutt	@Gábor Hojtsy (he/him) and what about real time updates? People will have to wait months or longer to get updated info on new modules, or latest versions, or stability, etc. That data would need to be refreshed or updated on a regular basis inside the Drupal site. Thats overhead
mglaman	This is a lot to read and take in, and I’m a being pulled in quite a few directions, right now. But I will say that views_remote_data is something than I plan on having 1.0.0 tagged and fully featured by end of Q1 and is a dependency on work I’m doing + something I want added to external_entities module for Views support. So if my job is to hammer that down a bit more, I’m available.I think adding more JS to core is a maintenance burden and won’t attract core devs. I’d lean on progressive enhancement and leverage web components over a specific framework
Gábor Hojtsy (he/him)	@Ron Northcutt who said people need to wait months?
chrisfromredfin	^ if we keep the fixture approach then people will need to continually update project browser (and PB will need to be continually updated) just to get latest data. (unless there's a mechanism to be updating that data, i.e. on cron or something)
Ron Northcutt	which means being tied to core updates in order to get the data.
Gábor Hojtsy (he/him)	So you assumed that :)
Aaron McHale	To clarify, this "big dump of data"/"fixture" that people need to get, I'm presuming this is a stop-gap until the necesary APIs are ready that would allow a site to directly pull on-demand from D.o?
Gábor Hojtsy (he/him)	As I wrote we have a meeting later this week to discuss exactly this.
Aaron McHale	Ah okay, great thanks @Gábor Hojtsy (he/him)

Hello one and all and welcome to the project browser meeting! I have a few top-level things to discuss. Some will be pointed to the above discussions! Let us know you're here.

shaal	:blob_wave:I'll assign this as the thread where we say "hi"
nod_	hi
lauriii	hey :wave:
AmyJune (volkswagenchick)	Good day…Lurking for some discovery issues for FLDC or the discover Drupal students.
Rajab Natshah	Hi
Aaron McHale	I guess I'm here now after the long comment I just posted :D
leslieg	Missed the meeting, but checking things out now
tim	hi

2️⃣ The current project browser iteration is wholly unusable. The work done thus far tries to pre-populate the data in a fixture when the module is installed, but it either doesn't work or doesn't work well enough, and so when we initially load the project browser it's trying to load EVERY PROJECT into the local cache from the API. This is also a MAJOR problem in advancing the prototype because people cannot put their hands on it. I need someone to volunteer to take on #3258248: onMount makes unpaginated API request, making PB unusable THIS WEEK. How can I bribe you? What will it take?

Gábor Hojtsy (he/him)	My understanding was that @nod_ is looking into re-prototyping with Views Ajax which might make this moot?
bnjmnm	It would still require an import
nod_	yep, import still needs to be fixed
bnjmnm	The importer is my fault - I'm not sure what my available time looks like this week but I can at least provide suggestions on how to address it. The easiest temp fix is to create a new fixture
bnjmnm	The longer term fix is to include the initial import between the fixture and presetnt-date in hook_install
Gábor Hojtsy (he/him)	Assuming this is a backend problem I can look into it later this week.
Gábor Hojtsy (he/him)	There was no guidance/indication on the issue where in the code is this.
bnjmnm	I'll mention it in the issue
nod_	i guess we could ship the module with sample data even in core, some sites have no internet access that would avoid some broken UIs
bnjmnm	Unfortunately this import is a necessary evil if we want certain filtering/sorting capabilities because D.O. does not offer them
Gábor Hojtsy (he/him)	I think we would work with a smaller data set and figure this out in the meantime while it would unblock other work :slightly_smiling_face:
drumm	No, that's not how to work with Drupal.org at all. We need to know what APIs are needed, then build them. Loading all data locally would be bad for both Drupal.org infrastructure and bad for people using it.
chrisfromredfin	Yeah, the biggest issue is no one can spin up a prototype on simplytest (fixture size issue, they mentioned they could fix it) or drupalpod (moreso just not usable once it spins up) so if there is a "quick fix" i would take it just so that site builders have an anchor point to reference.
chrisfromredfin	To be clear, the "load all data locally" issue is temporary. The real blocker here is that we are trying to build the D9 migrated site to build and define the APIs.
chrisfromredfin	but we're trying to currently build against the Drupal 8 API to have something to show
chrisfromredfin	@drumm
chrisfromredfin	so we're using this as a temporary workaround
drumm	It just amazes me that all data locally even made it into a prototype
chrisfromredfin	I gave some folks from Acquia commit access ¯\_(ツ)_/¯
chrisfromredfin	they sprinted, and that's what landed
bnjmnm	I'm right here
chrisfromredfin	I know. I'm saying I don't regret it. It was progress, and the most we've ever made since the initial prototype. :slightly_smiling_face:
chrisfromredfin	And I'm grateful :slightly_smiling_face:
bnjmnm	Excellent, that was the goal -- I think it's much easier to spec out what we'd need from D.O. if we have a working prototype.
bnjmnm	Even if there's some insanity under the hood
shaal	@chrisfromredfin sounds like there are 2 separate things/issues.1 - Quick fix for the current version of project_browser, so it doesn't try to load the complete d.o2 - The real fix - d.o api improvements, etc.
bnjmnm	I commented on the issue with my recommendation on how to improve the import experience
chrisfromredfin	@shaal that sounds right. I believe quickfix would apply under 3258248 and the real fix next step is 3249235
bnjmnm	I'm not sure how to get around the huge fixture thing... we could reduce that footprint if there was agreement that certain project types simply don't get added in the prototype
shaal	@bnjmnm about the huge fixture, would it help if there's always a filter active? or request just first page?
shaal	(sorry, I am not so familiar with how project_browser work yet)
bnjmnm	It wouldn't :disappointed: if you want to sort by date, for example, the earliest and most recent already need to be presesnt
bnjmnm	otherwise it will just sort by the modules that happened to be imported already
bnjmnm	We could get rid of sandbox projects -- with the understanding that the prototype isn't gonna have them -- and that would be a huge size reduction
shaal	:heart_hands: sounds like a win to me
bnjmnm	There may be other project types that don't HAVE to be there in order to properly demo PB functionality
bnjmnm	I did the firehose before I realized how much dang water is in the pipes
chrisfromredfin	I am 100% here for getting rid of sandboxes. Drupal's long term goal is to get rid of them entirely anyway.
bnjmnm	Just saw the issue and of course cron would help enormously. That plus culling some of the less necessary projects for proof-of-concept will probably make this way more manageable.
bnjmnm	And while this may not surprise anyone: the fixture creation was taking ~1hr to complete, so anyone updating that monster be aware.
chrisfromredfin	updated again so I could document my support for excluding sandboxes entirely
chrisfromredfin	If I could get internet running at the office again I'd be happy to try and start that process from your documentation :slightly_smiling_face:
chrisfromredfin	right now I'm tethered to 5G for the second day
shaal	@bnjmnm where is that update happening? is that in the project_browser itself? or that's somewhere else?
bnjmnm	@shaal in my comment on the issue there are instructions on how to update the fixture
tim	@bnjmnm First thank you for what you have doneThis isn't ideal but we need to reduce the numbers abit, What if we were to remove those with no security coverage. I ran some numbers and posted in the issue. Maybe we can get it down to 11113 Modules/ projects.It's only for a 'preview', right? When the real thing comes we can do it right in a different API .... so could include those with no security coverage.Ping @chrisfromredfin for your thoughts
chrisfromredfin	Mostly concerned on order of scale. If we exclude sandbox and that helps “a lot” then that's good. However there are some good modules that don't have security coverage that might look conspicuously absent in a demo. But if it gets down to it I think that's an ok plan
tim	AckThe preview will just haveto accept ....
bnjmnm	Another potential size reduction is being more strategic about what release data is stored. We currently store data on every release, but I’m sure only a small subset of those releases are actually relevant to Project Browser
tim	I would like to get my hands on some of that release data
tim	or in spreadsheet. How hard is that?
tim	Field collectionFeedsJob SchedulerConditional FieldsMasqueradeContent AccessContent Construction Kit (CCK)Are some of the top modules that don't have security covered that would be affected
bnjmnm	So here's an example of the release data for paragraphs https://updates.drupal.org/release-history/paragraphs/current
bnjmnm	This is literally EVERY release, betas, alphas, rc. The majority of data in there isn't really needed for PB.
bnjmnm	I forget what release data is needed... but it definitely isn't all that
bnjmnm	Even if it turns out you need access to all releases, there's a ton of metadata in there that isn't used by PB. I added a function that strips all that out for projects https://git.drupalcode.org/issue/project_browser-3258248/-/blob/1.0.x/sr.... That halved fixture size. I bet you could get some major reductions by adding a method to truncate release data.
tim	Yea you only need the relevant stuff. Thats alot of extra. I guess focus there?I was going to suggest to filter out those modules with low Total Usages (installs), but would get more returns get rid of storing / cache all that extra
tim	I can get you numbers for total Useage, but won't be as big as that release info probably

3️⃣ Same thing it's been every week. The last blocker to getting really useful data is that we need migrations to happen of release nodes into the Drupal 9 backend API. That's blocking us having useful data. @mandclu has pointed out some issues with the data - for example the Top 100 list I've provided from the current logic still shows projects that, for example, don't have a stable release for Drupal 9 only a -dev release. Those would normally be excluded but what's happening is that we're filtering by "has a stable release" and THEN filtering by "works with Drupal 9" - a robust Drupal 9 API where we can make the right requests here... goes a long way.

mandclu	I've volunteered to help add some manual review to the process the next time we refresh the list, until we have an endpoint that can provide more granular data.I doubt it needs a ton of work, but I think the quality of the recommendations will be a really important piece for the overall experience of using PB.
Rajab Natshah	I suggest that releases are fetched from the Gitlab API directlyOnly the latest release tag name is neededor the Gitlab API will feed the basic latest release in the Drupal 9 site
chrisfromredfin	release nodes are stored in Drupal, so would migrate to d.o afaik. And no, we kind of need all releases, because we need to infer "is there a stable release that's compatible with Drupal 9.3.4" ex.g.
chrisfromredfin	Also not sure if Gitlab's API will need authentication / tokens to access that data.. ?
Rajab Natshah	I mean to migrate releases from the current Gitlab API.and for sure the Drupal 9 site as the cache for that ..with a custom endpoint .. like /api/v9/releases/latest/[2, 332 ,23 , 33, 22]
Rajab Natshah	@chrisfromredfin I think you are right, It is better to migrate D7 to D9
dww	Is there an issue for this somewhere? I know something about d.o release nodes. ;) And migrations. I have some community time I can spend. If this is a big blocker for this initiative and no one else is going to get it done, I'll see what I can do. But I need some direction on where to start so I don't waste effort.
chrisfromredfin	@dww we'll be discussing our approach to this at the synchronous Wednesday meeting (see main channel) - hope you can make it then! My current thinking is that we write a migration for release data into the Drupal.org 9 site. We have already made effort porting module_project nodes to such a site. Then we can index data about releases on the projects in Solr, and use that site as the endpoint for project browser. That said, I suppose we could decide on a different approach on Wednesday, so we'll see what happens then. :slightly_smiling_face:
dww	Definitely can’t make that time. My TZ is hard for working with Europeans or even EST. But I’m happy to spearhead this migration if that’s what we end up needing
chrisfromredfin	Cool! I will keep that in mind at the meeting. :slightly_smiling_face: Appreciate your help!

I guess I have on on the data we expect from drupal.org. Currently we store taxonomy ids of various things. I think the drupal.org api should provide the data with no internal ids, so all taxonomy terms changed to their actual values. It would avoid exposing internal data structure to the outside

drumm	The quick plan has always been to “just use D9 core’s JSON:API” but I think there may be places where that falls short. This for example. And we need a full text search back by Solr or similar.
Gábor Hojtsy (he/him)	The second quickest plan could be to have a drupalprojectbrowser.org :stuck_out_tongue:
Gábor Hojtsy (he/him)	which redirects to whatever official place emerges later on, assuming that could provide BC for this interim service
chrisfromredfin	It's already here - http://drupalorgd9dev.prod.acquia-sites.com/ :wink: just incomplete. I think JSON:API is turned on, and I've set up a Solr server and connected it thus far... but then we were missing release data in order to generate a "compatibility" type field.
drumm	I meant quick as in I haven’t seen a lot of time spent finishing writing/refining API requirements/design. There’s an issue with a lot of ideas somewhere.
nod_	bonus question, should the project browser UI be the same in PB and in https://www.drupal.org/project/project_module or are they always going to be their own things?
chrisfromredfin	For now we're considering only inside of PB
chrisfromredfin	If it ends up being a great success, I could see us informing the d.o side later on
drumm	I had wanted to consider serving more-close to HTML from Drupal.org, but that wasn’t ever really taken up. That way we could introduce new features without being coupled to the core release cycle.
Gábor Hojtsy (he/him)	That would require some kind of post-processing approach on the Drupal site, eg. to mark projects already on the site and somehow inform the d.o side on what you have to suggest extensions based on that, etc.
Gábor Hojtsy (he/him)	I think it was not taken up because it raises a lot of those scary questions :smile:
drumm	Yep, that was touched on in a couple of the early calls. If we do personalization, either Drupal.org will need more data about your site, or your site will need at least an order of magnitude more data & compute. I think noting what’s already installed is a good call. Personalization, I’d keep that in mind for future expansion, we don’t want to be boxed in by API design, but I’d not consider it for initial functionality.
drumm	Thinking about it a bit more, I don’t see how client-side personalization could be practical. Even basic full-text search can’t really be done well with SQL as far as I know.
drumm	Back to the original question - we may want to consider migrating the categories into text fields with allowed values for the Drupal.org migration. We don’t really use the features of taxonomy. Would make renames harder, but we can work with that. Regardless, the project browser module can’t have hard-coded tables of ID → label; we can’t wait for a core release cycle to do things like add/change categories.
chrisfromredfin	We'd still be fetching the vocabulary over the API
chrisfromredfin	So we would be displaying near-real-time labels even if it were keyed by ID
drumm	If we store the category as a text field, that’d simplify everything, (at the expense of making future category renames into a mini-migration)
chrisfromredfin	I'm not opposed to that either.
chrisfromredfin	And certainly defer to you as the owner of that stuff.
drumm	Right, and details like the taxonomy IDs being non-ideal are the sort of API design input will help figure out what to do in the migration/etc.
Björn Brala (bbrala)	There is https://www.drupal.org/project/jsonapi_search_api which expose search api as jsonapi.
Björn Brala (bbrala)	Regarding internal it's. There are some relatively new issues in the queue to allow ooting out of internal metadata in jsonapi. Which does make sense in some cases
chrisfromredfin	The intention I have in my head thus far is in fact to leverage this module ^

:stop: New Threads Donezo. Thanks for showing up today, all! Great discussions around important topics. I'm always available outside of these meetings, please ping me here in #project-browser - I am on a limb saying same goes for @leslieg - though she's currently out. THANK YOU THANK YOU!

leslieg

Thanks everyone for your input today. And yes, feel free to ping me in this channel as well with any thoughts or questions.

Participants:

chrisfromredfin, Gábor Hojtsy, shaal, Aaron McHale, xjm, irinaz, Ron Northcutt, nod_, mglaman, lauriii, AmyJune (volkswagenchick), Rajab Natshah, leslieg, tim, bnjmnm, drumm, mandclu, dww, Björn Brala (bbrala)