Related initiatives
- #2958021: Proposal: Composer Support in Core initiative (Precursor for auto updates and site builder initiatives)
- #2940731: Automatic Updates Initiative overview and roadmap
- Provide a Site-Builder tool/Project browser ← this issue
- #2940737: Telemetry initiative
- #2940739: Project messaging channel in core initiative
Building a Drupal site has grown increasingly complex with the need to manage external dependencies using tools like Composer or NPM. Site-builders without command line access on their hosting providers are increasingly unable to take full advantage of Drupal
The Site-Builder Tool/Project Browser initiative
The goal is to implement a site builder tool, including a project browser, which abstracts away the manual process of installation, updates, and dependency resolution - handling those items behind the scenes for the site builder. The end product might be a code-base installed-in-place on the server, or it might be a consolidated tar.gz/zip with the complete codebase to be uploaded to a host.
This initiative also has the potential to support the sustainability of the Drupal Association and our community programs, perhaps through the promotion of featured projects (modules, distros, themes) or even hosting providers that the output of the tool could be deployed to.
These changes can help Drupal retain its lower to mid-market users, lower the barrier to entry, and lower the cost of ownership for a Drupal site.
---
Usecases we are targeting to improve:
- The experience of a site-builder or part-time maintainer who wants to seek out and try new projects on their Drupal installation.
- The ability for a site-builder without command line access to build Drupal projects that have external dependencies (through Composer, for example).
Usecases we are not targeting to improve:
- Automatic updates, though this is a related initiative
Specifications from comments:
- #11: Allow for tests during build process, creating a continuous delivery type system.
---
Sign-offs given:
@Dries has endorsed this idea in a call with DA staff, core maintainers, and community volunteers. The notion of a project browser has been endorsed by many in the community in the past, including @webchick.
Here is the slide deck from the check-in between Core, the DA, and some interested community members about these initiatives.
---
How we are working
Regular discussion is happening in the slack channel https://app.slack.com/client/T06GX3JTS/C01UHB4QG12.
Weekly meetings over zoom are also happening on Wednesdays at 2pm UTC (see https://www.google.com/search?q=utc+time+to+my+time for help with UTC time). Links to this meeting are posted in slack (sorry I don't have that link to post it here).
Join us :)
---
Current Status
Sitebuilders, designers, developers are collaborating (with not too much friction) via slack - join us! Read through the chat history, and checkout this MVP document. Yes....
We're defining the MVP, check it out https://docs.google.com/document/d/1oMI9CZMOheiie1-8Xej-WdQLMeG4cVWmxaOT...
And there's also a proof of concept module that's an experimental demonstration of what can be done with the existing drupal.org API, see that module here dgo.to/project_browser.
The API for serving project information (for the browser to consume/utilize) will need some improvements/enhancements. These are being documented at https://www.drupal.org/project/infrastructure/issues/3218285.
Comment | File | Size | Author |
---|---|---|---|
#58 | Screenshot 2021-04-24 at 21.08.34.png | 1.12 MB | ipwa |
#43 | Screen Shot 2021-04-14 at 8.30.03 PM.png | 377.51 KB | irinaz |
#36 | Screen Shot 2021-04-14 at 4.49.28 PM.png | 143.96 KB | mglaman |
#36 | Screen Shot 2021-04-14 at 4.49.09 PM.png | 191.29 KB | mglaman |
Issue fork ideas-2940733
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
hestenetComment #3
mherchelThis is a freaking fantastic initiative.
First of all, if I had to install, learn, and fight with Composer when I was first learning Drupal 5.3, I can tell you that I would not have learned Drupal. I think this has a huge benefit of helping site-builders (who IMO are Drupal's core constituency) with modern D8 development.
As far as the project browser, I think that's loooooooong overdue, and something that our competitors do very well. It's a solved problem that we just need to implement. It goes without saying that we need to make this functionality disable-able for our enterprise uses.
Woot!
Comment #4
geek-merlinThis may or may not help:
Embedded Composer (sflive portland 2013) // Speaker Deck
Comment #5
ipwa CreditAttribution: ipwa commentedSo happy to see this issue. Its hard enough to teach Drupal, but having to teach composer at the same time just to install certain Drupal modules is really crazy. Students just kind of give up.
Comment #6
droprocker CreditAttribution: droprocker commentedIt's a great idea and highly needed. I, as a sitebuilder, had a hard time to understand how Drupal 8 works. Almost everything was completely new to me. eg. CLI, Composer, Twig, ... not to mention the thousands of files I suddenly had to transfer via FTP (which was the way how I did it in Drupal 7). My complete, self-teached, Drupal 7 knowledge seemed to be worthless from one day to another.
I think a lot of sitebuilders out there have/had the same problems und probably just changed the platform to a more sitebuilder-friendly one.
We need to improve the sitebuilder experience in Drupal 8 and bring it back to them and not only support the developers.
Comment #7
ChandeepKhosa CreditAttribution: ChandeepKhosa at 2Toucans commentedI think this is great and I'm looking forward to seeing how this has progressed since speaking to Ryan Aslett & Greg Anderson about this at DrupalCon Vienna in this video interview I did with them "Improving the UX of updating Drupal sites with Composer"
Comment #8
Mile23Some useful chunks:
Define build needs for core + contrib. We currently build the tarball on d.o by running
composer install --no-dev
. What else would need to happen for a full build of a site codebase composed of extensions available on d.o?Define standard patterns for build-out of d.o projects. For example, maybe you've selected the Bootstrap theme, in which case part of building a working codebase would be to download the Bootstrap CSS. Extensions should have a standard scriptable point of entry where this can be performed automatically. We'd need to define what that point of entry is. We might expect that there is a drupal-builder.yml config, or maybe expect that there is a package.json script named 'drupal-builder.' Something like that.
These are some basic needs. Stretch goals could include emulating further benefits that are provided by drupal-composer/drupal-project:
Define build options, such as whether you want index.php to reside in the root or not.
Allow for patches.
And, of course, this could be a command-line tool available to those who want to use it that way.
Comment #9
cilefen CreditAttribution: cilefen as a volunteer commentedWhat you wrote is reminiscent of Symfony Flex recipes.
Comment #10
Mile23Well, maybe that's how it would be implemented, but the point is that if d.o is supposed to generate a codebase for us, we have to have a way to do things like add external CSS to the Bootstrap theme. There's currently no expressed standard way to do that. There's not even a single point of entry for doing these tasks for core even though it needs to do them.
So we'd need for core and contrib to ensure that their dependencies are buildable. A contrib project might require the flex plugin and then specify some build tasks... Would that conflict with the requirements of other contrib in the same build? That's the kind of question we'd need to answer.
Comment #11
ahillio CreditAttribution: ahillio commentedThis sounds like a great idea. I love seeing Drupal move towards empowering site builders more :)
How about having a development/testing site that the production site is aware of a part of this?
We always say updates should be tested before deploying them to production. It's not ideal to give someone an easier way of doing what we say should not be done (like applying updates directly to live/production site), so it seems like these tools should enable and encourage following that best practice.
Comment #12
Mile23Comment #13
Mile23Is anyone working on this?
Comment #14
hestenet@Mile23 - my thinking was that making Core's use of Composer sane would be an appropriate prerequisite for this (and other related issues like automatic updates) - does that seem right to you, or do you feel this can move in parallel?
Comment #15
Mile23@hestenet Which issues are those?
Comment #16
hestenetI was thinking 'vendorize core' and any of the related issues there. I think we(I) need to pull together some of the issues that have been open in the queue along with notes from email chains and DrupalCon sessions/conversations.
Comment #17
David_Rothstein CreditAttribution: David_Rothstein commentedThere is a lot of work on something similar (including code that was once functional) at #1841788: Add project browser. It's not necessarily the exact same thing being proposed here, but it's at least worth linking to it.
Comment #18
hestenetComment #19
ressa CreditAttribution: ressa at Ardea commentedThe recent release of Composer 2 saw big performance improvements, and many web hosts now offer managed VPS hosting solutions with 2GB memory from around €30 per month.
So perhaps this project is more realistic now?
Composer improvements
Composer 1 (
composer self-update --1
)Memory usage: 379.04MiB (peak: 737.02MiB), time: 7.21s
Composer 2
Memory usage: 17.06MiB (peak: 109.35MiB), time: 1.44s
Comment #20
mandclu CreditAttribution: mandclu at Northern Commerce commentedInterested in helping
Comment #21
frobVery interested. We have some site builders over here that can help with testing as well.
Comment #22
ThoreauinSF CreditAttribution: ThoreauinSF commentedThis is an excellent idea. how would this affect shops that use version control - if you used project browser to add to a site in the site - then you'd export configurations to make your local match? it just makes me think.
Comment #23
bbytyqi CreditAttribution: bbytyqi commentedI'm interested in helping too!
Comment #24
mglaman👋 I'd be interested in helping out.
Comment #25
mcgovernm CreditAttribution: mcgovernm as a volunteer commentedI'm interested in helping with this!
Comment #26
lhockleyI am 110% interested in this!
Has anyone created a slack channel, or should I start one?
Comment #27
hedrickbt CreditAttribution: hedrickbt as a volunteer commentedI am interested in helping.
Comment #28
lhockleyI went ahead and created a slack channel, in case we don't have one already. #project-browser
Feel free to join for real-time, threaded discussion on this initiative.
Comment #29
irinaz CreditAttribution: irinaz as a volunteer and at Fibonacci Web Studio commentedInterested in helping
Comment #30
cellearCame here because Dries talked about it in the DriesNote at DC2021. I'd like to be involved! I have experience advocating (successfully!) for the addition of a Project Browser in BackdropCMS, so I'm a bit of a veteran.
Comment #31
phenaproximaI'd love to help with this!
Comment #32
makurj CreditAttribution: makurj commentedComment #33
makurj CreditAttribution: makurj commentedDriesnote DrupalCon North America 2021 request for serving on this initiative team:
I'd like to help out.
Comment #34
webchickDefinitely interested. :) Here's a "stream of consciousness" I wrote up as I was attempting to go through this experience as a new user and it is... not pretty. ;) https://docs.google.com/document/d/1-X0aOG-79f3V-uzQhgfU6jHjONMfaGa9wr3x... (I'll try and turn this into a blog post at some point...)
Comment #35
crasx CreditAttribution: crasx as a volunteer and at Bounteous commentedThanks for putting that together @webchick!
I plan on putting more thought into this later, but based on the document provided I wonder if the better solution might be to focus on improving d.o documentation and UX.
I'm not a huge fan of having a "marketplace" for modules (within a given drupal installation). I think arbitrarily downloading zips to a server for execution will end up bad regardless of the rails we put in place. Even if we go with the composer route we need to deal with dependency conflicts and potentially patching which can be frustrating even with a ton of composer knowledge. I also don't think it is unreasonable to require site builders to have some technical knowledge to install a new module.
That being said there is a difference between that required technical knowledge and the knowledge currently required to effortlessly build drupal sites. Might an easier solution be to give drupal.org a UX facelift? An easy example is the modules page - the zip downloads are no longer relevant in drupal 8 (except in edge cases). If we replaced the download area with more clear information about available versions, their differences, and how to install them, I feel like that would be more valuable.
Comment #36
mglamanper @crasx on facelift, I wanted to quick link to what remains of the module/extension marketplace old CG build for Drupal Commerce
https://drupalcommerce.org/extensions
https://drupalcommerce.org/extensions/module
Comment #37
amykhailova CreditAttribution: amykhailova at Kalamuna commentedI would like to help!
Comment #38
lhockleyA few initial thoughts I have about this initiative, from video calls this afternoon and and from experience with similar tools in WordPress.
On another note, there is already some discussion happening over on the slack channel, #project-browser. Now is a great time to get in there and share your thoughts, brainstorming, experiences, etc. Let's chat!
Comment #39
bsnodgrass CreditAttribution: bsnodgrass at net2Community, Inc. commentedThere have been a number of related discussions and ideas as we have been talking about Drupal Recipes and a number of folks who would be interested in helping with this. Count me in!
Comment #40
makurj CreditAttribution: makurj commentedI would love to help.
Comment #41
andrewozoneI'd love to help with this initiative!
Comment #42
rubyjiAs a site builder since 2005 (D4.7) I'm thrilled to learn about this initiative. I'm even giving a talk at DrupalCon this year about what developers need to better understand about site building!
Not sure yet how much I can responsibly commit to, but I would love to help.
Comment #43
irinaz CreditAttribution: irinaz as a volunteer and at Fibonacci Web Studio commentedIt is great to see that name of initiative now includes "site building" :) Here is example of what UX for adding module site builders love
Comment #44
kevinquillen CreditAttribution: kevinquillen at Velir commentedWanted to add a quick note that it would be great if a module could add more in its info.yml file to help users out by linking to critical issues, or other general things to be aware of from the browser or on the Extend page.
I tried to do something like this years ago building on an module Dave Reid created:
https://www.drupal.org/project/module_supports
A module can suggest complimentary modules, or link to known conflicts, etc.
Example:
Comment #45
Simon-PWould love to be involved somehow. Not sure exactly sure how/when as yet, but started out as a Site Builder. Would find it easy to (pretend) to be the person who doesn't know anything about how Drupal works, which is who I think this initiative is geared towards helping.
Comment #46
zsofi.major CreditAttribution: zsofi.major at Cheppers commentedI'd love to help with this, I love the idea :)
Comment #47
AjitSGreat idea! Would love be involved.
Comment #48
sashainparisSounds cool :-)
Comment #49
saschaeggiI can help with Design
Comment #50
troutdale CreditAttribution: troutdale as a volunteer commentedI am excited by this initiative and would like very much to participate. I am THAT site builder persona that Dries described in his DriesNote. I'm not sure where or how I can help because I've never contributed before, but I can't think of a better place to start than with this project.
Comment #51
mahmoud-zayed CreditAttribution: mahmoud-zayed as a volunteer and at ImageX for ImageX commentedGreat idea! I'd love to contribute to it :)
Comment #52
Manjit.SinghIt would be my pleasure to help out in this in any possible way.
Comment #53
shaalI'd love to help with this!
Comment #54
codemannI'd love to contribute as well, I've been watching this issue for a while now.
Comment #55
droprocker CreditAttribution: droprocker commentedI'd like to join as well!
Comment #56
Mile23Wow, 36 new comments out of 55 since last time I looked at my dashboard. :-)
Comment #57
dreambubbler CreditAttribution: dreambubbler as a volunteer commentedI am keen to help this initiative.
Comment #58
ipwa CreditAttribution: ipwa commentedAdding a screenshot of the design Dries showed in his keynote when he talked about this initiative.
@saschaeggi I would love to see your take on this following your work from the Future UI.
Comment #59
rakesh.gectcrI love to help with this. Looks like a great Idea!
Comment #60
zviryatko CreditAttribution: zviryatko as a volunteer and commentedHello, I'd like to help.
Comment #61
Kir Lazur CreditAttribution: Kir Lazur at EPAM Systems commentedI'd like to participate in this initiative.
We even did the PoC of this idea together with team https://www.drupal.org/project/drupal_marketplace
Comment #62
ahillio CreditAttribution: ahillio commentedFrom #44:
that sounds absolutely essential. Modules need to be classified, for example... field related modules which fall into two categories of field type and field display.
For a use-case, consider a sitebuilder with a gallery node type that uses "flexslider" module for image galleries. They should be able to go into this tool and look for a way to change their existing gallery which would mean looking for a module that provides a different image display formatter... which would introduce them to the "slick" module and numerous others.
Comment #63
kevinquillen CreditAttribution: kevinquillen at Velir commentedYes, that is where that concept was going. The module also let you link to project pages (drupal.org filtered only), specific issues, and gave a place to let someone know right there (and not buried in readme or help files) what other modules to pair, or which ones to avoid (feature overlap or breakage).
Comment #64
AaronMcHale@ahillio @kevinquillen Re #62 and #63:
I agree, I think these would be good pieces of metadata to provide. Quite a lot of that metadata mentioned in #63 can already be provided in the composer.json, as well as extra metadata. As we are moving in that direction it would make sense to further explore how composer.json can be used, it might even replace info.yml for most cases in the future. See #1398772: Replace .info.yml with composer.json for extensions, this initiative might be a good opportunity to kickstart that issue. Of course this is all further down the road as the initial goal appears to be just to get a proof of concept first.
Comment #65
varunagarwal2004 CreditAttribution: varunagarwal2004 as a volunteer commentedI would love to help with this initiative.
Comment #66
Gábor HojtsyDries posted https://dri.es/join-the-first-drupal-project-browser-initiative-meeting
Comment #67
nonom CreditAttribution: nonom as a volunteer commentedExciting Innitiative. I would be around.
Comment #68
saki007sterLooks like a great Idea!
I love to help with this.
Comment #69
thejimbirch CreditAttribution: thejimbirch at Kanopi Studios commentedJumping in!
Comment #70
AjitSComment #71
GrienauerHi! I now have collected and saved some Web views from multiple CMS systems and their Module/Plugin browsers. So we have some inspirations, what others are doing. I added some info in the filename. CMS Name. Start: Frontpage of the project browser, Section: is if there is filtering for a section active. Details: info about a single “module”. I think this could help to get some more perspective. https://drive.google.com/drive/folders/1SRHHGCNKyj6sggOBjbmaVZF9vH0_FZOy maybe this helps for some decisions.
Comment #72
jungleIn addition to #71, maybe we can learn from NextCloud's apps management/App store, which is an open-source PHP project as well https://apps.nextcloud.com/
Comment #73
webchickAMAZING work, @Grienauer!! Thank you SO much for all of your work compiling this!! :O
Comment #74
webchickActually, that's a good point. I wonder if there's a way to make this open to others to add screenshots from other products..?
Comment #75
fhaeberleI would love to help with this idea, can provide design & dev.
Comment #76
Grienauer@webchick it was not a big thing. if there is a need for even more screenshots from other systems, I can jump on that too. I am just curious how other systems do it so we can combine the best essence from all into ours ;P
also said, please be aware, that this is only the web view, similar to our drupal.org/project/project_module and not from inside of the cms system.
I will think about it, how we could open this so other people could add screenshots too… google drive was just an easy/quick solution…
but if someone want to add stuff, just ask me for permission for now and I will add that person!
Comment #77
phpsubbarao CreditAttribution: phpsubbarao commentedI am interested to be part of Project Browser Initiative
Comment #78
ahillio CreditAttribution: ahillio commented@phpsubbarao and anyone else coming along... I just updated the description with details, here are the links I added for quickreference:
https://app.slack.com/client/T06GX3JTS/C01UHB4QG12
https://docs.google.com/document/d/1oMI9CZMOheiie1-8Xej-WdQLMeG4cVWmxaOT...
https://www.drupal.org/project/infrastructure/issues/3218285
https://dgo.to/project_browser
Comment #79
ahillio CreditAttribution: ahillio commentedProject Browser Initiative meeting for July 14, 2021: https://www.drupal.org/project/drupal/issues/3224533
Comment #80
effulgentsia CreditAttribution: effulgentsia at Acquia commentedI opened #3228567: Decide on consistent naming for future core modules that implement Automatic Updates and Project Browser functionality to discuss naming and would love some feedback on it.
Comment #81
nod_probably already an "approved plan" but not getting ahead of the workflow here.