Related initiatives

Building a Drupal site has grown increasingly complex with the need to manage external dependencies using tools like Composer or NPM. Site-builders without command line access on their hosting providers are increasingly unable to take full advantage of Drupal

The Site-Builder Tool/Project Browser initiative

The goal is to implement a site builder tool, including a project browser, which abstracts away the manual process of installation, updates, and dependency resolution - handling those items behind the scenes for the site builder. The end product might be a code-base installed-in-place on the server, or it might be a consolidated tar.gz/zip with the complete codebase to be uploaded to a host.

This initiative also has the potential to support the sustainability of the Drupal Association and our community programs, perhaps through the promotion of featured projects (modules, distros, themes) or even hosting providers that the output of the tool could be deployed to.

These changes can help Drupal retain its lower to mid-market users, lower the barrier to entry, and lower the cost of ownership for a Drupal site.

---

Usecases we are targeting to improve:

  • The experience of a site-builder or part-time maintainer who wants to seek out and try new projects on their Drupal installation.
  • The ability for a site-builder without command line access to build Drupal projects that have external dependencies (through Composer, for example).

Usecases we are not targeting to improve:

Specifications from comments:

  • #11: Allow for tests during build process, creating a continuous delivery type system.

---

Sign-offs given:

@Dries has endorsed this idea in a call with DA staff, core maintainers, and community volunteers. The notion of a project browser has been endorsed by many in the community in the past, including @webchick.

Here is the slide deck from the check-in between Core, the DA, and some interested community members about these initiatives.
---

How we are working

Regular discussion is happening in the slack channel https://app.slack.com/client/T06GX3JTS/C01UHB4QG12.

Weekly meetings over zoom are also happening on Wednesdays at 2pm UTC (see https://www.google.com/search?q=utc+time+to+my+time for help with UTC time). Links to this meeting are posted in slack (sorry I don't have that link to post it here).

Join us :)

---

Current Status

Sitebuilders, designers, developers are collaborating (with not too much friction) via slack - join us! Read through the chat history, and checkout this MVP document. Yes....

We're defining the MVP, check it out https://docs.google.com/document/d/1oMI9CZMOheiie1-8Xej-WdQLMeG4cVWmxaOT...

And there's also a proof of concept module that's an experimental demonstration of what can be done with the existing drupal.org API, see that module here dgo.to/project_browser.

The API for serving project information (for the browser to consume/utilize) will need some improvements/enhancements. These are being documented at https://www.drupal.org/project/infrastructure/issues/3218285.

Issue fork ideas-2940733

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

    Support from Acquia helps fund testing for Drupal Acquia logo

    Comments

    hestenet created an issue. See original summary.

    hestenet’s picture

    Issue summary: View changes
    mherchel’s picture

    This is a freaking fantastic initiative.

    First of all, if I had to install, learn, and fight with Composer when I was first learning Drupal 5.3, I can tell you that I would not have learned Drupal. I think this has a huge benefit of helping site-builders (who IMO are Drupal's core constituency) with modern D8 development.

    As far as the project browser, I think that's loooooooong overdue, and something that our competitors do very well. It's a solved problem that we just need to implement. It goes without saying that we need to make this functionality disable-able for our enterprise uses.

    Woot!

    geek-merlin’s picture

    ipwa’s picture

    So happy to see this issue. Its hard enough to teach Drupal, but having to teach composer at the same time just to install certain Drupal modules is really crazy. Students just kind of give up.

    droprocker’s picture

    It's a great idea and highly needed. I, as a sitebuilder, had a hard time to understand how Drupal 8 works. Almost everything was completely new to me. eg. CLI, Composer, Twig, ... not to mention the thousands of files I suddenly had to transfer via FTP (which was the way how I did it in Drupal 7). My complete, self-teached, Drupal 7 knowledge seemed to be worthless from one day to another.
    I think a lot of sitebuilders out there have/had the same problems und probably just changed the platform to a more sitebuilder-friendly one.

    We need to improve the sitebuilder experience in Drupal 8 and bring it back to them and not only support the developers.

    ChandeepKhosa’s picture

    I think this is great and I'm looking forward to seeing how this has progressed since speaking to Ryan Aslett & Greg Anderson about this at DrupalCon Vienna in this video interview I did with them "Improving the UX of updating Drupal sites with Composer"

    Mile23’s picture

    Some useful chunks:

    Define build needs for core + contrib. We currently build the tarball on d.o by running composer install --no-dev. What else would need to happen for a full build of a site codebase composed of extensions available on d.o?

    Define standard patterns for build-out of d.o projects. For example, maybe you've selected the Bootstrap theme, in which case part of building a working codebase would be to download the Bootstrap CSS. Extensions should have a standard scriptable point of entry where this can be performed automatically. We'd need to define what that point of entry is. We might expect that there is a drupal-builder.yml config, or maybe expect that there is a package.json script named 'drupal-builder.' Something like that.

    These are some basic needs. Stretch goals could include emulating further benefits that are provided by drupal-composer/drupal-project:

    Define build options, such as whether you want index.php to reside in the root or not.

    Allow for patches.

    And, of course, this could be a command-line tool available to those who want to use it that way.

    cilefen’s picture

    What you wrote is reminiscent of Symfony Flex recipes.

    Mile23’s picture

    Well, maybe that's how it would be implemented, but the point is that if d.o is supposed to generate a codebase for us, we have to have a way to do things like add external CSS to the Bootstrap theme. There's currently no expressed standard way to do that. There's not even a single point of entry for doing these tasks for core even though it needs to do them.

    So we'd need for core and contrib to ensure that their dependencies are buildable. A contrib project might require the flex plugin and then specify some build tasks... Would that conflict with the requirements of other contrib in the same build? That's the kind of question we'd need to answer.

    ahillio’s picture

    This sounds like a great idea. I love seeing Drupal move towards empowering site builders more :)

    How about having a development/testing site that the production site is aware of a part of this?

    We always say updates should be tested before deploying them to production. It's not ideal to give someone an easier way of doing what we say should not be done (like applying updates directly to live/production site), so it seems like these tools should enable and encourage following that best practice.

    Mile23’s picture

    Issue summary: View changes
    Mile23’s picture

    Is anyone working on this?

    hestenet’s picture

    @Mile23 - my thinking was that making Core's use of Composer sane would be an appropriate prerequisite for this (and other related issues like automatic updates) - does that seem right to you, or do you feel this can move in parallel?

    Mile23’s picture

    @hestenet Which issues are those?

    hestenet’s picture

    I was thinking 'vendorize core' and any of the related issues there. I think we(I) need to pull together some of the issues that have been open in the queue along with notes from email chains and DrupalCon sessions/conversations.

    David_Rothstein’s picture

    There is a lot of work on something similar (including code that was once functional) at #1841788: Add project browser. It's not necessarily the exact same thing being proposed here, but it's at least worth linking to it.

    hestenet’s picture

    Issue summary: View changes
    ressa’s picture

    The recent release of Composer 2 saw big performance improvements, and many web hosts now offer managed VPS hosting solutions with 2GB memory from around €30 per month.

    So perhaps this project is more realistic now?

    Composer improvements

    composer create-project drupal/recommended-project:8.9.1 && cd recommended-project
    COMPOSER_DISABLE_NETWORK=1 composer update --with-all-dependencies -v --dry-run --profile

    Composer 1 (composer self-update --1)
    Memory usage: 379.04MiB (peak: 737.02MiB), time: 7.21s

    Composer 2
    Memory usage: 17.06MiB (peak: 109.35MiB), time: 1.44s

    mandclu’s picture

    Interested in helping

    frob’s picture

    Very interested. We have some site builders over here that can help with testing as well.

    ThoreauinSF’s picture

    This is an excellent idea. how would this affect shops that use version control - if you used project browser to add to a site in the site - then you'd export configurations to make your local match? it just makes me think.

    bbytyqi’s picture

    I'm interested in helping too!

    mglaman’s picture

    👋 I'd be interested in helping out.

    mcgovernm’s picture

    I'm interested in helping with this!

    lhockley’s picture

    I am 110% interested in this!

    Has anyone created a slack channel, or should I start one?

    hedrickbt’s picture

    I am interested in helping.

    lhockley’s picture

    I went ahead and created a slack channel, in case we don't have one already. #project-browser

    Feel free to join for real-time, threaded discussion on this initiative.

    irinaz’s picture

    Interested in helping

    cellear’s picture

    Came here because Dries talked about it in the DriesNote at DC2021. I'd like to be involved! I have experience advocating (successfully!) for the addition of a Project Browser in BackdropCMS, so I'm a bit of a veteran.

    phenaproxima’s picture

    I'd love to help with this!

    makurj’s picture

    makurj’s picture

    Driesnote DrupalCon North America 2021 request for serving on this initiative team:

    I'd like to help out.

    webchick’s picture

    Definitely interested. :) Here's a "stream of consciousness" I wrote up as I was attempting to go through this experience as a new user and it is... not pretty. ;) https://docs.google.com/document/d/1-X0aOG-79f3V-uzQhgfU6jHjONMfaGa9wr3x... (I'll try and turn this into a blog post at some point...)

    crasx’s picture

    Thanks for putting that together @webchick!

    I plan on putting more thought into this later, but based on the document provided I wonder if the better solution might be to focus on improving d.o documentation and UX.

    I'm not a huge fan of having a "marketplace" for modules (within a given drupal installation). I think arbitrarily downloading zips to a server for execution will end up bad regardless of the rails we put in place. Even if we go with the composer route we need to deal with dependency conflicts and potentially patching which can be frustrating even with a ton of composer knowledge. I also don't think it is unreasonable to require site builders to have some technical knowledge to install a new module.

    That being said there is a difference between that required technical knowledge and the knowledge currently required to effortlessly build drupal sites. Might an easier solution be to give drupal.org a UX facelift? An easy example is the modules page - the zip downloads are no longer relevant in drupal 8 (except in edge cases). If we replaced the download area with more clear information about available versions, their differences, and how to install them, I feel like that would be more valuable.

    mglaman’s picture

    per @crasx on facelift, I wanted to quick link to what remains of the module/extension marketplace old CG build for Drupal Commerce

    https://drupalcommerce.org/extensions
    https://drupalcommerce.org/extensions/module

    amykhailova’s picture

    I would like to help!

    lhockley’s picture

    A few initial thoughts I have about this initiative, from video calls this afternoon and and from experience with similar tools in WordPress.

    • Modules in the "app store" should be verified. At minimum, passing the Drupal Security Team verification.
    • A star rating system may be desirable, along with number of installs. These data points would need to be stored somewhere.
    • We ought to not consider this a replacement for existing "manual" installs. Contrib, and custom modules will probably not be using this browser.

    On another note, there is already some discussion happening over on the slack channel, #project-browser. Now is a great time to get in there and share your thoughts, brainstorming, experiences, etc. Let's chat!

    bsnodgrass’s picture

    There have been a number of related discussions and ideas as we have been talking about Drupal Recipes and a number of folks who would be interested in helping with this. Count me in!

    makurj’s picture

    I would love to help.

    andrewozone’s picture

    I'd love to help with this initiative!

    rubyji’s picture

    As a site builder since 2005 (D4.7) I'm thrilled to learn about this initiative. I'm even giving a talk at DrupalCon this year about what developers need to better understand about site building!

    Not sure yet how much I can responsibly commit to, but I would love to help.

    irinaz’s picture

    It is great to see that name of initiative now includes "site building" :) Here is example of what UX for adding module site builders love

    screenshot

    kevinquillen’s picture

    Wanted to add a quick note that it would be great if a module could add more in its info.yml file to help users out by linking to critical issues, or other general things to be aware of from the browser or on the Extend page.

    I tried to do something like this years ago building on an module Dave Reid created:

    https://www.drupal.org/project/module_supports

    A module can suggest complimentary modules, or link to known conflicts, etc.

    Example:

    module info

    Simon-P’s picture

    Would love to be involved somehow. Not sure exactly sure how/when as yet, but started out as a Site Builder. Would find it easy to (pretend) to be the person who doesn't know anything about how Drupal works, which is who I think this initiative is geared towards helping.

    zsofi.major’s picture

    I'd love to help with this, I love the idea :)

    AjitS’s picture

    Great idea! Would love be involved.

    sashainparis’s picture

    Sounds cool :-)

    saschaeggi’s picture

    I can help with Design

    troutdale’s picture

    I am excited by this initiative and would like very much to participate. I am THAT site builder persona that Dries described in his DriesNote. I'm not sure where or how I can help because I've never contributed before, but I can't think of a better place to start than with this project.

    mahmoud-zayed’s picture

    Great idea! I'd love to contribute to it :)

    Manjit.Singh’s picture

    It would be my pleasure to help out in this in any possible way.

    shaal’s picture

    I'd love to help with this!

    codemann’s picture

    I'd love to contribute as well, I've been watching this issue for a while now.

    droprocker’s picture

    I'd like to join as well!

    Mile23’s picture

    Wow, 36 new comments out of 55 since last time I looked at my dashboard. :-)

    dreambubbler’s picture

    I am keen to help this initiative.

    ipwa’s picture

    Adding a screenshot of the design Dries showed in his keynote when he talked about this initiative.

    @saschaeggi I would love to see your take on this following your work from the Future UI.

    screenshot of Driesnote showing design for the project browser initiative

    rakesh.gectcr’s picture

    I love to help with this. Looks like a great Idea!

    zviryatko’s picture

    Hello, I'd like to help.

    Kir Lazur’s picture

    I'd like to participate in this initiative.
    We even did the PoC of this idea together with team https://www.drupal.org/project/drupal_marketplace

    ahillio’s picture

    From #44:

    it would be great if a module could add more in its info.yml file

    that sounds absolutely essential. Modules need to be classified, for example... field related modules which fall into two categories of field type and field display.

    For a use-case, consider a sitebuilder with a gallery node type that uses "flexslider" module for image galleries. They should be able to go into this tool and look for a way to change their existing gallery which would mean looking for a module that provides a different image display formatter... which would introduce them to the "slick" module and numerous others.

    kevinquillen’s picture

    Yes, that is where that concept was going. The module also let you link to project pages (drupal.org filtered only), specific issues, and gave a place to let someone know right there (and not buried in readme or help files) what other modules to pair, or which ones to avoid (feature overlap or breakage).

    AaronMcHale’s picture

    @ahillio @kevinquillen Re #62 and #63:

    I agree, I think these would be good pieces of metadata to provide. Quite a lot of that metadata mentioned in #63 can already be provided in the composer.json, as well as extra metadata. As we are moving in that direction it would make sense to further explore how composer.json can be used, it might even replace info.yml for most cases in the future. See #1398772: Replace .info.yml with composer.json for extensions, this initiative might be a good opportunity to kickstart that issue. Of course this is all further down the road as the initial goal appears to be just to get a proof of concept first.

    varunagarwal2004’s picture

    I would love to help with this initiative.

    Gábor Hojtsy’s picture

    Dries posted https://dri.es/join-the-first-drupal-project-browser-initiative-meeting

    First meeting will be May 24th between 14:30 UTC and 15:15 UTC. We'll share a Zoom-link on the Project Browser Slack channel (#project-browser) before the meeting starts.

    nonom’s picture

    Exciting Innitiative. I would be around.

    saki007ster’s picture

    Looks like a great Idea!
    I love to help with this.

    thejimbirch’s picture

    Jumping in!

    AjitS’s picture

    Grienauer’s picture

    Hi! I now have collected and saved some Web views from multiple CMS systems and their Module/Plugin browsers. So we have some inspirations, what others are doing. I added some info in the filename. CMS Name. Start: Frontpage of the project browser, Section: is if there is filtering for a section active. Details: info about a single “module”. I think this could help to get some more perspective. https://drive.google.com/drive/folders/1SRHHGCNKyj6sggOBjbmaVZF9vH0_FZOy maybe this helps for some decisions.

    jungle’s picture

    In addition to #71, maybe we can learn from NextCloud's apps management/App store, which is an open-source PHP project as well https://apps.nextcloud.com/

    webchick’s picture

    AMAZING work, @Grienauer!! Thank you SO much for all of your work compiling this!! :O

    webchick’s picture

    Actually, that's a good point. I wonder if there's a way to make this open to others to add screenshots from other products..?

    fhaeberle’s picture

    I would love to help with this idea, can provide design & dev.

    Grienauer’s picture

    @webchick it was not a big thing. if there is a need for even more screenshots from other systems, I can jump on that too. I am just curious how other systems do it so we can combine the best essence from all into ours ;P
    also said, please be aware, that this is only the web view, similar to our drupal.org/project/project_module and not from inside of the cms system.
    I will think about it, how we could open this so other people could add screenshots too… google drive was just an easy/quick solution…
    but if someone want to add stuff, just ask me for permission for now and I will add that person!

    phpsubbarao’s picture

    I am interested to be part of Project Browser Initiative

    ahillio’s picture

    Issue summary: View changes
    ahillio’s picture

    Project Browser Initiative meeting for July 14, 2021: https://www.drupal.org/project/drupal/issues/3224533

    effulgentsia’s picture

    nod_’s picture

    Component: Idea » Proposed Plan
    Issue tags: +Needs product manager review

    probably already an "approved plan" but not getting ahead of the workflow here.