Promote drupal endpoint to full / default source plugin

Postponed until we have the live endpoint.

Update given in slack channel: https://drupal.slack.com/archives/C01UHB4QG12/p1717668171381089

I wanted to give an update on where we are at in regards to the future Project Browser endpoint to read data from www.drupal.org.

PB side
Note that Project Browser is already prepared to consume that data:

• You can see the plugin that will consume the endpoint here: DrupalDotOrgJsonApi.php
• We also have an issue to track what the next steps would be: #3371084: Promote drupal endpoint to full / default source plugin

www.drupal.org side
We had some blockers that were finally solved and we could pivot back to working on this.

So, we have:

• A Drupal 10 site with jsonapi module enabled, it is available here: https://drupalorg.dev.cluster.drupalsystems.org/jsonapi (that is a throwaway copy where we constantly destroy/recreate the database to test things. Do not use.)
• We use the above server to test the migrations (https://www.drupal.org/project/drupalorg_migrate). As a reference, the user migration alone takes longer than a full day to run, so we need to. This is still WIP, regarding the access from the dev servers to a D7 copy of the datatabase.

We still don’t have:

• Elastic Search remote server to test the search functionality which leverages search_api. We’ve tried locally with database server, solr server and elastic search server and it all works well. Adding this to the remote server is our next priority.
• And staging or live endpoint.

More in the coming weeks 🙂

Current situation:

• ✅ Dev and prod servers created
• ✅ Drupal 7 database replicas accessible by the D10 projects
• ⚠️ Copy of the D7 files folder. Nearly there on dev and prod. Migrations have a fallback to the public URL.
• ⚠️ Migrations: Struggling to run the +2 million user migration. When it does, then it's files, projects and project usage, each depending on the previous.
• ⚠️ Elastic search: set up in the dev server but not yet in the prod server (It's meant to be quick once we finetune permissions).

Some of the ⚠️ are nearly ✅ but going into the cautious side.

Hi all,

As of today, we have:

🟢 Migrations
- Finally got the first full user migration to run.
- It now follows files (done)
- And then all the projects migrations (running)
- Finally, there is the project_usage migration. This last one has got +40 million rows so we are expecting it to take a long time.

🔴 Elastic Search
- We are awaiting on the Elastic Search server for the production environment.
- We have set it up on development and the infra team is fine-tuning permissions and settings, but it’s not set up in production yet.
- This is currently the main blocker as we cannot index and search the data migrated otherwise.

🟠 PB promoting endpoint issue (aka this issue)
- I will work tomorrow and the day after on it, hoping to have something to be reviewed / tested via drupalpod by the end of the week.

If/when the Elastic Search part is solved, we will be in a position to deliver a “live” endpoint. It will be live in the sense of being a endpoint reading from the d.o database, but note that currently, migrations take “hours” to run and that this is the very first week ever that we have these running on the new infra for the production environment. This also means that we are learning about server requirements, real volume of data, Drupal limitations, etc.

⚠️ So, our ask is that you treat the endpoint as an “alpha” or “beta” version of it, and as such, Project Browser could not be considered stable under we are happy with the stability of the endpoint. I think this makes sense but please let us know if you don’t understand any of the above.

Having said all that, the “alpha” live endpoint will be: https://drupalorg.prod.cluster.drupalsystems.org/jsonapi 🎉🎉
This will change to be www.drupal.org/jsonapi but I think this makes it more obvious that it’s still not the final version.

WIP in the MR.

Not sure why but when adding some filters the result is 0.

Works:

https://drupalorg.prod.cluster.drupalsystems.org/jsonapi/index/project_modules?
  filter[status]=1&
  filter[type]=project_module&
  filter[project_type]=full&
  page[limit]=12&
  page[offset]=0&
  include=field_supporting_organizations,field_supporting_organizations.field_supporting_organization,field_module_categories,field_maintenance_status,field_development_status,uid,field_project_images
  filter[security_coverage][value][0]=covered
  filter[security_coverage][operator]=IN
  filter[security_coverage][path]=security_coverage
  filter[n_security_coverage][value][0]=revoked
  filter[n_security_coverage][operator]=NOT IN
  filter[n_security_coverage][path]=security_coverage

When adding any of the following, it does not:

  filter[maintenance_status_uuid][value][0]="9125cb6-2f35-451b-922d-3042cb1b4391"
  filter[maintenance_status_uuid][value][1]="de457d5a-2ce3-45b4-b88b-1c54e5e6d0e2"
  filter[maintenance_status_uuid][value][2]="fd8b539f-a5e4-4577-9367-f119a252327b"
  filter[maintenance_status_uuid][operator]=IN
  filter[maintenance_status_uuid][path]=maintenance_status_uuid
  sort=-active_installs_total

Is it related to sort somehow? As when I hit https://drupalorg.prod.cluster.drupalsystems.org/jsonapi/index/project_modules?filter[status]=1&filter[project_type]=full&filter[type]=project_module&page[limit]=12&page[offset]=0&include=field_supporting_organizations,field_supporting_organizations.field_supporting_organization,field_module_categories,field_maintenance_status,field_development_status,uid,field_project_images&sort=title it is working but not when https://drupalorg.prod.cluster.drupalsystems.org/jsonapi/index/project_modules?filter[status]=1&filter[project_type]=full&filter[type]=project_module&page[limit]=12&page[offset]=0&include=field_supporting_organizations,field_supporting_organizations.field_supporting_organization,field_module_categories,field_maintenance_status,field_development_status,uid,field_project_images&sort=-active_installs_total

May be it's because field_active_installs_total has null value everywhere. Just a guess 🤔.

Yup. It’s related and due to that. We’re working on fixing the endpoint issue. It’s not because it’s null as it happens with other fields with data.

It’s related to Elastic vs Open Search and the Drupal modules we use.

WIP.

https://drupalorg.prod.cluster.drupalsystems.org/jsonapi/index/project_m...

works now. Will continue work on this tomorrow.

Live endpoint plugin in action (recording): https://www.drupal.org/files/issues/2024-07-04/Screen%20Recording%202024...

The failing tests on CI are passing locally (using the ddev contrib plugin).
I've tried a few things but I'm stuck on why the CI run is not getting results.

Local run:

$ php web/core/scripts/run-tests.sh --types PHPUnit-FunctionalJavascript --sqlite "sites/default/files/.sqlite" --verbose --class \\Drupal\\Tests\\project_browser\\FunctionalJavascript\\ProjectBrowserUiTestJsonApi

Drupal test run
---------------

Tests to be run:
  - \Drupal\Tests\project_browser\FunctionalJavascript\ProjectBrowserUiTestJsonApi

Test run started:
  Thursday, July 4, 2024 - 14:03

Test summary
------------

Drupal\Tests\project_browser\FunctionalJavascript\ProjectBro  15 passes                                      

Test run duration: 3 min 20 sec

Detailed test results
---------------------


---- Drupal\Tests\project_browser\FunctionalJavascript\ProjectBrowserUiTestJsonApi ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      ProjectBrowserUiT   50 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT   79 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT   90 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  109 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  152 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  167 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  217 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  263 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  305 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  312 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  325 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  332 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  350 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  425 Drupal\Tests\project_browser\Functi
    
Pass      Other      ProjectBrowserUiT  453 Drupal\Tests\project_browser\Functi
    

All tests are green now. ALL 🙂

The changes in tests/fixtures are just replacing contants/URLs, so no need to review those. Review all the other changes.

I also created a follow up to decide what to do with the "mock" plugin, as it is expected in many tests, but it's out of the scope for this issue:
#3458780: [PP-1] Remove drupalorg_mockapi or move it to "test" folder.

Please review.