Problem/Motivation

For large sites, it's useful to undertake a planning stage for updating modules to the latest next-major compatible version. To that end, it's useful to have a CSV export of upgrade status' report.

Proposed resolution

Add an "Export as CSV" button to the Upgrade Status Form.

Remaining tasks

Review patch

User interface changes

Adds an "Export as CSV" button to the Upgrade Status Form

API changes

None.

Data model changes

None.

Release notes snippet

Adds the ability to export the Upgrade Status report to CSV

CommentFileSizeAuthor
#10 issue-3152580--Export-to-CSV-for-8.x.patch8.36 KBBetoAveiga
#6 csv_support-3152580-6.patch9.62 KBmtift
#4 interdiff.txt1.62 KBm4olivei
#4 csv_support-3152580-4.patch9.49 KBm4olivei
#4 single_drop_button.png13.55 KBm4olivei
#4 double_drop_buttons.png6.15 KBm4olivei
#2 csv_support-3152580-2.patch8.54 KBm4olivei

Issue fork upgrade_status-3152580

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

m4olivei created an issue. See original summary.

m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented
Status: Active » Needs review
FileSize
csv_support-3152580-2.patch8.54 KB
Gábor Hojtsy’s picture

Gábor Hojtsy
he/him
Primary language Hungarian
Location Hungary
CreditAttribution: Gábor Hojtsy at Acquia commented

Thanks for the patch!

I think exporting in CSV is a good idea. We should discuss what is exported and how it appears on the UI. The HTML and ASCII export outputs detailed reports of the components selected. Looks like this CSV export would export only data directly visible in the form. I think this could get confusing. Also three export buttons is starting to get unwieldy so maybe we'll need to go to a select list or somesuch.

m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented
Assigned: Unassigned » m4olivei
Status: Needs review » Needs work
FileSize
double_drop_buttons.png6.15 KB
single_drop_button.png13.55 KB
csv_support-3152580-4.patch9.49 KB
interdiff.txt1.62 KB

Thanks for taking a look. Glad to work on it.

Here is a sample CSV export imported into a Google sheet of all the contrib modules on a site I help maintain:

https://docs.google.com/spreadsheets/d/1HGBNKx8IHViVA31Fl5mhA2ygvel2w9_K...

Comparing the CSV output to the HTML output, whats missing is the detailed list of errors including what group the errors fall into, eg. errors that can be fixed with automation, fix manually, check manually, etc. I suppose we can cram that into a single cell in the spreadsheet. Would that work?

As for the buttons getting crowded, agreed. First attempt at a fix for that is attached. This uses a single drop button for all of the buttons. Here is what that looks like:

I wanted to just group the export ones, but core is weird and wants to put drop buttons first :/ (https://git.drupalcode.org/project/drupal/-/blob/8.9.x/core/lib/Drupal/C...). I can get around that by making "Scan selected" a drop button as well, but then the styling gets a little weird:

Gábor Hojtsy’s picture

Gábor Hojtsy
he/him
Primary language Hungarian
Location Hungary
CreditAttribution: Gábor Hojtsy at Acquia commented

Re what should be in the csv I think it would be great to have some consistency between the different exports. The HTML and text export does not have a summary of error numbers for a project, just the list, while your proposed CSV only has the summary numbers. I think we should have some consistency. It may be that two kinds of CSV output would be in order, a summary and a detailed list.

mtift’s picture

mtift
he/him
Primary language English
Location Minnesota, USA
CreditAttribution: mtift at Lullabot commented
Version: 8.x-2.x-dev » 8.x-3.x-dev
FileSize
csv_support-3152580-6.patch9.62 KB

Here's a re-roll. It isn't working, but I ran out of time and I wanted to post my progress.

m4olivei opened merge request !8

m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented
Assigned: m4olivei » Unassigned

I opened up a merge request that applies the patch from #6, and then makes the necessary adjustments to get to a workable patch again. It's not complete, in the previous patch from #4 which was against 8.x-2.x, I was using the releaseStore to put some data into the CSV export and that doesn't seem to be working as expected.

I'll put a bit more time in today on this to see if I can get better output for the CSV. It's been awhile so I'll have to review the HTML and ASCII exports to see whats included there on the latest 3.x.

m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented

Oh it actually does work on par with what I did in #4 for 8.x-2.x. I was just forgetting to have update status module check available updates.

I think we're now on PAR with what we had in 8.x-2.x. Except I went back to the three buttons for simplicity. That leaves all of @Gábor Hojtsy suggestions outstanding. I'm spent for now, so happy for someone else to carry this if I don't get back first.

BetoAveiga’s picture

BetoAveiga
Primary language Spanish
Location Daule, Guayas
CreditAttribution: BetoAveiga at Lullabot for Dartmouth commented
FileSize
issue-3152580--Export-to-CSV-for-8.x.patch8.36 KB

I'm just adding a patch for @m4olivei MR https://git.drupalcode.org/project/upgrade_status/-/merge_requests/8 so I can add it to my composer JSON file.

Gábor Hojtsy’s picture

Gábor Hojtsy
he/him
Primary language Hungarian
Location Hungary
CreditAttribution: Gábor Hojtsy at Acquia commented

Based on https://www.lullabot.com/services/support-maintenance/drupal-9-readiness... I think there must be a working version of this patch? Would be great to get it in the base module. Thanks all!

eelkeblok’s picture

eelkeblok
him/his
Primary language Dutch
Location Netherlands 🇳🇱
CreditAttribution: eelkeblok at iO commented

Currently, the patch produces an empty "CSV" for me (as in, zero bytes). The export CSV button also shows up in a weird place. Investigating.

eelkeblok’s picture

eelkeblok
him/his
Primary language Dutch
Location Netherlands 🇳🇱
CreditAttribution: eelkeblok at iO commented

Well, mostly down to clearing the cache 😅. I started by doing my own merge from the patch with the latest 3.x and the results were slightly different from the last state of the MR. I ended up merging in the latest 3.x into the MR and applying my changes where they made sense.

eelkeblok’s picture

eelkeblok
him/his
Primary language Dutch
Location Netherlands 🇳🇱
CreditAttribution: eelkeblok at iO commented
Status: Needs work » Needs review

Added exporting the messages. Putting to review. Note that I did not venture into the dropbutton again. I wonder if it makes sense to refactor the export format so that it becomes a plugin type. That could also be a good time to implement that drop button, since there would then be an arbitrary number of export buttons. Obviously that should be a follow-up to this, if we choose to do that.

m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented
Title: Export to CSV for 8.x » Export to CSV for 4.x
Version: 8.x-3.x-dev » 4.x-dev
Issue summary: View changes
m4olivei’s picture

m4olivei
he/him
Primary language English
Location Grimsby, ON
CreditAttribution: m4olivei at Lullabot commented

Thanks @eelkeblok for taking up the torch!

I had occasion to use this for a 9.x -> 10.x upgrade, so I'm adding my tweaks as well. To summarize my commits:

  • Merged the latest 4.x branch
  • Adust the base branch of the merge request to 4.x
  • Adjustments to make the CSV export look more like the table data on the upgrade status UI
Gábor Hojtsy’s picture

Gábor Hojtsy
he/him
Primary language Hungarian
Location Hungary
CreditAttribution: Gábor Hojtsy at Acquia commented
Title: Export to CSV for 4.x » Add UI option to export Upgrade Status results as CSV
Gábor Hojtsy’s picture

Gábor Hojtsy
he/him
Primary language Hungarian
Location Hungary
CreditAttribution: Gábor Hojtsy at Acquia commented
Related issues: +#3391323: Add options to provide drush analyze output in JSON and CSV

I've been looking at how to integrate this more seamlessly. There is also #3391323: Add options to provide drush analyze output in JSON and CSV which adds JSON on the drush CLI (and a little script to further process it which we could also add to the module as an option). It has different logic from your setup and less involved in how it builds the data. I think a middle ground would be good as well as ideally as much code reuse as possible between the output formats IMHO.