Problem/Motivation

There are some differences between the format of CSVs exported from farmOS (using the "Export CSV" bulk action added in this PR: https://github.com/farmOS/farmOS/pull/783) and the format of CSVs that can be imported into farmOS (using the default CSV importers added to farmOS v2 by #3382539: farmOS v2 CSV import module).

This makes it difficult to use the CSV export/import to move data from one farmOS to another.

Steps to reproduce

Export a CSV of assets from farmOS and try to re-import it.

Proposed resolution

Use this issue to document that differences and think through next steps that might bring the two formats closer to compatibility with one another.

Remaining tasks

TBD

Old issue summary (from farmOS v1)

There are some major differences between the format of CSVs exported from farmOS and the format of CSVs that can be imported into farmOS. This makes it difficult to use the CSV export/import to move data from one farmOS to another.

Some of these differences are documented here: http://v1.farmOS.org/guide/export

Due to these differences, it is not recommended to use CSVs for moving data from one farmOS to another.

We may be able to improve upon this...

One thing we can do is override the fields provided by the CSV exports, and differentiate them from the output that is displayed in the View's Page display. Some examples:

* Do not truncate description fields
* Output numerical asset IDs (instead of names)
* ...

Comments

m.stenta created an issue. See original summary.

m.stenta’s picture

Version: 7.x-1.x-dev » 2.x-dev
Issue tags: +Needs issue summary update
Parent issue: » #3151244: [META] farmOS 2.x Importers

Bumping this to 2.x. We will need to review and update the issue description...

m.stenta’s picture

Version: 2.x-dev » 7.x-1.x-dev
Status: Active » Closed (won't fix)
Issue tags: -Needs issue summary update

I'm actually going to set this back to 7.x-1.x and close it as "won't fix".

We aren't maintaining 7.x-1.x anymore, and we don't have CSV importers in 2.x (see #3151244: [META] farmOS 2.x Importers). If we have the same issue after we add CSV importers, we can open a new issue.

m.stenta’s picture

Version: 7.x-1.x-dev » 3.x-dev
Category: Feature request » Task
Issue summary: View changes
Priority: Minor » Normal
Status: Closed (won't fix) » Postponed
Related issues: +#3382539: farmOS v2 CSV import module

Update: We now have CSV importers in farmOS v2+ (#3382539: farmOS v2 CSV import module).

I also just opened a pull request that will make the asset and log CSV exporters more closely aligned with the CSV importers: https://github.com/farmOS/farmOS/pull/783

Regarding CSV export/import compatibility (issue (5) above), this is step in the right direction. It changes the CSV columns headers to be field IDs instead of field labels, which match what the CSV importers expect, and it includes more potential data. It may be worth reopening https://www.drupal.org/project/farm/issues/2900239 and documenting what all the differences are, so we can consider whether or not we want to continue the effort of bringing them together.

There are still differences that prevent exports from being directly used in imports. Once that PR is merged, we can pick up this issue where we left off and document what all the differences are, and explore next steps to bring the two closer together.

Reopening this issue, updating the summary, and marking it as "postponed" pending that PR...

m.stenta’s picture

m.stenta’s picture

Status: Postponed » Active
m.stenta’s picture

Issue summary: View changes
m.stenta’s picture

farmOS 3.1.0 and 3.3.0 changed the way that we generate CSV exports for assets, logs, and quantities (notably data stream exports have not changed, but are not relevant to this issue).

See pull requests:

To summarize, we used to provide CSV exports as View displays with a limited set of columns (the same columns as the page displays). Now, we provide an "Export CSV" bulk action which uses the CSV Serialization module to create CSV rows for selected entities.

I referenced this issue in the first pull request:

The CSV exports are not compatible with the CSV importers, and we basically decided they never will be (see https://www.drupal.org/project/farm/issues/2900239). This will still be the case after this PR, but it brings us a step closer to supporting this.

Regarding CSV export/import compatibility (issue (5) above), this is step in the right direction. It changes the CSV columns headers to be field IDs instead of field labels, which match what the CSV importers expect, and it includes more potential data. It may be worth reopening https://www.drupal.org/project/farm/issues/2900239 and documenting what all the differences are, so we can consider whether or not we want to continue the effort of bringing them together.

There are a couple of notable changes between the CSV files that were produced by the old code and this new code:

  1. CSV headers are field IDs instead of field labels.
  2. More columns are included.
  3. Quantity columns are NOT included (see "Next steps" below).
  4. Timestamp fields are returned in RFC3339 format.
  5. Multi-value cells use a pipe (|) separator instead of a comma (,).

There may be others... those were the main ones I noticed/worked on personally. We get most of this output "for free" from the Drupal content entity normalizer, as opposed to specifying each one explicitly in Views, so we may find that there are bugs/issues that went unnoticed during my testing which we can address in follow-ups.

m.stenta’s picture

Version: 3.x-dev » 4.x-dev