farmOS v2 CSV import module

This effort is supported by OpenTEAM's ACTION initiative (see https://gitlab.com/OpenTEAMAg/feedback/support_requests/-/issues/146), and builds upon the initial work and investigations of @pcambra in #3151244: [META] farmOS 2.x Importers.

I am working on this in my 2.x-csv branch on my GitHub fork: https://github.com/mstenta/farmOS/tree/2.x-csv

I filled in the work that has already been done in the "Remaining tasks" above, but there is more to add. I will organize my notes and update the list as I go.

With the "default" importers we are going to provide on behalf of all asset/log/term types we will need to strike the right balance so that they are useful for general import use cases. But ultimately there are tradeoffs, and we'll need to make some decisions about what to support and what to defer to "custom" importers.

An example of this is log quantities. In my first pass I simply added 4 columns to the generic CSV templates:

• quantity measure
• quantity value
• quantity units
• quantity label

So currently only a single quantity is supported. I discussed this @paul121 in chat:

mstenta: can we support multiple quantities?
mstenta: we could provide a fixed number of "slots" for them - eg: ...,quantity1 value,quantity2 value,quantity3 value...
paul121: What about multiple being comma(?) separated in the same column?
paul121: With a convention for the order of label, value unit and measure?
paul121: No good solution really
mstenta: oh interesting
mstenta: yea could be error prone tho
mstenta: could just keep it limited to 1 quantity for MVP, and then see what kinds of requests come up
mstenta: ultimately i hope we'll identify more specific requirements that can be built as config-based importers
mstenta: so maybe these generic ones won't need multiple
paul121: Could it maybe auto-detect `quantity # value` columns? Then for each # found, check if relevant label, measure and units are provided?
mstenta: i wondered that... i don't know if migrate api supports it
paul121: I agree this is a good start but that would be cool
mstenta: i think all migration mappings are "pre-determined" at runtime
mstenta: we would need some way to hook in after the file is uploaded to alter it i think
paul121: yeah that makes sense

Another tricky decision is how to handle asset references, including the parent field on assets, and the asset, location, equipment, group, etc fields on logs.

I added support for taxonomy terms to set their parent by name. This is easy enough and makes sense for terms because in theory they should be unique.

For assets it's trickier. You may have multiple assets with the same name. Or archived assets that had the same name as a current asset (eg: if you use an ID tag as a name, and reuse ID tags after an animal is archived).

We had the same exact quandary in farmOS v1 (see #3086611: Allow CSV imports to reference assets by name), and ultimately decided to support asset IDs and asset names. That led to another issue (#3086844: Issues importing BOTH asset IDs AND asset names), but generally worked OK.

We may just need to take the same approach in v2, unless there's a better idea...

Also, asset references need to support multiple values, so a comma-separated list of IDs/names needs to be supported.

We may also want to support lookup by ID Tag...

This is making me thing we might want to provide a custom asset_lookup migration process plugin, which encapsulates all of this custom logic.

Add "Import [type]" local action buttons to view.farm_asset.page_type, view.farm_log.page_type, and entity.taxonomy_vocabulary.overview_form routes.

I looked into this briefly, and it wasn't as trivial as I was hoping, so I'm going to punt on it for the time being and focus on more important things. Users can still get to these importers through the menu. These action links are really "nice to haves" - not critical. But if we have time left at the end I might look into this again.

I opened a pull request for review: https://github.com/farmOS/farmOS/pull/722

This is a big one, but the commit history should describe the changes in a logical order. There are a lot of potential next-steps, but I think this is a good first step, and it's already better than the v1 importers in a number of ways.

Merged!