I tried now by two ways to import CSV data into the ERPAL platform. But none works.

First (failing) solution: Feeds importer with CSV parser, CRM contact processor (eg for individual bundle), the mapping fails, because there is no way to set address and field collections (mail, phone, url) as target. - Reason: The Address field is set to commerce entity bundle custumer profiles without and mail, phone, url are fields are linke to field collection bundles - My open question: How to make those fields recognized by the feeds import mapping?

The second (failing) solution: CRM Core Data Import (crm_core_data_import) - Here the mail, phone, url fields are recognized, but not the address field in Customer profiles Bundle Billing information - Additionally the CRM Core Data Import
solution, even without address import comes with variating error messages I still could not identify, if they are linked to the modules CRM Core, Migrate or Erpal Platform ...

For the ability and easy handle to import data into the Erpal platform are essential I wold like to know how other working on the ERPAL platform are dealing their imports. Thanks in advance for feedback.

CommentFileSizeAuthor
#12 democontact1.txt396 bytesbechtold
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

manuelBS’s picture

Please have a look at
https://www.drupal.org/project/commerce_feeds
and
https://www.drupal.org/project/field_collection_feeds
and may be
https://www.drupal.org/node/1831004

Currently we created mostly custom importers. I am open to help someone conntribute an importer just as we did with ERPAL for Service Providers at https://www.drupal.org/project/erpal_contacts_importer where we have a plugin that exposed the field collection fields to feeds. But in general this should be possible with the modules linked above, but I have not tried it yet.

ChoY’s picture

Installing commerce_feeds and field_collection_feeds is not the solution for data import. - at least not the easy solution, installing the modules doesn't change nothing to the unavailability of address details and field_collection bundle fields by feeds importers.
Probably there is no other way than some additional coding or eventually rules. I would have preferred a modular solution ... just as it use to work easily with CRM core feeds import under condition that all contact fields are defined in the contact bundle ....

At all, I wonder, if it woldn't have been a better solution not use field collection for mail, phone, url and a an address defined at contact with a sync rule to customer profile?

We will continue to find a solution, the next days, but it will take some time with the given ERPAL/CRM/Commerce integration. However, I think an integrated data import tool will ease switching to ERPAL platform ...

manuelBS’s picture

Thanks for your investigation. I agree this would be easier to have a feeds importer. So if we got the beta done and we find other users that are interesrted, we will care about this.
The idea behind using field collections is the categorisation of phone numbers and email adresses. With a single field there is no chance to mark a phone number as private, public or with any other attribute.

ChoY’s picture

When dealing only with two fields https://www.drupal.org/project/double_field can be good solution - I did so in my former CRM Core installations.
An other solution can be https://www.drupal.org/project/composed_field - On the project site you find as well a good short about composed_field vs double_field vs field_collection -

I was thinking about to desactivate the ERPAL field collection solution for one of these other options, but I was not sure yet if the ERPAL platform configuration would allow it without code modifcations.

However more tricky is still the identification of the commerce linked billing address details for feeds contact bundle import ...

Well let's see ... I am not a coder, next week my magic coding partner will have a look on it...

joe_pixelrow’s picture

The lack of a working contact importer solution is a significant deterrent for those considering adopting the ERPAL Platform, I hope a solution is released soon so we can give this application a tryout.

ChoY’s picture

I agree fully with you - to make ERPAL fully functional an import solution is needed.
A best way to solve the import problem seems to be CRM Core Data Import tool
I work on it, but still with satisfying results. There are still too much fields not recognized or not found through the importer tools.
Is anyone else working on an import solution for ERPAL data import?

manuelBS’s picture

So the solution from my perspective would be to create a feeds importer as we did with the ERPAL for Service Provider Distribution that exposes the CRM Core contact entity with the referenced commerce profile entitiy as a feeds target so that you can map you CSV fields to these datastructures. Anything missing to match all of your requirements?

ChoY’s picture

As I described above - The feeds importer through CRM Core processors out of the box only recognize the fields directly defined in the entity bundle. Referenced bundles the field collection bundles and the commerce address are not recognized as feed importer destination. - Which means, a own Feeds ERPAL importer module has to be written or rewritten, taking the ERPAL service as template ...

One essential difference between ERPAL Service and ERPAL Platform is the implementatin of CRM Core. Since CRM core 7.x-0.980 comes now with a migrate module handle through submodule CRM Core Data Import, I think it would make more sense to search this way a new solution. It will offer a more flexible importer solutions than the ERPAL service contact importer did. But if someone has a better proposal through fed importer - Any solution is welcome, but the solution is needed ... Because it is a fact: ERPAL platform without a an import solution out of the box isn't very convincing for end users. -

ChoY’s picture

After checking though all possibilities of a csv import with the given means (feeds or csv_data_import) we came to the conclusion that the architecture of erpal with crm core and commerce will demand the development of a own module to make csv import possible.
The reasons: The use of field collection for phone/mail/url is not compatible with the standard feeds csv importer - eventually a patch my help. See https://www.drupal.org/node/1063434
But even more difficult looks the import address field into the contact address entity reference.

To make csv import into erpal platform less complex without need of an extra module development it may be a solution to replace the field collections bundles phone/mail/url with phone/mail/url fields directly integrated into crm core. - Same for the address field. -

However, an ERP/CRM without a minimal import function don't make much sense ....

SocialNicheGuru’s picture

can contacts be imported now with alpha12

ChoY’s picture

No there is still no import solution, or better said no easy solution.
The complications are with the commerce referenced address fields and the field collections: phone, mail, url
The content for this four fields is to gather from various tables in database, which so far no given drupal module can solve. It has to be written by a own script or module.
With other words: with given drupal feeds or migrate modules you cannot import, address, phone, mail, url
Due to a opinion one of the biggest disadvantages of the current erpal platform versions

bechtold’s picture

FileSize
396 bytes

Hey,
we focused on getting the BETA out and finalize functionality in the last couple of weeks.
Now we will take on the topic of importing.

I just tried this patch https://www.drupal.org/node/1063434 and it seems to go in the right direction. But somehow the text fields for url and mail didn't work. But for phone and the types it did work.

Address should be importable as well as described here https://drupalize.me/videos/importing-data-address-field-using-feeds?p=1503
Here are more ideas: https://drupalcommerce.org/questions/8534/how-import-users-and-customer-...
And there is https://www.drupal.org/project/commerce_feeds
But I haven't tried this yet.
We will evaluate this further. Maybe a custom importer for feeds might make sense.

One question is still open though. How can we import multiple field collections/addresses?
I don't think this works with the patch mentioned above.
And it is not easy to store in csv either.

So we should investigate if feeds makes sense at all and is possible or if we need migrate which is way more powerful than feeds.
@choy have you tried migrate? I know that it is very powerful and should be able to import field_collections and address fields.
About migrate and field_collections here: https://www.drupal.org/node/1900640
and here: https://www.drupal.org/node/1175082
Migrate and address field: https://www.drupal.org/project/migrate_extras
or https://www.drupal.org/project/commerce_migrate
and https://www.deeson.co.uk/labs/how-drupal-migrate-data-addressfield
I haven't tried it yet, but just started the investigation.

I have created a sample csv file. Would that be a format that would suit you? (Given that you only import one phonenumber and email)
Then we could use it together when trying to make it work.

ChoY’s picture

Hi Bechthold,
great that you are now working on the import tool! :-)
as it is unavoidable to built a own import solution, I think the better import solution will be with migrate, for the simple reason: CRM Core have yet built on crm_core_data_import, based on the migrate module.
I had also the impression during my import trials with feed and migrate, that I come closer to a possible result with migrate. Additionally it may be of advantage to join the import with the module commerce_migrate.
The disadvantage of feeds is, that you need an own import solution for every entity bundle (eg Individual, organization, household not to talk about the complexity with the relationships and activities linked to these contacts). With migrate I think it should be possible to cover all bundles and relations.
Regarding your demand which fields to import, I would say, just all fields that are defined by default contact fields in the Erpal platform.
Concerning the multiple fields (adress/phone/mail/url) I think it is unavoidable to find a solution to import all.
Due to my experience with SMEs (the main ERPAL target group), the typical address lost is an excel-sheet or acces-database which give you in export to CSV something like
ID; Type; Company Name; Person Name; Address, Phone1; Phone2; Email1, Email2; URL; Notes; Relationships;
An import template tool should cover this basics + a relatively easy way to modify the import via UI to the CSV template of the importer.
If we can help you anyhow for a best practice import don't hesitate to contact me and my Drupal coding expert partner orfils

bechtold’s picture

Ok, thanks again for the feedback.
I guess we will have another short look into feeds, since with it's mapping through the ui it would be easier to use for non programmers than migrate .
But I think we will end up with migrate and provide a basic importer that can be extended for further use cases as you mentioned.

ChoY’s picture

with migrate ui you have a mapping as well
Before I usded ERPAL platform I made once migration from civicrm to crm core with feeds.
It was possible but very restricted: I could only import either individuals or organisations and the main pont; the relations between them would have been an extra import to configure -
I presume it has some good reasons why the crm core team use migrate instead of feeds for import. Probably it should be checked with them, how the plan to proceed their import, if it isn't done yet.

Solid Builders Inc.’s picture

Question? I see importing Customers metadata is important, but what about the heavy lifting data like "Tasks"? Will the importer solutions above cater to this Goliath? Is there a way to do this and I just overlooked it?

bechtold’s picture

Hi,
at the moment we are just talking about a basic importer for contacts from csv files. So this will not cover the tasks.

And since tasks come from various sources and have different structures and sources it would not even be possible to create a generic solution. You might also want more fields and customizations on the tasks that we can't support by default.

Since tasks are implemented as regular nodes they should be well supported by feeds and migrate.
I would not recommend feeds in this case though, as references to parent tasks and projects might no work so well.
But with migrate you can easily create a very detailed importer that matches your exact source data.

ChoY’s picture

is their any progress on the import issue? I am confronted with the wish to merge to erpal platform sites content in one. But how to? With the usual feeds or migrate tools it won't work. Is their any other (half) manual bulk import solution as long as their is now erpal platform import module written? Thanks for suggestions.

PS via migrate it os possible to import the content types project and task - but no entities such as crm core entities and their related customer profiles.

bechtold’s picture

Hey,
sorry so far we didn't make any progress. Somehow crm core seems to be very hard to import via migrate.

If you only want to merge two plain ERPAL Platform installations you could try to manually copy the dataset from the crm_core_contact table and adjust the auto increment value.
If you have custom fields on the contacts it would be more effort as you would also need to copy those values and probably adjust the contact ids by hand. But it still should be possible.

Yuri’s picture

Priority: Normal » Major

This issue really needs more priority. I needed to reinstall the ERPAL platform a few times in the last week only, just because its just not stable enough..with all respect to the developers and I admire this great module and platform, but I assumed it would be production ready and am using it for a running real-world project. It appears to be very hard to update each time without reinstalling the complete ERPAL platform.
That said, each reinstall of ERPAL requires me to manually recreate 250 customers..and that amount is growing fast.. including recreating activities and sales processes...
I really admire the ERPAL platform and applaud its developers, but having no way to import/export the content and entities with their relations makes it as good as impossible to continue using the platform for production sites. It also discourages Drupal contributers to continue their support for this module suite.
I think we don't necessarily have to restrict such feature to CSV files, so maybe I should open this as a new feature request or we may change the title of this issue if you all agree.
Anyhow, please let me know if any serious efforts to implement an import/export feature is planned soon.

ChoY’s picture

The missing import/export compatibility with any standard importer/exporter modules is definitively the weakest point of the ERPAL platform distribution. As described above it is possible to export part of the entities via views erporter, but import anything is simply impossible, because the database composition of ERPAL platform is too complex. There will be no other way than to create an own import/export module, and it won't be a short walk to develop it.
As far as I learned the developer team behind ERPAL platform has currently no free capacities to develop such a project. With other word: The Drupal community is asked to take the things in hand or stay patient until one of the developer better familiar than us with the ERPAL platform coding will find some days to work on this missing module.
For my part, as well working with a production site, I am not happy with, but I see no other chance than waiting until some of the developers may find again the time to close this yawning gap in the ERPAL platform.