Import data from CSV files that CES exports. Build it in a separated module. We can add new fields to our database tables so more data can be imported from CES. Other fields may be ommited and lost.

CommentFileSizeAuthor
#4 testdata.tgz3.08 KBesteve.badia

Comments

hongpong’s picture

This could probably be done by building up some kind of presets with the Migrate module. See http://drupal.org/project/migrate

"Migrate 2.6 beta1 is now available. The main theme of this release is UX - the existing UI has been significantly rearranged, and a new API has been introduced to allow other modules to easily developed wizard-style UIs targeted to non-developers." Nice!

esteve.badia’s picture

Interesting! Have in mind that the main goal of this issue is to import data from CES, though in future we could have to import an export from/to other platforms, o a generic approach like this sounds good to me. Moreover, I tried to have minimum dependencies (only to d7 core) for critical features, but for this kind of feature It seems right to me to get support from other projects if it really helps.

Fortunately, and administrator of a CES exchange group can download the database records in a CSV format. There are several files: users.csv, transactions.csv, offers.csv, etc.

I have access to these files because I'm currently administrating such a community, but I can't post them here because of privacy issues. Perhaps we should anonymize on of such set of files in order to have a public test suite. Do you find it really useful? Do anyone have an idea how such a thing is usually done?

The main thinks we have to import are:

  • Users
  • Exchanges, bank accounts, transactions: All of these objects are handled by the API in ces_bank.logic.inc but can also be directly written to the DB. Since they are kind a special (but easy) objects and almost unrelated to Drupal, I think that the migrate module cannot help.
  • Offers and Wants: They are entities handled by the ces_offerswants submodule.

Several thinks to take in count:

  • Users and bank accounts is the same in CES, but different concepts in Integral CES. Thus, for each CES user we should produce a drupal user, a bank account and a link between the two.
  • Related to the previous, users in CES don't have name, but account number. We could use the acount number (like "ABCD0123") as acount name
  • Password is plainly stored in CSV, so we could import it without problems.
  • Account limits are handled differently from CES to IntegralCES: In CES every exchange has a default credit and debit limit. And each account have its own credit and debit limit. Account limits are the default ones if they haven't been edited. In IntegralCES the same concept is represented by the LimitChain object. A LimitChain is a set of atomic Limits with a name. Each atomic limit defines something like "you cannot have minus than -100$" or "you cannot have more than 500$" or others depending on available implementations. Then each account is linked with one of the defined LimitChains, and the exchange also have the default LimitChain for new accounts.
  • Trades with other exchange groups. While CES has a virtual account ABCDVIRT that represents all other exchanges, IntegralCES has a virtual account for each exterior exchange. We could have to create a ghost exchange and "redirect" all old exterior exchange to this.

That's all that I have in mind now. The issue isn't easy and my explanation is neither clear nor exhaustive, but we are on it!

hongpong’s picture

that is quite informative!! If you can create and make available some kind of smaller fake version of all the CSV files, with pseudo data in the same format, that would be excellent. Think like 50 lines for each file perhaps.

I think that using Migrate module should definitely suffice - would be much better to use migrate code than re-implement. However also, we should look at making an 'entity controller class' so that things like transactions can be dealt with as entities in Drupal. I have been researching this as I have not done it before.

esteve.badia’s picture

StatusFileSize
new3.08 KB

Attached file users.csv with the users database export from CES. Soon I'll try to add more csv with balances, transactions, offers, etc. But we can start working with this. Of course it is not real data. There are 25 fictitious CES user accounts.

edumag’s picture

Assigned: Unassigned » edumag
Issue summary: View changes

  • Commit 435224f on 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit b9cacb6 on 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    

  • Commit e6dd2bf on 1916476-Import_data_from_CES by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...

  • Commit 9181cb4 on 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...
  • Commit a3cf6a4 on 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Code cleanup.
    

  • Commit 435224f on 7.x-1.x, 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit 9181cb4 on 7.x-1.x, 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...
  • Commit a3cf6a4 on 7.x-1.x, 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit b9cacb6 on 7.x-1.x, 1916476-Import_data_from_CES by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit e6dd2bf on 7.x-1.x, 1916476-Import_data_from_CES by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
esteve.badia’s picture

Status: Active » Fixed

  • Commit 435224f on 7.x-1.x, 1916476-Import_data_from_CES, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit 9181cb4 on 7.x-1.x, 1916476-Import_data_from_CES, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...
  • Commit a3cf6a4 on 7.x-1.x, 1916476-Import_data_from_CES, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit b9cacb6 on 7.x-1.x, 1916476-Import_data_from_CES, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit e6dd2bf on 7.x-1.x, 1916476-Import_data_from_CES, 2215169-interop, 2215171-interop by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...

  • Commit 435224f on 7.x-1.x, 1916476-Import_data_from_CES, 2215167-interop, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit 9181cb4 on 7.x-1.x, 1916476-Import_data_from_CES, 2215167-interop, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...
  • Commit a3cf6a4 on 7.x-1.x, 1916476-Import_data_from_CES, 2215167-interop, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit b9cacb6 on 7.x-1.x, 1916476-Import_data_from_CES, 2215167-interop, 2215169-interop, 2215171-interop by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit e6dd2bf on 7.x-1.x, 1916476-Import_data_from_CES, 2215167-interop, 2215169-interop, 2215171-interop by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...
edumag’s picture

Status: Fixed » Closed (fixed)

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers, 1996122-Module_code_documentation by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers, 1996122-Module_code_documentation by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers, 1996122-Module_code_documentation by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers, 1996122-Module_code_documentation by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 2271629-Comments_in_offers, 1996122-Module_code_documentation by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...

  • Commit b9cacb6 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 1996122-Module_code_documentation, 2283781-accept-payments by esteve.badia:
    Issue #1916476: Properly import echange admin user XXXX0000.
    
  • Commit 435224f on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 1996122-Module_code_documentation, 2283781-accept-payments by esteve.badia:
    Issue #1916476: Let the user upload files by user interface.
    
  • Commit e6dd2bf on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 1996122-Module_code_documentation, 2283781-accept-payments by esteve.badia:
    Issue  #1916476: Properly import limit chains from CES users.csv file (...
  • Commit a3cf6a4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 1996122-Module_code_documentation, 2283781-accept-payments by esteve.badia:
    Issue #1916476: Code cleanup.
    
  • Commit 9181cb4 on 7.x-1.x, 2215167-interop, 2215169-interop, 2215171-interop, 1401884-Notifications_b, 2271149-opentransact, 1996122-Module_code_documentation, 2283781-accept-payments by esteve.badia:
    Issue #1916476: Import currency value from Euro-based and time-based...