Great idea for module, good to be able to export content and import it on onto another version of the same site (e.g. dev to live and vice versa, or live to test and so on).

But can you advise on what the recommended practice would be for storing and transporting data between platforms, please?

Common scenarios (use-cases) would be:

  • e.g. Say you're a developer and you want to use the latest content on live with your local development instance of the site. What would you do, here?
  • e.g. Say you're developing new functionality and are using new test content to test how the functionality works. Following limited testing, you want to pass this onto a QA tester so that they can reuse the content as part of their testing suite and strategy for their testing. What would you do here?
  • e.g. Say you've been working with a client on a new section and content for the site and they have reviewed it with their colleagues. You now want to get this specific content onto live. What would you do here?

One thing I don't think people would do is store the content in git as, depending on the type of site, it may have sensitive information.

But apart from avoiding that, we still do want something like git, whereby we can push and pull content to and from a remote source (e.g. bitbucket, github or own-hosted solution) as this is simple to be able to transport items between the site at various stages: dev, test, qa, uat, live.

My guess is that there isn't a single recommended practice but several possible bespoke approaches:

  • rsync or (s)ftp of the exported content in the content sync folder between, for example, live down to test ('pull' initiated from test), dev up to test ('push' initiated from dev).
  • Via an intermediary (acting like a git remote). Examples might be cloud storage services such as amazon S3, google drive, dropbox, set to point to the sync folder (or perhaps a separate folder and then locally copied from - to save accidental changes). The usual clients (google backup and sync, dropbox) can offer the syncing functionality. For a fully cross-platform solution (windows, mac and linux) for google drive there's insync (see insynchq.com)

Tools such as Jenkins and other continuous integration tools might also play a part in co-ordinating the above approaches.

Great module - certainly needed, thank you.

What are your thoughts on the above, please?

Comments

therobyouknow created an issue. See original summary.

therobyouknow’s picture

Title: Where to store exported/imported content? Recommended best practice please? » Where to store/how to transport exported/imported content between platforms? Recommended best practice please?
therobyouknow’s picture

Issue summary: View changes
blanca.esqueda’s picture

Issue summary: View changes
Status: Active » Closed (works as designed)

But can you advise on what the recommended practice would be for storing and transporting data between platforms, please?

Every project have different needs, I would suggest a similar approach to Effective D8 Configuration Management Workflow with Git and Drush (http://slides.com/besqueda/d8-configuration-workflow#/3)

Common scenarios (use-cases) would be:
e.g. Say you're a developer and you want to use the latest content on live with your local development instance of the site. What would you do, here?

Multiple options, depending on restrictions/security level/etc...
If there is access to drush then maybe a similar Drush/Git approach as the previous question.
let's say that for some reason there is no drush access or git is not an option, then I would export using the interface and move the tar file to my local environment to import the tar file.
It can be a combination of both - and there are multiple alternatives on how to move the content from one environment to another.

e.g. Say you're developing new functionality and are using new test content to test how the functionality works. Following limited testing, you want to pass this onto a QA tester so that they can reuse the content as part of their testing suite and strategy for their testing. What would you do here?

for export/import similar ideas as before, but I would include in the source and target sites an extra content folder for QA/Testing (update the setting of your content folder path) to not mix it with the real content. And, if using git I will use a QA branch too.

e.g. Say you've been working with a client on a new section and content for the site and they have reviewed it with their colleagues. You now want to get this specific content onto live. What would you do here?

Assuming the configuration of the new section has been imported, then following the answer to the previous question: if there are different content folders depending on their content (real, qa, specific multisite, etc..) then I would export the content that has been reviewed and accepted to the folder used for the live content.
Note: with Drush (still need to include the option for UI) you can export per entity type, bundle, etc.. and even include dependencies, so for example if you export menu links that point to all the pages on the new section then automatically all of the pages (content entities) attached to those menu links will be exported including the dependencies of those pages (taxonomies, files, etc..)
Also, you can create a module to use the functionality of this one to export specifically what you want (hook with drush, hook using the import/export service, etc..)

One thing I don't think people would do is store the content in git as, depending on the type of site, it may have sensitive information.

Depending on the type of site, if security is a concern you can have private repositories so only the people allowed can see the content.
Or if security is a bigger concern then after exporting I would encrypt either the generated yml files or the tar file, move them and then decrypt before importing on the target site.
Currently, the module doesn't handle encrypt/decrypt, but not a bad idea for a future feature. But even when currently doesn't handle the encrypt/decrypt, using the approach of having a custom module to trigger the import/export can include the encrypt/decrypt functionality.

But apart from avoiding that, we still do want something like git, whereby we can push and pull content to and from a remote source (e.g. bitbucket, github or own-hosted solution) as this is simple to be able to transport items between the site at various stages: dev, test, qa, uat, live.
My guess is that there isn't a single recommended practice but several possible bespoke approaches:

rsync or (s)ftp of the exported content in the content sync folder between, for example, live down to test ('pull' initiated from test), dev up to test ('push' initiated from dev).
Via an intermediary (acting like a git remote). Examples might be cloud storage services such as amazon S3, google drive, dropbox, set to point to the sync folder (or perhaps a separate folder and then locally copied from - to save accidental changes). The usual clients (google backup and sync, dropbox) can offer the syncing functionality. For a fully cross-platform solution (windows, mac and linux) for google drive there's insync (see insynchq.com)
Tools such as Jenkins and other continuous integration tools might also play a part in co-ordinating the above approaches.

I've seen different projects using this module in different ways:

  • using the UI interface to import/export
  • using Drush with git
  • using a custom code that uses the import/export services of this one for specific needs like:
    • managing content when AWS deploys a new Drupal instance
    • selecting the content that goes to specific instances of the same site
    • - using a bash file to run the export/import commands.
    • filtering the content to move to a completely different site but with same configuration for some functionality (slider, site sections, etc..) so content can be shared for that functionality
    • even overriding data (as replacing users, duplicating data, update content titles, etc..) after exporting or before importing.
  • etc..

Just as you mentioned there could be several approaches, it depends on many factors, the tools available for you (git, drush, sftp, etc..), the stages of the content (live vs qa, etc..), the level of security that your site requires (git private, encrypt/decrypt, rsync, etc.. ), the way the content needs to be split (across environment content vs different content for multisite vs custom code to filter or even override the content being exported/imported), etc...