Speed up migration process [#1905326]

Hi,

We have a phpbb 3 database with around 6'000 users, 26'000 topics and 530'000 posts. The overall db is around 1 Gb. What is the usual speed that we can expect when migrating users / topics / posts?

We have tried different setups: shared hosting, MAMP, VPS machine, but the behavior is always identical: the import starts at a fair good rate (around 2'500/min for topics and posts) but after a while, the speed falls down to something like 50/min and even less.

It would take ages to process 500K posts at this rate... Is there anything we can do about that?

I have tried increasing the max_execution_time and memory_limit but that doesn't seem to make a big difference (on MAMP and a VPS server).

Thanks in advance for any advice!

Comments

Comment #1

JeremyFrench CreditAttribution: JeremyFrench commented 5 February 2013 at 16:45

Hi There

This will depend on the speed of your Drupal server, and it will get slower over time. Just think how your server would cope if it had to process years of posts in an hour. However there are some ways to speed things up, and in the process you will get a faster Drupal server.

APC and memcached are pretty helpful for this, I have also found that the entity cache module helps to speed up the processing.
For this it may be worth turning off query caching in the DB, as you are doing more writes than reads.
Having a separate physical database for phpBB and for Drupal seems to help.
Turn off modules that you don't need, even db log, the migration uses Drupal calls to save items so more hooks on x_save will add time.
try to minimize error messages
Finally use Drush drush mi -u 1 --all you can also add a --feedback="1000 items" flag, make sure your CLI php has plenty of memory, it may fail but you can start again where it left off and it is faster and more reliable than the web interface.

Even with all this you are going to be looking at the best part of a day with that many posts. You can debug with a smaller subset to make sure your data looks good before doing the main migrate.

Good luck with the migration and let us know how you get on

Comment #2

mrupsidown CreditAttribution: mrupsidown commented 7 February 2013 at 21:20

Thanks for this helpful information. When you say "separate physical database", you mean on the same server, right?

Yes, we already planned for something like 5 or 6 hours. We were able to migrate the whole half-a-million posts already as a test, but we had to watch it and restart the script a couple of times, that's right.

We used the web interface. I don't know too much about drush. Does drush let you know if the import fails / do you have a way to check the import status?

Thank you.

Comment #3

JeremyFrench CreditAttribution: JeremyFrench commented 12 February 2013 at 09:03

Thanks for this helpful information. When you say "separate physical database", you mean on the same server, right?

Separate servers if you can manage it. DB can take a lot of memory and disk IO there is potential for contention if they are both on the same box. May only be a few % difference though.

We used the web interface. I don't know too much about drush. Does drush let you know if the import fails / do you have a way to check the import status?

So drush can tell you the status periodically the flag --feedback="1000 items" tells it to notify you every 1000 items, any migrate errors will also be displayed on the console, but they are also displayed still as messages in the migrate screen.

You can go to the migrate screen while drush is running to see how far the migration is along as well. I really would recommend it in terms of stability and performance.