Migration doesn't process all items

*Sigh*. That buffering may be more trouble than it's worth...

I'll see if I can reproduce and debug this. Thanks for reporting it, I'd hate to get 2.7 out and then have to do a 2.8 right away...

Oh, I see it - the premise of the LIMIT clause is that each time we execute the query, the base query (without the LIMIT) returns exactly the same set of rows in the same order. However, if we're joining with the map table, which we're populating as we go, then each batch of 2000 rows the base query is 2000 rows shorter. So, on our second batch we're expecting to get rows 2001-4000, but since rows 1-2000 have been eliminated from the base query, the rows we want to fetch are now rows 1-2000 and we skip them.

This will take a bit of thought - how do we prevent this situation without reintroducing the previous problem? At a deeper level, I'm not sure we really want to trust the premise - if migrating from a live database where more rows might be added during the migration process, things are going to get messed up. It might be best to give up on the batching for now, or at least make it optional (documented with plenty of caveats about scenarios like this).

OK, the attached patch makes batching of the query optional (by defaulting batch_size to 0 and taking that to mean no batching), as well as setting map_joinable FALSE when batching is used. I've also added documentation at https://www.drupal.org/node/1152150#batch_size.

Great, committed, thanks!