Not sure how we never ran into this before now but we must deal with the issue in our other sources so its flown under the radar.

The fatal "Error: Unsupported operand types" happens in SourcePluginBase::next() when this code runs:

      $row_data = $this->getIterator()->current() + $this->configuration;

That code is actually fine because we require iterators to return arrays and configuration is the plugin configuration so that should always succeed. But when you inspect the values you'll find that $this->getIterator()->current() is an object.

So, what is the iterator creating that object? There's a lot of code around it but it boils down to this line in SqlBase::initializeIterator()

    return new \IteratorIterator($this->query->execute());

Looks pretty good again but there's a failed assumption here and that's that the statement returned by execute is set to return objects:
http://cgit.drupalcode.org/drupal/tree/core/lib/Drupal/Core/Database/Sta...

  protected function __construct(Connection $dbh) {
    $this->dbh = $dbh;
    $this->setFetchMode(\PDO::FETCH_OBJ);
  }

So now the PHP Iterator is wrapping the PDO iterator which is listing Objects and the migration blows up.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

neclimdul created an issue. See original summary.

neclimdul’s picture

Patch which forces the fetch type on the statement sent to the migrate iterator.

neclimdul’s picture

Status: Active » Needs review
quietone’s picture

How is this produced? I ask because commerce_migrate_ubercart extends SqlBase and in several source plugins and runs without Fatals.

Status: Needs review » Needs work

The last submitted patch, 2: 2918837-fix-sqlbase-fatal.patch, failed testing. View results

neclimdul’s picture

very strange... just a super simple source causes this.

Test failures btw are a test violating the database interface to mock results into an iterator. Not an actual failure but a troublesome tests.


use Drupal\migrate\Plugin\migrate\source\SqlBase;

/**
 * ...
 *
 * @MigrateSource(
 *   id = "basic_migration",
 *   source_module = "doc_migration"
 * )
 */
class Docs extends SqlBase {

  /**
   * {@inheritdoc}
   */
  public function fields() {
    return [
      'name' => 'Title',
      'description' => 'Description',
    ];
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    return ['id' => ['type' => 'string']];
  }

  /**
   * {@inheritdoc}
   */
  public function query() {
    return $this->getDatabase()
      ->select('other_docs', 'docs')
      ->fields('docs', ['id', 'name', 'description']);
  }

}
neclimdul’s picture

So resolved _why_ it works for commerce. The "solution" is in SqlBase::select() but here's the problem. Neither query() nor select() have any documented requirement on the state of the statement they return but we've implemented a requirement that they setup the query options with a fetch mode. (query isn't even documented at all other then a return type so double bad.)

I don't think this is a documentation problem though. This setup smells. Most other interact with the result set from the query, the developer will fetch from one of the many fetch methods on the statement that exist explicitly to solve the fetch mode issue. Think fetchAssoc. The only reason we have this particular issue is we're trying to be clever/efficient and using the iterator behavior of the statement. That is fine but we need to enforce the fetch mode where we build the result set explicitly instead of relying on an implied behavior of other methods.

neclimdul’s picture

Thinking about this more, specifically because the developer doesn't have the ability to modify SelectInterface, this could exclude a use case where the developer consumes a select query from some other query builder to populate their source. If that query builder has an object fetch mode the developer would have to do some sort of really complicated munging or entirely rebuild the query, or replace initializeIterator which is a very large complicated bit of code they should probably not touch. All bad options.

neclimdul’s picture

Status: Needs work » Needs review
FileSize
1.88 KB

So... SqlBaseTest::testHighWater makes no assertions and doesn't really test anything. The only reason phpunit hadn't warned us about it is a random statement calls a mock method that counts as an assertion but has nothing to do with highwater.

I was tempted to delete the whole test but since we're not actually asserting anything about the $query_result this should work and is less invasive.

phenaproxima’s picture

Nice find.

This, in my opinion, blocks Migrate API stability. Not sure if that means we should upgrade it to critical...I leave that decision to the committers. But I think it should be.

phenaproxima’s picture

Status: Needs review » Needs work

I'm glad to see the test passes. Great catch, and good fix. Thank you, @neclimdul.

Let's do a few things before this is RTBC:

  1. Let's move the test from testHighWater() to its own method in SqlBaseTest.
  2. In that method, let's assert that $statement->setFetchMode(\PDO::FETCH_ASSOC) was called. (We can do this if we mock the query, which testHighWater() already does, so we can build from that.)
  3. Let's add a comment on initializeIterator() that each item returned by the iterator is expected to be an array.
  4. Fail patch!
rakesh.gectcr’s picture

Assigned: Unassigned » rakesh.gectcr

Assigning to myself for the roll

anpolimus’s picture

Issue tags: +CSKyiv18
heddn’s picture

heddn’s picture

Assigned: rakesh.gectcr » Unassigned
Status: Needs work » Needs review
FileSize
1.33 KB
2.65 KB
1.69 KB

Here's what I think was asked for in #11. Tests pass on local.

The last submitted patch, 15: 2918837_tests_only-14.patch, failed testing. View results

phenaproxima’s picture

Status: Needs review » Needs work

One small thing:

+++ b/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
@@ -216,7 +216,7 @@ public function __construct(array $configuration, $plugin_id, $plugin_definition
+   * @return array|\ArrayIterator

initializeIterator() is returning an \IteratorIterator, not \ArrayIterator. Can this simply be changed to array|\Traversable? That should cover all bases.

heddn’s picture

Status: Needs work » Needs review
FileSize
1.13 KB
3.12 KB

Updated the docs.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Looks great. Ready to go when Drupal CI passes it.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
@@ -278,7 +278,7 @@ public function prepareRow(Row $row) {
-   * @return \Iterator
+   * @return array|\Traversable
    *   The iterator that will yield the row arrays to be processed.
    */
   protected function getIterator() {

Hm, now getIterator() will not get you an \Iterator?

heddn’s picture

Status: Needs review » Needs work

Good catch. I did some research of \Iterator vs \Traversable. We are actually still returning \Iterator. I'll fix that.

heddn’s picture

Status: Needs work » Needs review
FileSize
2.64 KB
1.12 KB

It is very obvious once you look at the docs of http://php.net/manual/en/class.iterator.php. We actually don't even support simple arrays; they must be wrapped in ArrayIterator or some such. Look at calls to getIterator(). It assumes things like next, rewind, current all exist. Or look at doCount. Where it obviously assumes an \Iterator.

Updated docs attached.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Okay, good point. Back to RTBC once Drupal CI goes green.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Thanks for fixing that, looks good to me now. Yay!

  • Gábor Hojtsy committed 358989e on 8.5.x
    Issue #2918837 by heddn, neclimdul, phenaproxima, quietone, Gábor Hojtsy...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.