Remove references to nonexistent functionality from migrate.api.php

+++ b/core/modules/migrate/migrate.api.php
@@ -38,7 +38,7 @@
+ * @ref sec_migrations below for details. To migrate an entire site, you'll need to

Lets do sec_migration so its singular like the rest. Also this line is > 80chars.

Rest looks good to me.

Looks good.

+++ b/core/modules/migrate/migrate.api.php
@@ -73,25 +73,24 @@
+ * 'core/modules/action/migration_templates'.

This reminds me, we should create an issue to rename migration_templates to migrations now that is our preference?

+++ b/core/modules/migrate/migrate.api.php
@@ -73,25 +73,24 @@
+ * \Drupal\migrate\Plugin\MigrationInterface. Migration plugins of IDs and
+ * configuration for the source, process, and destination plugins, as well as
+ * information on dependencies. Process configuration consists of sections, each

Can't figure out this sentence.

I think it means:

IDs and configuration for source, process and destination plugins, as well as information on dependencies.

But wondering if it should be a list, or needs a @code block as well.

Also it's a bit odd that it goes from plugins, to dependencies, back to the process plugin sections definition again.

+++ b/core/modules/migrate/migrate.api.php
@@ -38,8 +38,8 @@
  * The configuration of migrations is stored in configuration entities, which
  * list the IDs and configurations of the plugins that are involved. See

Shouldn't this part of the paragraph be updated too?

Had a read of this, for comprehension as I don't know the doc standards well enough to comment on that, and it reads well. But, I did have two problems with it.

1. +++ b/core/modules/migrate/migrate.api.php
   @@ -26,24 +26,24 @@
     * Each row is handed off to one or more series of process plugins, where each
     * series operates to transform the row data into one result property. After all
   

   Every time I read this I think it means that the entire row is transformed to one property. Obviously, that is not the case. Even a small change like 'some row data' instead of 'the row data' would be clearer to me. But maybe that is just my interpretation.

2. +++ b/core/modules/migrate/migrate.api.php
   @@ -70,34 +72,14 @@
   + * https://www.drupal.org/node/2127611 has more complete information on the
   + * Migration API, including information on load plugins, which are only used
   

   I could not find information about the load plugins in that documentation. Did I miss it?

Load plugins are long gone.

Yes, that is more accurate in my mind. And reads better too. Thanks.

1. +++ b/core/modules/migrate/migrate.api.php
   @@ -70,34 +72,13 @@
   - * You can run a migration with the "drush migrate-manifest" command, providing
   

   Were manifests removed as a feature too? Though obviously the core docs should not refer to Drush.

2. +++ b/core/modules/migrate/migrate.api.php
   @@ -70,34 +72,13 @@
   + * https://www.drupal.org/node/2127611 has more complete information on the
   + * Migration API.
   

   This could use @link/@endlink.

+++ b/core/modules/migrate/migrate.api.php
@@ -26,24 +26,24 @@
+ * @section sec_migrations Migration plugins
+ * The definition of how to migrate each type of data is stored in a migration
+ * plugin. The plugin class is \Drupal\migrate\Plugin\Migration, with interface
+ * \Drupal\migrate\Plugin\MigrationInterface. Migration plugins consist of
+ * configuration for @ref sec_source, @ref sec_process, and
+ * @ref sec_destination, as well as information on dependencies. A migration
+ * will configure a single source and destination plugin. The process
+ * configuration consists of sections, each of which defines the series of
+ * process plugins needed for one destination property. Migration plugin
+ * definitions are stored in a module's 'migrations' directory. For backwards
+ * compatibility we also scan the 'migration_templates' directory too. Examples
+ * of migration plugin definitions can be found in
+ * 'core/modules/action/migration_templates'. Migration plugins are managed by
+ * the \Drupal\migrate\Plugin\MigrationPluginManager class.

This is a wall o' text and a bit of a runon; can we split it up into paragraphs or a list?

+++ b/core/modules/migrate/migrate.api.php
@@ -24,26 +24,26 @@
  * Source, process, and destination phases are each provided by plugins. Source
  * plugins extract data from a data source in "rows", containing "properties".
  * Each row is handed off to one or more series of process plugins, where each

Also, this section is referring to plugins, before we get to a section about plugins.

(Sorry for the multiple comments.)

Thanks @alexpott.

I applied the patch and read the whole docblock locally but I am still very confused: Are "migration plugins" a separate plugin type from source, process, and destination? If so, I think that we should explicitly state that Migrate uses four different types of plugins. Because on first read I thought "migration plugin" was a general header summarizing all three types, but the relationship between "migration plugins" and the source, process, and destination plugins is not clear.

I think this first paragraph is fine still:

 * @section overview Overview of migration                                      
 * Migration is an                                                              
 * @link http://wikipedia.org/wiki/Extract,_transform,_load Extract, Transform, Load @endlink                                                                  
 * (ETL) process. For historical reasons, in the Drupal migration tool the      
 * extract phase is called "source", the transform phase is called "process",   
 * and the load phase is called "destination". 

I do not think it needs to be directly followed by this exact paragraph:

 * Source, process, and destination phases are each provided by plugins.        
 * @ref sec_source extract data from a data source in "rows", containing        
 * "properties". Each row is handed off to one or more series of                
 * @ref sec_process, where each series operates to transform the row's          
 * properties in order to prepare them for the @ref sec_destination. After all  
 * the properties are processed, the resulting row is handed off to a           
 * destination plugin, which saves the data. Source, process and destinations   
 * plugins are brought together to extract, transform and load a specific type  
 * of data by @ref sec_migrations. Migration plugins also contain information on                                                                               
 * dependencies.

The information about plugins can easily be removed from that paragraph. What is important for the initial section is to define the terms "source", "process", "destination", "rows", and "properties". Whether they are plugins or not is implementation.

And if "migration plugins" are a separate category from the other three, then we need a much clearer definition of what they are, and I'd suggest a different name because it would seem to encompass all types of plugins involved in Migrate. If they are not a separate thing, then we should instead say that they are divided into three types.

I can try to help rewrite the docs too once I understand it.

I think it is shame that we didn't change the other plugins to match the ETL (extract, transform, load) language as that is the lingua franca of tools that perform similar tasks.

I think this was considered, keeping similarity with the D7 module was one reason not to, another key reason was Drupal already used the term "load" to mean the exact opposite of what L means in ETL.

I think this is actually major given how out of date it is. :)

+++ b/core/modules/migrate/migrate.api.php
@@ -21,29 +21,25 @@
+ * "properties". Each row is handed off to one or more series of process
+ * plugins, where each series operates to transform the row's
+ * properties. After all the properties are processed, the resulting row is

This sentence is kind of confusing to me. Could we simplify it to something like: "Each row is handed off to one or more process plugins which transform the row's properties."

+++ b/core/modules/migrate/migrate.api.php
@@ -21,29 +21,25 @@
+ * A source plugin, one or more process plugins and a destinations plugin are

"destinations" should be singular here I think.

I've attached a patch with these two minor changes. But otherwise I think this does a good job of explaining the Migration ecosystem at a high level and providing enough direction for someone to get started without being overwhelming.

Per benjy's https://www.drupal.org/node/2687003#comment-10991539 above, and my https://www.drupal.org/node/2513398#comment-11053513, I suggest some clarification on the word "load".

I just ran into this again while trying to look up some information about writing a migration. I think the text here is good to go, and the sooner we can get the docs updated the better. Right now when you land on https://api.drupal.org/api/drupal/core%21modules%21migrate%21migrate.api... it just really confusing.

Moving this to 8.2.x-dev since I think it should be fixed there first. I also just tested, and FWIW, this same patch applies cleanly to both 8.2.x and 8.1.x.

Finally, marking as RTBC because in my opinion this is ready to roll. Thanks.

One last gasp for the random fail? Retest queued...

Restoring RTBC, "needs work" was just our old random test fail...

I applied this patch just so I could read through everything easily, looking great. +1000 from me, the current state of the API doc has been stale since 8.0.x and we should get this in ASAP.

Random test fail - #2749955: Random fails in UpdatePathTestBase tests, apparently.

So I'm a bit frustrated that the work in #19 and #20 was totally ignored, because it represents a significant amount of work. I discussed this with @alexpott and he created #2755865: Describe Migration plugins and their relationship to source, process and destination plugins for that needed work. Thus, that we can at least remove stale information about manifests and such here. I've rescoped this issue to that only.

So, for this patch, I've reviewed whether the individual sections are clear on their own after being changed. Here are some small typos I fixed on commit:

1. +++ b/core/modules/migrate/migrate.api.php
   @@ -17,33 +17,30 @@
   + * A source plugin, one or more process plugins and a destination plugin are
   

   Oxford comma.

2. +++ b/core/modules/migrate/migrate.api.php
   @@ -17,33 +17,30 @@
   + * brought together to extract, transform and load (in the ETL sense) a specific
   

   Oxford comma.

3. +++ b/core/modules/migrate/migrate.api.php
   @@ -17,33 +17,30 @@
   + * For backwards compatibility we also scan the 'migration_templates' directory
   + * too. Examples of migration plugin definitions can be found in
   

   Also too.

Diff on commit:

diff --git a/core/modules/migrate/migrate.api.php b/core/modules/migrate/migrate.api.php
index cd7fcd2..d5e14f4 100644
--- a/core/modules/migrate/migrate.api.php
+++ b/core/modules/migrate/migrate.api.php
@@ -29,14 +29,14 @@
  * transform the row's properties. After all the properties are processed, the
  * resulting row is handed off to a destination plugin, which saves the data.
  *
- * A source plugin, one or more process plugins and a destination plugin are
- * brought together to extract, transform and load (in the ETL sense) a specific
+ * A source plugin, one or more process plugins, and a destination plugin are
+ * brought together to extract, transform, and load (in the ETL sense) a specific
  * type of data by a migration plugin.
  *
  * @section sec_migrations Migration plugins
  * Migration plugin definitions are stored in a module's 'migrations' directory.
- * For backwards compatibility we also scan the 'migration_templates' directory
- * too. Examples of migration plugin definitions can be found in
+ * For backwards compatibility we also scan the 'migration_templates' directory.
+ * Examples of migration plugin definitions can be found in
  * 'core/modules/action/migration_templates'. The plugin class is
  * \Drupal\migrate\Plugin\Migration, with interface

The documentation still needs to introduce the migration plugins and their relationships to other kinds of plugins in a more clear way, because the new order does introduce new confusion. We will address that in the followup issue. Additionally, this new paragraph is very difficult to understand:

+++ b/core/modules/migrate/migrate.api.php
@@ -17,33 +17,30 @@
+ * Migration plugin definitions are stored in a module's 'migrations' directory.
+ * For backwards compatibility we also scan the 'migration_templates' directory
+ * too. Examples of migration plugin definitions can be found in
+ * 'core/modules/action/migration_templates'. The plugin class is
+ * \Drupal\migrate\Plugin\Migration, with interface
+ * \Drupal\migrate\Plugin\MigrationInterface. Migration plugins are managed by
+ * the \Drupal\migrate\Plugin\MigrationPluginManager class.

It's unclear what the relationship between the sentences in this paragraph is. That will also be addressed in the followup issue.

Meanwhile, committed 0bbcdf4 and pushed to 8.2.x and cherry-picked to 8.1.x. Thanks!