The Migration API provides services for migrating data from one place to another (usually, importing it into Drupal entities). The migrate module implements the general-purpose framework, while migrate_drupal builds on that foundation to provide an upgrade path from Drupal 6 and Drupal 7 to Drupal 8.
See the Update API section for information on how to update between minor versions (staying within Drupal 8.x that is).
The easiest way to understand migrate is to start with import: the source plugin provides rows. Each row is handed over to a series of process plugins and finally to the destination plugin for saving.
For each row, both source and destination have properties and values. For example, when importing nodes,
sticky is a property with possible values of 0 and 1. Some of the properties are identifiers, for example uid for users or retailer and SKU for a product import.
Identifiers enable us to track changes and more. Once the destination plugin saves the row and the destination identifier values are known, we save the values of the source identifiers, the values of the destination identifiers, and the row hash, into an identifier (id) map. The source identifier and the hash in the map allows for tracking changes for continuous migrations. The source identifier and the destination identifier allows for looking up values based on a previous migration -- if a source had user uid 44 which became user uid 56 on the destination, a source node authored by uid 44 needs to be able to look up that 56.
Highwater marks offer another way to track changes. This requires the source to have a property (the highwater property) telling when did the record change last, such as the node
changed property. If there is a highwater property, we can import in changed order and save the changed timestamp of the last successfully imported record.
All the configuration is kept in a migration configuration entity. Although Core ships these as templates, unless you need user input to create the entity, it's an overkill, just ship them as configuration.
A configuration contains four keys typically: id, source, process, destination. The first is a simple string, the identifier of the migration. The rest will each be covered in detail on their respective child pages.
There is a tutorial on writing entity migrations.
The Migrate Plus module comes with several helpful submodules, a couple of which provide example code for migrations from both Drupal and non-Drupal sources.