Creating and Editing Import Feeds

Last updated on
10 February 2022

Content

Overview

Creating a new import feed in Drupal 8 takes place in three steps.

  1. Create a "Feed type" (admin/structure/feeds) that describes parameters of import such as source type ( for example, csv), frequency, etc.
  2. Map "sources" from the import data structure to "targets" of the content/user/taxonomy/entity that you are importing to (admin/structure/feeds/*/mapping)
  3. Create a "Feed" of the "Feed type" you want (feed/add), select source (file or url) and import actual data.

You can then run the import from the Feed page itself (instead of from a separate import page like in Drupal 7). This means that creating a Feed in Drupal 8 is a lot like creating nodes (i.e., you first create the content type and then create new content of that type). 

Creating a Feed type

On the Feed types overview page (admin/structure/feeds), you can see all of the Feed types that have been created so far. If this is a new install, there won't be any. To create a new Feed type, select "Add feed type."

This brings you to the Feed type creation page where you have four configuration options:

  1. Basic settings (name, description, import period)
  2. Fetcher (where the content is coming from)
  3. Parser (the feed format, e.g., RSS/Atom, CSV, OPML, XML)
  4. Processor (what type of content the feed is creating, e.g., nodes, users, taxonomy terms)

These options are described more fully below.


The creation page for a new Feed type.

Basic Settings

  • Name: The name of the Feed type you're creating
  • Description: A short description of what the Feed type does
  • Import period (found in secondary settings tabs): How often the import should run; choose "Off" to only have the import run manually

Fetcher

Where you're importing (or "fetching") content from.

  • Choose from DirectoryDownload, and Upload
    • Directory: import content from a file or from a series of files already on your website (when creating the Feed later, you'll enter a directory or file path)
    • Download: import content from a URL
    • Upload: import content from a file
  • Based on the fetcher you choose, you'll have different secondary settings in the tabs:
    • Directory
      • Allowed file extensions (e.g., csv, xml, txt, etc.)
      • Allowed schemes (public files, search through sub-directories)
    • Download
      • Auto detect feeds (if the URL you give is an HTML document, it will attempt to extract a feed from it)
      • PubSubHubbub
      • Request timeout
    • Upload
      • Allowed file extensions
      • Upload directory (where the file you upload will get stored)

Parser

What the feed format is.

  • Choose from OPML, CSV, Sitemap XML, and RSS/Atom
    • OPML: "Outline Processor Markup Language". See OPML on Wikipedia
    • CSV: See Comma Separated Values on Wikipedia
    • Sitemap XML:  See Sitemaps on Wikipedia
    • RSS/Atom: See RSS on Wikipedia
  • Only the CSV parser has secondary settings:
    • Delimiter: comma, semi-colon, tab, etc.
    • No headers: check if you don't have a header row on the file you import, during mapping you then describe the source columns with numbers: 0, 1, 2, etc.

Processor

The type of content the feed will create.

  • Choose from Node, User, and Term
    • Node: you'll need to choose which content type you want
    • User: creates users
    • Term: you'll need to choose which vocabulary to add terms to
  • The processors have similar secondary settings in the tabs:
    • Update existing contents/users/terms: whether to update or do nothing when the feed contains the same item the next time it runs; this requires you to provide a unique ID in the mapping
      • Do not update: items in the feed with unique IDs that already exist will be ignored
      • Replace: does the same as update, this option will be removed in the future
      • Update: if different, existing items with the same unique IDs as items in the feed will be updated with new field data from the corresponding items
    • Previously imported items: choose what to do with previously imported items that are no longer in the feed (e.g., delete, unpublish, etc.)
    • Expire contents/users/terms: choose how long to keep imported items before deleting them (the default is "never" which means no items will be deleted)
    • Owner (only for entities that can be "owned", for example "Node"): choose which user will be listed as the author of the node/entity

Mapping Sources to Targets

After you click save, you'll notice several new primary tabs at the top of the Feeds type page you just created.


A newly created Feed type.

These include:

  • Mapping
    • Choose how to map data from the selected feed into the content created by your import.
  • Manage fieldsManage form display, and Manage display
    • These fields mimic the functionality and purpose they have on other content in Drupal 8. They can be edited to change the form and display of the feed, but they are not necessary for the import functionality. Take a look at the content documentation for more details on these sections.

Mapping

You'll need to edit the mapping of your Feed type before a Feed of that type will import content successfully. The parser you selected when creating the Feed type will affect the how the mapping will work. If you selected CSV, you'll define the "sources" based on the headers in your CSV file and map them to the target fields of your content type (or user or term). If you selected RSS/Atom, OPML, or XML Sitemap, you'll map predefined "sources" to the target fields.

  • RSS/Atom
    • Predefined sources: Title, Description, Published date, Item GUID, Item URL (link), Item latitude, Item longitude, Author email, Author name, Categories, Content, Feed URL (link), Feed description, Feed image, Feed title, and Enclosures
  • OPML
    • Predefined sources: Title, URL, Categories, Site URL, and Feed: Title of the OPML file
  • XML Sitemap
    • Predefined sources: Item URL (link), Change frequency, Priority, and Last Modification Date
  • CSV
    • There are no predefined sources for the CSV parser. You define them all based on the headers of the file you upload. If you selected "No headers" in the parser settings when creating the Feed type, the source for each target field will depend on the order of columns in your file (starting with 0, 1, 2, etc.).

Example 1: CSV File with Headers

If your CSV file looks like this...

id title author pubdate
1 Cat GIFs Alex Smith 2018-03-12
2 A Few More Cat GIFs Jesse Taylor 2018-03-20
3 Even More Cat GIFs Alex Smith

2018-04-02

...you could set up your mapping like this:

id => Feeds item: Item GUID
title => Title
author => Authored by
pubdate => Authored on

IMPORTANT: header name is case sensitive! If your mapping name is "product_name" then your CSV file must use "product_name" as header, not "Product_name" or "PRODUCT_NAME"

Example 2: CSV File without Headers

If your CSV file looks like this...

1 Cat GIFs Alex Smith 2018-03-12
2 A Few More Cat GIFs Jesse Taylor 2018-03-20
3 Even More Cat GIFs Alex Smith

2018-04-02

...you could set up your mapping like this:

0 => Feeds item: Item GUID
1 => Title
2 => Authored by
3 => Authored on

Selecting a Target

To start mapping, select a target from the drop-down menu.


An empty Mapping page for a Feed.

These targets are all Drupal data fields and are pulled in based on the processor you selected when creating the Feed type (e.g., the user processor pulls in fields like name and email, while the node processor pulls in all the fields from the content type you selected). After selecting a target, text and options may appear in the "source", "summary", "configure", "unique", and "remove" columns for that row.

  • Source: as noted above, this is the data you'll be importing to the selected field
  • Summary: contains information about the format of the field or how a reference will work (variable per field)
  • Configure: allows you to change the format of the field or how a reference will work (variable per field)
  • Unique: indicate that this field should be used to define the uniqueness of a feed item (this can be stored in the "Feeds item" target, the title of the item, or a custom field on the targeted content type--only certain types of custom fields are eligible though)
    • Note: Make sure to identify a unique field if you want to update or ignore items when an import is run or when you plan to do continuous imports. If you do not, you might create many duplicates of the same items.
  • Remove: selecting this will remove the row


An example of the unique field boolean for a mapped attribute.

Creating and Deleting Custom sources

You can select existing source or add a custom source. You can create new custom source in the Mapping Tab. From the source selector, choose "New [type] source...", where "[type]" is the type of source to add.



All custom sources are displayed in tab "Custom Sources" and can be edited or deleted. You cannot create new custom source in this tab, use Mapping tab as described above.



Using Blank Sources to Tamper Data

Blank source allows you to create a placeholder to tamper incoming data with Tamper Plugins using Feeds Tamper module

Suppose you have a CSV column 'SOURCE' which you want to use for several fields in Drupal. This is how this can be done.

Mappings:

SOURCE -> Temporary target 1
Blank source 1 -> field 1
Blank source 2 -> field 2

The order of the mappings is important.
You can add as much Blank source * as you want. Just make sure it starts with 'Blank source ' and all is well.

Tamper:

First, for each Blank source x -> field x use the Copy source plugin to copy from SOURCE to field x, or use the Rewrite plugin to use tokens to populate the field.

Then add tamper plugins as you wish.

Cloning a Feed Type

You can use module https://www.drupal.org/project/entity_clone to clone a feed.

Creating a Feed

After you've mapped the fields you want (you can return and edit the Feed type and mapping later if you need to change anything), you'll need to create a Feed (admin/content/feed). Similarly to the Feeds type page, on this Feed overview page, you'll see all of the Feeds you've already created.


The 'Add feed' on the Feeds tab under 'Content'.

Click "Add feed" to create a new Feed.


Selecting a Feed.

If you've created multiple Feed types, you'll be asked to select which Feed type to use. If you've only created one Feed type, you'll be taken directly to that Feed creation page.


The Feed Creation page.

On the Feed creation page, you'll need to enter a title for the Feed and either a URL (if the Feed type has a Download fetcher), a directory path/server file (Directory fetcher), or upload a file (Upload fetcher). If using a CSV Feed type, your selections from the default secondary parser settings (delimiter and headers) will also display so that you can override them. You can also check a box to deactivate the Feed, which will prevent a recurring importer from running (it will still run if you do it manually). When activating a feed you previously deactivated, you'll also need to manually run the import in order to reset the import schedule. If you added any other custom fields on the "Manage fields" page (not the "Mapping" page), they'll show up here as well.

After you've added those fields, you can select "Save and import" or just "Save." Either way, you'll be brought to the page for the Feed you just created, where you can manually run the import and/or delete items created with that Feed.


Processing the creation of new entities from a Feed.


The Feed entity page post import.

Notes

  • Screenshots and some import process documentation were reused form the work of #2861994.

Help improve this page

Page status: No known problems

You can: