Last updated May 20, 2013. Created on December 22, 2011.
Edited by drunken monkey, becw. Log in to edit this page.

This document refers to the 1.0 release of the Sarnia module. Sarnia allows a Drupal site to interact with and display external data from Solr, mainly by building views of data from Solr. This is useful for large external datasets that either aren't practical to store in Drupal or that are already indexed in Solr.

Sarnia is also the name of a town in Ontario, Canada, home of the largest photovoltaic power plant on the planet.

Table of contents

  1. Installation
  2. Generating a Solr core for testing
  3. Configuring Search API
  4. Creating Views of Solr data
  5. Advanced Solr
  6. Advanced Entities

Installation

Sarnia depends on Search API, Search API Solr, and Search API Views. The full list of dependencies includes:

The Apache Solr PHP Client library should be installed according to the instructions included with the Search API Solr (search_api_solr) module--ie, it should be unpacked and placed inside Drupal's sites/all/libraries directory.

Sarnia depends on the latest 1.x releases of Search API and Search API Solr. The included drush makefile, sarnia.make, may help with downloading all of the dependencies.

After downloading the required modules, installing Sarnia will enable its dependencies. Enabling the "Views UI" module (included with Views) is also recommended.

Generating a Solr core for testing

In order to use Sarnia, you need a populated Solr core to work with. Sarnia does not care what sort of data is in the core, as long as the Solr schema specifies that some fields are stored as well as indexed. You may want to use a separate Drupal site with the ApacheSolr module and content generated using Devel Generate (a module that accompanies the Devel module) to populate a Solr core for basic testing. QA testing against your own data will better reveal any issues that relate to searching and displaying your particular data set.

For generating sample Solr data, ApacheSolr is preferred over Search API. When indexing data, Solr can be configured to index data without storing it; Search API makes the decision to index most data using Solr but to not store it (make it retrievable from Solr), while ApacheSolr stores all of the data that it indexes. In short, a Solr core generated using Search API will contain very little retrievable data, while a core generated using ApacheSolr will allow you to retrieve all properties from the core--the use case that Sarnia was built to address.

Configuring Search API

To connect your Solr core to Drupal, create a Search API server configuration.

Visit the Search API configuration section:

Admin > Configuration > Search and metadata > Search API
(path: admin/config/search/search_api)

This page lists the configured Search API servers and indexes. Normally, servers and indexes are independent, but Sarnia's purpose is to use a Search API server as a data source. Instead of the normal process of creating an index and linking it up to a server through configuration, we will create a server and then let Sarnia create and manage the index:

Search API servers correspond with Solr cores, not Solr servers. If you want to use multiple Solr cores, you will create multiple "Search API servers", even though you may have a single multi-core Solr server set up.

Add a Search API server by visiting the "Add server" link. Give the server a name:

Then select the "Sarnia Solr service" service class and fill out your Solr connection information:

Clicking "Create server" will finalize your configuration, and you will be taken to an overview of your settings:

At this point, if you were to visit the Search API overview page again, you would see your new server listed:

Instead of going back to the overview page, visit the "Sarnia" tab (highlighted in image-4.png). This page allows you to create a new entity type based on your server.

The "ID field" select box contains a list of all the Solr fields that may be suitable for use as an entity id:

You must choose a field with unique integer values; however, Sarnia has no way to determine which fields have unique values, so this choice requires some knowledge of your Solr core. This can not be changed after creating the entity type. If you are only reading from the core and not creating data or links based on Sarnia entities, it is not destructive to delete and re-enable the Sarnia entity for a particular server. Clicking "Enable" will save your configuration:

When you save your configuration, Sarnia will create a Search API index for you. You can see this index when you visit the Search API overview page:

At this point, your Drupal site is connected to Solr and can retrieve Solr data.

Creating Views of Solr data

Visit the Views UI:

Admin > Structure > Views
(path: admin/structure/views)

Create a new View using the "Add new view" link.

In the "Show" section, select the name of the index that Sarnia created; it will be titled "Sarnia: [your server name]". The form will refresh, and you can click "Continue & edit":

By default, the View's "Display format" will be "(unformatted list) of (Entity)". You can change "Entity" to "Fields" in the wizard, or after clicking "Continue & edit". The "Entity" display is provided by Entity API, but since Sarnia knows very little about the meaning of the data in each entity, it can't display the information intelligently. What you'll see is a direct printout of all of the values available from Solr for each entity. You can also access this display as an administrator at URLs like:

http://yoursite.com/sarnia/%sarnia_entity_type/%entity_id

If you do end up with a View displaying entities, you'll want to change it to display fields. Find the link in the first column that says "Show: Entity":

Change the selection from "Entity" to "Fields", and click "Apply".

All of the Solr data is available through a single field, named "Sarnia: [your server name]: Solr property". At the time that Sarnia was designed, the Views UI lacked the ability to filter fields, and long lists of poorly labeled fields are not usable. The Sarnia field bundles all Solr fields together into a single field with a combobox select element.

Find the "Solr property" field by clicking "add" in the Fields section and selecting "Sarnia: [your server name]: Solr property":

Solr property Views fields have a "Formatter" option:

This can be used to provide basic formatting options for a property. Most text fields will benefit from using the "Filtered text" formatter with the "Plain text" option, which will translate plain text line breaks into HTML breaks and URLs into links:

If you add filters, sorts, or advanced contextual filters (formerly known as an "argument"), you will again see "Sarnia: [your server name]: Solr property" as an option. When you select it, you can choose the Solr property to filter, sort, or use as context:

You may add multiple instances of the field, filter, sort, or contextual filter, which will let you combine and arrange your data according to various Solr properties.

Advanced Solr

Often in Solr, the same piece of data will be indexed multiple times for different purposes; some fields will not be suitable for search or display. Sarnia provides some "Solr Schema" configuration to manage these behaviors.

Naming conventions for these behaviors are not standard across Solr schemas, and fields aren't described in a way that is intelligible to Sarnia (ie, nothing in the schema.xml explicitly declares the relationship between ss_* fields and sort_* fields, even they are generally different indexes of the same data), so Sarnia assumes certain conventions when applying schema rules. For example:

  • Content is often aggregated into a single content field for use in fulltext search, so the content field is not available for display.
  • Content is often aggregated and heavily tokenized in the spell field for spelling suggestions or corrections, so the spell field is not available for display.
  • The dynamic base sort_* is used for fields that are processed as a single token for sorting. There may be a duplicate version of this field for search, so sort_* fields are not available for fulltext search.
  • Solr fields containing more than one token are not suitable for sorting, since they are essentially multi-value. Sorting is disabled on content and spell fields.
  • sort_* fields that correspond with ss_* fields are used instead of the ss_* when sorting; this allows click sorting on display fields in Views.

If you crafted your Solr schema yourself, you may want to check out the "Solr Schema" tab on your Sarnia Search API server configuration; otherwise, you probably want to stay far, far away :)

Advanced Entities

In the Search API server configuration for Sarnia servers/entities, you can "manage fields" on Sarnia entities. It is possible to add fields here, but there is no corresponding interface for editing field content; saving content has not been tested, even programmatically. Sarnia's relationship with Solr is read-only, so even if an editing interface were built out, it would not be possible to edit data stored in Solr.

AttachmentSize
image-1.png107.75 KB
image-2.png152.92 KB
image-3.png185.46 KB
image-4.png139.5 KB
image-5.png100.52 KB
image-6.png159.45 KB
image-7.png156.65 KB
image-8.png120.47 KB
image-9.png197.65 KB
image-10.png194.26 KB
image-10a.png300.23 KB
image-11.png190.9 KB
image-12.png223.98 KB
image-13.png262.01 KB
image-14.png261.48 KB
image-15.png275.47 KB

Looking for support? Visit the Drupal.org forums, or join #drupal-support in IRC.

Comments

dbolser’s picture

I've worked through this very nice guide, and everything is working more or less as expected. However, my view (page) just lists all items in the index. How do I actually pass a search term to the view (page)?

Sorry for the dumb question, and thanks for the very nice module.

Dan.

Hi

dbolser’s picture

Also... (How) can I configure facets to appear on the view?

Hi

dbolser’s picture

I see an answer to this question under 'bugs', but it doesn't cover much detail... for example, my search doesn't appear to be sorted by relevance (the relevance field always reads 1).

Hi

ee1’s picture

Thanks for a great module. I am a newbie to drupal and got my custom solr data easily connected to my drupal7. However after adding a new field to my solr schema I cant get it to be reflected in the sarnia index.
The sarnia-server discoveres them after I do a refresh on the solr properties though , but not the index.
My sarnia index is not marked as read only and the cron is executed.
When I create a new server the new index gets the new fields and works OK,
this introduces alot work when the schema is update though.
I have tried using both solr 3.6.1 and 4.0 but get the same error.

ee1’s picture

Finally got this to work. I didn't realise I had to go to search api and reindex :-) Thanks for a great module!

ee1’s picture

I have not been able to get the date popup search to work on my sarnia views by using of the date module.
Is this at all possible ?

cdmo’s picture

Might work for you, might not, but it worked for me to just create a Filter Criteria of "Fulltext search" and then expose that to users. With that you should be able to search the index.

-cdmo

gadgetb92’s picture

I have been trying to use Sarnia for a couple of days now but without success, and I was wondering if you would know the cause of the problem.

Installed:
Search API Solr 7.x-1.2

Sarnia 7.x-1.1-beta2 (+ patch @ https://drupal.org/files/1884572-protected_to_public.patch)
(2013134-8--update_to_latest_parent_module_version.patch)

Search API 7.x-1.8

And I have tried many new versions of Search API and Search API SOLR. And using Drupal 7. But Sarnia is not working. I have tried to create server and server was successfully created. But after creation of server its not showing the SARNIA tab on the right.

Can anybody tell me what's going wrong with my process.

And if your sarnia is working properly, which version are you using for sarnia, search API and Apache Solr.

Please HELP.
Thank you.

gadgetb92’s picture

Hi......

I am trying to configure and try to use sarnia module of drupal. But, I am not getting any documentation or handbook about sarnia.

I need to know how sarnia index non- drupal data. So, anybody can help with this problem.

Installed:
Search API Solr 7.x-1.2

Sarnia 7.x-1.1-beta2 (+ patch @ https://drupal.org/files/1884572-protected_to_public.patch)
(2013134-8--update_to_latest_parent_module_version.patch)

Search API 7.x-1.8

Need help
Thank you.

strainyy’s picture

This seems like the perfect module for pulling in external data into Drupal for display. Only issue is it's poorly supported.

So you need to play around with the versions and patches to make it work.

I'll let you know my configuration... since I went through the set up and it'll probably save somebody a lot of time.

Install these search modules:

  • Sarnia 7.x-1.1-beta2
  • Search API 7.x-1.11
  • Solr Search 7.x-1.4

Along with the other dependencies listed on this page.

Apply the following patch: https://www.drupal.org/node/2187869
(For details on how to apply a patch check out https://www.drupal.org/node/60818 - if you're on a mac)

You should be good to go. Having issues with the views filters now though :/