Port Sarnia to Drupal 8/9 (UPDATE: Now part of search_api_solr 8.x)

Yes, there are plans to port this to Drupal 8. As far as I've tried, the dependent Search API and Search API Solr modules are still under development.

I'm in the middle of porting another module at the moment, but I was planning on helping with this when I'm free. It might not be soon though. The other port is pretty complex and it's taking up most of my free time.

I was really hoping to add some tests to Sarnia before porting it. It makes me concerned to do major changes to a module, like what has to happen during the upgrade, without tests. Of course, we don't have to do that. We could just port it as-is.

It kind of sounds like you've already been working on a port, @jmdeleon. If so, is it on your GitHub account?

On a tangential note, have you thought of making an official release of the 7.x branch? I think we got most/all of the known bugs fixed. The last beta has been available for nearly 3 months. I think it's probably good to go.

I took a whack at getting a working and relatively stable Search API Solr stack (without Sarnia) going for Drupal 8, a couple of days after the official D8 release, and was finally successful. So work on a D8 version of Sarnia can now move forward a bit more. This is taking more of a back-seat to implementation work with Sarnia I am doing in D7, but my schedule in this respect is clearing up, so I can tackle Sarnia for D8 in the early part of December.

Re - official release: I will probably bump up the beta7 to an official release shortly, with no new functions added from beta7.

Hi, I am very interested in using Sarnia for a D8 project, do you have a schedule or a timeline for when Sarnia will be ported to D8? Thanks!

I can't speak for @jmdeleon, but I'm still porting the other module I talked about in #3. I don't think I'll be done until closer to the end of January. At that point I should be available to work on the Sarnia port.

@dcam That sounds great. I'm a Drupal newbie, but if I can help out in any way please let me know (I have a lot of experience with Solr).

Maybe run an initial pass through the Drupal module upgrader.

https://www.drupal.org/project/drupalmoduleupgrader

The search_api family is close to beta. So that seems to be a good starting point for the port.

@sylus: I did do a run thru the module upgrader very early this year and decided that the result wasn't in any way usable. @mkalkbrenner's note that search_api is much further along in development would seem to be an opportunity to take a shot at this again...

I'd welcome anyone who'd be interested in taking this up!

I want to give it a shot, but I've been putting off finishing another D8 port. I'm trying to wrap it up now. Anyway, I don't think that I should start another project until that's done.

Thanks @dcam..! I have plenty of paying work with Sarnia 7.x that I simply don't have the time to pursue an 8.x port, as much as I'd like to to do it...

Just a little tease.

In case you're wondering, doing this port is not for the faint of heart. So don't get too excited. My rough estimation is that I've only completed about 5% of the work required.

That is awesome, I'd love to help as well with this, I don't suppose you have a sandbox? I will be probably freed up in a few weeks to tackle some of the port ^_^

Sure, I'll get it pushed to my GitHub account later and post the link.

BTW, I promptly re-broke that form not long after I took that screenshot when I started converting Sarnia entities into configuration. lol

Woot yeah I'd love to see it, I ran Sarnia through Drupal Module Upgrader myself and fixed some minor things but ended up getting stuck at $server->getRemoteFields since the Sarnia Entity class hasn't been moved over.

For my own edification did you declare a whole new backend extending from the search_api_solr?

/**
 * @SearchApiBackend(
 *   id = "sarnia_external",
 *   label = @Translation("Sarnia External"),
 *   description = @Translation("Index items using Solr External Index")
 * )
 */
class SarniaExternalBackend extends SearchApiSolrBackend implements SolrExternalBackendInterface {
...
}

RemoteFields() was fun to figure out. I had to work out what changes have happened to search_api_solr in order to fix it. When they made the switch to Solarium, they got rid of all the custom connection classes. Sarnia relied heavily on them to be able to communicate with the Luke request handler, specifically in order to get field info. At first I didn't see any helper methods in Solarium for communicating with Luke and I became concerned I'd have to write them myself. I could have done it, but I'm no Solr wizard and having to write that would have required extra learning beyond doing the module upgrade that I wasn't expecting.

Anyway, turns out Solarium does have one method for getting info from Luke. I think it returns all of its info, but that was totally ok with me. I wrote one or two methods to parse out the field info and was back in business. Well, I also had to update Sarnia's SolrField class. In 7.x it extended search_api_solr's SolrField class, but that went away too. So, I just copied in the missing methods from 7.x.

As for creating a backend, yep, that's what I did. I wouldn't consider it "new." I just ported the old class in service.inc, which extended search_api_solr's class in 7.x. I don't know if we're talking about the same thing.

Ah cool thanks for addressing my question! That was exactly what I wanted to know.

Since your way ahead of where I am, I will stop my port and work on yours when available. :) I just wanted to see what the work entailed and your quite right there is a lot.

I pushed my repo to a sandbox on GitHub.

One thing you should be aware of is that I've made a few changes that haven't been committed yet. At least one of these is probably critical to making it work. Should you decide to install the module and try it out, you may find that it's broken without that change. It's a one-line edit and I'm honestly hoping that whole function can just be removed at some point in this process, which is why I hadn't bothered yet. If you want to install it, then I guess it won't hurt to commit the change. I've just been trying to avoid making small one-off commits like that in order to keep the log clean.

Speaking of the log, it should be able to act as something of a CHANGELOG for you. I've tried to note in the commit messages which function I've moved and what class it got moved into. So if you're trying to find something, check the log first. Of course, functions like form submits simply got moved along with their form builders and aren't specifically noted.

I didn't use any sort of conversion tool. My process has been to start with one page, load it in the browser, see what breaks, and fix it. I started with the Search API Add Server form and went from there. I've ported a few of the Sarnia pages that are children of the Search API server page, including:
/admin/config/search/search_api/server/{search_api_server}/sarnia
/admin/config/search/search_api/server/{search_api_server}/sarnia/properties
/admin/config/search/search_api/server/{search_api_server}/sarnia/schema

I'm currently working on the rules configuration form at /admin/config/search/search_api/server/{search_api_server}/sarnia/schema/{behavior}. It's partly working at this point but nowhere near ready for committing.

I haven't worked on the Field API pages yet. Also, I haven't enabled the Enable Sarnia form to auto-create an index yet.

One last thing to note is that I've made a significant change from the 7.x version. In 7.x, an entire new entity type is created for each different Sarnia index. I have changed that so that 8.x creates a new bundle on a single Sarnia entity type for each index. I've only seen one vague reference to why it was done that way in 7.x, so I don't yet know the implications of making this change. It just made sense to me. It could end up being a mistake.

Actually, I haven't committed the Sarnia entity class yet, so that's probably another way in which the sandbox won't work yet. Sorry, I'm not accustomed to working on a project like this with others. I typically work alone. I'll see if I can't get it to an installable state for you, but I may not have time until next week. This will be a busy weekend for me.

No worries I likely will only be looking over the commits this weekend so no rush at all at getting it back to installable state. Just a learning process for me :)

Thanks for pushing the repo! If and when I do any commits will just submit a P.R. from a feature branch. :)

As an aside and maybe something to consider after the port. Does the name "Sarnia" really make sense for 8.x anymore?

I would personally prefer some a bit more SEO friendly and a namespace that is in line with the other solr module namespaces. Perhaps something like "search_api_solr_external" or something?

Anyways just thinking out loud for now :)

I share your sentiment. I've had the exact same thought. Facet API updated its name during the 8.x cycle. If we want to make the change, now is an ideal time to do it. I just hadn't brought up the subject yet.

I've had some personal stuff come up. I haven't had any time to work on this over the last week and a half. Sorry. I'll try to do more this week. =(

Hey no worries at all, honestly the work you did was amazing, I ended up learning alot for other D8 ports just by following your well structured commits. :)

I did end up having a problem because the main sarnia bundle isn't present maybe we can have a dev branch with all of the extra commits so can keep the main branch clean while working away.

I'd love to tackle the views handlers once the main bundle is going again :)

I'm glad you found it helpful. Your idea about another branch is good. I'll try to remember to do that this evening.

I committed three small or partial changes to the branch temp_fixes and pushed it to GitHub.

Is the d8 port usefull at the moment?

i was trying it out but it cant connect to the external solr.

No. It's not usable at all. Only a small percentage of the upgrade work has been completed. I haven't had much time to work on it in recent weeks.

I was on vacation last week and bored most of the time, so I got some good work done on the port. That's not to say that everything is working and happy now. I just got some additional parts of the conversion done. I've pushed the changes to the 8.x-1.x branch to GitHub, but there are still some parts of it that won't work without some uncommitted stuff (sorry). I haven't updated the temp_fixes branch with those uncommitted changes yet, but I'll try to get that done sometime this week if I can.

Here's a short list of things I did:
1. The schema rules form was partially converted. The form looks good, but isn't functional yet. I was working on it without Internet access, so I couldn't look up API changes that were necessary to upgrade the validation and submit functions. Future changes include upgrading those functions. Eventually, the rules need to be converted to configuration too. I looked into it, but decided not to do it at this time.

2. Sarnia entity types will now be listed and managed at admin/structure/sarnia-types. This is different from the D7 version where each entity type and its Field UI pages were listed under the corresponding Search API server page. I tried for more than a day to get it all to work the same as it has in the past, but eventually decided to give up and go this different route (no pun intended). Getting the Field UI pages to appear correctly was fraught with problems. It might be possible to resolve them with unnecessarily complex alteration of the Field UI routes, but after doing a lot of investigation I got tired of it.

So, I made the decision to move the entity management out of the Search API, which meant that it could be done quickly and easily like any other entity type that's not nested inside the management of another entity type. I can see this being slightly less convenient for site builders. The Field UI pages won't be right there to see once the entity type is created. On the other hand, it might be more convenient because lower-level site admins can be given permission to make changes to the entity type in the new structure page without also needing permission to access the Search API pages. So it might be a decent trade-off.

3. The Solr Document field type and empty widget have been converted to plugins. Conversion of the field formatters has begun.

On the subject of renaming things, calling the entity types "Sarnia type" started to get on my nerves while I was building out the new entity management pages. I didn't do anything to rename them because I wanted to consult the community first.

My initial thought was to call them "Solr documents." That term is already in use by the module as the type of field that we provide for displaying Solr content. So I thought it might be familiar as well as relatively descriptive. If there's a better term for content that lives in Solr, then that might be the best name to use. Maybe "Solr record" would also fit. I don't know. Please give me some feedback on this.

Note that I'm not asking about renaming the module as a whole. At the moment I'm just concerned with providing a descriptive name for the entity type that we provide, since this name is actually going to be visible to admins now.

Was taking a look at this today and is coming along great! Thanks for all of your amazing work on this @dcam. I did a small P.R. with some possible improvements but am still just getting oriented tbh.

As for the renaming of Sarnia type to Solr documents, I am definitely fine with eventually renaming it as I can attest there has been some confusion for me. ^_^

I mentioned in the P.R. I was able to see records at least being returned in views by adding a dpm to the postQuery() so once we add those fields to the Search API index things are really going :)

GitHub P.R.: https://github.com/dcameron/sarnia/pull/1/files

I noticed one thing while working on the SarniaStorage.php class (updated my P.R. with work) and this is in regard to an earlier comment:

In 7.x, an entire new entity type is created for each different Sarnia index. I have changed that so that 8.x creates a new bundle on a single Sarnia entity type for each index. I've only seen one vague reference to why it was done that way in 7.x, so I don't yet know the implications of making this change.

I noticed one area where this might cause an issue but hopefully we can work around it:

In the SarniaControlller::load (D7) controller we have calls like:

      $index = search_api_index_load($this->entityInfo['bundles'][$this->entityType]['search_api_index']);
      $id_field = $this->entityInfo['bundles'][$this->entityType]['id_field'];

I have attached an image showing the entityType sarnia_pd_ati:

You can see we are simply able to get the entity type as a key, but in the D8 implementation our "$this->entityType" would be Sarnia and and potential bundles. I am, not sure how we can get the correct bundle information. I was thinking in our stash we could add this information? I still think having multiple bundles on a sarnia entity is the correct way to go though!

I am hoping it was only done this because in the past Search API Indexes weren't bundle sensitive but that has since long been addressed. Although it would be create to have @drunken_monkeys statement on this.

My assumption has been that the historical reason for a unique entity type per index has been that Drupal usually expects entity ID numbers to be tied to the entity type, not the bundle. With Sarnia, the potential certainly exists to have multiple entities with the same ID if they come from different indexes or if the same index gets added twice. I don't actually know what would happen in that case, which is why I pushed forward with the 1-bundle-per-index strategy, figuring that eventually I would test it to see what happens.

Ah that makes sense :) I also think that multiple bundles on a single sarnia entity is a much cleaner approach so really glad we going this direction. ^_^

Okay I am having one annoying issue and wanted to see if anyone had any idea.

Looking at the code below:

    // Create the field storage if it doesn't already exist.
    if (empty(FieldStorageConfig::loadByName('sarnia', 'solr_document'))) {
      $field_storage = FieldStorageConfig::create([
        'field_name' => 'solr_document',
        'entity_type' => 'sarnia',
        'type' => 'sarnia_solr_document',
        'locked' => TRUE,
      ]);
      $field_storage->save();
    }

    // Create the field if it doesn't already exist.
    if (empty(FieldConfig::loadByName('sarnia', $this->id, 'solr_document'))) {
      try {
        $field = FieldConfig::create([
          'field_name' => 'solr_document',
          'entity_type' => 'sarnia',
          'bundle' => $this->id,
          'label' => 'Solr Document',
          'widget_type' => 'sarnia_no_input',
          'widget' => [],
          'required' => TRUE,
          'locked' => TRUE,
          'display' => [],
        ]);
        $field->save();
      }
      catch (FieldException $e) {
        drupal_set_message('An error occured while attempting to create a Solr Document field for bundle ' . $this->id . ': ' . $e->getMessage(), 'error');
      }

The FieldStorageConfig operation does work but the FieldConfig save returns:

Drupal\Core\Field\FieldException: Attempt to create a field solr_document that does not exist on entity type sarnia. in Drupal\field\Entity\FieldConfig->getFieldStorageDefinition() (line 286 of core/modules/field/src/Entity/FieldConfig.php).

Yet when I do the FieldStorageConfig + FieldConfig on a node entity type + page bundle everything works. Are we sure we should be using @EntityType and not @ContentEntityType? Or is something else missing?

<?php

namespace Drupal\sarnia\Entity;

use Drupal\Core\Entity\ContentEntityBase;
use Drupal\Core\Entity\EntityTypeInterface;
use Drupal\Core\Field\BaseFieldDefinition;
use Drupal\field\Entity\FieldStorageConfig;

/**
 * Defines Sarnia entities based on Solr data.
 *
 * @EntityType(
 *   id = "sarnia",
 *   label = @Translation("Sarnia"),
 *   bundle_label = @Translation("Sarnia type"),
 *   handlers = {
 *     "storage" = "Drupal\sarnia\SarniaStorage",
 *     "view_builder" = "Drupal\Core\Entity\EntityViewBuilder"
 *   },
 *   entity_keys = {
 *     "bundle" = "type"
 *   },
 *   bundle_entity_type = "sarnia_type"
 * )
 */
class Sarnia extends ContentEntityBase {

  /**
   * {@inheritdoc}
   */
  public static function baseFieldDefinitions(EntityTypeInterface $entity_type) {
    $fields = parent::baseFieldDefinitions($entity_type);

    return $fields;
  }

  /**
   * {@inheritdoc}
   */
  public function getType() {
    return $this->bundle();
  }

}

Actually, we should be using @ContentEntityType. I've had that change waiting uncommitted locally for a few months now, I think.* I'm not sure that it will fix the error and some more testing needs to be done. I saw it on a fresh install tonight too, but that was with your merged branch - the changes I mentioned were stashed. Still, I think the error will probably appear anytime a new Sarnia entity bundle is created. That means either on a fresh install or after the bundle has been deleted and the cache has been rebuilt to clear out any lingering config.

*edit: I committed the change a few minutes before posting this and it's available on GitHub.

FYI, I moved forward with renaming the Sarnia entity and bundle to Solr Document. Accordingly, I changed the Solr Document field type and widget to Solr Field. The field name is slightly confusing, mostly because "field" describes the type of thing we're creating and the type of thing it displays. Solr calls pieces of data within a document "fields" too, like any database does. I'm open to suggestions for a better name.

Ah yeah great news the @ContentEntityType and that should automatically get us a working datasource in search api! Also definitely agree with the field name changes as at least more helpful :) Thank you so much! ^_^

I should mention that @drunkenmoneky took the time to comment in the related issue (datasources) and he made some architectural points related to sarnia explicitly.

#2852627-8: DataSources + @EntityType support

If you just want to make data searchable with the Search API, you should use a datasource – that's what they're for.

However with our field level additions / configuration and with there being a well laid path to upgrade following entity api maybe this can be looked into in a subsequent branch.

Ok. Good info. I'll take a look at the datasources.

I took a look at the datasource myself and honestly I think it would have helped if we were told about this later as it is just delaying our work and the refactor seems to be more confusing at least to me. I think I recommend we continue the same path we are going. Did you have better luck @dcameron at taking a look the datasource route?

I think the last thing we need to do is get the SolrFields to be properly mapped, which is proving difficult.

Another issue with using just the datasources at least to my mind is say if I wanted to be able to rate some of the solr documents. This would work as a simple field based on the way we are currently mapping, but I don't see how it would work with a datasource.

Unfortunately I am still blocked on getting the Solr Fields properly mapped. Would appreciate any guidance in that regard ^_^

Sorry I hadn't gotten back to you. I've been working on the datasource plugin and I've had some success. I've also run into a few roadblocks, but I've worked around them. I think I'm pretty close to being able to display Solr data with it. My project manager mentioned last week that management wants the various Sarnia-entity-displaying sites to be consolidated into one (they all pull data from the same index). I told her that if we want to do that in D8 then I would need to be able to spend work time on developing these modules, so I might get to do it soon. We'll see. Otherwise, I've been too swamped to be able to work on any of this during work hours.

As for attaching other fields to Sarnia-type entities, did you know that all fields on an entity must share the same storage backend in D8? I think the only way you could accomplish that would be to build out an actual CRUD storage backend for Solr.

I'll have to look into what you mean about mapping the Solr fields. I've shifted focus for long enough that I don't remember we were at with the port.

Thanks @dcam for the status update that is great news that you got it working with the datasource. I was a bit worried about how complicated it would be to get it to work, glad you were able to work around.

When you get a chance hopefully you can update your github repo and I'll take a look :)

Thanks again for all of the great work!

At precisely midnight today I refreshed my D8 test site to see this:

That is data from Solr being displayed by Views inside of Drupal 8 via a Search API datasource. It's not perfect - far from it in fact, but it does have some minimal functionality now. I wouldn't be surprised if it's also pretty fragile right now.

To be perfectly honest, even I don't entirely understand how it works. I pretty much started implementing classes wherever it asked for them until I got to this point.

I should also mention that this is not a port of Sarnia. Although I did copy a little bit of Sarnia's code along the way, this is an entirely new module that was built from scratch. It does not create entities from Solr data. Consequently, it's also a smaller module right now because it doesn't have to contain all of the entity code.

I had to make a lot of additions to the module in order to get it from the "configurable in the Search API admin pages" stage to the "actually outputting data through Views" stage. I didn't make my usual incremental commits because I had no clear path that let me know what I was doing was correct. As a result, I have a several uncommitted files in the repo right now which is why I'm not providing you with a link to the module at this time. I'll get those squared away tonight and get back to you with it.

Thank you so much @dcam for your hard work, very excited to see all the amazing progress. Will wait to hear back from you and take a look.

Definitely curious to see the custom datasource logic :)

Here it is: https://github.com/dcameron/search_api_solr_datasource.

I put some basic setup instructions (also warnings) in the README file. Let me know what issues you run into with it.

Hazaa!

Yeah I might have noticed it last night and was playing around, and it works great! I only just quickly connected it and checked views but the fields were all displayed to be added to the interface :)

I will give it a more thorough run-through this weekend but results look great so far! I also love how much simpler it is by comparison to the sarnia module :)

Thank you so much for the great work!

Okay I gave it a thorough review and overall works great! I think the only problem I have is if I specify a field and it is empty or not present I get a weird default display in the view.

That's the output of the Search API Field object's __toString() method. It does that if it can't render any output. Not sure why. I just accepted it and moved on without investigating.

Fields have to be added to the index before they can be displayed. I added that info to the README file on Friday and created an issue to see if the Views field list can be limited to the index's fields.

Yeah the fields are displaying perfectly and I followed your steps to the letter, thanks for providing them ^_^. The only one that almost got me was:

- It may be necessary to check "Retrieve result data from Solr."

At least for me it was mandatory that this was checked then all I needed to do was add the fields I wanted then proceed to the view.

I will be giving the module some more review during this week as I will be starting to make some more core displays with it. I'll try to fix / provide patches over in your github issue.

Hopefully once your comfortable with the improvements and where we are at in a few weeks, could promote it to a full drupal project :)

Thanks again!

I also tryed the search_api_solr_datasource project and i can't get it working properly

I can connect to Solr; I can select the fields i want to 'Index'. I can select the fields in the view, but i don't get any output
Using Solr 6.3.0 and a clean drupal 8 install.

Any thoughts?

So you're not even getting the "[field_name]: indexed as type xxx" output in Views?

The one other instance I've seen where I haven't gotten any output at all is when I have Views try to output a multivalued field. So that's one issue I know about. I'm guessing that not all of your fields are multivalued though. Are they?

I will say that I haven't tested it on Solr 6.x yet. The environments that I have set up for testing are all 5.x. If the version difference is causing a problem then that's a bigger issue. The datasource module doesn't do any direct communication with the Solr server. Everything is sent through search_api_solr's backend and Solarium.

Oh! In the Search API server's settings did you check "Retrieve result data from Solr"? That does seem to be a necessary step at this time according to @sylus.

Edit: See two comments down for an explanation.

If none of that helps, then please reply back and post the output of Solr's query log for one query (or one set of queries if the search query is sent and then individual queries for each record). I've been using the query log a lot for debugging problems with the datasource. Seeing exactly what request is being made to the server has been critical in development.

So the whole multivalued field thing was me being a little bit clueless. Investigating it did reveal another problem though. If your field machine names have all been edited in the index's fields page, then that would have caused them to not appear in Views. That issue has been patched.

FYI, there are only two critical alpha-release-blocker issues now that I know about. One of them has already been figured out; I just need to edit the documentation.

Ok sorry for my late reaction;

I updated the code from git. Still same problem.

I get the following error:

Caused by: org.apache.solr.search.SyntaxError: Cannot parse 'accommodation_id:': Encountered "" at line 1, column 17

Sounds like the Solr server is throwing an error. I'll look into it.

Something is wrong with the query that's being sent to Solr. I think I'll need to see the query in order to know exactly what the issue is.

whats the easiest way to see that?

There's an option in the Search API server advanced config form to log queries. I've never used it, but I assume it works. Personally, I just check the Solr server log. In my 5.6 installation they're stored in %solr_root%/server/logs.

There's an option in the Search API server advanced config form to log queries.

That one doesn't exist (anymore).

Oh, ok. Thanks for letting us know, @mkalkbrenner.

See also: #2880587: Consider merge with Search API Solr 8.x-2.x.

Is there an update on this? We are planning our migration from Drupal 7 to 9 and I would like to know if sarnia will be ported to drupal 9 or if the latest version of Search API offers similar features now?

@Menthos The features of Sarnia have been integrated into Search API Solr.

@Menthos: The functionality of Sarnia to read any arbitrary Solr index was merged into the Drupal 8.x port of Search API Solr, and has been pretty stable as of that module's version 3.x (it's currently at version 4.x, which supports Drupal 9).

Setting the status of this issue to Postponed, basically indefinitely, so as to point the way to the 8.x version of Search API Solr which now includes the "any index" functions that Sarnia provided in 7.x.

It is unnecessary to port Sarnia to Drupal 8.x and beyond. This was actually posted on the project front page some time ago but it seems it was missed.