Override a non-indexed bundle on a per-entity basis

After applying your patch, entities marked to be indexed but belonging to non-indexed bundles are not added to the sitemap. The reason is in StimeapGenerator.php on line 176:

        foreach ($bundles as $bundle_name => $bundle_settings) {
          if ($bundle_settings['index']) {
            $data_sets[] = [
              'bundle_settings' => $bundle_settings,
              'bundle_name' => $bundle_name,
              'entity_type_name' => $entity_type_name,
              'keys' => $keys,
            ];
          }
        }

So only bundles are added to $data_setswhich are to be indexed. Later on in EntityUrlGenerator.php, getBatchIterationEntities() does not fetch the entities of non-indexed bundles.

I do not see many use cases for this feature, hence marking low priority. I will gladly commit it when we get this to work though.

Thanks for the patch. I have read through it without applying it and I do not understand why you are implementing a new parameter "active" considering the parameter "index" is already in place? The "index" parameter is supposed to define if an entity type is to be indexed, why the need for another parameter?
Tests also need to pass or need to be adjusted.

Tell me if I am wrong.

The goal is to have the possibility to enable the indexation per bunble of enabled entity type. And for each entities (node, taxonomy term, etc.) where the indexation is enabled, having the possibility to have the entity not indexed by default

The above is actually already implemented; let me check if we are on the same page: You can enable the sitemap settings for entity types and then enable indexation for certain bundles of that entity type. Then you can override this setting for a single entity of that bundle.

With a priority of 0 the entity is still present in the sitemap.

The priority doesn't have anything to do with whether the links end up in the sitemap. That's just info for search engines.

In regards to the feature of having bundles set to 'no index' and then the possibility to override this on a per-entity basis, this would be doable but would require some major refactoring and UI changes. I don't think there is enough interest from the community for me to implement this, but I am happy to review working patches.

@Grimreaper

The functionality we are discussing here has not changed a bit since you created this issue.

However you just found a bug I introduced in the last couple of commits. If you go a few commits back you will see everything is working as it used to. This is due to some refactoring I've been doing while preparing for a new release.

Thanks a ton for catching this, apparently no test covers this UI element. Will take a look now.

That's fixed.

As this issue came to life yesterday in the dev version and now it's fixed, I don't think we need a record of it, so feel free to revert the title/opening post/status/category of this issue to what it originally was.

Reverting the issue to its original topic and values. I am closing this for now because as mentioned earlier, I don't think there is enough interest from the community for me to implement this. I am however going to review and possibly commit patches.

Thanks for your patch. I've tested it, unfortunately it didn't work as expected.

• Unticking 'Index by default' does nothing. All entities of that bundle are still getting indexed. There seems to be still logic missingin EntityUrlGenerator::getDataSets(). EntityMenuLinkContentGenerator::getDataSets() would have to be adjusted as well.
• UI is a bit confusing, as there now are 'index' and 'index by default' checkboxes. Maybe there should be something like 'Enable for bundle' and 'Index by default'.

Having said all that, I am still not sure if this feature should be implemented. The use case for having entities of a bundle not indexed by default is rare. But I will look into it if we have a working patch.

Hi Grimreaper,

sorry for getting back to you this late. The number of subscribers to this issue tells me that this is a very esoteric use case. This is why I have decided to leave it out of the 2.x branch of the module which just received its final feature release. I am now focusing on branch 3.x and #2961233: Allow for multiple sitemap types on one Drupal instance. I do not plan to add this feature to 3.x either, but you are welcome to port this patch to 3.x for others to benefit from your work.
I Hope you will understand.

Hey Grimreaper,

I do believe that in a project of this size, the community voices their interests through participating in an issue, like you did. While I appreciate the work you have put into this patch, I have not heard another voice confirming the usefulness of this feature. Furthermore, I did not find the end result elegant. This is not because of your coding, but probably because of the constraints of the current data model. On top of that the patches did not work for me as expected and required additional time on my part.

Regarding the last bit, I believe I know what it means to use my free time in order to keep this project relevant, as I am not getting paid to participate in any of the issues in this project. I also believe I am doing a good job considering there are no open bug reports. I am sure you can appreciate, that being he only maintainer, you have to pick your battles and do what seems to be most relevant to most.

Having said this, the project benefited from you patching the tests and I am thankful for this contribution!

In regards to updating this patch for your client, there should be no struggle as long as you stay on the 2.x branch. If no critical bugs get discovered, 2.12 will have been the last release for 2.x.

Just wanted to voice a +1 for this feature. We would like to switch a large site from XMLSitemap to this module, and this is going to be necessary for us to make the transition.

Thanks for your input. Could you please elaborate on the why? Just so we are on the same page in terms of the necessity of this feature.

Sure, yeah. Our situation is that we have a single content type that for the most part we don't care whether Google indexes it or not, but certain pieces of content are much higher value. So we would like the default to be not to include it in the sitemap, with an option to include it. Using XMLSitemap, we can do it. With Simple Sitemap, it's currently impossible.

Version isn't terribly important to us since we don't have any existing config - we're fine with using 8.x-3.x as long as there's a timeline for getting that version to a stable release.

Hello,

Updating the patch to be appliable on the 8.x-3.x branch.

I know that the issue is closed but we still have the need on the project.

Thx for the review.

Regards, Quentin.

Sorry I forgot the modification in schema.yml file.

Regards,

Looks like there is a patch that can be reviewed by the community. I don't agree that number of followers is a good indication of the impact to the community. Some people may also look the other way because it is marked as closed (won't fix).

At the end of the day, it seems like the community is asking for a default setting as opposed to completely locking down the content type, which is the current problem.

comment 35 describes our use case as well.

I agree, and I am surprised this issue hasn't been resolved yet. Here's my problem.

Default content type setting: "include items in sitemap"
-- individual nodes can be set to be excluded

Default content type setting: "exclude items from sitemap"
-- individual nodes CANNOT be set to be included, despite the actual widget specifically saying "Settings for this entity can be overridden here."

If you want to make this editable on a per-node basis, then you have to make it editable no matter what the default setting is.

I have tested the patch in comment #37 running Drupal 8.7.6

I also added granular permissions with a patch on this other issue https://www.drupal.org/project/simple_sitemap/issues/3033792

I've updated the patch to be applicable for 3.5.0. #37 didn't work for me.

When the sitemap gets updated by cron runs, only the frontpage, and entities of a non-indexed bundle, where a per entity override is in place are indexed.
Very likely connected to this patch, because the issue didn't occur previously.
Also tests are failing.

Re-rolled patch for D8.8.0 core and 3.7.0 simple_sitemap release.

Re-rolled patch. Did not notice issues reported in #43

The UI for this fix (workaround) is not intuitive to me. I would have expected something like:

If the Simple XML Sitemap bundle setting "Do not index entities of type @bundle in variant Default" is selected,
the sitemap entity settings radio-buttons shouldn't be disabled anymore to allow the selection of "Index this @bundle entity in variant Default".

Probably this would have required some mayor refactoring as mentioned by gbyte at #8, otherwise it would be much cleaner. At least there should be a description to explain how this Option "Enable indexing for this bundle" option works and what is the difference compared to "Index entities of type @bundle category in variant Default".

For what it is worth -- we have a lot of interest in this functionality.

Here is a specific use case that without a viable solution/patch, a lot of extra manual work is created and we are exposed to an SEO penalty for thin content.

We mantain a faculty/staff directory which has nodes created and unpublished via an integration with Workday. That directory allows for users to submit bios. If a faculty/staff member doesn't have a bio, their node essentially is a blank page. This is regarded as "thin" content by Google and an excess of these could negatively affect our SEO.

We could solve this very easily by defaulting the content type's inclusion in a sitemap to NO and then when users add their bios, manually allowing that inclusion. But this module only works in the reverse. You include by default and then manually remove.

The net effect of this is to have to monitor an automated process constantly for new data -- in effect defeating the purpose of the automation in the first place.

Hopefully you'll reconsider this in your 4.x development.

Any updates here? Allowing individual entities to be indexed even if the bundle is excluded is quite a common need. Seems like the latest patch breaks module functionality.