Problem/Motivation

When attempting to write values to the loc column that are greater than 256 characters long, a DatabaseExceptionWrapper exception.

Proposed resolution

Increase the loc field to support longer locations, or at a minimum, catch the exception so the sitemap can still be built (albeit missing the entry that is too long).

Remaining tasks

  1. Write Patch
  2. Review Patch

User interface changes

None.

API changes

None.

Data model changes

Loc column lenght extended to 512.

Original Report

Everytime I try and build the sitemap for the first time, I get this error:

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'loc' at row 1: INSERT INTO {xmlsitemap} (type, id, loc, subtype, access, status, status_override, lastmod, priority, priority_override, changefreq, changecount, language) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12); Array ( [:db_insert_placeholder_0] => file [:db_insert_placeholder_1] => 1085411 [:db_insert_placeholder_2] => https://example.com/sites/default/files/video/thumbnail/1430780407.984--1070.373c1430781645.042--804.037c1430782570.534--19.42c1430782742.072--396.964c1430783321.252--242.409c1430783686.717--24.491c1430783862.493--672.305_1280x720_439541315986.mp4 [:db_insert_placeholder_3] => file [:db_insert_placeholder_4] => 1 [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => 0 [:db_insert_placeholder_7] => 1430788801 [:db_insert_placeholder_8] => 0.5 [:db_insert_placeholder_9] => 0 [:db_insert_placeholder_10] => 0 [:db_insert_placeholder_11] => 0 [:db_insert_placeholder_12] => en ) in Drupal\xmlsitemap\XmlSitemapLinkStorage->save() (line 138 of modules/contrib/xmlsitemap/src/XmlSitemapLinkStorage.php).

Drupal\Core\Database\Statement->execute(Array, Array) (Line: 615)
Drupal\Core\Database\Connection->query('INSERT INTO {xmlsitemap} (type, id, loc, subtype, access, status, status_override, lastmod, priority, priority_override, changefreq, changecount, language) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12)', Array, Array) (Line: 86)
Drupal\Core\Database\Driver\mysql\Connection->query('INSERT INTO {xmlsitemap} (type, id, loc, subtype, access, status, status_override, lastmod, priority, priority_override, changefreq, changecount, language) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12)', Array, Array) (Line: 37)
Drupal\Core\Database\Driver\mysql\Insert->execute() (Line: 376)
Drupal\Core\Database\Query\Merge->execute() (Line: 138)
Drupal\xmlsitemap\XmlSitemapLinkStorage->save(Array) (Line: 1582)
xmlsitemap_xmlsitemap_process_entity_links('file', Array) (Line: 2549)
xmlsitemap_rebuild_batch_fetch('file', Array)
call_user_func_array('xmlsitemap_rebuild_batch_fetch', Array) (Line: 252)
_batch_process(Array) (Line: 144)
_batch_progress_page() (Line: 72)
_batch_page(Object) (Line: 60)
Drupal\system\Controller\BatchController->batchPage(Object)
call_user_func_array(Array, Array) (Line: 128)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 577)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 129)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 102)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 139)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 62)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 62)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 53)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 55)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 31)
Drupal\gc_api\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 637)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

davidwbarratt created an issue. See original summary.

davidwbarratt’s picture

Issue summary: View changes
davidwbarratt’s picture

davidwbarratt’s picture

I realized that if a xml site map link cannot be saved, it probably should not hold up entity saving. Ideally this should never happen, but it's certainly possible. If it does happen the user can always fix the problem and rebuild the site map at a later date.

davidwbarratt’s picture

The patch I provided in #4 stops the rebuild procedure from throwing an Exception which allows the site map to be built with missing entries (if it could not be saved in the database).

amateescu’s picture

I would prefer to fix the 'loc' column to handle larger strings. Do you still have the location that couldn't be saved by any chance?

The last submitted patch, 3: db_exception-2747927-3.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 4: db_exception-2747927-4.patch, failed testing.

The last submitted patch, 3: db_exception-2747927-3.patch, failed testing.

The last submitted patch, 4: db_exception-2747927-4.patch, failed testing.

jatinkumar1989’s picture

Getting the same issue, Any help ?

fomenkoandrey’s picture

the same error
https://image.prntscr.com/image/BZm54guoRoqsLRbf5UEHQQ.png

Произошла ошибка.
Перейдите на страницу ошибки

Возникла AJAX HTTP ошибка.
Полученный код HTTP: 500
Следует отладочная информация.
Путь: /batch?id=1257&op=do_nojs&op=do
Текст Состояния: Internal Server Error
Текст Ответа: {"message":"A fatal error occurred: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column \u0027loc\u0027 at row 1: INSERT INTO {xmlsitemap} (type, id, loc, subtype, access, status, status_override, lastmod, priority, priority_override, changefreq, changecount, language) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12); Array\n(\n [:db_insert_placeholder_0] =\u003E file\n [:db_insert_placeholder_1] =\u003E 555\n [:db_insert_placeholder_2] =\u003E https:\/\/motokalendar.com\/sites\/default\/files\/images\/2017\/%D0%9D%D0%B0%D0%BA%D0%BB%D0%B5%D0%B9%D0%BA%D0%B0%20%D0%B7%D0%B0%D0%BB%D0%B8%D0%B217%D0%B2%D0%B5%D1%80%D1%81%D0%B8%D1%8F3%D1%81%D0%B0%D0%BC%D1%8B%D0%B9%20%D0%B2%D0%B5%D1%81%D0%B5%D0%BB%D1%8B%D0%B9%20%D0%BF%D0%B0%D1%80%D0%B5%D0%BD%D1%8C%20%D0%B7%D0%B0%D0%BB%D0%B8%D0%B2%D0%B0.jpg\n [:db_insert_placeholder_3] =\u003E file\n [:db_insert_placeholder_4] =\u003E 1\n [:db_insert_placeholder_5] =\u003E 0\n [:db_insert_placeholder_6] =\u003E 0\n [:db_insert_placeholder_7] =\u003E 1493061174\n [:db_insert_placeholder_8] =\u003E 0.5\n [:db_insert_placeholder_9] =\u003E 0\n [:db_insert_placeholder_10] =\u003E 0\n [:db_insert_placeholder_11] =\u003E 0\n [:db_insert_placeholder_12] =\u003E ru\n)\n"}

jonnyhocks’s picture

I have tried applying the patch in #4 which did stop the batch errors; however now my xml sitemap is empty apart from two empty entries if there is an error within a certain content type. eg, I have a fully formed sitemap if I include all content types apart from basic page (the problem content type) but then if I include basic page then the sitemap gets wiped.

I get the error in the log that stopped the sitemap being generated without this patch.

pianomansam’s picture

Title: Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'loc' at row 1 » Loc column limited to length of 256
Issue summary: View changes
Status: Needs work » Needs review
FileSize
950 bytes

This patch increases the loc column to 512.

pianomansam’s picture

davidwbarratt’s picture

That's still an arbitrary value that could be exceeded.

Perhaps we should use 2,000 characters?
https://stackoverflow.com/a/417184

pianomansam’s picture

@davidwbarratt I simply doubled the value, but you are correct - it is arbitrary. My only concerns going up to 2,000 characters are 1) Any DB-level detriments to such an increase, and 2) This is for a sitemap, which might not include the entire possible URL. Yes, 2k is only half the 4k limit of some browsers, but it's still a 10x increase from the previous length. Are there good reasons why it was previously only 256?

Dakwamine’s picture

Hello all. Just giving my thoughts about this issue. Hope to see this fixed someday. :)

Maybe it's still the best to stick with the standards about maximum URL length, as this will handle all valid possibilities. Or maybe we could use another column type if not too slow regarding performance?

Still, the exception handling in #4 is not a bad idea at all to ensure nothing wrong will break the sitemap build.

Dakwamine’s picture

Status: Needs review » Needs work

Also, the last patch errors on update because the Database class is not found.

Using the fully qualified class name fixes the error.

andralex’s picture

Hello all. Here is an updated patch using imported Database class and most recent hook_update version.

I've also increased the number of characters to 2,048 according to https://www.sitemaps.org/protocol.html#locdef specification.

andralex’s picture

andralex’s picture

Status: Needs work » Needs review
ysamoylenko’s picture

Hello everybody.
It seems the issue has been already resolved for the 8.x-1.x branch by the module maintainer.
Referenced git commit: https://git.drupalcode.org/project/xmlsitemap/-/commit/7c72014b288ffbc16...
Currently links stores in DB as internal paths.
For sitemap file generation module uses a mechanism which calls path_alias module to get SEO-friendly URLs if they are specified.

ysamoylenko’s picture

Status: Needs review » Reviewed & tested by the community
renatog’s picture

Status: Reviewed & tested by the community » Fixed

Exactly, you right. For now the loc is using internalPaths: https://git.drupalcode.org/project/xmlsitemap/-/blob/7c72014b288ffbc164d...

So it someone get this error, please upgrade for the last version and it'll be fixed

Thank you so much everyone.

If you have any other issue related to this feel free to let us know

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.