In latest version of s3fs (8.x.30-alpha10) nor any other s3fs_file table has index and it performs poorly with larger filesystems.

Comments

irmantasp created an issue. See original summary.

jansete’s picture

Priority: Normal » Major
jansete’s picture

Issue tags: +alpha target
jansete’s picture

jansete’s picture

mikeryan’s picture

Well, it does have an index on the timestamp column - but yeah, why is uri not a primary key?

mikeryan’s picture

Looking at the code, in s3fs_schema() we have:

    'indexes' => [
      'timestamp' => ['timestamp'],
    ],
    //'primary key' => ['uri'],
    // As mentioned on http://drupal.org/node/2193059, a bug in Drupal core's
    // MySQL driver prevents this setting from actually being applied. So we
    // manually fix that in hook_install().
    'collation' => 'utf8_bin',

and in s3fs_install() we have:

    case 'mysql':
      // As stated here:
      // http://forums.mysql.com/read.php?103,19380,200971#msg-200971
      // MySQL doesn't directly support case sensitive UTF8 collation.
      // Fortunately, 'utf8_bin' collation works for our purposes.
      \Drupal::database()->query("ALTER TABLE {s3fs_file} CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin");
      \Drupal::database()->query("ALTER TABLE {s3fs_file} ADD PRIMARY KEY (uri)");
      break;

It's attempting to make uri work as a primary key, but for some reason it's not taking...

mikeryan’s picture

Ah-ha! When looking at #2900237: PDOException: SQLSTATE[22001]: String data, right truncated, I noticed that my local system (which did not have that problem) had this problem - no PK, and collection not set to utf8_bin. But, the remote server, which did have that problem and thus never successfully populated the s3fs_file_temp table and swapped it for s3fs_file, has a PK on s3f3_file. What happens is when you refresh the cache, an s3fs_file_temp table is created by copying the schema of s3fs_file - but, that schema lacks the collation and key settings, so s3fs_file_temp does not receive them. Assuming the cache is successfully refreshed, the temp table replaces s3fs_file, which thus ends up with no PK.

Man, it'd be nice if #2193059: DatabaseSchema_mysql::createTableSql() can't set table collation got fixed...

mikeryan’s picture

Title: "s3fs_file" table has no index » "s3fs_file" table has no primary key
Status: Active » Needs review
StatusFileSize
new849 bytes

This should do the trick.

jansete’s picture

Hi @mikeryan!

It looks very well!

It's an important problem that we have to solved. I hope review and test tomorrow, if everything is fine, I'll create new release.

Thanks a lot!

  • jansete committed 4612aed on 8.x-3.x authored by mikeryan
    Issue #2972251 by mikeryan: "s3fs_file" table has no primary key
    
jansete’s picture

Nice work @mikeryan, now is in dev branch!

jansete’s picture

StatusFileSize
new3.78 KB

I've added a new function to fix indexes and now is used in hook_install, refresh-cache command and in a new hook_update to fix primary index for existing tables.

  • jansete committed e66819a on 8.x-3.x
    Issue #2972251 by jansete: "s3fs_file" table has no primary key
    
jansete’s picture

Status: Needs review » Fixed
mikeryan’s picture

Thanks!

arunverma’s picture

It's been really required. Thanks for it.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Anitha Ravi’s picture

Hi,

s3fs-2972251-9.patch does not working as expected. It gives the error as follows,

1.if i use patch -p1 < s3fs-2972251-9.patch
Hunk #1 FAILED at 229.
1 out of 1 hunk FAILED -- saving rejects to file /path/to/directory/modules/s3fs/src/S3fsService.php.rej

2. IF i use git apply -v s3fs-2972251-9.patch

Checking patch src/S3fsService.php...
error: while searching for:
try {
\Drupal::database()->schema()->dropTable('s3fs_file_temp');
\Drupal::database()->schema()->createTable('s3fs_file_temp', $schema['s3fs_file']);
}
catch (SchemaObjectExistsException $e) {
// The table already exists, so we can simply truncate it to start fresh.

error: patch failed: src/S3fsService.php:229
error: src/S3fsService.php: patch does not apply