I've attached a patch against the alpha1 version that implements a two-tier directory structure within the repository. i.e. a file with ID acc34176f7f83d7bc0a1a91250dd4506a18ff8dd would be stored in:

/a/c/acc34176f7f83d7bc0a1a91250dd4506a18ff8dd

rather than

/acc34176f7f83d7bc0a1a91250dd4506a18ff8dd

It also includes a 'rehash' method in the Bitcache_Repository class that can move a flat repository to the new tiered structure.

The tiered structure shows better performance (at least on some filesystems) with very large repositories.

Thanks.

CommentFileSizeAuthor
bitcache_two_tier.patch1.32 KBsonnen

Comments

Arto’s picture

Arto commented
Assigned: Unassigned » Arto

Thanks, Richard. I definitely intend to integrate multi-tier directories before cutting a first beta release. I will review and test your solution for the next alpha.

Arto’s picture

Arto commented
Title: Two-tier directory storage » Multi-tier repository structure
Status: Needs review » Postponed

I've now implemented initial internal support for multi-tier repository structures in the 6.x-1.0-alpha3 release. I proceeded from your patch but expanded on it as follows:

There is not yet any UI-level support for creating multi-tier repositories, but there are two Drupal system variables (named bitcache_depth and bitcache_width) that you can set manually in order to get support for any conceivable multi-tier repository configuration. In the next release, I will make it possible to configure these on a repository-by-repository basis via the administrative interface. (I plan to keep the zero-tier structure as the default as most people using Bitcache on Drupal are likely to have modest needs, and this also avoids the need for any potentially risky data migration updates.)

By default, both variables are set to zero which results in the familiar zero-tier, or "flat", repository. To get the exact two-tier structure that you wanted, simply set bitcache_depth to 2 and bitcache_width to 1. For the time being, you can modify these at the top of boost.module or in your variable MySQL table as follows:

INSERT INTO variable VALUES ('bitcache_depth', 'i:2;');
INSERT INTO variable VALUES ('bitcache_width', 'i:1;');

To demonstrate the flexibility these variables provide for configuring repository structures, consider a case where you wanted something like the object store structure that Git uses internally; to achieve that, you would just swap the two variables, setting bitcache_depth to 1 and bitcache_width to 2; this would yield up to 16x16 (= 256) two-character top-level directories such as e.g. /ac/acc34176f7f83d7bc0a1a91250dd4506a18ff8dd. (In fact, this would in principle allow you to plug in an existing Git object repository into Bitcache.)

Hopefully this information suffices to make use of this functionality for the time being, should you need it. I'll write more documentation in time for the next release...

jvieille’s picture

jvieille commented
Issue summary: View changes

I have a significant performance issue with Bitcache.
For example, this simple 10K image takes soemtimes 7 seconds to load - the minimum is 0.5 secondes.
The (fileframework) repository has currrntly 20,000 items for 11 Gb

Would this funtionality help?
Would it work with FF?
How migration from flat to multi-tiers would be handled?

Thanks for help