Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
By on
ShopCircuit is a product search engine and price comparison startup from Amsterdam, The Netherlands. ShopCircuit aggregates product catalogs from online merchants and offers hi-tech search tools for consumers.
Why Drupal was chosen:
Drupal was chosen for its widespread community support. ShopCircuit uses a flat data storage to manage millions of product records which we built on top of Drupal 7's flexible Entity API. The product catalog is stored in a custom database backend (MongoDb). The Drupal community offers a MongoDb API for it.
Because ShopCircuit is a search engine we cannot cache full pages, since there are infinite combinations of results. Therefore we depend on Drupal's granular caching framework that enables us to cache individual content entities instead of whole pages.
As an import project requirement, Drupal integrates well with third party technology. Drupal allows ShopCircuit to scale. We can:
- move the filesystem to a CDN
- scale the cache backend through server RAM using Memcache
- scale search across a Solr cloud
- scale our product catalog data using MongoDb
Drupal ties it all together nicely. It does require fine-tuning and developer expertise, but the result is scalable and performs well. Drupal 7 with Search API (special thanks to drunken monkey!) was a big selling point for going with Drupal. Going forward, I am very excited about Drupal 8 and its Symfony framework integration.
Describe the project (goals, requirements and outcome):
ShopCircuit is a product search engine and has several core requirements.
ShopCircuit is a Netherlands-based startup, but our Drupal technology is prepared to scale across multiple countries. The desire was to use a single Drupal core across multiple languages and domains, in order to reduce maintenance, development and server hosting cost. ShopCircuit offers different content to specific "cultures" (we define an online culture as a country and its language, such as en_US, en_GB or nl_NL).
Besides The Netherlands we have launched in several countries (Germany, United Kingdom, United States and France). Drupal should separate content from each domain. Even in the year 2013 cross-border e-commerce is uncommon, mainly due to shipping and handling costs, but also due to lack of cross-border trust. Therefore, ShopCircuit needs to offer separate content for each country. We decided to use a single Drupal Core to minimize developer maintenance cost. Thankfully, Drupal has a solution in 'Domain Access'; a set of modules that allow you to configure multiple domains.
Different countries have different languages. ShopCircuit offers a localized experience for each country using Drupal's built-in Translation Interface. The Drupal community offers excellent translations.
Importantly, ShopCircuit needs to scale millions of product catalog rows.
Technical specifications
Key modules/theme/distribution used:
Why these modules/theme/distribution were chosen:
Domain Access
Domain Access offers an interface to manage multiple country-specific site configurations within a single Drupal installation. Domain Access lets you manage your top-level domains (TLDs), including domain aliases. Using this module you can set country-specific content, themes, users and configurations.
Solutions used: Domain Access, Domain Views, Domain Context.
Internationalization
The suite of 'i18n' modules allows us to translate every part of Drupal: content, custom entities, blocks, views, pages and so on. The interface is easy to use.
Solutions used: Internationalization, i18n_views, Localization update, Translation overview.
Entity API
Drupal 7's core node entity has a flexible Fieldable API, which allows site builders to put together custom page input forms and displays. Drupal 7 stores each field in a separate MySQL table. This posed a problem for ShopCircuit. Storing millions of datapoints in this sprawled out manner was a major performance bottleneck. ShopCircuit ties Drupal and MongoDb together using the Entity API. A custom "flat entity" is stored, read and updated through the Drupal Entity API and the Drupal MongoDb backend. MongoDb allows us to scale up to billions of entities if we need to. We still use MySQL for everything else. This can leave some people confused, but MongoDb is only intended for storing "big data" and scaling it. MongoDb is not intended to be a SQL-replacement at all. In general you should use a dual approach; using MongoDb alongside SQL.
We built our custom 'product' entity by looking at the examples and learning from the node.module. Our product data is directly saved to MongoDb, simply by writing custom entity_load() and entity_save() hooks.
Solution used: MongoDb for Drupal, MongoDb, Entity API.
Search API & Solr
Faceted search means drilling down content using "facets" such as brands, categories, colors, materials, etc. In MySQL this would require expensive JOIN queries Which would not scale well. MongoDb obviously does not offer JOINs. So we use another technology layer: Apache Solr. Apache Solr is a Java application we run in Tomcat 7 for Ubuntu.
Solutions used: Search API, Search API solr, Tomcat 7, Apache Solr.
Context
Using the built-in Blocks API Drupal gives you the tools to decide where you want to show certain 'blocks' of content. But the blocks interface was not powerful enough for ShopCircuit. We switched to the Context UI (with plugins for i18n, domain etc.). The Context module allows you to define a 'section ' of your site by language, domain, path or any other external factor. It is then possible to define content for each section. Some examples to help illustrate this:
- we have a 'Global' context for each country, which displays the headers and footers in appropriate languages;
- we have a 'Front' context for each country, in order to set homepage content;
- we have a 'Search' context to decide what content to show on our main /search pages;
- we have 'Custom 404' and 'Custom 403' contexts to modify error page content (for example: http://www.shopcircuit.nl/try-this-error-page)
Solutions used: Context, Domain Context, Context Block Cache Alter, Context Error.
Community contributions:
Developed by ShopCircuit for Drupal:
- Search API Ranges - a price range slider now also used in Drupal Commerce
- Search API Ranges - a custom sorts block for Search API, now also used in Drupal Commerce
- Search API Ajax - to ajaxify search pages
Several patches were submitted during the development of ShopCircuit.
Project team:
Design by Rem-art.
Sectors:
E-Commerce
Comments
Project URL
Can you please give me your project URL? Is this available now?