Subscribe to Propeople Blog feed
Updated: 2 hours 8 min ago

A Content Staging Solution for Drupal 8 (and more)

February 26, 2015 at 6:02pm

Moving content between different environments is a need for many big companies that have a Drupal site and content that should be created, reviewed, edited and published on different environments. Although Drupal 8 is not yet released, a content staging solution is already in the works. Dick Olsson (dixon_ on drupal.org) and I are working on this solution together and we aim to release an alpha version soon.

The content staging solution for Drupal 8 is based on a re-designed version of the Deploy module. This solution consists of some contrib modules and depends on three Drupal core modules. The core dependencies are Entity API, Serialization and Restful Web Services. The contributed module dependencies are Relaxed Web Services, Multiversion, Key-value Extensions and (soon) Deploy.

The Relaxed Web Services module provides a Restful/Relaxed JSON API and endpoints for entities, file attachments, administrative tasks like revisions comparison, starting/stopping replication, etc. It extends the core REST API with better support for handling UUID references, revisions, file attachments, etc. This module is borrowing the API interface from CouchDB and its Replication API. CouchDB is a NoSQL, document-oriented database.

The Multiversion module provides revision support for all content entities. It also tracks update sequences to make dependency management easier and tracks revision trees (similar to Git) in order to support conflict detection. With Multiversion, entities are never deleted, they are just flagged as deleted. This is needed in order to replicate deletions and for conflict handling.

Key-value Extensions provides an extension of the core key-value API with a backend for lists and sorted sets that you can do range queries on. This module is needed because of the way the Multiversion module stores its sequence indexes.

Deploy (will be implemented soon) - provides a simple user interface to manage replication and conflicts.

Replication

At the moment, we are using the CouchDB replicator to test content replication between different systems. The CouchDB Replication protocol is a protocol for synchronizing documents between 2 peers over HTTP. This protocol will be used to implement the Replication Web Service module for Drupal 8.

The Replication Web Service module will provide the possibility to replicate content between different systems and Drupal 8, it will also have a Drush plugin for running the replication. Furthermore, it will be possible to run live replications in order to synchronize applications.

Offline applications

By using a standardized HTTP replication protocol for Drupal, such as the one CouchDB is using, the same solution will be applicable to other very interesting use cases as well.

The Offline First principle is quite new in web development, but it has many benefits for users and their experience. A website designed after these principles will continue to work, even if there is no Internet connection available. Now it’s possiblel to create Offline First applications with Drupal 8! We can build offline applications using the same suite of modules that we introduced earlier. To do this we need a remote database - represented by a Drupal 8 site and a local browser-based database, for example PouchDB.

The content staging suite provides all necessary features, such as synchronization, revisioning and file attachments, to create an offline application. At the moment, it’s working with PouchDB version 3.2.1. I’ve created a video to demonstrate how synchronization between PouchDB and Drupal 8 works

Video of Test pull and push replication using Drupal 8 and PouchDB

To test this I use an application based on ToDoMVC and PouchDB 3.2.1.

Headless Drupal

In Drupal 8, we have integrated the Twig template framework, a very good thing, especially for front-end developers. However, sometimes we want to create an absolutely custom frontend using the power of libraries and frameworks like AngularJS and Hoodie, combined with Drupal 8 on the back-end.

The solution we implement provides a lot more possibilities than the Restful Web Services module from Drupal 8 core. This will make possible to create awesome applications using frameworks like AngularJS, a PouchDB database and Drupal 8.

Other Systems

The content staging suite will have many different use-cases, allow for replication between different systems and database, not just between Drupal sites.

Currently we have test suites for replication between Drupal 8 and CouchDB (using the CouchDB replicator, but later you will be able to use the Replication Web Services module). We also have test suites for replication between Drupal 8 and PouchDB.

In the future, this solution may be used to integrate Drupal 8 with other libraries and frameworks.

For more information, check out:

Tags: Drupal 8content stagingCheck this option to include this post in Planet Drupal aggregator: planetTopics: Tech & Development
Categories: Planet Drupal

Varnish Tips and Tricks

February 18, 2015 at 5:02am

In this article we would like to share some use cases and recipes for configuring Varnish.

Use custom data for building varnish cache hash

By default, Varnish uses URL as a parameter for caching. In the VCL file, it is also possible to add custom data (for example: location, or custom header value) to hashing by using the sub vcl_hash{} configuration part. But there is yet another solution to be used that Varnish comes equipped with out of the box. We can set a Vary header that is respected. So, if we want our cache to be based on header X-MyCustomHeader in Drupal, then we can set the header to

Vary: Cookie,Accept-Encoding,X-MyCustomHeader. This way, different values of our custom header will have different cache records in Varnish.

Limit access to the site by ip address

When we build an intranet website, we can limit access to it on the Varnish level. This can be done in following way:

First we define list of allowed IP addresses:

acl offices {

   "localhost";

   "127.0.0.1";

   "1.2.3.4";`

   "5.6.7.8";

}

Then we restrict access to non matching addresses:

sub vcl_recv {

   if ( req.http.host ~ "(intranet\.example\.com)$" && !(client.ip ~ offices) ) {

        error 403 "Access denied";

   }

}

SSL termination

As Varnish is not handling https traffic, we need to terminate SSL before it hits Varnish. For that we can use nginx. Here is a list of links to articles that dive deeper into this topic:

https://www.digitalocean.com/community/tutorials/how-to-configure-varnish-cache-4-0-with-ssl-termination-on-ubuntu-14-04

http://edoceo.com/howto/nginx-varnish-ssl

http://mikkel.hoegh.org/2012/07/24/varnish-as-reverse-proxy-with-nginx-as-web-server-and-ssl-terminator/

https://wiki.deimos.fr/Nginx_%2B_Varnish_:_Cache_even_in_HTTPS_by_offloading_SSL

ESI

On a recent Propeople project, we had the requirement to include a block with data from an external website without any caching. The tricky part was that the external site was providing XML with the data. The solution we implemented was to use ESI block pointing to the custom php file that was pulling that XML and parsing it on the fly.

Hiding js requests to external domains

If we need to do some CORS (http://en.wikipedia.org/wiki/Cross-origin_resource_sharing) requests instead of our Javascript doing requests directly to external domain, we can do requests to our site, but with a specific URL. Then, on the Varnish level, we can redirect that request to external domain. In this case, Varnish will act like a proxy. This can be achieved with backend options.

backend google {

 .host = "209.85.147.106";

 .port = "80";

}

sub vcl_fetch {

 if (req.url ~ "^/masq") {

   set req.backend = google;

   set req.http.host = "www.google.com";

   set req.url = regsub(req.url, "^/masq", "");

   remove req.http.Cookie;

   return(deliver);

 }

}

This is an example from a brilliant book: https://www.varnish-software.com/static/book/

Multiple backends, load balancing

It is possible to define multiple backends for Varnish and switch between them. Most basic implementation is round robin or random. Here is an example:

backend web01 {

   .host = "example1";

   .port = "80";

   .connect_timeout = 120s;

   .first_byte_timeout = 300s;

   .between_bytes_timeout = 60s;

   .max_connections = 50;

   .probe = {

    .url = "/";

    .timeout = 10s;

    .interval = 20s;

    .window = 5;

    .threshold = 3;

   }

}

 

backend web02 {

   .host = "example2";

   .port = "80";

   .max_connections = 100;

   .connect_timeout = 120s;

   .first_byte_timeout = 300s;

   .between_bytes_timeout = 60s;

   .probe = {

       .url = "/";

       .timeout = 10s;          

       .interval = 20s;       

       .window = 5;

       .threshold = 3;

   }

}

 

director apache round-robin {

 { .backend = web01; }

 { .backend = web02; }

}

 

sub vcl_recv {

 set req.backend = apache;

}

 

It is also possible to set a specific backend for visitors coming from specific IP addresses. This can have a number of helpful uses, such as making sure that the editors team has  adedicated backend server.

if (client.ip ~ offices) {

 set req.backend = web03;

}


I hope you have enjoyed our tips regarding Varnish configuration. Please feel free to share your own thoughts and tips on Varnish in the comments below!

Tags: VarnishService category: TechnologyCheck this option to include this post in Planet Drupal aggregator: planetTopics: Tech & Development
Categories: Planet Drupal