Voting starts in March for the Drupal Association Board election.
On 30 June 2014 The BMJ became the first academic general medical journal to launch a fully responsive mobile first version of its website, thebmj.com.
- The redesign was in response to a steady rise in traffic from tablet computers and smartphones, which recently reached 25% of all page impressions for the first time. In total The BMJ site gets up to 1.5 million unique visitors and 6 million page views every month.
- The project also involved the migration of the site from Drupal 6 to Drupal 7. The later version offers more support and better tools for responsive websites.
- In addition, the redesign included a major rebranding of the journal (both in print and online) with a new logo and url (thebmj.com). Its original url, bmj.com, is now the publishing company’s corporate site, with clear signposts to guide users to The BMJ’s new website.
- One of the biggest open source communities, widely used by other journals, magazines, and newspapers, a good collaborative ethos and ideological fit for The BMJ.
- High quality contributed modules which enable it to scale well to support very high traffic.
- Out of the box rich content management features and can be extended to match the very specific requirements of our website.
- An intuitive admin interface, enabling the The BMJ’s web editorial team to update the site quickly and easily
The project marks the latest in a tradition of digital innovation at the BMJ that began in 1995, when the journal became the first medical title to launch a website-- and made history again three years later, when it became the first to load its full archive online dating back to 1840.
The BMJ’s responsive redesign and migration to Drupal 7 was a complex project: it involved the migration of a huge online content archive made up of almost 3 million content items, as well as more than 30 custom Drupal 6 modules that needed to be migrated to Drupal 7.
The BMJ site runs on top of the JCore (Journal Core) platform, a Drupal-based solution for scholarly journals provided by HighWire Press based in Stanford, California.
While the HighWire Drupal team provided most of the JCore backend development, The BMJ’s in-house Drupal team focused more on front end development.
Features and implementation
One of the main aims of the redesign project was to create a more responsive site that would work across a wide range of browsers and mobile devices.
A key principle guiding our front-end technology choices was that the site should resize well, be quick to load, and be straightforward to maintain and modify. It also needed to work reasonably with IE7 and be usable in IE6 (see “Device coverage” in the Testing section).
We initially started with a bootstrap sub-theme and as the project progressed we progressively stripped out what was not needed to optimise load time.
The new site design was informed by the journal’s new logo, in Duplicate Slab font. It is described as having a warm and informal texture, a friendly but forceful feel, and an antique but not old fashioned tone, which seemed a good fit for a journal that champions digital innovation and possesses a proud heritage in print dating back 144 years.
The font’s serif version in lower case is set within a distinctive rounded blue “lozenge” so that it works across all digital resources at any size.
As users navigate their way through our site, they see consistency and rounded corners in a variety of pages and components. The combination of rounded corners and straight lines gives balance to our site, breaking up white spaces.
This cleaner feel was also achieved by removing many of the links and widgets that showcased The BMJ sister products (specialty journals, BMJ Careers, BMJ Learning modules and Masterclasses etc) found on the previous version of The BMJ site. User feedback following our last redesign in 2011 was that the site was cluttered; removing these features helped deliver a cleaner website, which is particularly important as traffic continues to migrate from large-screened desktop computers to smaller-screened tablet devices and smartphones.
The BMJ website displays every article dating back to 1840, many of them research papers with complex headlines. The new design allowed them more space across a wider portion of the page, so they appear less dense.
Front end development
Following the SMACSS approach enabled us to define our colours, font sizing, line heights, padding, and so forth in a simple and constructive way. As an added benefit, The BMJ can reuse this architecture for future projects that incorporate The BMJ journal branding.
TheBMJ.com uses a 12 column grid system with Susy Grid. With the power of Susy we can control everything via CSS, removing the need to add more classes to Drupal templates. As all Drupal developers know, Drupal does inject lots of its own CSS classes.
TheBMJ.com has 4 breakpoints to accomodate all mobile devices:
- iPhone (max-width 320px)
- Tablets (max-width 768px)
- Desktops (max-width 980px)
- Large screens (max-width 1200px)
To achieve these screen sizes we used breakpoint-sass. By defining our breakpoints in sass variables, we could insert media queries in css simply and cleanly.
The BMJ home page
The BMJ article page
FontAwesome 4 provides a wide range of fully scale-able icons which can be inserted via CSS or via markup. This meant we could insert them into different parts of the site where needed via whichever method worked best. It also meant we could keep the number of images we needed to load as low as possible.
Front-end page load performance optimisation
There are very well established rules, best practice guidance, and tools for speeding up websites. These include Yahoo Yslow and Google PageSpeed Insights, both of which were utilised to test page load speed.
Article content is edited by The BMJ editorial staff and published into HighWire’s XML ATOM store database. HighWire deploys a service that automatically feeds article metadata (article title, authors, publication data, citation information, etc..) to the Drupal database, while the body of the article content is kept in Markup Service.
Panels 3 module and Ctools plugin API were used to integrate the article content page to the Markup service. This makes it possible to serve and cache the body content in the Drupal panel layer.
The diagram below details HighWire services, BMJ Group and Drupal platform integration
The BMJ pioneered post-publication peer review when it launched pre-moderated article based “rapid responses” in the late 1990s. Soon afterwards it broke with tradition by ending the practice of publishing print based letters to the editor. Instead, the “cream of the crop” is chosen each week to populate the print and online letters section.
We decided to create our own commenting feature for this high-value, user-generated, scholarly content by using the CCK module. We felt the default Drupal commenting system did not have the required flexibility and features, such as the ability to add custom fields and specific moderation features, for example. The rapid response content type has custom fields (name, email, address, affiliation, terms and conditions and competing interests etc), plus the ability to upload supplementary files (images, tables etc).
In 2011 the journal introduced the ability to rank responses by introducing a ‘like’ button. This feature is popular both with readers and authors, so a key requirement of the project was to migrate these to the new site. We used the Drupal Migrate module along with custom classes written for votes (aka ratings) migration. A total of 1,211,611 votes were transferred.
Editorial workflow for home, channel, archive and specialties pages
The BMJ is an international publication with editorial and commercial hubs in the UK, US, and India. In 2008 the journal adopted a fully “online first" publication model. New content (up to 20 articles) gets published each day, with a subset chosen for print (and its predominantly UK readership) each week.
In 2012 the journal launched dedicated, IP-based home and channel pages for strategically important territories that currently include the UK, US, and India. There is also an international edition for all other countries. Users can switch between country editions via a dropdown box at the top of all pages.
Updates to the home and channel pages serving these four territories are handled by The BMJ’s web editorial team. We now have three main content channels: research, education, campaigns, and news and views.
These pages include “channel highlight” and “channel features” panels that are updated manually a few times daily.
The Drupal Nodequeue module allows editorial staff to pick any article content they want to be featured in these panels. The rest of the home and channel pages are updated automatically as Drupal database gets updated via an automated feed from the ATOM store.
Although a general medical journal, The BMJ is widely accessed by specialist clinicians. This audience is served by 15 specialty “portals” covering main disease areas (see http://www.bmj.com/specialties), underpinned by dozens of sub-specialty clinical topic collections.
All article content is tagged by a taxonomy system during the publication process. The Drupal taxonomy module in conjunction with Views is used to query and filter by the taxonomy term and Panel 3 is used to assemble the specialty portal pages. As new articles gets published and added to the Drupal database, they automatically appear in the relevant specialty page.
Video, Podcasts and feeds
The BMJ has been producing video and audio regularly since 2008, and the project included a migration of all video content from Mio’s platform to BrightCove, which is easy to integrate to Drupal and works better on tablet computers and smartphones. The BrightCove Smart Player API handled article-video integration.
Immediately prior to the project beginning all BMJ journals started serving audio via the SoundCloud platform. We have used the basic SoundCloud API widget to integrate the SoundCloud player into The BMJ’s pages. There is also a server PHP side script which retrieves latest Podcasts from the BMJ and generates feeds which is integrated to iTunes.
The BMJ site has a simple search and an advanced search system. For the simple search (aka quick search), an Apache Solr backend provides an index of the content accessible to Drupal.
The advanced search is powered by a HighWire based service called FAST and can provide search results from data held in HighWire’s content services.
Team and processes
Software development process
BMJ’s Drupal development team used Scrum during the development process to self-organise and collaborate with team members and business colleagues. This was part of a department-wide transition to Scrum as a software development process framework.
The BMJ redesign Scrum team comprised 3 back-end developers, 2 front-end developers/designers, 2 testers, the business owner and additional business support.
The back end team concentrated on three areas: migration of content from Drupal 6 to Drupal 7, migration of old modules to Drupal 7 and improving integration to HighWire services, and building a new continuous integration system using Jenkins.
The front end and design team concentrated on making all pages and components of the site responsive across all mobile devices and ensuring the site works across all browser versions, particularly IE7 and IE6. This is because some strategically important customers have yet to move to later versions.
The testing team had to meet two important QA requirements: implement a BDD (Behaviour Driven Development) based approach to QA, and iteratively create a full regression test pack for The BMJ website.
Our Scrum process worked around fortnightly sprints, each containing a planning session, at least one backlog refinement meeting to prioritise and size upcoming tasks, a show-and-tell to present each sprint’s progress to key internal stakeholders, and a retrospective.
We worked closely with The BMJ’s business owner and representatives throughout the process. Having a quick and effective line of communication was helpful in resolving issues quickly within a sprint, so one or two people from the business would attend sprint planning and backlog refinement sessions. A larger number of The BMJ’s editorial team routinely attended the show-and-tell and provided important feedback.
Rather than appoint a single Scrum-master to manage the Scrum process, the position was rotated around the team. This gave colleagues first hand experience of leading a scrum team and provided an important career development opportunity. It also resulted in the project not swallowing up one person’s time up disproportionately.
Source control management, workflow, and deployment
One of the aims of Scrum is to have releasable code at the end of each sprint. This was a challenging timescale. In the early stages of the project we released the site to an internal staging environment. As the project progressed we began releasing to a ‘pre-live’ environment, which was switched to become the live environment on 30 June, the day of the site’s first public release. This meant we were very familiar with a release’s potential impact by the time end users got to see the first publicly released version of the site.
The project used Git and Github for source control management, with the redesign project divided into three repositories: Drupal core, HighWire JCORE, and BMJ redesign. The JCORE and Drupal core repos are maintained by our co-development partners in the Highwire Drupal team in Stanford, California.
We used an amended version of the Git branching model, Git-Flow, to create a flexible branching strategy for managing our Git repositories, and for creating dev, test, and stable branches. Following our Git-Flow model, two days before every release, all new incremental code feature changes were frozen before we switched to the test branch. On the test branch only bugs fixes were allowed.
By using Vagrant, we were able to create single-click disposable and portable local development environments with consistent workflow. Developers had the capability to do code tweaks and do testing on their local development environment before sharing (pushing and pulling) their commits in to the centralised Github repo.
The QA server is integrated to the dev branch where Jenkins automatically pulls code changes from Github and runs automated build scripts and tasks for BDD testing.
Launch day deployment
The team prepared a list of launch tasks with directives for developers, testers and ops teams to ensure that the transition from the old to new site went smoothly, avoiding surprises and lengthy downtimes. We categorised directives into three parts: pre-launch, launch and post launch tasks.
Examples of the pre-launch tasks included content and new feature code freezes, final full regression pack testing, performance analysis and content load. As for every previous release, new incremental code feature changes were frozen and only bugs fixes were allowed two days before each release.
During the launch day, we had about a dozen operation tasks to perform. These ranged from temporarily stopping robots from crawling the site, pausing downstream processes and site monitors, modifying F5 and Varnish settings, putting the old D6 site into maintenance mode, and switching DNS records to the new D7 site.
For the post launch phase, we performed quick smoke tests to verify that important functionalities were working. We started monitoring the new site immediately using Google Analytics real-time features. This provided an immediate insight into the status of visitor activity across important geographical locations. From the real-time page reports, we saw visitors accessing some problem pages identified as 404, 500 and 501 HTTP responses. We were able to troubleshoot these problem and fix them in our dev branch before our customers reported them. We’ve also found Google Webmasters Tools to be very useful as it gave us a summary health check of the massive content migration we’ve done.
Overview of the HighWire infrastructure
The chart below illustrates HighWire’s back end infrastructure, beginning with the F5 load balancer, to the Varnish accelerator, Apache instances, CDN, database server etc...
Continuous delivery and continuous integration
To automate and improve the process of software delivery, we implemented a continuous delivery system for The BMJ redesign project. The aim was to deliver high standard software by automating our continuous integration build process to dev, test and staging sites.
The team was encouraged to make frequent code commits and push/pull their changes to our centralised Github repo multiple times a day to find problems quickly, fix them and deliver better quality software.
Jenkins was used to provide a continuous integration service for our development and test environments. We used the Phing project build system to write custom scripts to handle packaging, deploying and testing of our application.
One of the first build tasks was to assure the quality control of our code and report some important metrics. To achieve this we used some common tools like Drupal Coding Standard, PHP Mess Detector, PHP Copy Paste Detector, PHP Depend, CSS lint and JS lint.
Jenkins made it also very easy to assess the global performance of the main sections of our website. The load time of the most important pages were continuously evaluated with Jmeter to ensure each release will not slow down the site. Yslow was also used to keep track of the front-end page load speed performance.
Finally, Jenkins allowed us to automate our regression pack which was written in BDD. Most of our stories were supported by a set of acceptance criteria written in Behat. Whenever bugs occurred they were converted into stories and were included into our regression pack.
Profiling and performance optimisation
For back-end optimisation, we used XHprof profiling of important business components and pages to identify performance bottlenecks in our code and configurations. As a result, we used iterative code changes and configuration tweaks to reduce the PHP memory footprint and global CPU time to improve page load performance. Most of the issues were resolved by making use of static caching for external service calls, enabling panel pane caching and optimising Drupal Views’s queries.
Alongside XHprof profiling, the JMeter plugin for Jenkins was used to automate and measure page load performance. This allowed us to draw page load performance graphs in our project site screen monitor for various user roles (anonymous, institutional users, individual subscribers) and track performance trends as we were doing code optimisation improvements.
To achieve better quality software development, the project used a Behaviour Driven Development (BDD) approach. During the initial phase of the project, there was a steep learning curve for the team to adopt BDD best practices and test driven development.
Behat, Mink, and The BMJ custom steps were used as the sole BDD framework for the duration of the project. The tests were written in Gherkin syntax to ensure that they could be reviewed by non-tech business owners. The tests primarily focused on functional GUI testing that emulated a typical end users actions. Areas of scope included verify any editorially controlled as well as feed generated content on the website, verify rapid responses, search, alerts, validate any forms and validate any authentication required for access controls by different users.
The BMJ provides a mix of free open access content (including its research), and content behind access controls. Most of its paid usage comes from institutional subscribers. Access to The BMJ is a membership benefit of The British Medical Association. BMJ is a wholly owned subsidiary of the BMA.
All tests were triggered to run periodically every hour as well as when a change is pushed to GitHub. Tests are grouped and tagged according to the release type. For deployment to the development and staging environment, tests would have the tag @Regression, for deployment to the pre-live and live environment, tests would have the tag @Staging. For specific browser test management, tests will be tagged @Chrome to run specifically on chrome browsers or @firefox to run on firefox browsers. Some key testing figures include:
- 168 test scenarios
- 1600 test steps
- 119 test feature files tagged @Smoke
- 146 test feature files tagged @Regression
The site was verified across a number of OS and devices. These included:
- iOS 6&7
- Android v4 and upwards
- Windows 7 (Sanity check only)
The site was verified across a number of browser both on desktop and mobile. These included:
- Google Chrome
- Internet Explorer 6/7/8/9/10/11
- Windows mobile
A number of emulators were used where actual physical devices couldn’t be sourced. These included:
- Android sdk
- iOS simulator
Jira was the proprietary defect management platform used internally to record all defects. Defects would have a defect ID, severity level, title, steps to reproduce, screenshot/video where required, environment and browser the defect was found on as well as any additional supporting documentation.
In addition to Jira, the defects were recorded in a Google drive spreadsheet managed by the product owners. This gave added visibility and tracking of issues both on the server side (HighWire) and client side (Drupal).