Course-Notes.org

1) Clean up the database and organize the file structure of the existing Drupal 6 site

When I first started working on the original Drupal 6 site, I knew it was going to be a challenge. There were essentially two main problems with site: 1. hundreds of database errors and issues and 2. thousands and thousands of files that had been stored in default/files directory.

The site had obviously seen a ton of evolutions (especially problematic were lingering issue from one of the early version with the fusion site of vBulletin/Drupal). The site was use db_prefixes since it was a hybrid site/database which made it incompatible with Aegir hosting where I do most of my development and hosting. It was also a rather big database with lots of unused tables that had been lingering with either new version, even if the module had ceased to be used. A few development tools checks revealed there were several hundred “table errors” in that initial site.

My initial goal was see if I could get some version of site migrated to Drupal 7. So my initial cleaning effort was done via direct SQL manipulations and disabling and uninstall as many modules as possible. Since we were aiming for a Drupal 6 to 7 migration, I figured we should use core and contrib to do this. (Arguably I could have (and maybe should have!) used Migrate module.) So slowly but surely (emphasis on slowly) I got rid of enough tables and modules so that finally a version of the site landed in Drupal 7—warts and all.

Once I had gotten an initial version of the site in Drupal 7, I began recreating content types and Views structures in Drupal 7 and exporting them via ctools Features. By this point, I had collected enough of the site’s essence in Drupal 7 that I could drop the current database and start developing the Drupal 7 as such, including checking on ported modules and working on the site redesign. This allowed us to work on the Drupal 7 site, even though the database clean up and true migration wasn’t done yet.

While I had gotten a version of the site into Drupal 7, it had been done by removing anything that broke the migration/upgrade path. The task now was get the database cleanup, pre-upgrade preparation, migration and site re-launch into a series of more or less automated steps. To do this, I started creating several bash scripts that used SQL and Drush to break the whole process into several “cleansing” processes so I could debug in methodic manner.

The initial cleansing script fixed the db prefix issue and dropped various unused tables. A subsequent script used various SQL queries to fix table errors. And a final “cleaning” script disabled and uninstall any modules and their tables that were no longer used. At this point, we reached a point where the Drupal 6 site was still working and the database was “fixed” in a valid format.

With the database fixed, the next problem to fix was the files directory. As a notes exchange site, Course Notes gets thousands of document submissions, which unfortunately weren’t properly uploaded into any structured directories. FTP and SSH had issues even accessing the files directory.

My initial fix was to move all of the files from default/files to default/files/past and then rewrite the database accordingly. I then used a wonderful module File (Field) Paths (http://drupal.org/project/filefield_paths) which not only allows you to use tokens to set paths but provides a way to “retroactively update” files. I made a few adjustments and then saved the file fields. A batch process started and I watched as file and file was moved into subject-specific directories. Wow!

2) Upgrade to Drupal 7 using core modules whenever possible.

While the database had been cleansed, there were several problems in migrating the site but the major problems were:

a) Modules not yet ported or fully ported to Drupal 7

When we started the migration work, we created a list of modules that were “Must Have” for migrating to Drupal 7 and their Drupal 7 status or alternatives. As a bootstrapped project, Course Notes has tried a lot of modules and was using several rather small modules that weren’t ported. We prioritized and helped fix or port several modules like Text Extract for Drupal using Tika, Quizlet and helped fix a few others like quiz. Converting the site’s content types and Views to Features made the migration a bit easier since we could then apply changes to fields by enabling and reverting the Features modules.

Obviously this step with functional migration is a delicate one and in a few cases we needed to completely disable and reinstall the module in the new site to avoid errors. In other cases, like Premium module, which is a popular module in Drupal 6 but had no port to Drupal 7, we were forced to use an alternative module nopremium along with some SQL queries to update preexisting content.

b) Multiple-use file references that breaks upgrades

One of the biggest migration issues we had was with file path conflicts in file fields in Drupal 7 and CCK field conversions. The problem was the fact that multiple fields were referencing the same file in Drupal 6, which wasn’t a problem. There are specific changes in Drupal 7 core that require unique file references so even though there were technical solutions and hacks to core to make this work in spite of this problem, I didn’t like the idea of a “patched” core. So I used various SQL queries to determine the duplicates, then use some bash code to copy and rename the duplicate files and then redo the database references. This is likely not a solution for all sites but for me, this fixed the integrity errors and moved the migration process along.

These file path fixes were run as part of a final cleanup and prepare for migration bash script. This script fixed some of the miscellaneous errors that remained but couldn’t be applied without breaking the Drupal 6 site in some way. This script also made a bunch of Drush commands disabling and/or uninstalling all non-core modules. I also used Variable Cleanup module to check on unused variables and deleting several hundred variables from the variable table.

At this point, the functional essence of the site could migrate to Drupal 7. For those of you that haven’t done a Drupal 6 to 7 migration it’s important to do it in several stages when you have a lot of modules. So first migrate just the core modules, then slowly added more code and contrib back into your site and run update.php (better to use Drush!). It was a bit of trial and error to figure out which modules needed upgrading first but eventually the order worked. After I, of course, used CCK field migration to help migrate all the CCK field to Drupal 7 fields.

At this point the site, I used an additional bash script that applied all of the new Features and enabled various modules and changes that were built. A few changes were needed here and there for the UI but it was mostly seamless.

c) Converting core profile stuff to fieldable user entity

The final step, which was more about future compatibility than purely needed for migration, was converting core profile to fieldable user entity. Depending on your experience and history with Drupal, you’ll notice two user profile modules in Drupal 7. This is because Drupal 7 moved to fields everywhere approach and there were unfortunately a few stragglers, namely core profile module which doesn’t use fields and provides a nearly exact equivalent to Drupal 6 version. So while the core profile stuff migrated more or less fine to Drupal 7 (excluding some of the minor modules that we dropped), the core profile stuff is quite functionally limited compared to the user entity. So we decided to convert this to Drupal 7.

There are a few approaches to how to do this and discussions around Profile2 are quite helpful. In the end, I created a new user feature with the user entity fields I wanted and used some custom code that transferred field values. Since the site has so many users, this script took some time to perfect and optimize in terms of speed.

3) Redesign / rebrand the website and introduce a responsive theme.

With the database and migration work taking so much time, we hesitated about leaving the new design for post-migration work, but finally we decided (and rightly so, I think) to include a new design and theme in the new site. After spending many hours looking at websites for inspiration, taking hundreds of screenshots of design elements, and mocking up the site, we came up with a basic wireframe of “needs” and “page elements”. These initial “page elements” were reworked into a new front page design, logo and branding. This became the inspiration and base palette used for various internal pages.

For better or worse, we didn’t take a “mobile first” approach since our design was clearly for desktop. This obviously created later problems. With the PSD design, we created an initial HTML/CSS version in order to get the basic elements working and looking correct.

We opted to use Omega base theme from the beginning. Omega provides a very solid, responsive framework and creating multiple regions is extremely easy. The other key omega module that we use in several places in the site is Delta module which allows you to create an alternative layout that can be triggered with context module. This has allowed us to create major layout changes in different sections from frontpage to dashboard or book pages to forum pages.

4) Improve the user experience.

High school students are under a lot of pressure these days. There once was a time where you could simply score highly on standardized exams and earn good grades and you could get into the college of your dreams. Now, in addition to getting a perfect score on your SAT and earning straight As in all of your classes, you’re also expected to be the captain of varsity soccer team, volunteer at the local animal shelter, and take every Advanced Placement course under the sun. Students have been forced to learn how to study more efficiently, which is where Course Notes comes in.

Course Notes utilizes Drupal’s Book module to organize content that has been curated by Course-Notes.Org administrators. This content typically consists of entire sets of chapter outlines for a textbook or other study material that needs to be grouped together. Unfortunately, this meant that students were at the mercy of the Course-Notes.org administrators for finding and adding new content and content was only added when there was a complete set of outlines available.

To solve this problem, we introduced a ‘Premium’ content section allows students to exchange individual notes, flashcards, essays, and other materials with each other. In order to access these materials, we require students contribute back to the community by sharing their own notes, participating on the forum, writing articles for the blog, or providing feedback on other student’s work. Each of these activities yields the user a certain number of user points, or c-note$ as we call them. The User Points module handles the granting of points and automatically promotes users into the Premium content role as soon as they meet the minimum point requirement.

Once control of the content curation was relinquished to the community, there were a number of steps taken in order to ensure this user generated content remained well organized, easy to find, and the user experience didn’t suffer.

a) Organization
In an ideal world, users would tag their content with the most descriptive phrases making it very easy for users to find related documents. Unfortunately, this rarely happens. Integrating the OpenCalais/Social Tags module allows user generated content to automatically be scanned and related tags inserted. For example, if a user uploaded a document related to the ‘Revolutionary War’, the document might be tagged with terms like ‘George Washington’, ‘Boston Massacre’, ‘King George III’, and ‘Declaration of Independence’.

b) ‘Searchability’
Students have the option of either pasting text into the body field from a document, or attaching the document itself. A majority of the students opt to upload the document which results in a blank field and users only able to rely on the title field to determine how useful the attachment will be to them. To solve this problem, a custom module utilizing Tika to extract the text elements from the documents was developed. This gives unregistered users to view the teaser text, premium users to view the full text without downloading the attachment, and increases the likelihood of search engines indexing the page.

c) User Experience
Students are able to upload a variety of different file types, such as Word Docs, PDFs, PowerPoints, etc. Some of these file types play better than others when formatting the extracted text from them. To provide the best user experience possible, uploaded attachments are embedded into the page using the Google Docs Viewer module. This allows users to preview the document in its original formatting prior to downloading it to their computer.

5) Optimize site performance and speed.

From a performance perspective, we decide that post-migration we’d need to think about possible moving servers. The previous server lacked many optimizations and did not quite have the Drupal “awesome sauce” we wanted.

For anyone thinking about Drupal site performance and hosting, it’s not an easy decision. It’s not an easy decision since there are a number of great Drupal hosting choices available today. The major players are Pantheon, Acquia and Omega8.cc. There models and methods are different. If you are interested in the details, check out this Quora discussion which layouts

In July, we decided to migrate servers to Omega8.cc hosting. Their high performance Aegir/Nginx hosting along with various technical suggestions have significantly improved page loads. We also switched from Drupal's default search to Apache Solr which significantly improved server load.

While all of the above changes improved the page speed, we hit a glass ceiling (or a floor, rather?) due to the advertising blocks blocking the loading of the remaining page elements until the ad unit had completely loaded. Unfortunately, simply removing the advertisements from the site is not an option as they fund the site’s hosting and the development efforts. By utilizing Google DFP’s asynchronous ad tags to serve our impressions, we were able to drastically reduce the page load time.