Achieving sub 5ms performance for authenticated users with Drupal on shared hosting

Disclaimer: This is a work in progress, and as such probably not *quite* suitable for production sites unless you understand the issues and want to “chip in”. However the instructions are pretty simple and will work on most hosting set ups from a modest shared hosting account to a server farm.

We all know the problem: Drupal has an excellent caching system that serves pages extremely quickly to *anonymous* users. Unfortunately, for authenticated (ie logged in) users, caching becomes a *much* more complicated problem. The current OOTB solution to this complication is to simply disable caching for authenticated users.

The situation is pretty dire. With stock Drupal a page can be served to an anonymous user in around 30ms, for an authenticated user on a site with a reasonable selection of modules, this time can easily increase to 650ms or even 900ms. Worse, because an authenticated user page request uses so much more system resource, it’s much more difficult to scale sites to large numbers of authenticated users. Even a  fairly beefy dedicated server can probably only manage somewhere between 2 and 10 authenticated requests per second and as the number of concurrent requests increases, the server quickly gets swamped and page generation time increases to between 5 and 10 seconds per page!!

Fortunately, there is a solution ! After much investigation I discovered the authcache module. This looked to be close to the solution we’re all looking for but there was no D7 version and didn’t seem to be any progress from the module maintainer in upgrading.

What you need
If you’d like to see your Drupal site *flying* along with a <5ms page generation time, this is what you’re going to need:

  • Drupal 7 or Pressflow – I’m using Pressflow but this approach will achieve comparable results with stock Drupal.
  • Authcache module.
  • (Memcached module no longer needed :) )
  • A small core hack – I’m sure there’s a way round needing this but I haven’t found it yet
  • An understanding of the caveats

General Instructions

1. Install authcache module – carefully follow the fairly straightforward instructions in the README.txt

2. At this point you should find authcache is working and your page generation times (for cached pages) are very much faster (in the area of 10ms for me). If not, it may help to run “clear cache” and also log out and log back in again. Also note that caching is still disabled for the “Admin” user (ie uid=1) so you won’t notice any improvement whilst running as “Admin”.

Caveats

1. Whilst providing very fast response times in many circumstances, this solution is not a “magic bullet”. Very fast response times are possible under all sorts of “personalisation use cases” but it requires work and understanding. Sometimes *much* work and understanding.

2. The authcache module works by generating a hash of all the groups a user belongs to and using that hash as a prefix to the “cache key”. The effect of this is that a user with an obscure or unique combination of groups will be much less likely to get a “cache hit” and will therefore get a slow (ie 650ms) page generated from scratch.

3. This is currently a work in progress. At some point (soon) I should be able to make the Authcache module a “point and click”, zero configuration install, but for now, you early adopters will have little work and understanding to do. Any site owners out there with high traffic authenticated user Drupal sites that would like to help me “remain alive” whilst working on this project would be very welcome to get in touch.

4. If you edit a node / page you’ll notice you need to clear the cache to publish the changes. I’ll solve this problem (easily) but there are likely to be other similar issues waiting to be discovered.

5. The authcache module uses javascript to personalise the page for individual users (ie “hello, user name” and all that). You should read the authcache documentation fully to understand the potential implications on how you should develop your site.

6. one issue that has become apparent in porting the module is that there is currently a “security consideration”. Authcache potentially makes all cached content available to anyone who has the appropriate cache_key cookie set. This makes it somewhat easy for a “curious” person to fake the appropriate cookie value and gain access to “private” content from the cache.

This means that authcache is currently only suitable if you want to speed up sites with roles used for “personalising” the content. Authcache currently only provides very “casual” protection to private information so if you want to keep certain content totally protected you will need to wait for a subsequent release of this module.

Posted in Drupal | Leave a comment

Running Large Scale Web Sites

Below are links to a couple of articles on Slashdot about the hardware and software required to run a massively high traffic website.

Hardware Requirements
Software Requirements

Posted in Content Management, Drupal | Leave a comment

Moving Drupal modules from sites/all/modules to sites/all/modules/contrib

If you move your local repository of Drupal modules from sites/all/modules to sites/all/modules/contrib you may find that your site dies with error messages like “failed opening required”. This is because Drupal stores the location of the module files in system and registry tables in the database.

If this sounds like you, here is the SQL you need:


update system set filename = replace(filename,'sites/all/modules','sites/all/modules/contrib');
update registry set filename = replace(filename,'sites/all/modules','sites/all/modules/contrib');
update registry_file set filename = replace(filename,'sites/all/modules','sites/all/modules/contrib');

Posted in Drupal | Leave a comment

Launch of ShopGreenwich.co.uk site

I’m delighted to announce the launch of my latest customer project ShopGreenwich.co.uk.

The site was built with Drupal 7 and makes extensive use of taxonomies (categorisation), google maps. It also implements the Apache Solr search engine combined with Drupal’s “faceted search”.

Posted in Content Management | Leave a comment

3d Graphics Revolution

For a number of years the goal of graphics technology has been to create 3d images and environments that are “indistinguishable from reality”. That is, when you look at a computer generated image, it’s hard or impossible to tell if it’s real or not.

Euclideon have made quantum leap towards achieving this goal. The company are apparently 18 months or so away from releasing any kind of product but take a look at the video. It’s incredible.

Edit: Worth noting that This post by a well known games programmer suggests that the technology may not be the quantum leap that it appears to be. Apparently others have done similar work before and there are many drawbacks yet to be solved, but not mentioned in the video.

Edit 2: Here’s an interview with the CEO of Euclidean explaining a bit more about their software and defending claims that it’s just a scam. Aside from the slightly annoying delivery by the CEO, it looks pretty promising.

Posted in Computer Programming, Human Achievement, Technology | Leave a comment

Garra rufa Fish Pedicures

Fish Pedicure in Wimborne, DorsetOver the weekend I made this site for my girlfriends business, Fish Pedicure in Dorset

The site is still a little rough around the edges but we’ll get a graphic designer on the case in the near future.

For now though the priority is traffic generation using SEO, social media and guerilla marketing tactics!

The site is built in Drupal 6 (because that’s what I had “lying around”) so will be upgrading to Drupal 7 when time permits.

We also have a Facebook page so if you’re a fan of Garra rufa fish pedicures then check it out and “like” us :cool:

Posted in Content Management, e-commerce, Web Development | 2 Comments

Geocoding and Google Mapping with Drupal

Armour Auto Dealer NetworkJust thought I’d show you the “dealer locator” I built recently for Armour Automotive.

The dealer locator is built using the Drupal GMAP and Location modules in conjunction with my own bespoke module.

The bespoke module makes it possible to pull location data from a central source in a multi-site Drupal environment. Armour Automotive owns a number of audio and automotive brands and the ability to easily manage all their dealer location information in one central location and have it automatically published to the appropriate “brand” site has been both an eye opener and a time saver for Armour.

The past few months has seen heavy investment by Armour in re-vamping 10 brand websites using Drupal 7 and yours truly helping with the technical stuff. It’s been a very busy few months to say the least, but I’ll be publishing details of some great new Drupal 7 sites in hopefully the next few days.

Much of the credit for the ArmourAuto.com site has to go to James Hoskins and Stuart Ephgrave (both at Armour) who have put a lot of high quality work into making great content for the Armour site – not to mention, believing in the possibilities …

Edit: A link to a more recent (and more advanced) map I’ve built www.shopgreenwich.co.uk/map. Source code for the Drupal module is provided in the comments for anyone interested.

Posted in Content Management, e-commerce, Web Development | 11 Comments

Changing the collation of all tables in a mysql database

If you’re reading this, you’re probably wanting to change the collation of your mysql database to utf8_general_ci in order to support non-English characters.

There seems to be lots of methods out there for doing this, probably none of which are as quick and simple as my script below :cool:

This script changes the collation of the mysql database, every table and every column that has an existing collation.

Cut and paste the script below into a file called (say) change_collation.php on your server. Set the connection properties at the top of the script, then call the script via your browser.

*** TAKE A BACKUP BEFORE RUNNING THIS SCRIPT ***

If you don’t take a complete backup before running this script then you are extremely foolish and don’t come crying to me when all your data is gone. This script worked fine for me but changing collations might have all sorts of odd effects that I can’t predict on your system. Almost certainly everything will be fine, especially if you have a backup !!


<?php

// Configuration Section
$server = 'localhost'; //probably localhost but change if required
$username = 'your_username';
$password = 'your_password';
$database = 'your_database';
$new_charset = 'utf8'; // change to the required character set - you're probably changing to utf8 ?
$new_collation = 'utf8_general_ci'; // change to the required collation - you're probably changing to utf8_general_ci ?

// Connect to database
$db = mysql_connect($server, $username, $password); if(!$db) die("Cannot connect to database server -".mysql_error());
$select_db = mysql_select_db($database); if (!$select_db) die("could not select $database: ".mysql_error());

// change database collation
mysql_query("ALTER DATABASE $database DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci");

// Loop through all tables changing collation
$result=mysql_query('show tables');
while($tables = mysql_fetch_array($result)) {
$table = $tables[0];
mysql_query("ALTER TABLE $table DEFAULT CHARACTER SET $new_charset COLLATE $new_collation");

// loop through each column changing collation
$columns = mysql_query("SHOW FULL COLUMNS FROM $table where collation is not null");
while($cols = mysql_fetch_array($columns)) {
$column = $cols[0];
$type = $cols[1];
mysql_query("ALTER TABLE $table MODIFY $column $type CHARACTER SET $new_charset COLLATE $new_collation");
}

print "changed collation of $table to $new_collation<br/>";
}
print '<br/>The collation of your database has been successfully changed!<br/>';
?>

Posted in SQL programming | 7 Comments

Launch of Solatricity website

Solatricity Solar PV ElectrictyJust a quick announcement to the world that I’ve recently launched this site for one of the uk’s leading solar panel installation companies.

The site has been built using the Drupal 6 content management system and has a suite of modules to enable group based permissions, workflow and publishing approval, automatic image resizing, WYSIWYG content editing, friendly intuitive urls and lots more state of the art content management functionality.

There is also a secure area for customers to log in to report technical problems and a searchable knowledge base for common customer questions.

The site combines sophisticated tracking of user form submissions with an automated data feed directly into their back end CRM and business systems. This enables Solatricity to respond quickly and effectively to customer enquiries and also allows them to analyse which online and offline marketing activities have been most successful.

Credit to Zane Clements for the superb graphic design we used for the site. (Note: The logo design for the site changed significantly just before the site went live and so it doesn’t look quite as beautiful as it once did. Zane is being commissioned to do some more work to adjust the site to accommodate the new logo.

Posted in Content Management, Customer Service, News | Leave a comment

Scalextric Muscle Car

I’ve been getting interest in my Scalextric lap timer software recently, which reminded me about this project that I did a couple of years or so ago.

My friend had a few broken Scalextric cars lying around (not to mention a lot of non-broken ones) and I had a spare 12v electric motor from a HoneyBee King 2 remote control helicopter.

You can imagine the direction in which my thoughts began to wander …

It was more than a little fiddly to get the huge motor securely fixed to the chassis, the motor shaft nicely lined up with the drive gear on the rear wheels and the body shell back in place. The results are shown below.

Standard Weedy Engine
Standard "weedy" engine

As nature intended
HoneyBee King 2 RC Helicopter Engine

Look at the rotors on that …
HoneyBee King 2 RC Helicopter Engine in Scalextric Car

Completed Scalextric Muscle Car
Completed Scalextric muscle car

Sadly, the car never actually worked. It drew too much current and the short circuit protection on the Scalextric power supply kept cutting the juice. I did have plans to sort out some kind of booster / override for the power supply but annoyingly my friend decided that it would be interesting to take a blow torch to the car and photograph the flames melting the plastic.

Um, yeah.

In fairness, it was my fault for not making it clear how much work it had taken and the potential for one day solving the power supply problem.

Posted in Inventions, Scalextric | 2 Comments