Plone, Repoze, and the Dojo Toolkit

I was hoping it would be made public a couple weeks ago in September, but it just went live this weekend. It's the first of a series of sites Saugus.net has been working on to utilize cross-platform vector graphics via the Dojo Toolkit. To make it more interesting, most of these sites also use the Plone CMS.

Plone and Dojo don't coexist so easily. There is an old product called ZPDojo (with a slightly newer fork) that managed to pull it off a few years ago, but it was using an old version of Dojo. I decided to take a different approach and instead used Repoze and Deliverance to layer the Dojo portion in a logically separate portion from the Plone portion. By doing so it became fairly straightforward to use Dojo's excellent build system to build the requisite custom JavaScript package to insert into the Deliverance portion while continuing to use Plone's own excellent merge system to operate on Plone's requisite JavaScript (and even preserve KSS capabilities).

All is not perfect. KSS relies upon JQuery; theoretically this is somewhat wasteful as by pulling in Dojo's GFX package one is already getting pretty much all the capabilities that JQuery offers resulting in a slightly slower-than-necessary initial load time, and practically it breaks unless one removes JQuery from Plone's automerge system and pulls it in manually. The next version of Dojo has a JQuery compatibility package planned, so long term the theoretical solution is probably one of removing JQuery entirely and relying on the compatibility package (perhaps even gaining a bit of speed), but it's still early to say for certain. In the meanwhile, separating out JQuery from the rest of the Plone JavaScript source files and linking it directly solves the practical problem.

The site in question is the Shining Stars Learning Center. They wanted lots of stars on a light blue background, and they got them in spades. A single star drawing routine is looped through a number of times based upon the client's screen size, filling the current browser window (actually the maximum possible browser window should the window be resized) with stars of random size, rotation, and position. To add a bit of whimsy, all the stars can be independently dragged around the window.

The Shining Stars logo in the upper left hand corner is also drawn via vector graphics, although it can't be moved. I disallowed stars being drawn behind it in the initial star populating phase, but they can be dragged behind it after the window has been drawn. It was converted from SVG to Dojo GFX JSON using my fork of svg2gfx.xsl (which will soon hopefully go into the main trunk).

The whole site, including the logo and stars, is built around em size, so resizing the text causes the logo and stars to resize accordingly, too, and since they're defined using vectors they never get fuzzy.

Actually, for the fun of it I tried copying a few of the resulting vector images into bitmap images (one of which is used for the Shining Stars Twitter background). Besides being noticeably fuzzier, the image ended up being somewhat larger than the JavaScript used to define it, too. Obviously this won't always be the case depending upon specifics (and it certainly wasn't my driving motivation; I wanted a clean appearance with nice resizing) but I found it to be an interesting perk.

When I say it's a cross-platform solution I really mean it. Not only does it work on the usual suspects (Firefox, Safari, Opera, Chrome, MSIE, etc.) but it works on the Wii Internet Channel as well. Shining Stars has a Wii in their boardroom (how cool is that?) that they can use for browsing the Web (among other things) with the Wiimote as a pointer. I thus did as much testing of the site on the Wii as I did with the other more common browsers, and it works surprisingly well. It's a bit slow rendering the vector graphics, but not disastrously so, and the star dragging even works. The only problem I had with it is that when one switches into its so-called "Single Column Mode" almost nothing gets displayed at all. This could be simply alleviated by selectively turning off some JavaScript and CSS when the mode is enabled, but I've yet to find a way to reliably detect it. It's not a huge worry though as the site does work fine in the Wii's default display mode.

Microsoft Internet Explorer (as usual) provided its own set of problems. While on most browsers (including the Wii Internet Channel) the Dojo GFX stuff gets rendered as SVG, MSIE instead uses VML. Unfortunately Microsoft's recent crusade to catch up with other browsers' support of standards has not included standard vector graphics in the form of SVG, and to make matters worse they've allowed their own VML to rot with each new version of MSIE seeming to introduce new, unaddressed VML problems. In particular, some of the vector graphics the site uses cause VML issues in MSIE8, and the only way I was able to fix it was to use the Microsoft-specific X-UA-Compatible to explicitly request that MSIE8 render the site as if it were really MSIE7. Since I was forced to mess with this anyway, I took it one step further and added Chrome Frame support with the line:

<meta http-equiv="X-UA-Compatible" content="chrome=1;IE=7">

and dropping all MSIE-specific hacks when it is engaged.

It now uses Chrome Frame (if present) in preference to any of MSIE's native rendering modes, and uses MSIE7 rendering mode if in a newer version (that lacks Chrome Frame). Chrome Frame is visibly faster than all of the MSIE native rendering modes for this site, so much so that we (for the moment, anyway, we may remove it) added a notice to let regular users of the site know about it in order to improve their browsing experience. I'm assuming that the speed difference is mostly due to a combination of Chrome's faster JavaScript implementation and the fact that no extra JavaScript hacks are required to make up for CSS deficiencies, but regardless of its source it is significant.

Please note that the site is still in a beta mode of sorts and thus may not only still have bugs but will also be in fairly frequent flux. If things don't look as you'd expect from the above description, please try again in a few minutes; you probably just caught it during an update.

Antichrist Structure

Last time I mentioned how I'll post some articles discussing how some of the key technologies (like the Dojo Toolkit, PostgreSQL, Twisted, Zope, Python, and Repoze) are used by Antichrist Watch. In this post I'll discuss some of the general architecture behind the site, along with motivations behind some of the design decisions I made. In future posts I'll follow up with specifics related to some of the key technologies employed.

I've already covered the originally planned core features of the site, so I won't repeat them here. What's important for this discussion is that those features require fast access to live data on votes, tallies, and comments, and the ability to display that data in a number of ways, and that everything be done cleanly according to current industry standards (that is, using decent semantic HTML backed by appropriate CSS and RDF that'll not cause the respective W3C validators to choke). Obviously an AJaX design is called for, and the individual toolkits and frameworks used must be flexible enough to support reasonably standards-compliant code and mark-up. This may sound trivial on the surface, but oftentimes toolkits, frameworks, and even design tools have their own axes to grind and introduce deliberate incompatibilities or proprietary extensions in an attempt to promote their own stuff or denigrate others' stuff.

PostgreSQL for the back-end database was an easy choice. It performs well, has great support for standard SQL (and even XML) built-in, and has advanced capabilities like triggers, rules, and stored procedures that can be used to help ensure data integrity. I'm a big believer in keeping code that enforces data integrity as deep within a Web application as practically possible; the closer it is to the front-end the more space there is for something to go wrong. With PostgreSQL I was able to get most of the code that handles data integrity embedded in the database itself, so not only is there not even a theoretical way for a crafty user to bypass constraints, there's not even a theoretical way for misbehaving middleware to bypass them. I'll discuss this in more detail later.

The Dojo Toolkit for the front-end JavaScript will be surprising to some as many believe its recent (post 1.0) versions cannot be used within a site that validates. This is incorrect. What is true is that Dojo's Dijit widgets can not be used in a validating site via mark-up without modifying Dojo's parser. This is a completely different statement, as there are other (and better) ways to apply Dijit widgets to a site. I'll discuss my use of the Dojo Toolkit in quite a bit more detail later. Most of the site's code is actually Dojo code, so I suspect I'll dedicate a couple of posts to it.

Of course, the JavaScript needs to be tied into some mark-up, and for this I used a single Zope Page Template designed to output clean XHTML. This would be trivial for a typical Zope site, but in this case it's certainly not typical as only scattered Zope technologies are being used. I'll get back to this in a bit.

The Dojo Toolkit has the concept of an abstracted data store that it leverages heavily for most of its widgets. It has numerous concrete implementations of this store that make binding it to various data sources fairly trivial. The one it includes for dealing with relational databases requires data to adhere to a particular format that would be awkward to generate directly in PostgreSQL, so it was necessary to create some code to fetch data from PostgreSQL and format it for Dojo. Since I had already built a great deal of intelligence into the PostgreSQL database itself, it was actually fairly simple to create this middleware, and during development I tested out a couple of different approaches (including raw Python and PHP) before settling on one designed around Twisted Python. It makes good use of Twisted Enterprise and the psycopg2 database adapter (including its splendid DictCursor) to make database access fast, efficient, and comprehensible. I also tied the rendering of the main Zope Page Template into this Twisted app by virtue of Repoze Chameleon in lieu of making it a typical Zope site. This whole layer is probably also worthy of its own post.

That's a quick run-through of what it takes to reveal the Antichrist. I'll be posting more details on these individual components later; please feel free to point out areas of particular interest.

Who Is The Antichrist?

I know it's been a long time since I've written anything here. A really long time. I figured I'd break this silence with an overview of a project I've been working on lately that will hopefully prove interesting.

It's a site called Antichrist Watch. It's a site that attempts to answer this article's titular question who is the Antichrist? in a Web 2.0 sort of way using a fun mix of AJaX technologies.

I recently realized that much of the work that I do ends up buried within various intranets and extranets and is thus invisible to the world at large. While I've worked on all sorts of dynamic systems that do all sorts of things (both server-side and client-side), I really don't have any that I can show people. Most of the public sites that I work on tend to be simpler sorts of things, and people viewing them tend to mostly remember the graphics design elements (which in most cases I didn't actually work on anyway). With Antichrist Watch I have something that I can show both curious acquaintances and potential clients. The dynamic properties of it are fundamental to how the site works, and even people not all that familiar with the Web can get it.

Under the hood it uses a somewhat unusual combination of key technologies that not too many others use together: the Dojo Toolkit, Zope, Twisted, Python, and PostgreSQL. While people familiar with my company Saugus.net will not at all be surprised with Zope, Python, and PostgreSQL (we've been using them all pretty prominently for years), Dojo and Twisted may be unexpected. We've actually also been using these off and on for quite awhile too, but not in too many public projects. In fact, thinking back, there are currently only two public areas of Saugus.net sites that utilize Dojo, and no public areas at all that use Twisted. While we are hoping to change this in the relative near future, Antichrist Watch demonstrates how these technologies can work together in harmony now.

While I'm still working on the site fixing bugs and adding new minor features, all of my originally planned core features are in place:

1. The ability to vote on one's own choice of Antichrist, whether or not he or she is already in the list, with autocompletion based upon the live list of everyone's past choices.
2. Charts and tables showing current leaders for the past day, the past week, the past month, the past year, and all time.
3. Additional charts and tables showing votes as they come in with individual votes mapped to rough geographical areas, and the recent levels of voting activity over time.
4. A way to drill down and see the numbers for any particular Antichrist candidate.
5. A facility that allows one to argue for or against the likelihood of a particular Antichrist candidate being the Antichrist, respond to others' comments, and display it all in a tree.
6. The means to moderate the aforementioned comments.
7. A login system allowing people to establish identities within the site and voluntarily add personal information that will be displayed along with their comments to add personality.
8. Support for various icon types to go along with login identities, including at minimum PIcons, Gravatars, and .Mac icons.
9. Support for OpenID to ease the whole login process.
10. Basic support for forward and back buttons within the site.
11. A reasonably semantic design including a fully populated external RDF metadata file boasting appropriate Dublin Core terms and DOAP information (plus a few other useful nuggets).
12. Cleanly handle the full UTF-8 character set throughout.
13. Proper validation of pretty much everything according to the W3C's respective validators.

It does all of this in a way that almost totally avoids page reloads (the only exception is with OpenID authentication which by definition requires a redirect offsite and back). It also heavily makes use of lazy loading within its tables and trees, so that although access is provided to all data all the way back to day one, only data actually required for the current view gets downloaded keeping things fast on both client and server side. Of course, through the magic of Dojo it also works pretty well in all major browsers.

There are still lots of little details that need to be added. A means for automated password recovery quickly comes to mind, as well as more styling (what's there now is pretty close to being bare Dojo tundra), some more selective refreshing of data based on user input, better error handling (both in terms of weird user input and HTTP type errors like the dreaded 404), some friendly help for new users, and some more intelligence added to certain buttons and dialogs already within the site.

I'll post some more articles here discussing some of the key technologies and how I put them to use within the site. If anyone has any particular areas of interest, don't be afraid to speak up.