Wednesday, December 30, 2009

End-of-Year Miscellanea

I haven't posted too much lately. Here and there I have been working on a few interesting projects; I continue to add little bits of functionality to Antichrist Watch and have added dojox.gfx vector graphics to a handful of sites (including among others NEARTA, Keep Saugus A Town, and Shining Stars Learning Center). Using dojox.gfx vector graphics is really interesting; it enables the sizing of elements within a site to be completely based on em size and/or window size without significant increase in load times or loss of graphical clarity. My svg2gfx.xslt utility was recently promoted to Dojo proper (in time for the 1.4 release), too, and this should hopefully make creating dojox.gfx graphics a bit simpler for some. I'll try to write something up describing the general process of making a vector image logo for a site in the relative near future. I've also been messing around with getting egg-based builds of Zope 2 and Plone working under Repoze with varying degrees of success. I don't have a solid, simple enough solution for either of these to post much of anything yet.

I've also spent a not-insignificant time lately messing around with e-mail. I've been using Apple's for awhile now, and whenever I've needed encryption I've simply used its built-in S/MIME capabilities. With a free Thawte client certificate, it was extremely easy to set up and just worked everywhere. As you're probably aware, Thawte recently dropped its free certificate service. This meant that I had to change my ways. The simplest path was to start using the free CAcert service for client certificates and instruct those with whom I'd already been communicating via S/MIME to pull in the appropriate root certificates. I did this fairly painlessly (although I still have to hunt around for some locals willing to vouch for my ID; if you're in the North-of-Boston area and willing to do so, please drop me a line). As I'll be working more closely with the SitePen folks though in the near future and they all use PGP I wanted to get it working as well. Sadly, does not have the nice built-in support for PGP that it does for S/MIME; to make matters worse, the GPGMail plug-in to give PGP capabilities isn't quite yet ready for the current version of There are some unofficial releases available through the forums, and I got the newest of these to work. Thus I now (finally) have both S/MIME and PGP signing & encryption working for my e-mail. (Although I also have to hunt around for locals to sign my PGP key; if you're in the North-of-Boston area and willing to do so, please drop me a line.)

Sunday, October 04, 2009

Plone, Repoze, and the Dojo Toolkit

I was hoping it would be made public a couple weeks ago in September, but it just went live this weekend. It's the first of a series of sites has been working on to utilize cross-platform vector graphics via the Dojo Toolkit. To make it more interesting, most of these sites also use the Plone CMS.

Plone and Dojo don't coexist so easily. There is an old product called ZPDojo (with a slightly newer fork) that managed to pull it off a few years ago, but it was using an old version of Dojo. I decided to take a different approach and instead used Repoze and Deliverance to layer the Dojo portion in a logically separate portion from the Plone portion. By doing so it became fairly straightforward to use Dojo's excellent build system to build the requisite custom JavaScript package to insert into the Deliverance portion while continuing to use Plone's own excellent merge system to operate on Plone's requisite JavaScript (and even preserve KSS capabilities).

All is not perfect. KSS relies upon JQuery; theoretically this is somewhat wasteful as by pulling in Dojo's GFX package one is already getting pretty much all the capabilities that JQuery offers resulting in a slightly slower-than-necessary initial load time, and practically it breaks unless one removes JQuery from Plone's automerge system and pulls it in manually. The next version of Dojo has a JQuery compatibility package planned, so long term the theoretical solution is probably one of removing JQuery entirely and relying on the compatibility package (perhaps even gaining a bit of speed), but it's still early to say for certain. In the meanwhile, separating out JQuery from the rest of the Plone JavaScript source files and linking it directly solves the practical problem.

The site in question is the Shining Stars Learning Center. They wanted lots of stars on a light blue background, and they got them in spades. A single star drawing routine is looped through a number of times based upon the client's screen size, filling the current browser window (actually the maximum possible browser window should the window be resized) with stars of random size, rotation, and position. To add a bit of whimsy, all the stars can be independently dragged around the window.

The Shining Stars logo in the upper left hand corner is also drawn via vector graphics, although it can't be moved. I disallowed stars being drawn behind it in the initial star populating phase, but they can be dragged behind it after the window has been drawn. It was converted from SVG to Dojo GFX JSON using my fork of svg2gfx.xsl (which will soon hopefully go into the main trunk).

The whole site, including the logo and stars, is built around em size, so resizing the text causes the logo and stars to resize accordingly, too, and since they're defined using vectors they never get fuzzy.

Actually, for the fun of it I tried copying a few of the resulting vector images into bitmap images (one of which is used for the Shining Stars Twitter background). Besides being noticeably fuzzier, the image ended up being somewhat larger than the JavaScript used to define it, too. Obviously this won't always be the case depending upon specifics (and it certainly wasn't my driving motivation; I wanted a clean appearance with nice resizing) but I found it to be an interesting perk.

When I say it's a cross-platform solution I really mean it. Not only does it work on the usual suspects (Firefox, Safari, Opera, Chrome, MSIE, etc.) but it works on the Wii Internet Channel as well. Shining Stars has a Wii in their boardroom (how cool is that?) that they can use for browsing the Web (among other things) with the Wiimote as a pointer. I thus did as much testing of the site on the Wii as I did with the other more common browsers, and it works surprisingly well. It's a bit slow rendering the vector graphics, but not disastrously so, and the star dragging even works. The only problem I had with it is that when one switches into its so-called "Single Column Mode" almost nothing gets displayed at all. This could be simply alleviated by selectively turning off some JavaScript and CSS when the mode is enabled, but I've yet to find a way to reliably detect it. It's not a huge worry though as the site does work fine in the Wii's default display mode.

Microsoft Internet Explorer (as usual) provided its own set of problems. While on most browsers (including the Wii Internet Channel) the Dojo GFX stuff gets rendered as SVG, MSIE instead uses VML. Unfortunately Microsoft's recent crusade to catch up with other browsers' support of standards has not included standard vector graphics in the form of SVG, and to make matters worse they've allowed their own VML to rot with each new version of MSIE seeming to introduce new, unaddressed VML problems. In particular, some of the vector graphics the site uses cause VML issues in MSIE8, and the only way I was able to fix it was to use the Microsoft-specific X-UA-Compatible to explicitly request that MSIE8 render the site as if it were really MSIE7. Since I was forced to mess with this anyway, I took it one step further and added Chrome Frame support with the line:

<meta http-equiv="X-UA-Compatible" content="chrome=1;IE=7">

and dropping all MSIE-specific hacks when it is engaged.

It now uses Chrome Frame (if present) in preference to any of MSIE's native rendering modes, and uses MSIE7 rendering mode if in a newer version (that lacks Chrome Frame). Chrome Frame is visibly faster than all of the MSIE native rendering modes for this site, so much so that we (for the moment, anyway, we may remove it) added a notice to let regular users of the site know about it in order to improve their browsing experience. I'm assuming that the speed difference is mostly due to a combination of Chrome's faster JavaScript implementation and the fact that no extra JavaScript hacks are required to make up for CSS deficiencies, but regardless of its source it is significant.

Please note that the site is still in a beta mode of sorts and thus may not only still have bugs but will also be in fairly frequent flux. If things don't look as you'd expect from the above description, please try again in a few minutes; you probably just caught it during an update.

Friday, August 28, 2009

Maintaining Data Integrity

I promised several posts ago that I'd discuss some of the interesting internals of Antichrist Watch in more detail.

I've since gotten a bit off track, posting three entries about some of the Repoze stuff I've been doing lately. As I've been getting deeper and deeper into other projects I realize I'd better catch up and make good on my original promise or else I may never get back to it.

Today I figured I'd cover the principles behind Antichrist Watch's back-end database connection. It's using PostgreSQL and making heavy use of some of the features that many other database servers lack. In particular, PostgreSQL has great support for embedded logic (also known in database land as stored functions or stored procedures); not only does it have its own PL/pgSQL (similar to Oracle's PL/SQL) it also supports embedding code in other languages like Python, Perl, Tcl, and others besides. This embedded code can be tied into constraints that get executed (roughly) prior to a SQL command, rules that get executed (roughly) during a SQL command, and triggers that get executed after a SQL command. Additionally these stored procedures can be called directly allowing not only the effective batching of SQL commands but also multiple SQL commands bound together logically (that is, it's trivial to make a stored procedure execute one list of SQL commands under one set of conditions but execute another under a different set of conditions), or fashioned into views that behave like tables.

This combination gives the designer two big advantages over less capable database systems: first, it becomes possible to embed code to help guarantee data integrity within the database itself by careful use of constraints, rules, and triggers; and second it becomes possible to hide the database's implementation details from any client code by creating a public API via stored procedures. Thus we're basically using information hiding and restricting data access to certain controlled channels. I'm guessing that anyone who has read this far will understand in principle why these are good things. I'll carry it further though and list some specific benefits obtained by using this type of database design.

No raw tables are accessed in Antichrist Watch. It has a database access API fashioned from stored procedures and views. This level insulates the client from the raw tables; right now internally it is heavily normalized and even uses table inheritance to avoid repetition, but if at some point down the road it became necessary to duplicate some information for the sake of performance, the tables could be completely redesigned without requiring any change in client code whatsoever. Furthermore, with a clean interface it's easy to change clients. Antichrist Watch has had three completely different front-ends already. My original version was hacked together quickly from basic Python; my second version used PHP; the current version uses Twisted and Zope Page Templates (via Repoze Chameleon). At each stage making the transition was reasonably easy. It would have been a bear (possibly even unthinkable) if the interface were using raw tables and the client was responsible for data integrity.

On that note it must be said that data integrity is also improved. I'm a big believer in trying to enforce data constraints as close to the data itself as practically possible. The further away the constraint enforcement is from the data, the more space there is for loopholes, security holes, and logic errors to breed. Antichrist Watch has all data constraints and relationships enforced by the database itself. Even if someone with malicious intent were to somehow get beyond the client-side JavaScript and server-side Twisted Python (or some bugs were to crop up in my code, be it client-side or server-side), the allowed options on the database side are restricted limiting the potential damage that could be done. Basically it would be possible for fake data to get inserted, but it would not be possible for invalid data to get inserted, and this can mean the difference between a minor failure (accompanied by some embarrassment) and a critical failure (accompanied by a system crash). As an added bonus overall performance also often improves by moving such data constraint enforcement into the tightly optimized world of the database server itself.

Please note that I'm not advocating the moving of all logic out of things like Zope, Twisted, or PHP into the database server itself. This would be counterproductive. Tools for developing in languages like PL/pgSQL tend to be more primitive than tools for developing in languages like unembedded Python, and application logic definitely belongs outside the database. What I am advocating is making the database responsible for its own data checking while hiding its source internals.

Friday, July 31, 2009

Plone 3.3 and Repoze

While I've already discussed what's generally required for making Plone run with Repoze and mod_wsgi, I had been using Plone 3.3rc3. Anyone (myself included) who's tried to sub in anything more recent (like Plone 3.3rc4) has run into a problem with version conflicts due to changes between 3.3rc3 and 3.3rc4. Specifically the upgrade of five.localsitemanager from 0.4 to 1.1 sets off a zope.component version conflict, as any version of it 1.0 or beyond tries to pull in a whole bunch of eggs that aren't really necessary. Now obviously pinning five.localsitemanager to a version less than 1.0 will allow the buildout to complete properly, but this isn't really a good solution as it creates a mutant (and thus untested) version of Plone that may or may not work in random situations.

The real fix isn't that hard, though. It's simply a matter of utilizing fake Zope 2 eggs. The only catch here is that in order for them to work, they must be in place before the Plone eggs get pulled, but as always they cannot be created until after Zope 2 itself is in place. This necessitates splitting the one Zope part I showed last time into two and sticking a fake eggs part in between. The second part will have to reference the eggs of the first part. To make things just a little more interesting, z3c.recipe.fakezope2eggs expects a location to be provided by the [zope2] part. We thus have to create one. The changed section will look as follows:

recipe = zc.recipe.egg
dependent-scripts = true
location = ${buildout:directory}/parts
eggs =

recipe = iw.recipe.cmd
on_install = true
on_update = true
cmds =
    mkdir -p ${buildout:directory}/parts/lib/python
    ln -sh ${buildout:directory}/eggs/zopelib*/zope ${buildout:directory}/parts/lib/python/zope

recipe = z3c.recipe.fakezope2eggs

recipe = zc.recipe.egg
dependent-scripts = true
interpreter = zopepy
eggs =

It's probably obvious, but just in case it's not, the two new parts have to be added to the parts section near the top:

parts =

Don't forget that the order is important, as the fake Zope eggs have to be created after Zope proper but before Plone.

Deliverance and URISpace should continue to work without changes.

Also, I've created eggs for zopelib and ZODB 3.7.3 (the official versions that the Plone 3.3 builds are based upon; I've also made a zopelib 2.11.3 for people using straight Zope) and submitted them to the Repoze guys for inclusion in their distribution section. If you'd like to play around with these before they become official, you can find them in the distribution section.

Wednesday, July 15, 2009

Repoze, Zope, Plone, URISpace, and Deliverance

I've further developed the technique I outlined in my last post for making Plone run with Repoze and mod_wsgi. In particular I've included the ability to skin via Deliverance (including the ability to separate out different sections of the site for different treatments via URISpace) and enhanced the script to remove the limitations I'd indicated were there without going into details; it now both handles update cases better and is far more intelligent with regards to ensuring that egg path order is maintained regardless of whether Paste or mod_wsgi is being used. If you had trouble getting my prior instructions to work in your environment, you may just want to update to the smarter included below.

My use case is (although note that the current public version may not yet reflect the changes discussed in this post as they're obviously being done on a separate development server). Basically several different not-for-profit entities have their sites located on this domain, and while there is a significant overlap between members of the various sub-sites (probably 10% - 25% or so) it's certainly not absolute, and each sub-site has different administrators and different styles. While there are ways of making Deliverance theme things differently for different portions of a site, I didn't really care for any of them in this application as they'd make it too easy to break things for particular sub-sites and/or make it too difficult to actually do the theming for individual sub-sites. The obvious solution is repoze.urispace (which implements the W3C URISpace specification in a way that can be used to make Deliverance distinguish between different portions of a site). With repoze.urispace, it becomes easy to give each sub-site its own independent set of Deliverance rules and themes, and working on one will in no way affect another.

Making It Happen

Now that you understand the motivation it's time to see the implementation. Generally one needs to follow the instructions I outlined in my earlier post, but make a few changes:

  1. First, substitute its copy of with the following much improved version prior to running bin/buildout:

    import os
    from dircache import listdir
    # The bin directory holds all the executables for the buildout
    BinDir = 'bin'
    # The unadorned files aren't given any of the egg paths on creation
    UnadornedFiles = (os.path.abspath(BinDir+'/zope2.wsgi'),)
    # Most regular files are given the egg paths, but not the old-style products path
    RegularFiles = [os.path.abspath('%s/%s'%(BinDir,filename)) for filename in listdir(BinDir)]
    # Eggs can live in more than one location
    EggDirs = ('eggs',)
    # Old style products should all be contained in one location
    ProductsPath = os.path.abspath('products')
    # The sample file should be regular file with a regular egg path
    SampleFile = os.path.abspath(BinDir+'/paster')
    def main(options, buildout):
        # We have to ensure that the zopelib directory is earlier in the search path
        # than other possibly competing packages.  Unfortunately, we don't know its
        # full name a priori and have to hunt it down first.
        for eggDir in EggDirs:
            for filename in listdir(eggDir):
                if 'zopelib' in filename:
        # First we handle the regular files.  We both relocate the zopelib component
        # (if present) to the beginning of the list and prepend the old-style products
        # path.  Note that this will ignore unadorned files mixed in.
        for filename in RegularFiles:
            lines = open(filename, 'r').readlines()
            file = open(filename, 'w')
            for line in lines:
                if 'zopelib' not in line:
                if line.startswith('sys.path'):
                    if ProductsPath not in ' '.join(lines):
                        file.write("  '%s',\n"%ProductsPath)
                    if zopelibPath:
                        file.write("  '%s',\n"%zopelibPath)
        # The path list should now be perfect in our sample file.  Grab it.
        for line in lines:
            if line.startswith('import sys'):
            elif begLineNum and ']' in line:
        # Now we're ready to handle the unadorned files.  Simply replace any existing
        # sys path with the good one we've obtained from the sample, or add it if nothing
        # yet exists.  We're assuming that in a healthy file the import os statement will
        # occur after the import sys statement.
        for filename in UnadornedFiles:
            lines = open(filename,'r').readlines()
            for line in lines:
                if line.startswith('import sys'):
                elif begLineNum and ']' in line:
                elif line.startswith('import os') and not alreadyProcessed:
            file = open(filename,'w')
        # All done!  Let's just tell the user roughly what we've done.
        print "Egg paths added to %s" % ', '.join(UnadornedFiles)
        print "Product path added to %s" % ', '.join(RegularFiles)
  2. Next, a slightly modified buildout.cfg must be used (obviously also before running bin/buildout). The following should work:

    extends =
    versions = versions
    find-links =
    develop =
    parts =
    zopelib =
    recipe = z3c.recipe.staticlxml
    egg = lxml
    libxml2-url =
    libxslt-url =
    recipe = zc.recipe.egg
    dependent-scripts = true
    interpreter = zopepy
    eggs =
    recipe = collective.recipe.zcml
    zcml =
    recipe = iw.recipe.cmd
    on_install = true
    cmds =
       bin/mkzope2instance --use-zeo --zeo-port=${buildout:directory}/var/zeo.zdsock --zope-port=8888
       sed -i "" "s/server localhost:/server /" ${buildout:directory}/etc/zope.conf
       echo "Please run 'bin/runzeo -C etc/zeo.conf' and 'bin/paster serve etc/zope2.ini', then 'bin/addzope2user  '"
    recipe = z3c.recipe.runscript
    install-script =
    update-script =

    As before note the dummy mysite.policy product included to show the most general solution. If you are not developing any products and do not need any additional ZCML slugs, these can be totally omitted. Otherwise they should be changed accordingly.

  3. Next, add rules and themes subdirectories to the etc directory.

  4. Create a default.xml file in the rules directory. The following sample:

    <?xml version="1.0" encoding="UTF-8"?>
    <rules xmlns:xi=""
      <replace theme="/html/head" content="/html/head" />
      <replace theme="/html/body" content="/html/body||/html/frameset" />

    is basically a NOP and will get you started. There's more information on the Deliverance site on the sorts of commands you can add in here. Ultimately you'll be making more of these rules files with each one being used for a different section of your site.

  5. Create a default.html file in the themes directory. Something as basic as:

        <title>default.html theme for Deliverance</title>

    will likewise be enough to get you started. You can ultimately create this file however you'd like using any sorts of HTML generation tools that tickle your fancy. Ultimately you'll be making more of these theme files (probably one to go with each rules file) with each one being used for a different section of your site.

  6. Lastly you'll need to create a URISpace configuration file urispace.xml in the etc directory. While there's more info on the repoze.urispace site on the options that you can include in here, the following is a minimal one that directs everything to the default rules and theme we have already defined:

    <?xml version="1.0" ?>
     <!-- default theme and rules -->
  7. Finally, before trying to run the new site one must ensure that the WSGI pipeline is properly prepared. The following new zope2.ini will do so:

    debug = True
    paste.app_factory = repoze.obob.publisher:make_obob
    repoze.obob.get_root = repoze.zope2.z2bob:get_root
    repoze.obob.initializer = repoze.zope2.z2bob:initialize
    repoze.obob.helper_factory = repoze.zope2.z2bob:Zope2ObobHelper
    zope.conf = %(here)s/zope.conf
    use = egg:repoze.errorlog#errorlog
    path = /__error_log__
    keep = 20
    ignore = paste.httpexceptions:HTTPUnauthorized
    use = egg:deliverance#main
    theme_uri =
    rule_uri = file:///%(here)s/rules/rules.xml
    use = egg:repoze.urispace#urispace
    filename = %(here)s/urispace.xml
    pipeline = egg:Paste#cgitb
    #           egg:Paste#translogger
    # Note: replace egg:Paste#cgitb with egg:Paste#evalerror above to get
    # the browser to display eval'able traceback stacks (unsuitable for
    # production).
    # If you enable (uncomment) the translogger, it will show access log
    # info to the console.
    use = egg:repoze.zope2#zserver
    host =
    port = 8888

    The theme referenced in the Deliverance section will never actually be used as it'll be overridden by repoze.urispace.

Details To Note

First off, thanks to Tres Seaver not just for originally writing repoze.urispace and its companion repoze.dvselect, but for also putting up with (and acting on) both my bug reports and suggestions for making things easier to use. I had originally gotten this all working with an earlier version of repoze.urispace and and the original unreleased version of repoze.dvselect, but the method described here is much cleaner, and it wouldn't have been possible without Tres' numerous recent updates to both repoze.urispace and repoze.dvselect.

Second, as seems to happen a lot lately, just as I was starting to write this I was informed of another effort to do something similar. Wojciech Lichota's technique seems to do some things better and some things worse than the technique described above. Depending upon what you're doing, you may find that what he's doing more directly addresses your needs. Probably a hybrid technique will be better than either...

Also, one may wonder about the whole z3c.recipe.staticlxml used above to build lxml and whether or not it's really necessary. It may look like extra work, but this method allows the exact same buildout to work on Mac OS X in addition to regular more traditional UNIX and UNIX-like environments.

Finally, note that as presented here Deliverance and URISpace won't actually do anything... it's expected that you'll add rules and themes appropriate to your own project.