Wednesday, July 15, 2009

Repoze, Zope, Plone, URISpace, and Deliverance

I've further developed the technique I outlined in my last post for making Plone run with Repoze and mod_wsgi. In particular I've included the ability to skin via Deliverance (including the ability to separate out different sections of the site for different treatments via URISpace) and enhanced the addpaths.py script to remove the limitations I'd indicated were there without going into details; it now both handles update cases better and is far more intelligent with regards to ensuring that egg path order is maintained regardless of whether Paste or mod_wsgi is being used. If you had trouble getting my prior instructions to work in your environment, you may just want to update to the smarter addpaths.py included below.

My use case is Saugus.org (although note that the current public version may not yet reflect the changes discussed in this post as they're obviously being done on a separate development server). Basically several different not-for-profit entities have their sites located on this domain, and while there is a significant overlap between members of the various sub-sites (probably 10% - 25% or so) it's certainly not absolute, and each sub-site has different administrators and different styles. While there are ways of making Deliverance theme things differently for different portions of a site, I didn't really care for any of them in this application as they'd make it too easy to break things for particular sub-sites and/or make it too difficult to actually do the theming for individual sub-sites. The obvious solution is repoze.urispace (which implements the W3C URISpace specification in a way that can be used to make Deliverance distinguish between different portions of a site). With repoze.urispace, it becomes easy to give each sub-site its own independent set of Deliverance rules and themes, and working on one will in no way affect another.

Making It Happen

Now that you understand the motivation it's time to see the implementation. Generally one needs to follow the instructions I outlined in my earlier post, but make a few changes:

  1. First, substitute its copy of addpaths.py with the following much improved version prior to running bin/buildout:

    import os
    from dircache import listdir
    
    # The bin directory holds all the executables for the buildout
    BinDir = 'bin'
    
    # The unadorned files aren't given any of the egg paths on creation
    UnadornedFiles = (os.path.abspath(BinDir+'/zope2.wsgi'),)
    
    # Most regular files are given the egg paths, but not the old-style products path
    RegularFiles = [os.path.abspath('%s/%s'%(BinDir,filename)) for filename in listdir(BinDir)]
    
    # Eggs can live in more than one location
    EggDirs = ('eggs',)
    
    # Old style products should all be contained in one location
    ProductsPath = os.path.abspath('products')
    
    # The sample file should be regular file with a regular egg path
    SampleFile = os.path.abspath(BinDir+'/paster')
    
    def main(options, buildout):
        # We have to ensure that the zopelib directory is earlier in the search path
        # than other possibly competing packages.  Unfortunately, we don't know its
        # full name a priori and have to hunt it down first.
        for eggDir in EggDirs:
            for filename in listdir(eggDir):
                if 'zopelib' in filename:
                    zopelibPath=os.path.abspath('%s/%s/%s'%(os.getcwd(),eggDir,filename))
                    break
    
        # First we handle the regular files.  We both relocate the zopelib component
        # (if present) to the beginning of the list and prepend the old-style products
        # path.  Note that this will ignore unadorned files mixed in.
        for filename in RegularFiles:
            lines = open(filename, 'r').readlines()
            file = open(filename, 'w')
            for line in lines:
                if 'zopelib' not in line:
                    file.write(line)
                if line.startswith('sys.path'):
                    if ProductsPath not in ' '.join(lines):
                        file.write("  '%s',\n"%ProductsPath)
                    if zopelibPath:
                        file.write("  '%s',\n"%zopelibPath)
            file.close()
    
        # The path list should now be perfect in our sample file.  Grab it.
        lines=open(SampleFile,'r').readlines()
        lineNum=begLineNum=endLineNum=0
        for line in lines:
            lineNum+=1
            if line.startswith('import sys'):
                begLineNum=lineNum-1
            elif begLineNum and ']' in line:
                endLineNum=lineNum
        eggLines=lines[begLineNum:endLineNum]
    
        # Now we're ready to handle the unadorned files.  Simply replace any existing
        # sys path with the good one we've obtained from the sample, or add it if nothing
        # yet exists.  We're assuming that in a healthy file the import os statement will
        # occur after the import sys statement.
        for filename in UnadornedFiles:
            lines = open(filename,'r').readlines()
            alreadyProcessed=False
            lineNum=begLineNum=endLineNum=0
            for line in lines:
                lineNum+=1
                if line.startswith('import sys'):
                    alreadyProcessed=True
                    begLineNum=lineNum-1
                elif begLineNum and ']' in line:
                    endLineNum=lineNum
                elif line.startswith('import os') and not alreadyProcessed:
                    begLineNum=endLineNum=lineNum-1
            lines[begLineNum:endLineNum]=eggLines
            file = open(filename,'w')
            file.writelines(lines)
            file.close()
    
        # All done!  Let's just tell the user roughly what we've done.
        print "Egg paths added to %s" % ', '.join(UnadornedFiles)
        print "Product path added to %s" % ', '.join(RegularFiles)
    
  2. Next, a slightly modified buildout.cfg must be used (obviously also before running bin/buildout). The following should work:

    [buildout]
    extends =
        http://good-py.appspot.com/release/repoze.zope2/1.0
        http://dist.plone.org/release/3.3rc3/versions.cfg
    
    versions = versions
    
    find-links =
        http://dist.repoze.org/zope2/latest
        http://dist.repoze.org/zope2/dev
        http://dist.plone.org/release/3.3rc3
        http://download.zope.org/ppix/
        http://download.zope.org/distribution/
        http://effbot.org/downloads
    
    develop =
        src/mysite.policy
    
    parts =
        lxml
        zope2
        instance
        slugs
        addpaths
    
    [versions]
    zopelib = 2.10.7.0
    
    [lxml]
    recipe = z3c.recipe.staticlxml
    egg = lxml
    libxml2-url = http://xmlsoft.org/sources/libxml2-2.7.2.tar.gz
    libxslt-url = http://xmlsoft.org/sources/libxslt-1.1.24.tar.gz
    
    [zope2]
    recipe = zc.recipe.egg
    dependent-scripts = true
    interpreter = zopepy
    eggs =
        lxml
        repoze.zope2
        Plone
        PIL
        Products.DocFinderTab
        Products.ExternalEditor
        plone.openid
        deliverance
        repoze.urispace
        repoze.dvselect
        mysite.policy
    
    [slugs]
    recipe = collective.recipe.zcml
    zope2-location=${buildout:directory}
    zcml =
        mysite.policy
    
    [instance]
    recipe = iw.recipe.cmd
    on_install = true
    cmds =
       bin/mkzope2instance --use-zeo --zeo-port=${buildout:directory}/var/zeo.zdsock --zope-port=8888
       sed -i "" "s/server localhost:/server /" ${buildout:directory}/etc/zope.conf
       echo "Please run 'bin/runzeo -C etc/zeo.conf' and 'bin/paster serve etc/zope2.ini', then 'bin/addzope2user  '"
    
    [addpaths]
    recipe = z3c.recipe.runscript
    install-script = addpaths.py:main
    update-script = addpaths.py:main
    

    As before note the dummy mysite.policy product included to show the most general solution. If you are not developing any products and do not need any additional ZCML slugs, these can be totally omitted. Otherwise they should be changed accordingly.

  3. Next, add rules and themes subdirectories to the etc directory.

  4. Create a default.xml file in the rules directory. The following sample:

    <?xml version="1.0" encoding="UTF-8"?>
    <rules xmlns:xi="http://www.w3.org/2001/XInclude"
           xmlns="http://www.plone.org/deliverance">
      <replace theme="/html/head" content="/html/head" />
      <replace theme="/html/body" content="/html/body||/html/frameset" />
    </rules>
    

    is basically a NOP and will get you started. There's more information on the Deliverance site on the sorts of commands you can add in here. Ultimately you'll be making more of these rules files with each one being used for a different section of your site.

  5. Create a default.html file in the themes directory. Something as basic as:

    <html>
      <head>
        <title>default.html theme for Deliverance</title>
      </head>
      <body>
      </body>
    </html>
    

    will likewise be enough to get you started. You can ultimately create this file however you'd like using any sorts of HTML generation tools that tickle your fancy. Ultimately you'll be making more of these theme files (probably one to go with each rules file) with each one being used for a different section of your site.

  6. Lastly you'll need to create a URISpace configuration file urispace.xml in the etc directory. While there's more info on the repoze.urispace site on the options that you can include in here, the following is a minimal one that directs everything to the default rules and theme we have already defined:

    <?xml version="1.0" ?>
    <themeselect
       xmlns:uri='http://www.w3.org/2000/urispace'
       xmlns:uriext='http://repoze.org/repoze.urispace/extensions'
       xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'
       >
    
     <!-- default theme and rules -->
     <theme>file:///home/eric/WSGI/SaugusOrg/etc/themes/theme.xhtml</theme>
     <rules>file:///home/eric/WSGI/SaugusOrg/etc/rules/rules.xml</rules>
    </themeselect>
    
  7. Finally, before trying to run the new site one must ensure that the WSGI pipeline is properly prepared. The following new zope2.ini will do so:

    [DEFAULT]
    debug = True
    
    [app:zope2]
    paste.app_factory = repoze.obob.publisher:make_obob
    repoze.obob.get_root = repoze.zope2.z2bob:get_root
    repoze.obob.initializer = repoze.zope2.z2bob:initialize
    repoze.obob.helper_factory = repoze.zope2.z2bob:Zope2ObobHelper
    zope.conf = %(here)s/zope.conf
    
    [filter:errorlog]
    use = egg:repoze.errorlog#errorlog
    path = /__error_log__
    keep = 20
    ignore = paste.httpexceptions:HTTPUnauthorized
           paste.httpexceptions:HTTPNotFound
           paste.httpexceptions:HTTPFound
    
    [filter:deliverance]
    use = egg:deliverance#main
    theme_uri = http://www.example.com/
    rule_uri = file:///%(here)s/rules/rules.xml
    
    [filter:urispace]
    use = egg:repoze.urispace#urispace
    filename = %(here)s/urispace.xml
    
    [pipeline:main]
    pipeline = egg:Paste#cgitb
               egg:Paste#httpexceptions
    #           egg:Paste#translogger
               egg:repoze.retry#retry
               egg:repoze.tm#tm
               egg:repoze.vhm#vhm_xheaders
               errorlog
               urispace
               egg:repoze.dvselect#main
               deliverance
               zope2
    
    # Note: replace egg:Paste#cgitb with egg:Paste#evalerror above to get
    # the browser to display eval'able traceback stacks (unsuitable for
    # production).
    
    # If you enable (uncomment) the translogger, it will show access log
    # info to the console.
    
    [server:main]
    use = egg:repoze.zope2#zserver
    host = 127.0.0.1
    port = 8888
    

    The theme referenced in the Deliverance section will never actually be used as it'll be overridden by repoze.urispace.

Details To Note

First off, thanks to Tres Seaver not just for originally writing repoze.urispace and its companion repoze.dvselect, but for also putting up with (and acting on) both my bug reports and suggestions for making things easier to use. I had originally gotten this all working with an earlier version of repoze.urispace and and the original unreleased version of repoze.dvselect, but the method described here is much cleaner, and it wouldn't have been possible without Tres' numerous recent updates to both repoze.urispace and repoze.dvselect.

Second, as seems to happen a lot lately, just as I was starting to write this I was informed of another effort to do something similar. Wojciech Lichota's technique seems to do some things better and some things worse than the technique described above. Depending upon what you're doing, you may find that what he's doing more directly addresses your needs. Probably a hybrid technique will be better than either...

Also, one may wonder about the whole z3c.recipe.staticlxml used above to build lxml and whether or not it's really necessary. It may look like extra work, but this method allows the exact same buildout to work on Mac OS X in addition to regular more traditional UNIX and UNIX-like environments.

Finally, note that as presented here Deliverance and URISpace won't actually do anything... it's expected that you'll add rules and themes appropriate to your own project.

5 comments:

  1. Hi,
    good post.
    You can also let Plone control which theme and rules apply injecting into the response request the X-Deliverance-Page-Class and use deliverance page classes from the themed application. You can dynamically inject different X-Deliverance-Page-Class headers for different folders or types of objects serving your themes from plone with few hacks. This way you can avoid complex deliverance configurations and get site sections with different themes or different themes based on the type of obejct. Limitation of the X-Deliverance-Page-Class approach: you need control on the themed application.

    ReplyDelete
  2. Nice! Glad to see you are using http://good-py.appspot.com/release/repoze.zope2/1.0 :-)

    ReplyDelete
  3. Davide - Yes, I know of that technique and consider it a good match with the use case you describe. However in this case I essentially have different sub-sites with different administrators, and I don't think it's a good fit. I want to give the administrators the ability to work on their own templates, rules, etc. for their own sites, but I don't want to make it easy for them to accidentally drop the template or set it incorrectly. URISpace works well here because it handles the base theme assignment at a level above the individual sub-sites.

    Alex - Yes, I very much like the idea of Good-Py. Egg dependency graphs are getting complicated enough now that they're causing weird issues to spring up, and anything that helps manage them is welcome. That's actually the reason I bit the bullet and enhanced the old addpaths.py -- I was noticing that in certain cases the naive approach I was originally using to order egg paths would give different results (and sometimes errors) when used in different ways. The new approach used here addresses that (plus makes it handle updates a lot more intelligently).

    ReplyDelete
  4. Have you been able to add plone.app.blob to your setup?

    ReplyDelete
  5. I've not tried, but I don't know of any reason why it wouldn't work. It's possible though (depending upon dependent eggs) that one would have to split up the current [zope2] part into [zope2] and [plone] parts (both still using zc.recipe.egg) with a Zope 2 fake egg generator of some sort in between in order to avoid version conflicts.

    ReplyDelete