Plone 3.3 and Repoze

While I've already discussed what's generally required for making Plone run with Repoze and mod_wsgi, I had been using Plone 3.3rc3. Anyone (myself included) who's tried to sub in anything more recent (like Plone 3.3rc4) has run into a problem with version conflicts due to changes between 3.3rc3 and 3.3rc4. Specifically the upgrade of five.localsitemanager from 0.4 to 1.1 sets off a zope.component version conflict, as any version of it 1.0 or beyond tries to pull in a whole bunch of eggs that aren't really necessary. Now obviously pinning five.localsitemanager to a version less than 1.0 will allow the buildout to complete properly, but this isn't really a good solution as it creates a mutant (and thus untested) version of Plone that may or may not work in random situations.

The real fix isn't that hard, though. It's simply a matter of utilizing fake Zope 2 eggs. The only catch here is that in order for them to work, they must be in place before the Plone eggs get pulled, but as always they cannot be created until after Zope 2 itself is in place. This necessitates splitting the one Zope part I showed last time into two and sticking a fake eggs part in between. The second part will have to reference the eggs of the first part. To make things just a little more interesting, z3c.recipe.fakezope2eggs expects a location to be provided by the [zope2] part. We thus have to create one. The changed section will look as follows:

[zope2]
recipe = zc.recipe.egg
dependent-scripts = true
location = ${buildout:directory}/parts
eggs =
    lxml
    repoze.zope2

[zopeliblink]
recipe = iw.recipe.cmd
on_install = true
on_update = true
cmds =
    mkdir -p ${buildout:directory}/parts/lib/python
    ln -sh ${buildout:directory}/eggs/zopelib*/zope ${buildout:directory}/parts/lib/python/zope

[fakezope2eggs]
recipe = z3c.recipe.fakezope2eggs

[plone]
recipe = zc.recipe.egg
dependent-scripts = true
interpreter = zopepy
eggs =
    ${zope2:eggs}
    Plone
    PIL
    Products.DocFinderTab
    Products.ExternalEditor
    plone.openid
    deliverance
    repoze.urispace
    repoze.dvselect
    mysite.policy

It's probably obvious, but just in case it's not, the two new parts have to be added to the parts section near the top:

parts =
    lxml
    zope2
    zopeliblink
    fakezope2eggs
    plone
    instance
    slugs
    addpaths

Don't forget that the order is important, as the fake Zope eggs have to be created after Zope proper but before Plone.

Deliverance and URISpace should continue to work without changes.

Also, I've created eggs for zopelib 2.10.8.0 and ZODB 3.7.3 (the official versions that the Plone 3.3 builds are based upon; I've also made a zopelib 2.11.3 for people using straight Zope) and submitted them to the Repoze guys for inclusion in their distribution section. If you'd like to play around with these before they become official, you can find them in the Saugus.net distribution section.

Repoze, Zope, Plone, URISpace, and Deliverance

I've further developed the technique I outlined in my last post for making Plone run with Repoze and mod_wsgi. In particular I've included the ability to skin via Deliverance (including the ability to separate out different sections of the site for different treatments via URISpace) and enhanced the addpaths.py script to remove the limitations I'd indicated were there without going into details; it now both handles update cases better and is far more intelligent with regards to ensuring that egg path order is maintained regardless of whether Paste or mod_wsgi is being used. If you had trouble getting my prior instructions to work in your environment, you may just want to update to the smarter addpaths.py included below.

My use case is Saugus.org (although note that the current public version may not yet reflect the changes discussed in this post as they're obviously being done on a separate development server). Basically several different not-for-profit entities have their sites located on this domain, and while there is a significant overlap between members of the various sub-sites (probably 10% - 25% or so) it's certainly not absolute, and each sub-site has different administrators and different styles. While there are ways of making Deliverance theme things differently for different portions of a site, I didn't really care for any of them in this application as they'd make it too easy to break things for particular sub-sites and/or make it too difficult to actually do the theming for individual sub-sites. The obvious solution is repoze.urispace (which implements the W3C URISpace specification in a way that can be used to make Deliverance distinguish between different portions of a site). With repoze.urispace, it becomes easy to give each sub-site its own independent set of Deliverance rules and themes, and working on one will in no way affect another.

Making It Happen

Now that you understand the motivation it's time to see the implementation. Generally one needs to follow the instructions I outlined in my earlier post, but make a few changes:

1. First, substitute its copy of addpaths.py with the following much improved version prior to running bin/buildout:

   import os
   from dircache import listdir
   
   # The bin directory holds all the executables for the buildout
   BinDir = 'bin'
   
   # The unadorned files aren't given any of the egg paths on creation
   UnadornedFiles = (os.path.abspath(BinDir+'/zope2.wsgi'),)
   
   # Most regular files are given the egg paths, but not the old-style products path
   RegularFiles = [os.path.abspath('%s/%s'%(BinDir,filename)) for filename in listdir(BinDir)]
   
   # Eggs can live in more than one location
   EggDirs = ('eggs',)
   
   # Old style products should all be contained in one location
   ProductsPath = os.path.abspath('products')
   
   # The sample file should be regular file with a regular egg path
   SampleFile = os.path.abspath(BinDir+'/paster')
   
   def main(options, buildout):
       # We have to ensure that the zopelib directory is earlier in the search path
       # than other possibly competing packages.  Unfortunately, we don't know its
       # full name a priori and have to hunt it down first.
       for eggDir in EggDirs:
           for filename in listdir(eggDir):
               if 'zopelib' in filename:
                   zopelibPath=os.path.abspath('%s/%s/%s'%(os.getcwd(),eggDir,filename))
                   break
   
       # First we handle the regular files.  We both relocate the zopelib component
       # (if present) to the beginning of the list and prepend the old-style products
       # path.  Note that this will ignore unadorned files mixed in.
       for filename in RegularFiles:
           lines = open(filename, 'r').readlines()
           file = open(filename, 'w')
           for line in lines:
               if 'zopelib' not in line:
                   file.write(line)
               if line.startswith('sys.path'):
                   if ProductsPath not in ' '.join(lines):
                       file.write("  '%s',\n"%ProductsPath)
                   if zopelibPath:
                       file.write("  '%s',\n"%zopelibPath)
           file.close()
   
       # The path list should now be perfect in our sample file.  Grab it.
       lines=open(SampleFile,'r').readlines()
       lineNum=begLineNum=endLineNum=0
       for line in lines:
           lineNum+=1
           if line.startswith('import sys'):
               begLineNum=lineNum-1
           elif begLineNum and ']' in line:
               endLineNum=lineNum
       eggLines=lines[begLineNum:endLineNum]
   
       # Now we're ready to handle the unadorned files.  Simply replace any existing
       # sys path with the good one we've obtained from the sample, or add it if nothing
       # yet exists.  We're assuming that in a healthy file the import os statement will
       # occur after the import sys statement.
       for filename in UnadornedFiles:
           lines = open(filename,'r').readlines()
           alreadyProcessed=False
           lineNum=begLineNum=endLineNum=0
           for line in lines:
               lineNum+=1
               if line.startswith('import sys'):
                   alreadyProcessed=True
                   begLineNum=lineNum-1
               elif begLineNum and ']' in line:
                   endLineNum=lineNum
               elif line.startswith('import os') and not alreadyProcessed:
                   begLineNum=endLineNum=lineNum-1
           lines[begLineNum:endLineNum]=eggLines
           file = open(filename,'w')
           file.writelines(lines)
           file.close()
   
       # All done!  Let's just tell the user roughly what we've done.
       print "Egg paths added to %s" % ', '.join(UnadornedFiles)
       print "Product path added to %s" % ', '.join(RegularFiles)
   

2. Next, a slightly modified buildout.cfg must be used (obviously also before running bin/buildout). The following should work:

   [buildout]
   extends =
       http://good-py.appspot.com/release/repoze.zope2/1.0
       http://dist.plone.org/release/3.3rc3/versions.cfg
   
   versions = versions
   
   find-links =
       http://dist.repoze.org/zope2/latest
       http://dist.repoze.org/zope2/dev
       http://dist.plone.org/release/3.3rc3
       http://download.zope.org/ppix/
       http://download.zope.org/distribution/
       http://effbot.org/downloads
   
   develop =
       src/mysite.policy
   
   parts =
       lxml
       zope2
       instance
       slugs
       addpaths
   
   [versions]
   zopelib = 2.10.7.0
   
   [lxml]
   recipe = z3c.recipe.staticlxml
   egg = lxml
   libxml2-url = http://xmlsoft.org/sources/libxml2-2.7.2.tar.gz
   libxslt-url = http://xmlsoft.org/sources/libxslt-1.1.24.tar.gz
   
   [zope2]
   recipe = zc.recipe.egg
   dependent-scripts = true
   interpreter = zopepy
   eggs =
       lxml
       repoze.zope2
       Plone
       PIL
       Products.DocFinderTab
       Products.ExternalEditor
       plone.openid
       deliverance
       repoze.urispace
       repoze.dvselect
       mysite.policy
   
   [slugs]
   recipe = collective.recipe.zcml
   zope2-location=${buildout:directory}
   zcml =
       mysite.policy
   
   [instance]
   recipe = iw.recipe.cmd
   on_install = true
   cmds =
      bin/mkzope2instance --use-zeo --zeo-port=${buildout:directory}/var/zeo.zdsock --zope-port=8888
      sed -i "" "s/server localhost:/server /" ${buildout:directory}/etc/zope.conf
      echo "Please run 'bin/runzeo -C etc/zeo.conf' and 'bin/paster serve etc/zope2.ini', then 'bin/addzope2user  '"
   
   [addpaths]
   recipe = z3c.recipe.runscript
   install-script = addpaths.py:main
   update-script = addpaths.py:main
   

   As before note the dummy mysite.policy product included to show the most general solution. If you are not developing any products and do not need any additional ZCML slugs, these can be totally omitted. Otherwise they should be changed accordingly.

3. Next, add rules and themes subdirectories to the etc directory.

4. Create a default.xml file in the rules directory. The following sample:

   <?xml version="1.0" encoding="UTF-8"?>
   <rules xmlns:xi="http://www.w3.org/2001/XInclude"
          xmlns="http://www.plone.org/deliverance">
     <replace theme="/html/head" content="/html/head" />
     <replace theme="/html/body" content="/html/body||/html/frameset" />
   </rules>
   

   is basically a NOP and will get you started. There's more information on the Deliverance site on the sorts of commands you can add in here. Ultimately you'll be making more of these rules files with each one being used for a different section of your site.

5. Create a default.html file in the themes directory. Something as basic as:

   <html>
     <head>
       <title>default.html theme for Deliverance</title>
     </head>
     <body>
     </body>
   </html>
   

   will likewise be enough to get you started. You can ultimately create this file however you'd like using any sorts of HTML generation tools that tickle your fancy. Ultimately you'll be making more of these theme files (probably one to go with each rules file) with each one being used for a different section of your site.

6. Lastly you'll need to create a URISpace configuration file urispace.xml in the etc directory. While there's more info on the repoze.urispace site on the options that you can include in here, the following is a minimal one that directs everything to the default rules and theme we have already defined:

   <?xml version="1.0" ?>
   <themeselect
      xmlns:uri='http://www.w3.org/2000/urispace'
      xmlns:uriext='http://repoze.org/repoze.urispace/extensions'
      xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'
      >
   
    <!-- default theme and rules -->
    <theme>file:///home/eric/WSGI/SaugusOrg/etc/themes/theme.xhtml</theme>
    <rules>file:///home/eric/WSGI/SaugusOrg/etc/rules/rules.xml</rules>
   </themeselect>
   

7. Finally, before trying to run the new site one must ensure that the WSGI pipeline is properly prepared. The following new zope2.ini will do so:

   [DEFAULT]
   debug = True
   
   [app:zope2]
   paste.app_factory = repoze.obob.publisher:make_obob
   repoze.obob.get_root = repoze.zope2.z2bob:get_root
   repoze.obob.initializer = repoze.zope2.z2bob:initialize
   repoze.obob.helper_factory = repoze.zope2.z2bob:Zope2ObobHelper
   zope.conf = %(here)s/zope.conf
   
   [filter:errorlog]
   use = egg:repoze.errorlog#errorlog
   path = /__error_log__
   keep = 20
   ignore = paste.httpexceptions:HTTPUnauthorized
          paste.httpexceptions:HTTPNotFound
          paste.httpexceptions:HTTPFound
   
   [filter:deliverance]
   use = egg:deliverance#main
   theme_uri = http://www.example.com/
   rule_uri = file:///%(here)s/rules/rules.xml
   
   [filter:urispace]
   use = egg:repoze.urispace#urispace
   filename = %(here)s/urispace.xml
   
   [pipeline:main]
   pipeline = egg:Paste#cgitb
              egg:Paste#httpexceptions
   #           egg:Paste#translogger
              egg:repoze.retry#retry
              egg:repoze.tm#tm
              egg:repoze.vhm#vhm_xheaders
              errorlog
              urispace
              egg:repoze.dvselect#main
              deliverance
              zope2
   
   # Note: replace egg:Paste#cgitb with egg:Paste#evalerror above to get
   # the browser to display eval'able traceback stacks (unsuitable for
   # production).
   
   # If you enable (uncomment) the translogger, it will show access log
   # info to the console.
   
   [server:main]
   use = egg:repoze.zope2#zserver
   host = 127.0.0.1
   port = 8888
   

   The theme referenced in the Deliverance section will never actually be used as it'll be overridden by repoze.urispace.

Details To Note

First off, thanks to Tres Seaver not just for originally writing repoze.urispace and its companion repoze.dvselect, but for also putting up with (and acting on) both my bug reports and suggestions for making things easier to use. I had originally gotten this all working with an earlier version of repoze.urispace and and the original unreleased version of repoze.dvselect, but the method described here is much cleaner, and it wouldn't have been possible without Tres' numerous recent updates to both repoze.urispace and repoze.dvselect.

Second, as seems to happen a lot lately, just as I was starting to write this I was informed of another effort to do something similar. Wojciech Lichota's technique seems to do some things better and some things worse than the technique described above. Depending upon what you're doing, you may find that what he's doing more directly addresses your needs. Probably a hybrid technique will be better than either...

Also, one may wonder about the whole z3c.recipe.staticlxml used above to build lxml and whether or not it's really necessary. It may look like extra work, but this method allows the exact same buildout to work on Mac OS X in addition to regular more traditional UNIX and UNIX-like environments.

Finally, note that as presented here Deliverance and URISpace won't actually do anything... it's expected that you'll add rules and themes appropriate to your own project.