Buildout with Repoze, Zope, and Plone

I know in my last post I indicated that I'd post more about the various technologies I'm using for the Antichrist Watch site, but I've now had a few requests related to some of the Repoze work I'm doing with some of the local Saugus, MA sites, so I'll be making a slight detour. I will get back to the Dojo stuff. I promise.

I discovered Repoze when first dabbling around with what would eventually become the aforementioned Antichrist Watch, and in fact I directly made use of repoze.chameleon to handle its templating needs. I liked what I saw, and decided to try first experimenting a bit with it and then actually porting over some real-world sites to it. We were in the process of doing some hardware upgrades on the some servers anyway, so the sites they hosted seemed liked good candidates. To make things interesting, most of them came in basically two distinct flavors: straight Zope, and customized Plone.

The first group was largely composed of mostly non-technical customers who take advantage of the bottom of the so-called "Z shaped curve" to do basic site edits through the Web. Some of these customers enjoy having fairly current versions of Zope. The second group featured various degrees of customization, from the fairly vanilla to the strikingly different. Most of these were handled via custom Plone products designed more or less in the manner described in Martin Aspeli's Professional Plone Development. With this logical grouping, buildout seemed a logical choice for site construction as I could make just three buildouts and then trivially generate as many sites as desired. The only problems were that the Repoze guys themselves seem not to use it much and (understandably considering the drudgery involved) don't have prepackaged versions of the latest-and-greatest of either Zope or Plone currently available.

Ultimately I wanted all the sites running through mod_wsgi using as few physical servers as practical. I wanted to only have one supervisord instance per physical server monitoring all the ZEO servers it contained. I also wanted to minimize the number of ports in use in order to reduce the bureaucracy of port tracking we'd have to do afterwards.

Getting current versions of Zope and Plone running under Repoze turned out not to be so simple. I read a few articles on the topic in addition to the Repoze Quick Start, but due to differences in versions and/or environment none of them did what I needed.

Making It Happen

Enough introduction! Here's what I did for the variant with Plone:

1. paster create -t zope2_buildout targetname

   It doesn't much matter what answers are given, as buildout.cfg gets overwritten anyway.

2. cd targetname

3. Replaced buildout.cfg with the following:

   [buildout]
   extends =
       http://good-py.appspot.com/release/repoze.zope2/1.0
       http://dist.plone.org/release/3.3rc3/versions.cfg
   
   versions = versions
   
   find-links =
       http://dist.repoze.org/zope2/latest
       http://dist.repoze.org/zope2/dev
       http://dist.plone.org/release/3.3rc3
       http://download.zope.org/ppix/
       http://download.zope.org/distribution/
       http://effbot.org/downloads
   
   develop =
       src/mysite.policy
   
   parts =
       zope2
       instance
       slugs
       addpaths
   
   [zope2]
   recipe = zc.recipe.egg
   dependent-scripts = true
   interpreter = zopepy
   eggs =
       repoze.zope2
       Plone
       PIL
       Products.DocFinderTab
       Products.ExternalEditor
       plone.openid
       mysite.policy
   
   [slugs]
   recipe = collective.recipe.zcml
   zope2-location=${buildout:directory}
   zcml =
       mysite.policy
   
   [instance]
   recipe = iw.recipe.cmd
   on_install = true
   cmds =
      bin/mkzope2instance --use-zeo --zeo-port=${buildout:directory}/var/zeo.zdsock --zope-port=8888
      sed -i "" "s/server localhost:/server /" ${buildout:directory}/etc/zope.conf
      echo "Please run 'bin/runzeo -C etc/zeo.conf' and 'bin/paster serve etc/zope2.ini', then 'bin/addzope2user  '"
   
   [addpaths]
   recipe = z3c.recipe.runscript
   install-script = addpaths.py:main
   update-script = addpaths.py:main
   

4. Added the file addpaths.py to the buildout's top level directory with the following contents:

   import os
   from dircache import listdir
   
   BinDir = 'bin'
   UnadornedFiles = ('bin/zope2.wsgi',)
   RegularFiles = ['%s/%s'%(BinDir,filename) for filename in listdir(BinDir)]
   EggDirs = ('eggs',)
   ProductsPath = os.path.abspath('products')
   
   def main(options, buildout):
       for filename in UnadornedFiles:
           lines = open(filename,'r').readlines()
           file = open(filename,'w')
           alreadyProcessed=False
           for line in lines:
               if line.startswith('import sys'):
                   alreadyProcessed=True
               elif line.startswith('import os') and not alreadyProcessed:
                   file.write("import sys\nsys.path[0:0] = [\n")
                   for eggDir in EggDirs:
                       for filename in listdir(eggDir):
                           file.write("  '%s',\n"%os.path.abspath('%s/%s'%(eggDir,filename)))
                   file.write("  ]\n\n")
               file.write(line)
       for filename in RegularFiles:
           lines = open(filename, 'r').readlines()
           file = open(filename, 'w')
           for line in lines:
               file.write(line)
               if line.startswith('sys.path'):
                   file.write("  '%s',\n"%ProductsPath)
           file.close()
   
       print "Egg paths added to %s" % ', '.join(UnadornedFiles)
       print "Product path added to %s" % ', '.join(RegularFiles)
   

5. python2.4 bootstrap.py

6. bin/buildout

   Just ignore any 'return' outside function types of errors you see.

Once these steps have been completed, ZEO can be started with the command bin/runzeo -C etc/zeo.conf (which can be easily controlled via supervisord) and Zope can be started manually for testing with bin/paster serve etc/zope2.ini or in a more production-ready form via mod_wsgi using zope2.wsgi.

Details To Note

I borrowed but heavily modified the addpath.py concept already living in the Plone collective to add all missing paths to all the scripts in bin. The WSGI script as created lacks pretty much everything, and all the others lack the products directory used for old-style products. As written it's not very clever and could be greatly improved, but it serves my current needs.

The order of parts matters somewhat, as both slugs and addpaths rely on pieces created earlier.

I'm using direct sockets in lieu of ports since these sites are living on single physical servers anyway and it means I have less to manage afterwards. Unfortunately the mkzope2instance doesn't do exactly what one might want in this case, so the sed line is necessary afterwards to clean up.

The slugs section adds ZCML slugs for those products that need them, including the hypothetical product mysite.policy being actively developed in src.

I just recently discovered Martin Aspeli's Good-Py. I was originally individually pinning the versions of the pieces that mattered, and only made this simplification this morning... so far it seems to be fine, though.

Just before starting this post I spotted Alex Clark's post also discussing this topic. We're doing a lot alike, but a few things differently. Depending upon what you're doing, you may find that what he's doing more directly addresses your needs.

Earlier on I mentioned how I needed to handle two main groups of sites: Plone ones and straight Zope ones. The above instructions only cover how I handled the Plone ones. This post has gotten long enough already, and although my treatment of the straight Zope ones is similar it's different enough to make things confusing for those who don't need it. If you need it, let me know and I'll probably give it a similar treatment to what I did here.