Friday, June 19, 2009

Buildout with Repoze, Zope, and Plone

I know in my last post I indicated that I'd post more about the various technologies I'm using for the Antichrist Watch site, but I've now had a few requests related to some of the Repoze work I'm doing with some of the local Saugus, MA sites, so I'll be making a slight detour. I will get back to the Dojo stuff. I promise.

I discovered Repoze when first dabbling around with what would eventually become the aforementioned Antichrist Watch, and in fact I directly made use of repoze.chameleon to handle its templating needs. I liked what I saw, and decided to try first experimenting a bit with it and then actually porting over some real-world sites to it. We were in the process of doing some hardware upgrades on the some servers anyway, so the sites they hosted seemed liked good candidates. To make things interesting, most of them came in basically two distinct flavors: straight Zope, and customized Plone.

The first group was largely composed of mostly non-technical customers who take advantage of the bottom of the so-called "Z shaped curve" to do basic site edits through the Web. Some of these customers enjoy having fairly current versions of Zope. The second group featured various degrees of customization, from the fairly vanilla to the strikingly different. Most of these were handled via custom Plone products designed more or less in the manner described in Martin Aspeli's Professional Plone Development. With this logical grouping, buildout seemed a logical choice for site construction as I could make just three buildouts and then trivially generate as many sites as desired. The only problems were that the Repoze guys themselves seem not to use it much and (understandably considering the drudgery involved) don't have prepackaged versions of the latest-and-greatest of either Zope or Plone currently available.

Ultimately I wanted all the sites running through mod_wsgi using as few physical servers as practical. I wanted to only have one supervisord instance per physical server monitoring all the ZEO servers it contained. I also wanted to minimize the number of ports in use in order to reduce the bureaucracy of port tracking we'd have to do afterwards.

Getting current versions of Zope and Plone running under Repoze turned out not to be so simple. I read a few articles on the topic in addition to the Repoze Quick Start, but due to differences in versions and/or environment none of them did what I needed.

Making It Happen

Enough introduction! Here's what I did for the variant with Plone:

  1. paster create -t zope2_buildout targetname

    It doesn't much matter what answers are given, as buildout.cfg gets overwritten anyway.

  2. cd targetname

  3. Replaced buildout.cfg with the following:

    [buildout]
    extends =
        http://good-py.appspot.com/release/repoze.zope2/1.0
        http://dist.plone.org/release/3.3rc3/versions.cfg
    
    versions = versions
    
    find-links =
        http://dist.repoze.org/zope2/latest
        http://dist.repoze.org/zope2/dev
        http://dist.plone.org/release/3.3rc3
        http://download.zope.org/ppix/
        http://download.zope.org/distribution/
        http://effbot.org/downloads
    
    develop =
        src/mysite.policy
    
    parts =
        zope2
        instance
        slugs
        addpaths
    
    [zope2]
    recipe = zc.recipe.egg
    dependent-scripts = true
    interpreter = zopepy
    eggs =
        repoze.zope2
        Plone
        PIL
        Products.DocFinderTab
        Products.ExternalEditor
        plone.openid
        mysite.policy
    
    [slugs]
    recipe = collective.recipe.zcml
    zope2-location=${buildout:directory}
    zcml =
        mysite.policy
    
    [instance]
    recipe = iw.recipe.cmd
    on_install = true
    cmds =
       bin/mkzope2instance --use-zeo --zeo-port=${buildout:directory}/var/zeo.zdsock --zope-port=8888
       sed -i "" "s/server localhost:/server /" ${buildout:directory}/etc/zope.conf
       echo "Please run 'bin/runzeo -C etc/zeo.conf' and 'bin/paster serve etc/zope2.ini', then 'bin/addzope2user  '"
    
    [addpaths]
    recipe = z3c.recipe.runscript
    install-script = addpaths.py:main
    update-script = addpaths.py:main
    
  4. Added the file addpaths.py to the buildout's top level directory with the following contents:

    import os
    from dircache import listdir
    
    BinDir = 'bin'
    UnadornedFiles = ('bin/zope2.wsgi',)
    RegularFiles = ['%s/%s'%(BinDir,filename) for filename in listdir(BinDir)]
    EggDirs = ('eggs',)
    ProductsPath = os.path.abspath('products')
    
    def main(options, buildout):
        for filename in UnadornedFiles:
            lines = open(filename,'r').readlines()
            file = open(filename,'w')
            alreadyProcessed=False
            for line in lines:
                if line.startswith('import sys'):
                    alreadyProcessed=True
                elif line.startswith('import os') and not alreadyProcessed:
                    file.write("import sys\nsys.path[0:0] = [\n")
                    for eggDir in EggDirs:
                        for filename in listdir(eggDir):
                            file.write("  '%s',\n"%os.path.abspath('%s/%s'%(eggDir,filename)))
                    file.write("  ]\n\n")
                file.write(line)
        for filename in RegularFiles:
            lines = open(filename, 'r').readlines()
            file = open(filename, 'w')
            for line in lines:
                file.write(line)
                if line.startswith('sys.path'):
                    file.write("  '%s',\n"%ProductsPath)
            file.close()
    
        print "Egg paths added to %s" % ', '.join(UnadornedFiles)
        print "Product path added to %s" % ', '.join(RegularFiles)
    
  5. python2.4 bootstrap.py

  6. bin/buildout

    Just ignore any 'return' outside function types of errors you see.

Once these steps have been completed, ZEO can be started with the command bin/runzeo -C etc/zeo.conf (which can be easily controlled via supervisord) and Zope can be started manually for testing with bin/paster serve etc/zope2.ini or in a more production-ready form via mod_wsgi using zope2.wsgi.

Details To Note

I borrowed but heavily modified the addpath.py concept already living in the Plone collective to add all missing paths to all the scripts in bin. The WSGI script as created lacks pretty much everything, and all the others lack the products directory used for old-style products. As written it's not very clever and could be greatly improved, but it serves my current needs.

The order of parts matters somewhat, as both slugs and addpaths rely on pieces created earlier.

I'm using direct sockets in lieu of ports since these sites are living on single physical servers anyway and it means I have less to manage afterwards. Unfortunately the mkzope2instance doesn't do exactly what one might want in this case, so the sed line is necessary afterwards to clean up.

The slugs section adds ZCML slugs for those products that need them, including the hypothetical product mysite.policy being actively developed in src.

I just recently discovered Martin Aspeli's Good-Py. I was originally individually pinning the versions of the pieces that mattered, and only made this simplification this morning... so far it seems to be fine, though.

Just before starting this post I spotted Alex Clark's post also discussing this topic. We're doing a lot alike, but a few things differently. Depending upon what you're doing, you may find that what he's doing more directly addresses your needs.

Earlier on I mentioned how I needed to handle two main groups of sites: Plone ones and straight Zope ones. The above instructions only cover how I handled the Plone ones. This post has gotten long enough already, and although my treatment of the straight Zope ones is similar it's different enough to make things confusing for those who don't need it. If you need it, let me know and I'll probably give it a similar treatment to what I did here.