Setting up a ZEO Cluster for ZEO 2, Zope 2.6.1 and DBTab.

============================= DRAFT VERSION ================================

  Meta::

    Valid for:  Silva 0.9.x
    Author:     Jan-Wijbrand Kolman
    Email:      jw@infrae.com
    CVS:        $Revision: 1.3 $ 

  Introduction:

    These notes were collected during a ZEO2 Cluster setup proces to test
    Silva in a clustered environment. Most information can be found at the
    www.zope.org website, although either slightly outdated or targeting a
    specific platform or usage.

    The document assumes knowledge of and experience with setting up Zope
    from sources, using the "INSTANCE_HOME" [Ax] technique, using sessions,
    etc. It also assumes a UNIX-like OS for the installation. The
    instructions '$ ' for the command line prompt and commands or code
    snippets which really should be typed on one line, will be split with a
    '\'.

    I hope the notes do provide enough instructions to setup a ZEO2
    cluster, with DBTab and shared sessions and a highlevel overview and
    sufficient pointers to more information. Feedback and/or corrections are
    highly appreciated.

  Problem description, solution:

    A ZEO cluster provides building blocks to setup a scalable and highly
    available platform for CPU intensive webapplications.

    Typically a ZEO cluster consists of 1 or more clients (using a
    "ClientStorage" - one will often find the "CS" abbreviation) and 1
    server (the "ZEO StorageServer" - or ZSS). Each client, most probably
    running on a seperate network node, serves the actual webapplications.

    The server provides access to one or more object databases and is
    responsible for synchronizing the clients throughout database changes.
    Each client keeps a local cache in RAM for objects and a separate
    "storage layer" cache, to speed up object read actions. Objects in
    these caches get invalidation messages from the ZSS whenever an object
    changed on an other node. 

    To evenly spread the application load over the different clients a
    'front-end', or 'load balancing' server is put in place. End users
    connnect to the balancer, which transparently maps the requests to one
    of the clients. Typically the client and server nodes are kept on a
    private network, where the balancer acts as a "bridge" between the
    public and this private network.

    Most web applications - and Silva is certainly one of them - keep track
    of the end users "state" within the application. Webapplications may do
    so by "tagging" the end user web browser with a cookie. This cookie
    merely provides an identifying number to the webapplication. With this
    ID the application keeps a 'session' object per end user. This session
    object is a generic data container to store end user states within the
    application.

    Normally (in a non-clustered situation) this session object is not
    persisted into the ZODB (usually a FileStorage - the "Data.fs" file),
    but into a volatile storage type (TemporaryStorage), storing the
    objects in RAM.

    This is possible since session objects are generaly speaking small and
    only exist for a limited lifespan (typically 20 minutes). Yet another
    reason is to prevent the main ZODB from excessive growth, since for
    each change in object (and in case of session objects, this effectively
    is for each HTTP request), data is appended to the ZODB.

    In a clustered situation however, if the balancer maps a request to,
    say, client node A in the network, this will set a session object in
    its RAM cache. A consecutive request might be mapped to a client B,
    which does not have a session object in RAM identified by the end users
    cookie and a new empty sessions object is created. This will obviously
    break the application state tracking.

    A solution is to actually persist the session objects in a central
    database on the ZSS. To prevent the mentioned excesive growth, a
    seperate database is setup which does not append, but just replaces the
    data on each (session) object change.

    This separate database can be a TemporaryStorage, which means that the
    session objects are kept in a volatile RAM base storage on the ZSS
    itself. Since all clients connect to this storage, they are able to
    share the session objects. This setup also requires less system
    administration.

    An alternative approach (similar as described in the earlier version of
    this document) is to store these session objects in a BDBStorage -
    which is the successor of the bsddb3Storage - a Berkeley DB backed
    storage type. This approach is not (yet) described in this document.

    [DBTab details]

  Software versions used:

    * Python 2.1.3
        http://www.python.org

    * Zope 2.6.1
        http://www.zope.org/Products/Zope/2.6.1

    * ZEO version 2.0.2
        http://www.zope.org/Products/ZODB3.1

        ZEO is now part of the so called "Standalone ZODB" distribution.

    * DBTab 1.1
        http://hathaway.freezope.org/Software/DBTab

  Software setup (in high level instrcutions):

    Asumming the following directory layout:

      ../src (compiled Zope2.6.1 sources)
      ../client ('INSTANCE_HOME' for CS)
      ../server (location for ZSS)

    To use ZEO copy the ZEO directory from the ZODB3.1.1 distribution to
    src/lib/python (or any other means to get ZEO on the python search
    path).

    Place the 'StorageConfig.py' and 'CustomTemporaryStorage.py' python
    modules in the ../server directory. The 'StorageConfig' instructs the
    ZSS what storages should be served. The 'CustomTemporaryStorage' is a
    simple subclass of the standard Zope TemporaryStorage class.

    The ZSS should be able to start now [B3]. The ZSS will start in "debug"
    mode and provide many details in the logfile if the '-D' option is
    supplied (in the startup script). In case the ZSS does not (correctly)
    start, this output might provide hints. A common problem is configuring
    the 'PYTHONPATH'.

    Install the DBTab Product in ../client/Products directory and place the
    external method to delete the temp_folder [B6] in ../client/Extensions.
    As instructed by the DBTab documentation, copy (or symlink) the
    'custom_zodb.py' from the DBTab Product directory to the ../client
    directory. We will not alter this file.

    Copy the 'dbtab.conf.in' from the DBTab Product directory to the
    ../client directory and rename it to 'dbtab.conf'. This is the actual
    configuration file which will instruct the CS to connect to the ZSS
    [B4] and what storages to mount from the ZSS.
                                                    
    Now, the CS should be able to start. Again, when supplied with the '-D'
    option, detailed debugging information is provided in the logfile. This
    should show the CS actually connecting to the ZSS. The ZSS logs will
    also show this information.

    Now two more steps are required both using the Zope Management
    Interface (ZMI). First add an "External Method". This requires three
    options to to be filled in (and a title which is optional): 'id',
    'module name' and 'function name'. This external method will point to
    the 'delete_temp_folder.py' file. The filename is the 'module name'.
    The name of the method defined in there, is the 'function name'.

    Once added, we can use the 'test tab' of this object to remove the
    existing 'temp_folder' (mind the special icon of this object). This
    folder cannot be deleted with the normal Zope 'delete' button, since it
    is "protected".

    Now we can add a "DBTab mount point'. The form provided shows all
    available mountable storages. The '/temp_folder' as defined in the
    'dbtab.conf' should be available. Select it and create the mount point.

    After a CS restart, there should be a 'temp_folder' normal looking
    folder icon. Inside this folder there should be a 'session_data'
    object. This folder is actually a mount point to the TemporaryStorage
    on the ZSS.

  A: References:

      [Ax] Advanced Production Site Setup with ZEO:
            http://www.zope.org/Members/dshaw/AdvancedSiteSetup

      [Ax] Central Zope code base, multiple instances:
            http://www.zope.org/Members/4am/instancehome

      [Ax] ZEO setup:
            http://www.zope.org/Members/kedai/UseZeoZope                                              

      [Ax] ZODB:
            http://www.zope.org/Products/ZODB3.1

      [Ax] DBTab product:
            http://hathaway.freezope.org/Software/DBTab

  B: Code:

    [B1] StorageConfig.py - for storages to server to clients:

      # begin
      import os
      from ZODB import DB
      from ZODB.FileStorage import FileStorage
      from Products.Transience.Transience import TransientObjectContainer
      from OFS.Folder import Folder
      # Getting TemporaryStorage to work with ZEO2
      from CustomTemporaryStorage import CustomTemporaryStorage

      mainStorage = FileStorage('/path/to/Data.fs',)
      sessionStorage = CustomTemporaryStorage()
      # end

    [B2] CustomTemporaryStorage.py - subclass of TemporaryStorage:

      # begin
      from Products.TemporaryFolder.TemporaryStorage import TemporaryStorage

      class CustomTemporaryStorage(TemporaryStorage):
          def supportsTransactionalUndo(self): return 0
      # end

    [B3] ZSS start script:

      This bash script start the ZEO client. Should obviously be adjusted
      to local ZEO and INSTANCE_HOME setup.

      #! /bin/bash
      export SRC=../src
      export INSTANCE_HOME=$PWD
      export STUPID_LOG_FILE=$PWD/var/ZEO_EVENTS.log
      export PYTHONPATH=$PWD:$SRC/lib/python:$PYTHONPATH

      exec python2.1 $SRC/lib/python/ZEO/start.py \
        -t 10 \
        -p portnumber-to-serve-on \
        -D \
        -S main=StorageConfig:mainStorage \
        -S sessiondata=StorageConfig:sessionStorage &
      # end

    [B4] dbtab.conf for CS:

      The 'dbtab.conf.in' as distributed with the DBTab Prodcust contains a
      lot of documentation on required and optional parameters.

      # begin
      [Storage: Main]
      type=ClientStorage
      addr=zeo://url.to.zss:port
      client=zss-filesystemcache
      storage=main

      [Database: Main]
      cache_size=10000
      mount_paths=/

      [Storage: SharedSessions]
      type=ClientStorage
      addr=zeo://url.to.zss:port
      storage=sessiondata

      [Database: SharedSessions]
      #connection_class=LowConflictConnection # is the needed?
      cache_size=10000
      mount_paths=/temp_folder
      open_at_startup=1
      auto_create=1
      # end

    [B5] CS start script:

      This bash script start the ZEO client. Should obviously be adjusted
      to local ZEO and INSTANCE_HOME setup, the use of FastCGI, ProxyPass,
      etc. For more info and details, read [zopesrc]/doc/ENVIRONMENT.txt

      #!/bin/bash
      export FORCE_PRODUCT_LOAD=1 
      export INSTANCE_HOME=$PWD
      export CLIENT_HOME=/var/zeo2 # for storing caches and logs
      export EVENT_LOG_FILE=$CLIENT_HOME/zope.log

      exec python2.1 ../src/z2.py \
        -X \
        -w http-portnumber \
        -F /path/to/fcgi/socket/file \
        "$@" &

    [B6] Delete temp_folder external method:

      Place this external method in the Extensions directory of the clients
      INSTANCE_HOME.

      # begin
      def delete_temp_folder(self):
          self.getPhysicalRoot()._delObject('temp_folder')
          return 'Removed temp_folder successfully.'
      # end


<br />
Copyright &copy; 2002, 2003 Infrae. All rights reserved.<br />
See also "LICENSE.txt":license.txt
