<?xml version="1.0"?>
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V1.2//EN" "http://apache.org/forrest/dtd/document-v12.dtd">
<document>
  <header>
    <title>The ForrestBot</title>
    <authors>
      <person name="Marc Portier" email="mpo@outerthought.org"/>
    </authors>
    <abstract>This document explains how the Forrest project can be used to
      centrally build the documentation for different projects. It also explains
      which pieces in the Forrest distribution form this
      functionality, and how they could be extended in the future.</abstract>
  </header>
  <body>
    <section>
      <title>Foreword</title>
      <p>This document explains how the Forrest project can be used to
      centrally build the documentation for different projects.
      It also explains which pieces in the Forrest distribution form this
      functionality, and how they could be extended in the future.</p>
      <p>See some related documents:</p>
      <ul>
        <li><link href="site:forrest-contract">Our Contract</link>
      - explains the rules with which your project should comply
     </li>
        <li><link href="site:forrestbot-intro">Forrestbot introduction</link>
      - concise overview of Forrestbot
     </li>
      </ul>
    </section>
    <section>
      <title>Using the Forrestbot</title>
      <section>
        <title>Setting up the centralized service</title>
        <p>The goal is to publish a static website that can be updated
          in near-real-time to accomodate newly available content. Forrestbot
          can run on one machine, get content from another machine, and publish
          to yet another machine.</p>
        <p>Running the bot is as easy as calling the following in your shell.
          (executed in the root dir of the distribution.)</p>
        <source>build bot -Dbot.config=myOwn.conf.xml -Dtemplate.echo=[true|false]</source>
        <p>It is quite typical to perform this from a shell script that gets
          scheduled for execution on a regular basis.</p>
      </section>
      <section>
        <title>Describing your project to the bot</title>
        <fixme author="mpo">There is not yet a DTD for this configuration
          file.</fixme>
        <p>The actions the bot is going to take are defined in the XML
          file pointed at with the <code>bot.config</code> System
          property. (if not specified with <code>-Dbot.config</code>
          then the default config from the
          distribution is taken: <code>./forrestbot.conf.xml</code>) </p>
        <p>This configuration file has one (required, but optionally empty) child-element for
          setting cross-project defaults, followed by a list of
          project-elements that hold the configuration for the various projects
          that the bot is going to work on.</p>
        <source>
<![CDATA[<?xml version="1.0"?>
<forrest-config>
  <defaults>
     ...<!-- different fallback values for various workstages -->
  </defaults>

  <project name="project-name-1">
     ...<!-- different subelements configure the various workstages -->
  </project>

  <project name="project-name-2">
     ...<!-- different subelements configure the various workstages -->
  </project>
</forrest-config>
]]>
        </source>
        <p>During the bot-run, each project entry will start a parallel and
          independent process consisting of a number of so called 'workstages'.</p>
        <p>These workstages are listed in the
          <code>./src/resources/forrestbot/stages.conf.xml</code> file and their actual
          execution is handled by the various template tasks in the
          <code>./resources/forrestbot/ant/templates.build.xml</code> file. </p>
        <p>Currently the various workstages are:</p>
        <ul>
          <dl>
            <dt>prepare</dt>
            <dd> Make a clean Forrest-defined Cocoon context environment.</dd>
          </dl>
          <dl>
            <dt>get-src</dt>
            <dd>Slide in the project specific document sources in that Cocoon
            context environment.</dd>
          </dl>
          <dl>
            <dt>generate</dt>
            <dd> Actually generate the site using Cocoon</dd>
          </dl>
          <dl>
            <dt>deploy</dt>
            <dd>Publish the newly created site</dd>
          </dl>
          <dl>
            <dt>cleanup</dt>
            <dd>&lt;No real actions yet&gt;</dd>
          </dl>
        </ul>
        <p>Inside the project element the configuration allows to set some
          parameters that get consumed in the template targets. This workstage dependant
          configuration always takes the following layout: an element with the name of
          one of the tasks, and an optional type-attribute with nested elements that hold
          parameter names, with required name attribute holding the actual parameter
          value. Example:</p>
        <source>&lt;project name=&quot;xml-forrest&quot;&gt;
    &lt;prepare&gt;
        &lt;skin name=&quot;forrest-site&quot;/&gt;
    &lt;/prepare&gt;

    &lt;get-src type=&quot;cvs&quot;&gt;
      &lt;host name=&quot;cvs.apache.org&quot;/&gt;
      &lt;root name=&quot;/home/cvspublic&quot;/&gt;
      &lt;user name=&quot;anoncvs&quot;/&gt;
      &lt;passwd name=&quot;anoncvs&quot;/&gt;
      &lt;module name=&quot;xml-forrest&quot;/&gt;
      &lt;content-dir name=&quot;src/documentation&quot;/&gt;
      &lt;project-dir name=&quot;&quot;/&gt;
    &lt;/get-src&gt;

    &lt;generate&gt;
      &lt;debuglevel name=&quot;ERROR&quot; /&gt;
    &lt;/generate&gt;  

    &lt;deploy type=&quot;scp&quot;&gt;
      &lt;host name=&quot;krysalis.sourceforge.net&quot;/&gt;
      &lt;root name=&quot;/home/groups/k/kr/krysalis/htdocs&quot;/&gt;
      &lt;user name=&quot;forrestbot&quot;/&gt;
      &lt;dir name=&quot;forrest&quot;/&gt;
    &lt;/deploy&gt;
  &lt;/project&gt;</source>
        <note>Only the workstages that need configuration settings have an
          entry here. At all times there will be a workstage template called for each
          stage of the stages.conf.xml (even if it is not explicitly mentioned inside the
          &lt;project&gt; element.) </note>
        <p>Explained in detail, this will</p>
        <ul>
          <li>SET names properties (workstage(.@type)?.*) to appropriate values
            (*/@name)</li>
          <ul>
            <dl>
              <dt>prepare.skin</dt>
              <dd>forrest-site</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.host</dt>
              <dd>cvs.apache.org</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.root</dt>
              <dd>/home/cvspublic</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.user</dt>
              <dd>anoncvs</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.passwd</dt>
              <dd>anoncvs</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.module</dt>
              <dd>xml-forrest</dd>
            </dl>
            <dl>
              <dt>get-src.cvs.content-dir</dt>
              <dd>src/documentation</dd>
            </dl><dl>
              <dt>get-src.cvs.project-dir</dt>
              <dd></dd>
            </dl>
            <dl>
              <dt>generate.debuglevel</dt>
              <dd>ERROR</dd>
            </dl>
            <dl>
              <dt>deploy.scp.host</dt>
              <dd>krysalis.sourceforge.net</dd>
            </dl>
            <dl>
              <dt>deploy.scp.root</dt>
              <dd>/home/groups/k/kr/krysalis/htdocs</dd>
            </dl>
            <dl>
              <dt>deploy.scp.user</dt>
              <dd>forrestbot</dd>
            </dl>
            <dl>
              <dt>deploy.scp.dir</dt>
              <dd>forrest</dd>
            </dl>
          </ul>
          <li>and CALL the following templates
          (template.workstage(.@type)?):</li>
          <ul>
            <li>template.prepare</li>
            <li>template.get-src.cvs</li>
            <li>template.generate</li>
            <li>template.deploy.scp</li>
            <li>template.cleanup</li>
          </ul>
        </ul>
        <p>What each of these templates is going to do with the parameters can
          be found in the templates.build.xml itself (at this stage) and is put out to
          the console if the bot is run with the <code>-Dtemplate.echo=true</code>
          flag</p>
        <section>
          <title>Setting bot-wide defaults</title>
          <p>The <code>&lt;defaults&gt;</code> section in the
            <code>&lt;forrest-config&gt;</code> allows for one-time setting of parameters
            you want to reuse for different projects. The syntax (and semantic value) of
            the nested elements is identical to the use inside the
            <code>&lt;project&gt;</code> tags.</p>
        </section>
      </section>
      <note>Once you understand all of this you might decide NOT to run your
        own centralized service but rather ask your project to be taken up in the
        centralized service of the project team itself: just send a mail to
        forrest-dev <link href="site:mail-lists">mail list</link>
        defining the required parameters to be taken up in the
        process.</note>
      <section>
        <title>Checking logs and getting them by mail</title>
        <p>For each distinct project the Forrest bot is maintaining a separate
          working log of the different workstages being executed. These logs can be
          opened and viewed in the ./build/bot directory</p>
        <p>In addition to the @name attribute the &lt;project&gt; tag also can
          contain a @sendlogto attribute that must be holding a (comma separated list
          off) email address(es) where the project specific log needs to be sent to after
          all workstages have been completed (or as soon as building failed)</p>
      </section>
    </section>
    <section>
      <title>Current supported workstages and types.</title>
      <section>
        <title>
          <code>&lt;prepare&gt;</code>
        </title>
        <p>No specific type versions. (don't use type-attribute)</p>
        <p>Template arguments:</p>
        <ul>
          <li><code>&lt;skin&gt;</code> @name holds the name of the skin to use.</li>
        </ul>
      </section>
      <section>
        <title>
          <code>&lt;get-src ...&gt;</code>
        </title>
        <section>
          <title>
            <code>&lt;get-src type=&quot;cvs&quot;&gt;</code>
          </title>
          <p>Template arguments:</p>
          <ul>
            <li><code>&lt;host&gt;</code> @name holds the ipaddress or dns name of the  cvs serving host</li>
            <li><code>&lt;root&gt;</code> @name holds the cvs-root directory on that host</li>
            <li><code>&lt;user&gt;</code> @name holds the username to use on the cvs repository </li>
            <li><code>&lt;passwd&gt;</code> @name holds the password to use on the cvs repository</li>
            <li><code>&lt;module&gt;</code> @name holds the module name that holds the </li>
            <li><code>&lt;dir&gt;</code> @name holds the relative path to the {docroot} directory.  (This is the dir  that is holding the ./content/xdocs as specified in t. </li>
          </ul>
        </section>
        <section>
          <title>
            <code>&lt;get-src type=&quot;local-copy&quot;&gt;</code>
          </title>
          <p>Template arguments:</p>
          <ul>
            <li><code>&lt;content-dir/&gt;</code> @name holds the path pointing to the {docroot} directory. (See the <link href="site:forrest-contract">contract</link> to understand what should be there.)</li>
            <li><code>&lt;project-dir/&gt;</code> @name holds the path pointing to the {projecthome} directory where the xml project descriptors reside.</li>
          </ul>
        </section>
      </section>
      <section>
        <title>
          <code>&lt;generate&gt;</code>
        </title>
        <p>No specific type versions. (don't use type-attribute)</p>
        <p>Template arguments:</p>
        <ul>
          <li><code>&lt;debuglevel&gt;</code> @name holds the threshold-level for showing debug statements in the cocoon generation process.</li>
        </ul>
      </section>
      <section>
        <title>
          <code>&lt;deploy ...&gt;</code>
        </title>
        <section>
          <title>
            <code>&lt;deploy type=&quot;scp&quot;&gt;</code>
          </title>
          <p>Template arguments:</p>
          <ul>
            <li><code>&lt;host&gt;</code> @name holds the remote host where the file will be copied to.</li>
            <li><code>&lt;user&gt;</code> @name holds the username to be used when logging onto the remote host.</li>
            <li><code>&lt;root&gt;</code> @name holds the prefix part of where the content needs to be published.</li>
            <li><code>&lt;dir&gt;</code> @name holds the suffix part of where the content needs to be published.</li>
          </ul>
          <warning>Using this approach requires that the public key of the user executing the bot target is present in the authorized key file (<code>~/.ssh/authorized_keys2</code>) of the remote user (on the remote host).</warning>
          <note>Might be interesting to understand that the full process actually is bundling all files to copy in a *.tar.gz that is un-tar.gzipped on the remote host using ssh. </note>
        </section>
        <section>
          <title>
            <code>&lt;deploy type=&quot;local-copy&quot;&gt;</code>
          </title>
          <p>Template arguments:</p>
          <ul>
            <li><code>&lt;destination&gt;</code> @name holds the path where the generated content needs to be copied to.</li>
          </ul>
        </section>
        <section>
          <title>
            <code>&lt;deploy type=&quot;ftp&quot;&gt;</code>
          </title>
          <p>Template arguments:</p>
          <ul>
            <li><code>&lt;host&gt;</code> @name holds the hostname of the ftp server to publish to.</li>
            <li><code>&lt;user&gt;</code> @name holds the username to use for loging onto the ftp server.</li>
            <li><code>&lt;passwd&gt;</code> @name holds the password to use.</li>
            <li><code>&lt;destination&gt;</code> @name holds the path to the directory on the remote host where the generated content needs to be published.  The process is using the <code>ftp cd</code> command to get there.  This means this directory has to exist.</li>
          </ul>
        </section>
        <warning>Currently the ant distribution that is included in forrest cvs is not offering the required NetComponents.jar to actually support this type of deployment. If you want to use it anyway you should get the required jar from <link href="http://www.savarese.org/oro/downloads/index.html#NetComponents">http://www.savarese.org/oro/downloads</link> and drop that into your <code>{forrest-sandbox}/tools/antipede/lib</code> directory.  </warning>
      </section>
      <section>
        <title>
          <code>&lt;cleanup&gt;</code>
        </title>
        <p>No specific type versions. (don't use type-attribute)</p>
        <p>Template arguments: None.</p>
      </section>
    </section>
    <section>
      <title>ForrestBot design</title>
      <p>Most of us will just like things to work, and will be happy enough
        just using it, maybe even letting the forrest-dev
        <link href="site:mail-lists">mail list</link> know if we could improve
        this or the other. Some however might be drawn to the
      <link href="http://www.howstuffworks.com/">how stuff works</link> of things.
      For them is this section as a start in the rest of their own pursuit.</p>
      <section>
        <title>The Ant play</title>
        <p>All actions and targets related to the actual execution of the bot's
          work is separated over 3 distinct build files:</p>
        <ul>
          <dl>
            <dt>[main] <code>./build.xml</code></dt>
            <dd>the starting point where generation and preparation is initiated.
            </dd>
          </dl>
          <dl>
            <dt>[generated] <code>./build/bot/work.build.xml</code></dt>
            <dd>the one that reflects the configured tasks in the
            forrestbot.conf.xml</dd>
          </dl>
          <dl>
            <dt>[templates]
            <code>./src/resources/forrestbot/ant/templates.build.xml</code></dt>
            <dd>listing the different targets that actually depict the supported
            workstages (and types of)</dd>
          </dl>
        </ul>
        <p>The complete process runs as follows:</p>
        <ol>
          <li>'bot' target inside main build.xml depends on a number of targets
            that</li>
          <ol>
            <li>'bot.init': initialize the environment</li>
            <li>'bot.conf2build': use XSLT stylesheets to convert the
            forrestbot.conf.xml into the work.build.xml (and a properties file for the
            default-section)</li>
            <li>'bot.run': delegates to the 'work' target of the generated
            work.build.xml</li>
          </ol>
          <li>'work' target inside generated work.build.xml will then start in
          parallel (so failure of the one would not influence the other) the different
          project specific work-processes. </li>
          <li>each of these project runs (generated 'work.[projectname]' targets)
          depends on the sequential execution of the different workstage actions
          (generated '[workstage].[projectname(.[type])?]' targets) </li>
          <li>In those specific targets the pattern is as follows:</li>
          <ol>
            <li>set the project specific parameters for this project and
            workstage</li>
            <li>load the default parameter values for the ones that aren't set
            yet</li>
            <li>call the actual template.[workstage(.[type]?)]</li>
          </ol>
        </ol>
      </section>
      <section>
        <title>Adding features to the bot - The meta template</title>
        <p>Given the mechanism, the actual things the bot <strong>can</strong>
          do boil down to the template targets it can call. So extending the bot means:
          adding templates-targets.</p>
        <p>If you want to add your own tempales to
          <code>template.build.xml</code>, take in account the following guidelines:</p>
        <ul>
          <li> in the future there will be a DTD for forrestbot.conf.xml that
            should be updated probably after you added stuff to the
            <code>templates.build.xml</code>. Backwards-incompatible changes need a formal
            vote.</li>
          <li>build an echo target for each template target you make: </li>
        </ul>
        <source>
<![CDATA[
<target name="echo.workstage-name.optional-type" if="template.echo">
    ... here you put echo ant-tasks that somewhat tell what you will
    ... do with all the params you expect
</target>

<target name="template.workstage-name.optional-type"
            depends="shared.set-props, echo.workstage-name.optional-type">
    ... here you put whatever ant-tasks that actually do it
</target>          
    ]]>
        </source>
      </section>
    </section>
  </body>
</document>
