Regression Testing

Bacula 1.34 User's Guide |
Porting Bacula | Index | Implementing a GUI
# Bacula Regression Testing ## General This document is intended mostly for developers who wish to ensure that their changes to Bacula don't introduce bugs in the base code. You can find the existing regression script in the Bacula CVS on the SourceForge CVS in the project tree named regress. There are two different aspects of regression testing that this document will discuss: 1. Running the Regression Script, 2. Writing a Regression test. ## Running the Regression Script There are a number of different tests that may be run, such as: the standard set that uses disk Volumes and runs under any userid; a small set of tests that write to tape; another set of tests where you must be root to run them. To date, each subset of tests runs no more than about 15 minutes. ### Setting the Configuration Parameters Once you have the regression directory loaded, you will first need to create a custom xxx.conf file for your system. You can either edit prototype.conf directly or copy it to a new file and edit it. To see a real example of a configuration file, look at kern.conf. The variables you need to modify are:
    # Where to get the source to be tested
    BACULA_SOURCE="${HOME}/bacula/k"

    # Where to send email   !!!! Change me !!!!!!!
    EMAIL=your-email@domain.com

    # Full path where to find sqlite
    DEPKGS="${HOME}/bacula/depkgs/sqlite"

    TAPE_DRIVE="/dev/nst0"

    # if you don't have an autochanger set
    #   AUTOCHANGER to /dev/null
    AUTOCHANGER="/dev/sg0"

    # This must be the path to the autochanger
    #  including its name
    AUTOCHANGER_PATH="/bin/mtx"
BACULA_SOURCE should be the full path to the Bacula source code that you wish to test. EMAIL should be your email addres. Please remember to change this or I will get a flood of unwanted messages. You may or may not want to see these emails. In my case, I don't need them so I direct it to the bit bucket. SQLITE_DIR should be the full path to the sqlite package, must be build before running a Bacula regression, if you are using SQLite. This variable is ignored if you are using MySQL or PostgreSQL. To use PostgreSQL, edit the Makefile and change (or add) WHICHDB?="--with-postgresql". For MySQL use "WHICHDB?="--with-mysql". TAPE_DRIVE is the full path to your tape drive. The base set of regression tests do not use a tape, so this is only important if you want to run the full tests. AUTOCHANGER is the name of your autochanger device. Set this to /dev/null if you do not have one. AUTOCHANGER_PATH is the full path including the program name for your autochanger program (normally mtx. Leave the default value if you do not have one. ### Building the Test Bacula Once the above variables are set, you can build Bacula by entering:
    ./config xxx.conf
    make setup
Where xxx.conf is the name of the conf file containing your system parameters. This will build a Makefile from Makefile.in, then copy the source code within the regression tree (in directory regress/build), configure it, and build it. There should be no errors. If there are, please correct them before continuing. ### Running the Disk Only Regression Once Bacula is built, you can run the basic disk only non-root regression test by entering:
    make test
This will run the base set of tests using disk Volumes, currently (19 Dec 2003), there are current 18 separate tests that run. If you are testing on a non-Linux machine two of the tests will not be run. In any case, as we add new tests, the number will vary. It will take about 5 or 10 minutes if you have a fast (2 GHz) machine, and you don't need to be root to run these tests (I run under my regular userid). The result should be something similar to:
    Test results

      ===== Backup Bacula Test OK =====
      ===== Verify Volume Test OK =====
      ===== sparse-test OK =====
      ===== compressed-test OK =====
      ===== sparse-compressed-test OK =====
      ===== Weird files test OK =====
      ===== two-jobs-test OK =====
      ===== two-vol-test OK =====
      ===== six-vol-test OK =====
      ===== bscan-test OK =====
      ===== Weird files2 test OK =====
      ===== concurrent-jobs-test OK =====
      ===== four-concurrent-jobs-test OK =====
      ===== bsr-opt-test OK =====
      ===== bextract-test OK =====
      ===== recycle-test OK =====
      ===== span-vol-test OK =====
      ===== restore-by-file-test OK =====
      ===== restore2-by-file-test OK =====
      ===== four-jobs-test OK =====
      ===== incremental-test OK =====
and the working tape tests are:
    Test results

      ===== Bacula tape test OK =====
      ===== Small File Size test OK =====
      ===== restore-by-file-tape test OK =====
      ===== incremental-tape test OK =====
      ===== four-concurrent-jobs-tape OK =====
      ===== four-jobs-tape OK =====
Each separate test is self contained in that it initializes to run Bacula from scratch (i.e. newly created database). It will also kill any Bacula session that is currently running. In addition, it uses ports 8101, 8102, and 8103 so that it does not intefere with a production system. ### Other Tests There are a number of other tests that can be run as well. All the tests are a simply shell script keep in the regress directory. For example the "make test" simply executes ./all-non-root-tests. The other tests are: all_non-root-tests All non-tape tests not requiring root. This is the standard set of tests, that in general, backup some data, then restore it, and finally compares the restored data with the original data. all-root-tests All non-tape tests requiring root permission. These are a relatively small number of tests that require running as root. The amount of data backed up can be quite large. For example, one test backs up /usr, another backs up /etc. One or more of these tests reports an error -- I'll fix it one day. all-non-root-tape-tests All tape test not requiring root. There are currently three tests, all run without being root, and backup to a tape. The first two tests use one volume, and the third test requires an autochanger, and uses two volumes. If you don't have an autochanger, then this script will probably produce an error. all-tape-and-file-tests All tape and file tests not requiring root. This includes just about everything, and I don't run it very often. ### If a Test Fails If you one or more tests fail, the line output will be similar to:
      !!!!! concurrent-jobs-test failed!!! !!!!!
If you want to determine why the test failed, you will need to modify the script so that it prints. Do so by finding the file in regress/tests that corresponds to the name printed. For example, the script for the above error message is in: regress/tests/concurrent-jobs-test. In order to see the output produced by Bacula, you need only change the lines that start with @output to @tee, then rerun the test by hand. it is very important to start the test from the regress directory. To modify the test mentioned above so that you can see the output, change every occurrence of @output to @tee in the file. In rare cases you might need to remove the 2>&1 >/dev/null from the end of the bacula, bconsole, or diff lines, but this is rare. ## Writing a Regression Test Any developer, who implements a major new feature, should write a regression test that exercises and validates the new feature. Each regression test is a complete test by itself. It terminates any running Bacula, initializes the database, starts Bacula, then runs the test by using the console program. ### Running the Tests by Hand You can run any individual test by hand by cd'ing to the regress directory and entering:
    tests/<test-name>
### Directory Structure The directory structure of the regression tests is:
      regress                - Makefile, scripts to start tests
        |------ scripts      - Scripts and conf files
        |-------tests        - All test scripts are here
        |
        |------------------ -- All directories below this point are used
        |                       for testing, but are created from the
        |                       above directories and are removed with
        |                       "make distclean"
        |
        |------ bin          - This is the install directory for
        |                        Bacula to be used testing
        |------ build        - Where the Bacula source build tree is
        |------ tmp          - Most temp files go here
        |------ working      - Bacula working directory
        |------ weird-files  - Weird files used in two of the tests.
### Adding a New Test If you want to write a new regression test, it is best to start with one of the existing test scripts, and modify it to do the new test. When adding a new test, be extremely careful about adding anything to any of the daemons' configuration files. The reason is that it may change the prompts that are sent to the console. For example, adding a Pool means that the current scripts, which assume that Bacula automatically selects a Pool, will now be presented with a new prompt, so the test will fail. If you need to enhance the configuration files, consider making your own versions.
Porting Bacula | Index | Implementing a GUI
Bacula 1.34 User's Guide The Network Backup Solution | Copyright © 2000-2004 Kern Sibbald and John Walker