Director Configuration

Bacula 1.34 User's Guide |
Bacula Configuration | Index | Client/File daemon Configuration
# The Old FileSet Resource Note, this form of the FileSet resource still works but has been replaced by a new more flexible form in Bacula version 1.34.3. As a consequence, you are encouraged to convert to the new form as this one is deprecated and will be removed in a future version. The FileSet resource defines what files are to be included in a backup job. At least one FileSet resource is required. It consists of a list of files or directories to be included, a list of files or directories to be excluded and the various backup options such as compression, encryption, and signatures that are to be applied to each file. Any change to the list of the included files will cause Bacula to automatically create a new FileSet (defined by the name and an MD5 checksum of the Include contents). Each time a new FileSet is created, Bacula will ensure that the first backup is always a Full save. FileSet Start of the FileSet records. At least one FileSet resource must be defined. Name = <name> The name of the FileSet resource. This record is required. Include = <processing-options> { <file-list> } The Include resource specifies the list of files and/or directories to be included in the backup job. There can be any number of Include file-list specifications within the FileSet, each having its own set of processing-options. Normally, the file-list consists of one file or directory name per line. Directory names should be specified without a trailing slash. Wild-card (or glob matching) does not work when used in an Include list. It does work in an Exclude list though. Just the same, any asterisk (*), question mark (?), or left-bracket ([) must be preceded by a slash (\\) if you want it to represent the literal character. You should always specify a full path for every directory and file that you list in the FileSet. In addition, on Windows machines, you should always prefix the directory or filename with the drive specification (e.g. c:/xxx) using Unix directory name separators (forward slash). However, within an Exclude where for some reason the exclude will not work with a prefixed drive letter. If you want to specify a drive letter in exclusions on Win32 systems, you can do so by specifying:
      Exclude = { /cygdrive/d/archive/Mulberry }
where in this case, the /cygdrive/d &nsbp; is Cygwin's way of referring to drives on Win32 (thanks to Mathieu Arnold for this tip). Bacula's default for processing directories is to recursively descend in the directory saving all files and subdirectories. Bacula will not by default cross file systems (or mount points in Unix parlance). This means that if you specify the root partition (e.g. /), Bacula will save only the root partition and not any of the other mounted file systems. Similarly on Windows systems, you must explicitly specify each of the drives you want saved (e.g. c:/ and d:/ ...). In addition, at least for Windows systems, you will most likely want to enclose each specification within double quotes. The df command on Unix systems will show you which mount points you must specify to save everything. See below for an example. Take special care not to include a directory twice or Bacula will backup the same files two times wasting a lot of space on your archive device. Including a directory twice is very easy to do. For example:
      Include = { / /usr }
- on a Unix system where /usr is a subdirectory (rather than a mounted filesystem) will cause /usr to be backed up twice. In this case, on Bacula versions prior to 1.32f-5-09Mar04 due to a bug, you will not be able to restore hard linked files that were backed up twice. The <processing-options> is optional. If specified, it is a list of keyword=value options to be applied to the file-list. Multiple options may be specified by separating them with spaces. These options are used to modify the default processing behavior of the files included. Since there can be multiple Include sets, this permits effectively specifying the desired options (compression, encryption, ...) on a file by file basis. The options may be one of the following: compression=GZIP All files saved will be software compressed using the GNU ZIP compression format. The compression is done on a file by file basis by the File daemon. If there is a problem reading the tape in a single record of a file, it will at most affect that file and none of the other files on the tape. Normally this option is not needed if you have a modern tape drive as the drive will do its own compression. However, compression is very important if you are writing your Volumes to a file, and it can also be helpful if you have a fast computer but a slow network. Specifying GZIP uses the default compression level six (i.e. GZIP is identical to GZIP6). If you want a different compression level (1 through 9), you can specify it by appending the level number with no intervening spaces to GZIP. Thus compression=GZIP1 would give minimum compression but the fastest algorithm, and compression=GZIP9 would give the highest level of compression, but requires more computation. According to the GZIP documentation, compression levels greater than 6 generally give very little extra compression but are rather CPU intensive. signature=MD5 An MD5 signature will be computed for all files saved. Adding this option generates about 5% extra overhead for each file saved. In addition to the additional CPU time, the MD5 signature adds 16 more bytes per file to your catalog. We strongly recommend that this option be specified as a default for all files. signature=SHA1 An SHA1 signature will be computed for all The SHA1 algorithm is purported to be some what slower than the MD5 algorithm, but at the same time is significantly better from a cryptographic point of view (i.e. much fewer collisions, much lower probability of being hacked.) It adds four more bytes than the MD5 signature. We strongly recommend that either this option or MD5 be specified as a default for all files. Note, only one of the two options MD5 or SHA1 can be computed for any file. *encryption=<algorithm> All files saved will be encrypted using one of the following algorithms (NOT YET IMPLEMENTED): *AES verify=<options> The options letters specified are used when running a Verify Level=Catalog job, and may be any combination of the following: i compare the inodes p compare the permission bits n compare the number of links u compare the user id g compare the group id s compare the size a compare the access time m compare the modification time (st_mtime) c compare the change time (st_ctime) s report file size decreases 5 compare the MD5 signature 1 compare the SHA1 signature A useful set of general options on the Level=Catalog verify is pins5 i.e. compare permission bits, inodes, number of links, size, and MD5 changes. onefs=yes/no If set to yes (the default), Bacula will remain on a single file system. That is it will not backup file systems that are mounted on a subdirectory. In this case, you must explicitly list each file system you want saved. If you set this option to no, Bacula will backup all mounted file systems (i.e. traverse mount points) that are found within the FileSet. Thus if you have NFS or Samba file systems mounted on a directory included in your FileSet, they will also be backed up. Normally, it is preferable to set onefs=yes and to explicitly name each file system you want backed up. See the example below for more details. portable=yes/no If set to yes (default is no), the Bacula File daemon will backup Win32 files in a portable format. By default, this option is set to no, which means that on Win32 systems, the data will be backed up using Windows API calls and on WinNT/2K/XP, the security and ownership data will be properly backed up (and restored), but the data format is not portable to other systems -- e.g. Unix, Win95/98/Me. On Unix systems, this option is ignored, and unless you have a specific need to have portable backups, we recommend accept the default (no) so that the maximum information concerning your files is backed up. recurse=yes/no If set to yes (the default), Bacula will recurse (or descend) into all subdirectories found unless the directory is explicitly excluded using an exclude definition. If you set recurse=no, Bacula will save the subdirectory entries, but not descend into the subdirectories, and thus will not save the contents of the subdirectories. Normally, you will want the default (yes). sparse=yes/no Enable special code that checks for sparse files such as created by ndbm. The default is no, so no checks are made for sparse files. You may specify sparse=yes even on files that are not sparse file. No harm will be done, but there will be a small additional overhead to check for buffers of all zero, and a small additional amount of space on the output archive will be used to save the seek address of each non-zero record read. Restrictions: Bacula reads files in 32K buffers. If the whole buffer is zero, it will be treated as a sparse block and not written to tape. However, if any part of the buffer is non-zero, the whole buffer will be written to tape, possibly including some disk sectors (generally 4098 bytes) that are all zero. As a consequence, Bacula's detection of sparse blocks is in 32K increments rather than the system block size. If anyone considers this to be a real problem, please send in a request for change with the reason. The sparse code was first implemented in version 1.27. If you are not familiar with sparse files, an example is say a file where you wrote 512 bytes at address zero, then 512 bytes at address 1 million. The operating system will allocate only two blocks, and the empty space or hole will have nothing allocated. However, when you read the sparse file and read the addresses where nothing was written, the OS will return all zeros as if the space were allocated, and if you backup such a file, a lot of space will be used to write zeros to the volume. Worse yet, when you restore the file, all the previously empty space will now be allocated using much more disk space. By turning on the sparse option, Bacula will specifically look for empty space in the file, and any empty space will not be written to the Volume, nor will it be restored. The price to pay for this is that Bacula must search each block it reads before writing it. On a slow system, this may be important. If you suspect you have sparse files, you should benchmark the difference or set sparse for only those files that are really sparse. readfifo=yes/no If enabled, tells the Client to read the data on a backup and write the data on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet. In this case, you must have a program already running that writes into the FIFO for a backup or reads from the FIFO on a restore. This can be accomplished with the RunBeforeJob record. If this is not the case, Bacula will hang indefinitely on reading/writing the FIFO. When this is not enabled (default), the Client simply saves the directory entry for the FIFO. mtimeonly=yes/no If enabled, tells the Client that the selection of files during Incremental and Differential backups should based only on the st_mtime value in the stat() packet. The default is no which means that the selection of files to be backed up will be based on both the st_mtime and the st_ctime values. In general, it is not recommended to use this option. keepatime=yes/no The default is no. When enabled, Bacula will reset the st_atime (access time) field of files that it backs up to their value prior to the backup. This option is not generally recommended as there are very few programs that use st_atime, and the backup overhead is increased because of the additional system call necessary to reset the times. (I'm not sure this works on Win32). <file-list> is a space separated list of filenames and/or directory names. To include names containing spaces, enclose the name between double-quotes. The list may span multiple lines, in fact, normally it is good practice to specify each filename on a separate line. There are a number of special cases when specifying files or directories in a file-list. They are: Any file-list item preceded by an at-sign (@) is assumed to be a filename containing a list of files, which is read when the configuration file is parsed during Director startup. Note, that the file is read on the Director's machine and not on the Client. Any file-list item beginning with a vertical bar (|) is assumed to be a program. This program will be executed on the Director's machine at the time the Job starts (not when the Director reads the configuration file), and any output from that program will be assumed to be a list of files or directories, one per line, to be included. This allows you to have a job that for example includes all the local partitions even if you change the partitioning by adding a disk. In general, you will need to prefix your command or commands with a sh -c so that they are invoked by a shell. This will not be the case if you are invoking a script as in the second example below. Also, you must take care to escape wild-cards and ensure that any spaces in your command are escaped as well. If you use a single quotes (') within a double quote ("), Bacula will treat everything between the single quotes as one field so it will not be necessary to escape the spaces. In general, getting all the quotes and escapes correct is a real pain as you can see by the next example. As a consequence, it is often easier to put everything in a file, and simply us the file name within Bacula. In that case the sh -c will not be necessary providing the first line of the file is #!/bin/sh. As an example:
      Include = signature=SHA1 {
         "|sh -c 'df -l | grep \"^/dev/hd[ab]\" | grep -v \".*/tmp\" \
            | awk \"{print \\$6}\"'"
      }
  will produce a list of all the local partitions on a RedHat Linux system. Note, the above line was split, but should normally be written on one line. Quoting is a real problem because you must quote for Bacula which consists of preceding every \ and every " with a \, and you must also quote for the shell command. In the end, it is probably easier just to execute a small file with:
      Include = signature=MD5 {
         "|my_partitions"
      }
  where my_partitions has:
      #!/bin/sh
      df -l | grep "^/dev/hd[ab]" | grep -v ".*/tmp" \
            | awk "{print \$6}"
- If the vertical bar (|) is preceded by a backslash as in \|, the program will be executed on the Client's machine instead of on the Director's machine -- (this is implemented but not tested, and very likely will not work on Windows). Any file-list item preceded by a less-than sign (<) will be taken to be a file. This file will be read on the Director's machine at the time the Job starts, and the data will be assumed to be a list of directories or files, one per line, to be included. This feature allows you to modify the external file and change what will be saved without stopping and restarting Bacula as would be necessary if using the @ modifier noted above. If you precede the less-than sign (<) with a backslash as in \<, the file-list will be read on the Client machine instead of on the Director's machine (implemented but not tested). If you explicitly specify a block device such as /dev/hda1, then Bacula (starting with version 1.28) will assume that this is a raw partition to be backed up. In this case, you are strongly urged to specify a sparse=yes include option, otherwise, you will save the whole partition rather than just the actual data that the partition contains. For example:
      Include = signature=MD5 sparse=yes {
         /dev/hd6
      }
- will backup the data in device /dev/hd6. Ludovic Strappazon has pointed out that this feature can be used to backup a full Microsoft Windows disk. Simply boot into the system using a Linux Rescue disk, then load a statically linked Bacula as described in the Disaster Recovery Using Bacula chapter of this manual. Then simply save the whole disk partition. In the case of a disaster, you can then restore the desired partition. If you explicitly specify a FIFO device name (created with mkfifo), and you add the option readfifo=yes as an option, Bacula will read the FIFO and back its data up to the Volume. For example:
      Include = signature=SHA1 readfifo=yes {
         /home/abc/fifo
      }
if /home/abc/fifo is a fifo device, Bacula will open the fifo, read it, and store all data thus obtained on the Volume. Please note, you must have a process on the system that is writing into the fifo, or Bacula will hang, and after one minute of waiting, it will go on to the next file. The data read can be anything since Bacula treats it as a stream. This feature can be an excellent way to do a "hot" backup of a very large database. You can use the RunBeforeJob to create the fifo and to start a program that dynamically reads your database and writes it to the fifo. Bacula will then write it to the Volume. During the restore operation, the inverse is true, after Bacula creates the fifo if there was any data stored with it (no need to explicitly list it or add any options), that data will be written back to the fifo. As a consequence, if any such FIFOs exist in the fileset to be restored, you must ensure that there is a reader program or Bacula will block, and after one minute, Bacula will time out the write to the fifo and move on to the next file. The Exclude Files specifies the list of files and/or directories to be excluded from the backup job. The <file-list> is a comma or space separated list of filenames and/or directory names. To exclude names containing spaces, enclose the name between double-quotes. Most often each filename is on a separate line. For exclusions on Windows systems, do not include a leading drive letter such as c:. This does not work. Any filename preceded by an at-sign (@) is assumed to be a filename on the Director's machine containing a list of files. The following is an example of a valid FileSet resource definition:
    FileSet {
      Name = "Full Set"
      Include = compression=GZIP signature=SHA1 sparse=yes {
         @/etc/backup.list
      }
      Include = {
         /root/myfile
         /usr/lib/another_file
      }
      Exclude = { *.o }
    }
Note, in the above example, all the files contained in /etc/backup.list will be compressed with GZIP compression, an SHA1 signature will be computed on the file's contents (its data), and sparse file handling will apply. The two files /root/myfile and /usr/lib/another_file will also be saved but without any options. In addition, all files with the extension .o will be excluded from the file set (i.e. from the backup). Suppose you want to save everything except /tmp on your system. Doing a df command, you get the following output:
    [kern@rufus k]$ df
    Filesystem      1k-blocks      Used Available Use% Mounted on
    /dev/hda5         5044156    439232   4348692  10% /
    /dev/hda1           62193      4935     54047   9% /boot
    /dev/hda9        20161172   5524660  13612372  29% /home
    /dev/hda2           62217      6843     52161  12% /rescue
    /dev/hda8         5044156     42548   4745376   1% /tmp
    /dev/hda6         5044156   2613132   2174792  55% /usr
    none               127708         0    127708   0% /dev/shm
    //minimatou/c$   14099200   9895424   4203776  71% /mnt/mmatou
    lmatou:/          1554264    215884   1258056  15% /mnt/matou
    lmatou:/home      2478140   1589952    760072  68% /mnt/matou/home
    lmatou:/usr       1981000   1199960    678628  64% /mnt/matou/usr
    lpmatou:/          995116    484112    459596  52% /mnt/pmatou
    lpmatou:/home    19222656   2787880  15458228  16% /mnt/pmatou/home
    lpmatou:/usr      2478140   2038764    311260  87% /mnt/pmatou/usr
    deuter:/          4806936     97684   4465064   3% /mnt/deuter
    deuter:/home      4806904    280100   4282620   7% /mnt/deuter/home
    deuter:/files    44133352  27652876  14238608  67% /mnt/deuter/files
Now, if you specify only / in your Include list, Bacula will only save the Filesystem /dev/hda5. To save all file systems except /tmp with out including any of the Samba or NFS mounted systems, and explicitly excluding a /tmp, /proc, .journal, and .autofsck, which you will not want to be saved and restored, you can use the following:
    FileSet {
      Name = Everything
      Include = {
         /
         /boot
         /home
         /rescue
         /usr
      }
      Exclude = {
         /proc
         /tmp
         .journal
         .autofsck
      }
    }
Since /tmp is on its own filesystem and it was not explicitly named in the Include list, it is not really needed in the exclude list. It is better to list it in the Exclude list for clarity, and in case the disks are changed so that it is no longer in its own partition. Please be aware that allowing Bacula to traverse or change file systems can be very dangerous. For example, with the following:
    FileSet {
      Name = "Bad example"
      Include = onefs=no {
         /mnt/matou
      }
    }
you will be backing up an NFS mounted partition (/mnt/matou), and since onefs is set to no, Bacula will traverse file systems. However, if /mnt/matou has the current machine's file systems mounted, as is often the case, you will get yourself into a recursive loop and the backup will never end. The following FileSet definition will backup a raw partition:
    FileSet {
      Name = "RawPartition"
      Include = sparse=yes {
         /dev/hda2
      }
    }
Note, in backing up and restoring a raw partition, you should ensure that no other process including the system is writing to that partition. As a precaution, you are strongly urged to ensure that the raw partition is not mounted or is mounted read-only. If necessary, this can be done using the RunBeforeJob record. ### Additional Considerations for Using Excludes on Windows For exclude lists to work correctly on Windows, you must observe the following rules: Filenames are case sensitive, so you must use the correct case. To exclude a directory, you must not have a trailing slash on the directory name. If you have spaces in your filename, you must enclose the entire name in double-quote characters ("). Trying to use a backslash before the space will not work. You must not precede the excluded file or directory with a drive letter (such as c:) otherwise it will not work. Thanks to Thiago Lima for summarizing the above items for us. If you are having difficulties getting includes or excludes to work, you might want to try using the estimate job=xxx listing command documented in the Console chapter of this manual. ## Windows Considerations for FileSets If you are entering Windows file names, the directory path may be preceded by the drive and a colon (as in c:). However, the path separators must be specified in Unix convention (i.e. forward slash (/)). If you wish to include a quote in a file name, precede the quote with a backslash (\\). For example you might use the following for a Windows machine to backup the "My Documents" directory:
    FileSet {
      Name = "Windows Set"
      Include = {
         "c:/My Documents"
      }
      Exclude = { *.obj *.exe }
    }
When using exclusion on Windows, do not use a drive prefix (i.e. c:) as it will prevent the exclusion from working. However, if you need to specify a drive letter in exclusions on Win32 systems, you can do so by specifying:
      Exclude = { /cygdrive/d/archive/Mulberry }
where in this case, the /cygdrive/d is Cygwin's way of referring to drive d: (thanks to Mathieu Arnold for this tip). ## A Windows Example FileSet The following example was contributed by Phil Stracchino:
    This is my Windows 2000 fileset:

    FileSet {
      Name = "Windows 2000 Full Set"
      Include = signature=MD5 {
        c:/
      }
    # Most of these files are excluded not because we don't want
    #  them, but because Win2K won't allow them to be backed up
    #  except via proprietary Win32 API calls.
      Exclude = {
        "/Documents and Settings/*/Application Data/*/Profiles/*/*/
             Cache/*"
        "/Documents and Settings/*/Local Settings/Application Data/
             Microsoft/Windows/[Uu][Ss][Rr][Cc][Ll][Aa][Ss][Ss].*"
        "/Documents and Settings/*/[Nn][Tt][Uu][Ss][Ee][Rr].*"
        "/Documents and Settings/*/Cookies/*"
        "/Documents and Settings/*/Local Settings/History/*"
        "/Documents and Settings/*/Local Settings/
             Temporary Internet Files/*"
        "/Documents and Settings/*/Local Settings/Temp/*"
        "/WINNT/CSC"
        "/WINNT/security/logs/scepol.log"
        "/WINNT/system32/config/*"
        "/WINNT/msdownld.tmp/*"
        "/WINNT/Internet Logs/*"
        "/WINNT/$Nt*Uninstall*"
        "/WINNT/Temp/*"
        "/temp/*"
        "/tmp/*"
        "/pagefile.sys"
      }
    }
Note, the three line of the above Exclude were split to fit on the document page, they should be written on a single line in real use.
Bacula Configuration | Index | Client/File daemon Configuration
Bacula 1.34 User's Guide The Network Backup Solution | Copyright © 2000-2004 Kern Sibbald and John Walker