Bacula Developer Notes

Bacula 1.34 User's Guide |
Internal Component Designs | Index | Porting Notes
# Bacula Developer Notes ## General This document is intended mostly for developers and describes the the general framework of making Bacula source changes. ### Contributions Contributions from other programmers will be broken into two groups. The first are contributions that are aids and not essential to Bacula. In general, these will be scripts or will go into and examples or contributions directory. The second class of contributions are those which will be integrated with Bacula and become an essential part. Within this class of contributions, there are two hurdles to surmount. One is getting your patch accepted, and two is dealing with copyright issues. The rest of this document describes some of the requirements for such code. ### Patches Subject to the copyright hurdle described below, your patches should be sent in diff -u format, which is the easiest for me to understand. If you plan on doing significant development work over a period of time, after having your first patch reviewed and approved, you will be eligible for having CVS access so that you can submit your changes directly to the CVS repository. ### Copyrights To avoid future problems concerning changing licensing or copyrights, all code contributions more than a hand full of lines must be in the Public Domain or have the copyright assigned to Kern Sibbald and John Walker as in the current code. This double copyright ownership ensures that no single person controls the software. The author of the code should be clearly indicated, and you must be extremely careful not to violate any copyrights or use other people's code without acknowledging it. The purpose of this requirement is to avoid future problems copyright problems. To understand on possible source of future problems, please examine the difficulties Mozilla is having finding previous contributors at http://www.mozilla.org/MPL/missing.html. The other important issue is to avoid copyright, patent, or intellectual property violations as are currently (May 2003) being claimed by SCO against IBM. Although the copyright will be held by Kern and John, each developer is expected to indicate that he wrote and/or modified a particular module (or file) and any other sources. Although we have absolutely no plans for any commercial venture, it would be ashame to totally rule it out. Our desire is that if one day something commercial is developed around Bacula (as is the case with MySQL, RedHat, Ximian, and other organizations), each developer will have an opportunity to participate, at a minimum, proportional to his prior contributions. Again, we have no current plans for creating any such commercial enterprise. If you have any doubts about this, please don't hesitate to ask. Our (John and mine) track records with Autodesk are easily available; early programmers/founders/contributors and later employees had substantial shares of the company, and no one founder had a controlling part of the company. Even though Microsoft created many millionaires among early employees, the politics of Autodesk (during our time at the helm) is in stark contrast to Microsoft where the majority of the company is still tightly held among a few. Items not needing a copyright assignment are: most small changes, enhancements, or bug fixes of 5-10 lines of code, and documentation. ### Copyright Assignment Since this is not a commercial enterprise, and I prefer to believe in everyone's good faith, developers can assign the copyright by explicitly acknowledging that they do so in their first submission. ### Developing Bacula Typically the simplest way to develop Bacula is to open one xterm window pointing to the source directory you wish to update; a second xterm window at the top source directory level, and a third xterm window at the bacula directory <top>/src/bacula. After making source changes in one of the directories, in the top source directory xterm, build the source, and start the daemons by entering: make and ./startit then in the enter: ./console or ./gnome-console to start the Console program. Enter any commands for testing. For example: run kernsverify full. Note, the instructions here to use ./startit are different from using a production system where the administrator starts Bacula by entering ./bacula start. This difference allows a development version of Bacula to be run on a computer at the same time that a production system is running. The ./startit strip starts Bacula using a different set of configuration files, and thus permits avoiding conflicts with any production system. To make additional source changes, exit from the Console program, and in the top source directory, stop the daemons by entering: ./stopit then repeat the process. ### Debugging Probably the first thing to do is to turn on debug output. A good place to start is with a debug level of 20 as in ./startit -d20. The startit command starts all the daemons with the same debug level. Alternatively, you can start the appropriate daemon with the debug level you want. If you really need more info, a debug level of 60 is not bad, and for just about everything a level of 200. ### Using a Debugger If you have a serious problem such as a segmentation fault, it can usually be found quickly using a good multiple thread debugger such as gdb. For example, suppose you get a segmentation violation in bacula-dir. You might use the following to find the problem: <start the Storage and File daemons> cd dird gdb ./bacula-dir run -f -s -c ./dird.conf <it dies with a segmentation fault> where The -f option is specified on the run command to inhibit dird from going into the background. You may also want to add the -s option to the run command to disable signals which can potentially interfere with the debugging. As an alternative to using the debugger, each Bacula daemon has a built in back trace feature when a serious error is encountered. It calls the debugger on itself, produces a back trace, and emails the report to the developer. For more details on this, please see the chapter in this manual entitled What To Do When Bacula Crashes (Kaboom). ### Memory Leaks Because Bacula runs routinely and unattended on client and server machines, it may run for a long time. As a consequence, from the very beginning, Bacula uses SmartAlloc to ensure that there are no memory leaks. To make detection of memory leaks effective, all Bacula code that dynamically allocates memory MUST have a way to release it. In general when the memory is no longer needed, it should be immediately released, but in some cases, the memory will be held during the entire time that Bacula is executing. In that case, there MUST be a routine that can be called at termination time that releases the memory. In this way, we will be able to detect memory leaks. Be sure to immediately correct any and all memory leaks that are printed at the termination of the daemons. ### Special Files Kern uses files named 1, 2, ... 9 with any extension as scratch files. Thus any files with these names are subject to being rudely deleted at any time. ### When Implementing Incomplete Code Please identify all incomplete code with a comment that contains ***FIXME***, where there are three asterisks (*) before and after the word FIXME (in capitals) and no intervening spaces. This is important as it allows new programmers to easily recognize where things are partially implemented. ### Bacula Source File Structure The distribution generally comes as a tar file of the form bacula.x.y.z.tar.gz where x, y, and z are the version, release, and update numbers respectively. Once you detar this file, you will have a directory structure as follows:
    |
    |- depkgs
       |- mtx              (autochanger control program + tape drive info)
       |- sqlite           (SQLite database program)
    |- depkgs-win32
       |- pthreads         (Native win32 pthreads library -- dll)
       |- zlib             (Native win32 zlib library)
    |- bacula              (main source directory containing configuration
       |                    and installation files)
       |- autoconf         (automatic configuration files, not normally used
       |                    by users)
       |- doc              (documentation directory)
          |- home-page     (Bacula's home page source)
          |- html-manual   (html document directory)
          |- techlogs      (Technical development notes);
       |- intl             (programs used to translate)
       |- platforms        (OS specific installation files)
          |- redhat        (Red Hat installation)
          |- solaris       (Sun installation)
          |- freebsd       (FreeBSD installation)
          |- irix          (Irix installation -- not tested)
          |- unknown       (Default if system not identified)
       |- po               (translations of source strings)
       |- src              (source directory; contains global header files)
          |- cats          (SQL catalog database interface directory)
          |- console       (bacula user agent directory)
          |- dird          (Director daemon)
          |- filed         (Unix File daemon)
             |- win32      (Win32 files to make bacula-fd be a service)

          |- findlib       (Unix file find library for File daemon)
          |- gnome-console (GNOME version of console program)
          |- lib           (General Bacula library)
          |- stored        (Storage daemon)
          |- tconsole      (Tcl/tk console program -- not yet working)
          |- testprogs     (test programs -- normally only in Kern's tree)
          |- tools         (Various tool programs)
          |- win32         (Native Win32 File daemon)
             |- baculafd   (Visual Studio project file)
             |- compat     (compatibility interface library)
             |- filed      (links to src/filed)
             |- findlib    (links to src/findlib)
             |- lib        (links to src/lib)
             |- console    (beginning of native console program)
         |- wx-console     (wxWidgets console program)

    |- regress             (Regression scripts)
       |- bin              (temporary directory to hold Bacula installed binaries)
       |- build            (temporary directory to hold Bacula source)
       |- scripts          (scripts and .conf files)
       |- tests            (test scripts)
       |- tmp              (temporary directory for temp files)
### Header Files Please carefully follow the scheme defined below as it permits in general only two header file includes per C file, and thus vastly simplifies programming. With a large complex project like Bacula, it isn't always easy to ensure that the right headers are invoked in the right order (there are a few kludges to make this happen -- i.e. in a few include files because of the chicken and egg problem, certain references to typedefs had to be replaced with void ). Every file should include bacula.h. It pulls in just about everything, with very few exceptions. If you have system dependent ifdefing, please do it in baconfig.h. The version number and date are kept in version.h. Each of the subdirectories (console, cats, dird, filed, findlib, lib, stored, ...) contains a single directory dependent include file generally the name of the directory, which should be included just after the include of bacula.h. This file (for example, for the dird directory, it is dird.h) contains either definitions of things generally needed in this directory, or it includes the appropriate header files. It always includes protos.h. See below. Each subdirectory contains a header file named protos.h, which contains the prototypes for subroutines exported by files in that directory. protos.h is always included by the main directory dependent include file. ### Programming Standards For the most part, all code should be written in C unless there is a burning reason to use C++, and then only the simplest C++ constructs will be used. Note, Bacula is slowly evolving to use more and more C++. Code should have some documentation -- not a lot, but enough so that I can understand it. Look at the current code, and you will see that I document more than most, but am definitely not a fanatic. I prefer simple linear code where possible. Gotos are strongly discouraged except for handling an error to either bail out or to retry some code, and such use of gotos can vastly simplify the program. ### Indenting Standards I cannot stand code indented 8 columns at a time. This makes the code unreadable. Even 4 at a time uses a lot of space, so I have adopted indenting 3 spaces at every level. Note, indention is the visual appearance of the source on the page, while tabbing is replacing a series of 8 spaces with a tab character. Braces are required in all if statements (missing in some very old code). To avoid generating too many lines, the first brace appears on the first line (e.g. of an if), and the closing brace is on a line by itself. E.g.
       if (abc) {
          some_code;
       }
Just follow the convention in the code. Originally I indented case clauses under a switch(), but now I prefer non-indented cases.
       switch (code) {
       case 'A':
          do something
          break;
       case 'B':
          again();
          break;
       default:
          break;
       }
Avoid using // style comments except for temporary code or turning off debug code. Standard C comments are preferred (this also keeps the code closer to C). Attempt to keep all lines less than 85 characters long so that the whole line of code is readable at one time. This is not a rigid requirement. Always put a brief description at the top of any new file created describing what it does and including your name and the date it was first written. Please don't forget any Copyrights and acknowlegements if it isn't 100% your code. Also, include the Bacula copyright notice that is in src/c. In general you should have two includes at the top of the file. One is #include "bacula.h" and the second is an include for the particular directory the code is in, for example, #include "dird.h". Sometimes additional includes are needed, but this should be rare. In general (except for self-contained packages), prototypes should all be put in protos.h in each directory. Always put space around assignment and comparison operators.
       a = 1;
       if (b>= 2) {
         cleanup();
       }
but your can compress things in a for statement:
       for (i=0; i < del.num_ids; i++) {
        ...
Don't overuse the inline if (?:). A full if is preferred, except in a print statement, e.g.:
       if (ua->verbose && del.num_del != 0) {
          bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del,
             del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName);
       }
Leave a certain amount of debug code (Dmsg) in code you submit, so that future problems can be identified. This is particularly true for complicated code likely to break. However, try to keep the debug code to a minimum to avoid bloating the program and above all to keep the code readable. Please keep the same style in all new code you develop. If you include code previously written, you have the option of leaving it with the old indenting or re-indenting it. If the old code is indented with 8 spaces, then please re-indent it to Bacula standards. If you are using vim, simply set your tabstop to 8 and your shiftwidth to 3. ### Tabbing Tabbing (inserting the tab character in place of spaces) is as normal on all Unix systems -- a tab is converted into 8 spaces. My editor converts strings of spaces to tabs automatically -- this results in significant compression of the files. Thus, you can remove tabs by replacing them with spaces if you wish. Please don't confuse tabbing (use of tab characters) with indenting (visual alignment of the code). ### Don'ts Please don't use:
    strcpy()
    strcat()
    sprintf()
    snprintf()
These should be replaced by the Bacula safe equivalents:
    char *bstrncpy(char *dest, char *source, int dest_size);
    char *bstrncat(char *dest, char *source, int dest_size);
    int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...);
    int bvsnprintf(char *str, int32_t size, const char  *format, va_list ap);
See src/lib/bsys.c for more details on these routines. Don't use the %lld or the %q printf format editing types to edit 64 bit integers -- they are not portable. Instead, use %s with edit_uint64(). For example:
       char buf[100];
       uint64_t num = something;
       char ed1[50];

       bsnprintf(buf, sizeof(buf), "Num=%s\n", edit_uint64(num, ed1));
The edit buffer ed1 must be at least 27 bytes long to avoid overflow. See src/lib/edit.c for more details. If you look at the code, don't start screaming that I use lld. I actually use subtle trick taught to me by John Walker. The lld that appears in the editing routine is actually #define to a what is needed on your OS (usually "lld" or "q") and is defined in autoconf/configure.in for each OS. C string concatenation causes the appropriate string to be concatenated to the "%". ### Message Classes Currently, there are four classes of messages: Debug, Error, Job, and Memory. ### Debug Messages Debug messages are designed to be turned on at a specified debug level and are always sent to STDOUT. There are designed to only be used in the development debug process. They are coded as: DmsgN(level, message, arg1, ...) where the N is a number indicating how many arguments are to be substituted into the message (i.e. it is a count of the number arguments you have in your message -- generally the number of percent signs (%)). level is the debug level at which you wish the message to be printed. message is the debug message to be printed, and arg1, ... are the arguments to be substituted. Since not all compilers support #defines with varargs, you must explicitly specify how many arguments you have. When the debug message is printed, it will automatically be prefixed by the name of the daemon which is running, the filename where the Dmsg is, and the line number within the file. Some actual examples are: Dmsg2(20, "MD5len=%d MD5=%s\n", strlen(buf), buf); Dmsg1(9, "Created client %s record\n", client->hdr.name); ### Error Messages Error messages are messages that are related to the daemon as a whole rather than a particular job. For example, an out of memory condition my generate an error message. They are coded as: EmsgN(error-code, level, message, arg1, ...) As with debug messages, you must explicitly code the of arguments to be substituted in the message. error-code indicates the severity or class of error, and it may be one of the following:
M_ABORT | Causes the daemon to immediately abort. This should be used only in extreme cases. It attempts to produce a traceback.
M_ERROR_TERM | Causes the daemon to immediately terminate. This should be used only in extreme cases. It does not produce a traceback.
M_FATAL | Causes the daemon to terminate the current job, but the daemon keeps running
M_ERROR | Reports the error. The daemon and the job continue running
M_WARNING | Reports an warning message. The daemon and the job continue running
M_INFO | Reports an informational message.
There are other error message classes, but they are in a state of being redesigned or deprecated, so please do not use them. Some actual examples are: Emsg1(M_ABORT, 0, "Cannot create message thread: %s\n", strerror(status)); Emsg3(M_WARNING, 0, "Connect to File daemon %s at %s:%d failed. Retrying ...\n", client->hdr.name, client->address, client->port); Emsg3(M_FATAL, 0, "bdird<filed: bad response from Filed to %s command: %d %s\n", cmd, n, strerror(errno)); ### Job Messages Job messages are messages that pertain to a particular job such as a file that could not be saved, or the number of files and bytes that were saved. ### Memory Messages Memory messages are messages that are edited into a memory buffer. Generally they are used in low level routines such as the low level device file dev.c in the Storage daemon or in the low level Catalog routines. These routines do not generally have access to the Job Control Record and so they return error messages reformatted in a memory buffer. Mmsg() is the way to do this.
Internal Component Designs | Index | Porting Notes
Bacula 1.34 User's Guide The Network Backup Solution | Copyright © 2000-2004 Kern Sibbald and John Walker