@(#) $Id: CHANGES,v 1.33 2003/09/04 22:57:11 vern Exp $ (LBL)

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0.8a37 Wed Sep  3 23:20:21 PDT 2003

- A new attribute, "&persistent", indicates that a global variable's
  state should persist on disk (Robin Sommer).  Currently, they
  reside in the file "state.bst".

  For example, given the following script:

	global a = 0 &persistent;

	event bro_init()
		{
		print ++a;
		}

  then every time you run it, a increases by one.

  You can dump the state file using "bro -x state.bst <scriptname>".
  (There's also a partially-implemented XML dumper which you invoke via
  "bro -X state.bst <scriptname>".)

  If you send Bro a HUP signal, then it will synchronize all persistent
  state to disk (checkpoint) at that point.  More generally, the policy
  script can do so at any time by calling the new built-in checkpoint_state().

  By including the new policy script "checkpoint", you can have Bro
  automatically checkpoint every time checkpoint_interval elapses
  (default 15 minutes).

- You can also record events to disk by loading the "capture-events"
  policy script, or calling the new capture_events() built-in (Robin Sommer).  
  The events are written to the file events.bst.  You can dump these
  using bro -x (or -X), as above.  You can also replay them using
  "bro -R <file>".  By replaying them using the same policy script as
  generated them, you should get the same output.  For example:

	bro -r tracefile myscript capture-events

	...

	bro -R events.log myscript

- An experimental module facility has been added (Umesh Shankar).

  The module facility implements namespaces. Everything is in some namespace
  or other. The default namespace is called "GLOBAL" and is searched by
  default when doing name resolution. The scoping operator is "::" as in
  C++. You can only access things in the current namespace, things in the
  GLOBAL namespace, or things that have been explicitly exported from a
  different namespace. Exported variables and functions still require
  fully-qualified names. The syntax is as follows:

  module foo;  # Sets the current namespace to "foo"
  export { 
	int i;
	int j;
  }
  int k;

  module bar;
  int i;

  foo::i = 1;
  bar::i = 2;
  print i;    # bar::i (since we're currently in module bar)
  j = 3;      # ERROR: j is exported, but the fully qualified name
              #        foo::j is required
  foo::k = 4; # ERROR: k is not exported

  The same goes for calling functions.

  One restriction currently in place is that variables not in the "GLOBAL"
  namespace can't shadow those in GLOBAL, so you can't have

    module GLOBAL;
    global i: int;

    module other_module;
    global i: int;

  It is a little confusing that the "global" declaration really only means
  that the variable i is global to the current module, not that it is truly
  global and thus visible everywhere (that would require that it be in
  GLOBAL, or if using the full name is ok, that it be exported).  Perhaps
  there will be a change to the syntax in the future to address this.

  The "module" statement cuts across @load commands, so that if you say:

	module foo;
	@load other_script;

  then other_script will be in module foo. Likewise if other_script changes
  to module bar, then the current module will be module bar even after
  other_script is done.  However, this functionality may change in the future
  if it proves problematic.

  The policy scripts in the Bro distribution have not yet been updated to
  use it, but there is a backward-compatibility feature so that existing
  scripts should work without modification. In particular, everything is
  put in GLOBAL by default.

- The hooks are now in place for communicating events between running
  Bro's.  An experimental implementation of doing so (written by Robin
  Sommer) will be integrated shortly.

- A side-effect of getting those hooks in place is that event handlers must
  now be directly specified (by naming them) rather than indirectly
  (for example, by indexing a table whose value yields an event handler).
  This may be fixed soon.

- An experimental "match" expression scans a list of predicates to find
  the first one that's both true and has the highest priority (Umesh Shankar).

  The general form is:

	match some_record on { 
		[ priority, class_name, predicate ], 
		... 
	}

  where "predicate" is evaluated in the context of the value (and type) of
  some_record.  For example, if some_record has a field "num_alerts"
  then predicate could be "$num_alerts > 5".

  "priority" is a non-negative integer (i.e., of type "count"), and,
  for now, "class_name" is a string.

  For example,

	global c: conn_id;

	c = [ $orig_h = 0.0.0.0, $orig_p = 0/tcp,
	      $resp_h = 1.1.1.1, $resp_p = 1/tcp ];

	print match c on { 
		[ 2, "emptyweb", $orig_h == 0.0.0.0 && $resp_p == 80/tcp ], 
		[ 1, "emptyhost", $orig_h == 0.0.0.0 ], 
		[ 0, "should not match", 1 == 0 ]
	};

  will print "emptyhost".

  The initial intent behind this is to eventually provide more flexible
  customization of alert processing, though it clearly has broader
  applicable.  *It is very likely that the semantics and perhaps the syntax
  of "match" will change in the near future.*

- Bro's packet filter is now computed via pcap.bro (which is automatically
  included via bro.init).  It uses two new built-ins:

	precompile_pcap_filter(id: PcapFilterID, s: string):  bool
	install_pcap_filter(id: PcapFilterID): bool

  The first is for precompiling a pcap filter so it can be installed
  or turned off dynamically. Associating an ID with the filter, you can
  then activate the filter by calling the second function (installing a
  new filter replaces the current one).  (Robin Sommer)

  Be default, pcap.bro is responsible for building a pcap string based on
  the capture/restrict_filters defined by the various analyzers. It compiles
  and installs this filter, so there is no observable difference in usage
  to the old implementation, except capture/restrict_filter are now *tables*
  rather than strings, and are written as plural rather than singular.
  So the analyzers need to define something like this:

	[finger.bro]
	redef capture_filters += { ["finger"] = "port finger" };              

  This then allows "finger" to be used as the name for the corresponding
  filter element (see the next item).

- load-level.bro is an experimental policy script for allowing Bro to
  shed or add on load (in terms of which protocols it analyzes).  It
  provides three interface functions for setting the current loadlevel:

	# level is LoadLevel1, ..., LoadLevel10
	function set_load_level(level: PcapFilterID): bool

	function increase_load_level()
	function decrease_load_level()

  load-levels.bro defines ten different load levels (from 1 to 10, with 10
  being the one with the most load imposed on the system), which are
  configured by defining capture_load_level/restrict_load_levels: Example:

	redef capture_load_levels += {
		["dns"]          = LoadLevel1,
		["smtp"]         = LoadLevel2,
		["http-reply"]   = LoadLevel3,
		["http-request"] = LoadLevel8,
	};

  This means for example: "include the capture_filter associated with
  'http-reply' if the current load level is 3 or below".  There's a similar
  mechanism for restrict_filters:

	redef restrict_filters += {
		["cs-only"] = "net 131.159.0.0/16",
	};

	redef restrict_load_levels += {
		["cs-only"] = LoadLevel7,
	};

  This applies the given restrict_filter if the current load level is 7
  or *above*.

  The pcap filters belonging to the ten load levels are built and pre-compiled
  on startup. The three functions shown above just call install_pcap_filter()
  then.  (Robin Sommer)

- drop-adapt.bro tries to adjust the load level based on the current
  packet drop rate (Robin Sommer).

- synflood.bro is an experimental policy script for detecting SYN floods.
  It is not yet documented, other than brief comments in the script.
  (Robin Sommer)

- Subnet lookups now use Patricia trees instead of brute-force, which should
  make them significantly more efficient for large lists, e.g., local nets.
  (Robin Sommer)

- Due to the addition of modules, which use a C++-style "::" scope
  designator, you now need to use "0x" to introduce IPv6 address constants
  that use "::" to specify a series of empty octets.  For example, you
  used to be able to specify "deadbeef::cafe", but now this needs to be
  "0xdeadbeef::cafe".  Note that "1::2" can still be written without needing
  a "0x" prefix; it's just hex constants that start with letters that need
  the prefix.

- A new built-in, escape_string(), takes a string and returns a copy of
  it that uses escape sequences for any problematic characters (Robin Sommer).

- A number of low-level bug fixes and portability tweaks (Robin Sommer,
  Ruoming Pang, Christian Kreibich, Chema Gonzalez).

- A new timer, status_update_timer, fires for each ongoing connection
  every connection_status_update_interval seconds (default = 0 seconds,
  which means "don't fire).  (Robin Sommer)

- An additional Bro-level packet filter can filter/sample packets
  based on their src/dest ip/subnet (using a Patricia tree for
  efficiency; Robin Sommer). install_src_addr_filter(ip, flags, p) drops
  packets originating from ip with probability p (0..100) if none of the
  given TCP flags is set. install_src_net_filter, install_dst_addr_filter
  and install_dst_net_filter" work similarly.  The corresponding "uninstall_*"
  functions remove the filters again.

- The @if/@else/@endif mechanisms have been made more powerful (Robin Sommer).

- New configure option --enable-debug to compile without optimization
  (Robin Sommer).

- Small tweaks to the mpatrol support (Robin Sommer).

- SMTP is now one of the services which can use a small inactivity timeout
  in inactivity.bro (Robin Sommer).

- Alerts for signatures which are triggered by worms may be suppressed if
  we already know that the triggering host is indeed infected by a worm
  (Robin Sommer).

- Matches of a signature can now be counted per destination host
  (Robin Sommer).

- snort2bro now ignores sid-526 ("BAD TRAFFIC data in TCP SYN packet";
  Robin Sommer). Due to different semantics of Bro (matching stream-wise) 
  and Snort (matching packet-wise) this signature generates a lot of
  false positives.
  

0.8a34 Sun Jul 13 09:11:32 PDT 2003

- The new "subnet" type corresponds to a CIDR prefix (Robin Sommer).
  You can use the '/' operator to convert an address to a subnet.  The
  "in" operator tests whether an address matches a subnet (e.g., "1.2.3.4
  in 1.2.255.255/16" yields T).

  You can index tables with index type "subnet" using addresses to retrieve
  the table entry with the longest-matching prefix for that address, and
  a number of tables/sets in the default scripts have been converted to
  this form.  So for example the local_16_nets and local_24_nets variables
  have been replaced by local_nets, which has type "set[subnet]",
  is_local_addr() now refers to it, and skip_scan_nets_{16,24} have
  likewise been consolidated into skip_scan_nets.

  One present deficiency is that subnets can't be used in tables/sets
  with multiple indices.  Fixing this is going to take some time ...
  Another deficiency is that the longest-match lookup is not very
  efficient.

  Caution: this feature has been only briefly tested with IPv6.

- ALERT now generates an event "alert_action" which includes the alert_info
  and AlertAction associated with an alert.  You can define your own handler
  to further customize alert processing.

- The "snort2bro" conversion utility has been extended to recognize some
  of the new Snort options (depth, distance, within; Robin Sommer).  For
  example:

      alert .... ( ... content:"|00 01 86 B8|"; 
		       content:"|00 00 00 02|"; distance:4; within:4; 
		       content:"%x %x"; distance:16; within:256; 
		       ... )
		       
  is converted to:
  
      signature sid-1891 {
	  ...
	  payload /.*\x00\x01\x86\xB8.{4}\x00\x00\x00\x02.{16}.{0,251}%x %x/
	  }

  Note that not all of the new Snort options are supported yet.

- You can refer to script variables within signatures (Robin Sommer).
  For example, the following is now possible:

	signature sid-547 {
	  ip-proto == tcp
	  src-ip != local_nets
	  dst-ip == local_nets
	  dst-port == 21
	  event "FTP \"MKD  \" possible warez site"
	  tcp-state established
	  payload /[mM][kK][dD]  /
	  }

  This makes the signatures independent of the environment ("local_nets"
  in the above examle).  snort2bro now converts some well-known Snort
  variables into corresponding Bro variables, a number of which are listed
  in policy/site.bro.

- The default action for signature matches is now logging insted of only
  writing it to a file (Robin Sommer).

- You can now use the '^' and '$' regular expression operators inside
  subpatterns.  For example, /foo$/ | /^bar/ works now.

- You can now use "msec" for milliseconds (in addition to "usec" for
  microseconds, "sec" for seconds, etc).

- The log_file_name and open_log_file functions are now redef'able
  if you want to change their behavior.

- Bro now exits more cleanly upon exhaustion of memory (Robin Sommer).

- A bug was fixed for the case of getrlimit() returning RLIM_INFINITY for
  the maximum number of open files.

- Numerous additions of std:: scoping to address porting issues (Robin Sommer).

- gcc 3.X and Solaris portability fixes.

- A new event RemoteWorm is the complement of LocalWorm (Robin Sommer).

- A bug in which the FTP analyzer would complain about failing to be able
  to look up a connection has been fixed (Robin Sommer).

- You can now configure Bro using "--with-mpatrol" to activate MPatrol
  debugging hooks.  When built in this mode, -m dumps the leak table
  and -M shows unfreed parts of the heap.  These dumps can also be
  triggered using SIGUSR1 / SIGUSR2, respectively.  (Robin Sommmer)
  
- A script function get_contents_type() which returns the type of reassembling
  being done for a connection (none/orig-side/resp-side/both; Robin Sommer).

- A minor bugfix for the regular expression matcher (Robin Sommer).


0.8a32 Thu Jun 12 23:33:21 PDT 2003

- The low-level hash functions have been reimplemented to use HMAC-MD5 to
  counter the "algorithmic complexity attacks" discussed in the USENIX
  Security 2003 paper by Scott Crosby & Dan Wallach
  (http://www.cs.rice.edu/~scrosby/hash/) (Ruoming Pang and Vern Paxson)

- Ruoming Pang has made extensive changes to the anonymization/transformation
  framework.  A paper describing it will appear in SIGCOMM 2003, and will
  be included with the Bro distribution once the final copy is ready.

- Internal hash tables now resize incrementally, and more quickly.
  This makes a big difference in avoiding lengthy processing pauses when
  processing large volumes of traffic. (Craig Leres)

- gcc 3.1 and Linux portability tweaks.

- The calling sequence of http_request() has changed.  It now includes both
  the original URI and the escaped URI.  This was made explicit because
  we found that if only the original was passed in, it was too easy to forget
  to expand the escapes in it; and if only the escaped was passed in,
  some attacks could be missed. (Ruoming Pang)

- Signature rules can now refer to "finger" payload as well as HTTP and FTP
  (Robin Sommer).

- The signature engine now includes keywords "dst-ip", "dst-port",
  "ip-proto", "src-ip", "src-port".  (Robin Sommer)

- Packet sorting now defaults to off.

- The FTP analysis now attempts to track the current directory (Ruoming Pang).

- A number of scan detection additions have been added (Scott Campbell):

	activate_landmine_check = F
		if T, then access to more than landmine_thresh_trigger
		addresses in landmine_address (a set of addresses)
		constitutes a scan

	activate_priv_port_check = T
		if T, then inbound access to more than priv_scan_trigger
		privileged ports (i.e., ports < 1024) is considered a port
		scan.  You can exclude particular services from the count
		via troll_skip_service, which defaults to { smtp, ftp,
		ssh, 20/tcp, http }.

- The SMTP analysis now includes a new alert, HotEmailRecipient, which
  is triggered by mail sent to any of the addresses specified in
  the pattern hot_recipients (which defaults to a bare /@/).

- The new built-in cat_string_array_n() is like cat_string_array() except
  it takes starting and ending indices as additional arguments (Ruoming Pang).

- The new built-in sort_string_array() takes a string array and returns
  a sorted version (Ruoming Pang).  It currently uses an N^2 algorithm
  so shouldn't be used on large arrays.

- The new built-in subst() substitutes all instances of a given pattern
  in a given string with another string (Ruoming Pang).  For example,
  subst("fooboo", /oo/, "xxx") returns "fxxxbxxx".

- The new built-in cut_tail() trims characters from the end of a string
  (Ruoming Pang).  For example, cut_tail("fooboo", 2) returns "foob".

- sub_bytes() can now take a negative argument, which is with respect to
  the end of the string rather than the beginning (Ruoming Pang).

- The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string
  (Ruoming Pang).  The HMAC secret key is generated from available entropy
  when Bro starts up, or it can be specified for repeatability using
  the new -K flag.
 
- The new built-in split_all() is like split() except that the returned
  array also includes the parts of the string that match the pattern
  used for the splitting (Ruoming Pang).

- The new built-in split_n() splits up to a given number of instances,
  optionally returning the parts matching the split pattern (Ruoming Pang).

- The new built-in split_complete() is the most general of the split functions
  (the others are written in terms of it).  It splits a given string
  with separators that appear in either a given pattern or a given set
  of strings, optionally including the text matching the separators in
  its return value, and optionally limiting the number of matches to
  a specified maximum.  (Ruoming Pang)
 
- The new built-in to_string_literal() returns an escaped version of a string
  suitable to feeding into Bro's parser.  For example, calling it on
  "foo\nbar" (where '\n' is an embedded newline) returns "foo\x0abar".
  (Ruoming Pang)

- subst_substring() has been renamed subst_string (Ruoming Pang).

- unescape_URI() no longer takes the connection as an argument (Ruoming Pang).

- config.guess and config.sub updated

- String escape expansion has been refined (Ruoming Pang) and some bugs
  fixed.  It now supports a format that's consistent with Bro string literals.

- Scanning of octal and hex escape sequences now stops after 3 or 2
  characters, respectively.  For example, "\0007" now expands to a
  NUL followed by the character '7'.  (Ruoming Pang)

- Bug fixes for handling of bare <CR> and <LF>'s when <CRLF> expected
  (Ruoming Pang), and associated "weird" events.

- A bug in signature matching reassembly has been fixed (Robin Sommer).

- A bug in reporting "bad connection size" for connection sizes > 2GB
  has been fixed.

- A bug in computing sizes for large partial connections has been fixed.

- A bug in delayed generation of connection closed events has been fixed.

- A framework has been added for compression of some "weird" events
  to only be generated once-per-connection (Ruoming Pang).

- Some of the "weird"'s generated by the HTTP analyzer have been regularized.

- Some memory management fixes.

- A performance problem in allocating List objects was fixed.

- The copyright dates have been updated (Craig Leres).


0.8a22 Wed Jan 15 16:47:18 PST 2003

- There is now a "conditional compilation" feature similar to C's preprocessor
  (contributed by Robin Sommer).  "@if (expr)" will include the remaining
  input lines up to "@endif" only if the given expression evaluates to true.
  "@ifdef (id)" will do so only if the given identifier has been defined,
  and "@ifndef (id)" only if it has not been defined.  There's currently
  no "@else" directive, and conditionals cannot be nested.  Both of these
  will be added in the future.

- New built-in functions (contributed by Ruoming Pang): parse_dotted_addr
  takes a string in A1.A2.A3.A4 form an returns a corresponding "addr" value.
  unescape_URI takes a URI that possibly has embedded escape sequences
  ("%61" for 'a') and expands the sequences.

- The URIs in HTTP requests are no longer automaticaly escaped.  Instead,
  they need to be manually escaped using unescape_URI.  As this is likely
  error-prone (users forgetting to do so), the plan is to change the
  calling sequence of http_request in the near future so that *both* the
  escaped and the unescaped URI are passed in.

- A number of g++ 3.0 porting tweaks have been added (thanks to Sean Irvine).

- The term "rule" has been systematically changed to "signature" (Robin Sommer).

- The functionality of monitoring packet drops has been moved into its
  own policy script, netstats.bro (Robin Sommer).

- A number of rewriter and location bugs have been fixed.

- Some missing HTTP "weird"'s have now been included in weird.bro.
 

0.8a21 Thu Nov 28 23:31:38 PST 2002

- A new mechanism will reorder packets within a window of packet_sort_window
  if due to their sequence numbers they appear to be out of order.  This
  can help a great deal when reading from multiple NICs.  (Contributed by
  Ruoming Pang.)

- A bug in regular-expression matching in which for example
  "2a1" == /[0-9]+/ would evaluate as true was fixed (Ruoming Pang).

- There's now a rewriter/anonymizer for FTP (Ruoming Pang).

- The rewriter/transformation framework now supports a notion of
  delaying transformation decisions until later (Ruoming Pang).

- An incompatibility with bison 1.75 has been identified.  The problem
  is that bison 1.75 generates incomplete location information for empty
  grammar productions (those for which the RHS is empty).  This will
  be fixed soon.

- Some bugs in the signature engine have been fixed (Robin Sommer).

- The sources no longer use the not-fully-portable hash_map template
  (Umesh Shankar).

- Some bugs with the debugger getting confused about source line number
  locations, and also with the -t option, have been fixed (Umesh Shankar).

- If a content gap occurs wholly inside an HTTP entity, then the analyzer
  will skip over the entity and continue processing any subseqeuent HTTP
  items, rather than giving up on processing the connection (Ruoming Pang).

- The following new built-in functions have been contributed by
  Ruoming Pang:

	function cat_string_array(a: string_array): string
	function split_all(str: string, re: pattern): string_array
	function strstr(big: string, little: string): count
	function subst_substring(s: string, from: string, to: string): string
	function int_to_count(n: int): count
	function fmt_ftp_port(a: addr, p: port): string


0.8a20 Sun Nov 17 20:09:31 PST 2002

- This is the first "public" Bro release (one accessible directly via
  the Web).  The README has been correspondingly updated.

- The user manual has been significantly expanded, and source for it is
  now included in the release.

- Some "active mapping" bug fixes (contributed by Umesh Shankar).

- The configuration script now checks for the presence of the necessary
  BIND libraries/headers, and also for bogus --enable options (contributed
  by Craig Leres).

- backdoor.bro now includes a tcpdump filter for detecting (some) KaZaA
  traffic.

- http-reply.bro now tracks the sizes of request/response entities.

- http-request.bro now treats an URL accessing variants of /etc/netconfig
  as sensitive.


0.8a18 Sun Oct 27 15:28:23 PST 2002

- Improvements to the performance of the SMTP analyzer (Ruoming Pang).
  A new function, skip_smtp_data, skips over the content of an SMTP
  message.

- If you're doing trace rewriting and specify -w but don't specify -A,
  then the rewritten traffic goes to the -w save file, along with the
  usual non-transformed traffic (Ruoming Pang).  If a connection is
  being transformed (which is specified by the new rewriting_smtp_trace
  and rewriting_http_trace globals), then only its transformed packets
  are written to the -w file; not the original packets.  This allows
  a potentially large degree of trace compression.  There's also a
  new variable omit_rewrite_place_holder, which directs that when rewriting
  a trace, packets that are placeholders for now-missing packets (because
  you've excised content) are *not* generated.  This diminishes the
  timing fidelity of the transformed trace, but also can save a large
  amount of space.

- SMTP relay analysis is now standalone from regular SMTP analysis
  (Ruoming Pang).

- Some memory management and error propagation fixes (Ruoming Pang and
  Vern Paxson).


0.8a16 Wed Oct 23 23:48:40 PDT 2002

- "--enable-brov6" was broken by some recent changes.  It now works again.

- Some "make distclean" tweaks.

- Error checking for "redef enum".


0.8a15 Tue Oct 22 00:02:51 PDT 2002

- Fixed Makefile bug which left out wide-spread dependencies on
  event.bif.netvar_h.


0.8a14 Mon Oct 21 01:16:46 PDT 2002

- The "add" statement has been implemented.  This allows you to add
  elements to sets.  A bunch of policy scripts that used to use
  "table [] of bool"'s for this purpose have been modified to instead
  use sets.

- You can now extend the elements of an enum using "redef enum foo += { ... }"
  where the additional names are listed in the braces (contributed by
  Umesh Shankar).  A number of policy scripts have been tweaked to use
  this for the (fairly) new Alert enum.  This allows you to create new
  Alert's without alert.bro having to know about them.

- Some bugs in identifying error locations have been fixed.

- -A now supports anonymizing IP addresses (contributed by Ruoming Pang).
  This includes four new functions: preserve_prefix, preserve_subnet, and
  preserve_net (which direct that the corresponding prefix/net/subnet
  not be anonymized) and anonymize_addr (which returns the anonymized
  version of the given address.

- Some bugs in HTTP -A have been fixed (thanks to Ruoming Pang).

- The beginnings of support for CIDR prefixes has been added by
  Ruoming Pang.

- Porting tweaks (use of map's rather than hash_map's, courtesy of
  Umesh Shankar; libedit comments fixed).

- http-detail.bro has gone away.

- Some more copyright and $Header -> $Id fixes.

- There is now a function string_cat() which concatenates two strings
  and returns the result (contributed by Ruoming Pang).  This function
  is useful in place of the regular cat() because the latter will
  expand escape sequences etc.  It probably shouldn't, and that may
  change in the future.


0.8a11 Sun Oct 13 10:53:07 PDT 2002

- The framework for defining built-in functions has been extended
  so it's also now the way to specify the interface to event handlers.
  See event.bif.  (Contributed by Ruoming Pang)

- A new policy script, http-abstract.bro, can be loaded when doing HTTP
  transformation (via http-rewriter.bro), contributed by Ruoming Pang.
  It trims HTTP text responses to the first "http_abstract_max_length" bytes.

- A new built-in, skip_http_entity_data, specifies that the entity
  associated with the current HTTP request/response should be skipped
  (Ruoming Pang).

- More changes have been made to the mechanisms for tracking source
  file locations in order to pinpoint error messages.  If you encounter
  problems, please let me know.

- If you try to configure but your resolve library lacks res_mkquery(),
  this is now a fatal error (Craig Leres).  Ideally, someone will modify
  DNS_Mgr.cc to work even if only blocking DNS is available ...

- In most (not all, yet) of the sources, the copyright notices have
  been updated, $Header has been changed to $Id, and config.h is included
  (Craig Leres).


0.8a10 Tue Oct  8 16:05:42 PDT 2002

- The way that Bro tracks the source-file location associated with
  different objects has been tweaked, with a resulting savings of
  about 10% in Bro's memory use and a smidgen of CPU time, too.

- Built-in functions now are better about identifying source-file locations
  associated with errors.

- The http.$BRO_ID log file format has changed to no longer track
  individual request streams.  (So it is now closer to how it used
  to operate in 0.7.)

- The autoconf setup has been tweaked so that you use:

	--enable-brov6              enable IPV6 processing
	--enable-activemapping      enable active mapping processing
	--enable-expire-dfa-states  enable DFA state expiration

  rather than --with-XXX as was the case in 0.8a9.


0.8a9 Mon Oct  7 10:15:12 PDT 2002

- A bunch of configuration/autoconf/portability tweaks (Craig Leres).
  These include compiling under Solaris.

  When running ../configure, you can now specify:

	--with-brov6              enable IPV6 processing
	--with-activemapping      enable active mapping processing
	--with-expire-dfa-states  enable DFA state expiration

  You no longer set these via editing the Makefile.

- Some bugs with HTTP 1.0 keep-alive connections fixed (Ruoming Pang).

- The "hf" suite of utilities is no longer distributed in aux/.  Instead,
  get it directly from ftp://ftp.ee.lbl.gov/hf.tar.gz.

- bro_logchk has been renamed bro-logchk.pl and is now distributed in aux/
  (it was supposed to be before, but was inadvertantly left out of the list
  of distribution files).


0.8a7 Fri Oct  4 22:24:30 PDT 2002

- HTTP reply analysis has been split into a number of policy script files:

	http-body.bro
	http-detail.bro
	http-entity.bro
	http-event.bro
	http-header.bro
	http-reply.bro
	http-rewriter.bro

  so you can readily trade off how detailed the HTTP processing is vs.
  the CPU/memory it consumes (contributed by Ruoming Pang).

- Bro now generates login_prompt events when the $TTYPROMPT environment
  variable is passed during a login session, in order to detect the
  recently publicized Solaris remote /bin/login exploit.

- Ruoming Pang has extended the framework for defining "rewriter" functions
  to now also serve as the way to define general built-in functions.

- bro.init has been rearranged to have a more logical structure (courtesy
  of Ruoming Pang).

- Craig Leres contributed a number of portability & autoconf tweaks.

- Craig Leres has extended nb_dns.{h,c} to support IPv6 DNS lookups.
  Bro does not yet take advantage of these extensions.

- The beginnings of portability to gcc 3.2 were added.  There unfortunately
  is more work to do here!

- The README has finally been updated to have more correspondence with
  the 0.8 release.


0.8a6 Wed Oct  2 18:58:12 PDT 2002

- Upgrade to autoconf 2.53 (Craig Leres).


0.8a5 Tue Oct  1 19:04:53 PDT 2002

- The regular expression matcher how has a mechanism to stop scanning when
  no further match is possible (Robin Sommer).  If you find problems with
  regular expression matching, especially if you're using EXPIRE_DFA_STATES,
  please let us know!

- Rule/signature files are now searched for using $BROPATH (Robin Sommer).
  In addition, you can define a list of signature files to incorporate
  using the new global "signature_files".  For example,

	redef signature_files += "web-stuff"

  will look for web-stuff and web-stuff.sig along $BROPATH.

- The tcp_deliver_undelivered variable is now tcp_match_undelivered and
  only applies to delivering otherwise-undelivered data to the signature
  matcher; such data is *not* delivered to any associated analyzer.
  (Robin Sommer)

- The framework for tracking version numbers now allows negative as
  well as positive versions (Robin Sommer).


0.8a4 Tue Oct  1 15:54:58 PDT 2002

- Support for extracting the contents of HTTP replies (and POST's),
  and for transforming/anonymizing HTTP traffic, contributed by Ruoming Pang.

- Some minor internal tweaks to the timer management to help track patterns
  of timer expiration.


0.8a3 Mon Sep 23 22:48:07 PDT 2002

- HTTP reply handling refined in policy scripts.

- New built-in functions to_int(), to_count() convert strings to
  integers/counts.

- Bug fixes for DNS_Mgr and IPv6.

- AckAboveHole alerts now ignored (just written to alert.$BRO_ID) by default.

- Packets belong to ignored connections (for example, partial connections if
  partial_connection_ok is false) are no longer recorded to the save file.

- Some minor formatting/naming tweaks.


0.8a1 Sat Sep 21 22:09:23 PDT 2002

- IPv6 support enabled if you build using -DBROv6.  Deficiencies: Bro
  doesn't yet look up hostnames for AAAA records; no handling of extension
  headers (if you have traces of these, please send them to me!); no
  handling of FTP PORT/PASV w/ IPv6 addresses (again, if you have traces,
  please send them!); DNS analyzer doesn't understand AAAA yet (again,
  please send me traces!); you have to change the capture_filter line
  in tcp.bro (as indicated in the script) in order to process TCP traffic,
  due to deficiencies in libpcap's support for IPv6.

- Bro is migrating towards a more structured way of handling log messages /
  alerts.  Analyzers now @load alert.bro, which has a function ALERT()
  for processing alerts.  Soon this function will provide a variety of
  filtering/processing hooks; expect changes.

- Bro now has an HTTP response analyzer (contributed by Ruoming Pang).
  The HTTP policy scripts have been split up into http.bro (just general
  definitions), http-request.bro (handles requests; loaded by http.bro),
  http-reply.bro (handles replies; you need to explicitly load this), and
  http-detail.bro (handles individual headers).  http-reply.bro will be
  undergoing some significant reworking in the near future; probably the
  scripts will be merged back into a single http.bro plus http-detail.bro.

- ssl-worm.bro contains a prototype policy script for detecting the
  Scalper SSL worm (contributed by Robin Sommer).  It uses the signature
  file sig.ex.ssl-worm.bro.  If someone has traces of Scalper in action
  to send us, that would be great.

- A new policy script, contents.bro, extracts the contents of each
  Bro connection into its own pair of files (one file for each
  direction).  Use in conjunction with -f or discarder_XXX() to
  extract specific connections.

- A new built-in function, strcmp(), returns the usual comparison between
  two strings (contributed by Robin Sommer).

- A new event, content_gap(), is generated when Bro detects that it is
  forced to skip over data in a reconstructed TCP stream because it is
  missing from the packet input.

- BIND8 is no longer included with the distribution.  If this causes you
  problems, let me know.

- aux/scripts/bro_logchk is a Perl script for parsing Bro HTTP & FTP logs
  (contributed by Jim Barlow).

- You can now compare addresses to see which is larger.  a < b means
  that in network order, the octets making up 'a' are ordered before
  those for 'b'.  E.g., 128.2.3.4 < 128.2.3.5 < 129.0.0.1.  Note that
  IPv4 addresses are all < IPv6 addresses (other than IPv4 addresses
  that are embedded in IPv6 addresses, e.g., ::128.2.3.4 < 128.2.3.5).

- Serious bug in TCP option handling fixed.

- Some bugs in CRLF handling fixed (courtesy Ruoming Pang).

- Bug in the implementation of &optional fixed.

- Bug in computing memory statistics when not reading packets (from
  an interface or the trace file) fixed.

- You can now include a trailing comma after the last item in an
  "enum" enumeration list.

- port-name.bro now maps 389/tcp to "ldap".

- A bug has been fixed in loading files multiple times
 

v0.7a175 Thu Aug 29 21:14:34 PDT 2002

- bro -s <file> reads in a *signature* file to search for regular expressions
  in packet payloads or TCP byte streams (written by Robin Sommer).
  See policy/rules.bro for an example of a policy script for processing
  the matches.

  Note that this feature is experimental, and will be evolving in the
  near future.

- The python script "snort2bro" reads in Snort signatures and translates
  them into Bro signature rules, suitable for processing using -s.

  An example of its operation is seen by running

	python snort2bro < sig.ex.web-rules.snort

  which, after reading in sig.ex.classification.config and
  sig.ex.reference.config, generates the output given in
  sig.ex.web-rules.bro, which is suitable to use as input to
  bro -s.

- bro -d invokes a gdb-like debugger (written by Umesh Shankar).  You can
  set breakpoints and watchpoints, examine tracebacks, print Bro expressions,
  and the like.  Type "help" for on-line help.

- bro -t <tracefile> turns on tracing of the policy script execution,
  written to the given file.

- Bro now includes an SMTP analyzer, which includes processing MIME
  message bodies (written by Ruoming Pang).  See smtp.bro and mime.bro
  for related policy scripts.  smtp.bro includes several experimental
  techniques for detecting mail relaying.

- You can now define enumerated types, such as

	type scanner_type: enum {
		SCANNER_STEALTH, SCANNER_HIGH_SPEED, SCANNER_AMBIGUOUS,
	};

  Enumerated types can be compared for equality with one another, and used
  as table indices, but cannot be converted to/from integers.

- bro -A <file> invokes an experimental, general trace transformation/
  anonymization framework (written by Ruoming Pang) which writes a modified
  tcpdump trace file from the input (which can be the network or another
  trace file) with potentially extensive modifications to the recorded
  packets.

  Transformers are built from .rw files (currently, {finger,ftp,ident,smtp}.rw),
  which are processed by the utility "rwcl" to generate both event engine
  analyzer components and rewriter policy scripts (for example, after
  configuring and building Bro, you'll find the scripts
  policy/{finger,ftp,ident,smtp}.rw.bro).

  See policy/smtp-rewriter.bro for an example of a policy script that
  performs transformation/anonymization.

- New built-ins:

	split(s: string, p: pattern): string_array;

	  takes a string and splits it into pieces at each occurrence of
	  the regular expression pattern p.  (The functionality is like
	  that in awk.)  It returns a string_array, which is a table[count]
	  of string that is indexed starting at 1, giving the different
	  pieces.

	  For example,

		split("foobar", /o/)

	  returns a 3-element table, for which [1] is the string "f",
	  [2] is the string "" (empty), and [3] is the string "bar".

	split1(s: string, p: pattern): string_array;

	  split1() does the same thing as split(), but only performs splitting
	  at the first occurrence, so it returns either a one-element table
	  (if the pattern doesn't appear in the string) or a two-element
	  table.  split1("foobar", /o/) returns a 2-element table for which
	  [1] is "f" and [2] is "obar".

	md5_hash(s: string): string

	  returns (in human-readable form) the MD5 hash of a given string.

	  So, for example,

		md5_hash("foobar")

	  yields "3858f62230ac3c915f300c664312c63f".

	to_addr(s: string): addr

	  takes a string representing an address in "dotted quad" format
	  and returns the correponding "addr" value.

	set_buf(f: file, buffered: bool)

	  sets the given file to have its writes buffered or unbuffered
	  depending on the value of "buffered".  It does not return a value.

	connection_exists: function(c: conn_id): bool

	  returns T if the given connection identifier corresponds to a
	  currently instantiated connection (one for which the event engine
	  has state), F otherwise.

	lookup_connection(c: conn_id): connection

	  returns the "connection" record associated with the given
	  connection identifier, or a fatal run-time error if there
	  isn't one.

	set_inactivity_timeout(c: conn_id, timeout: interval): interval

	  sets the inactivity timeout for the given connection to the
	  given interval, returning the old interval.

	  If the interval is non-zero, then when no packets have been
	  processed for a connection after that much time has elapsed,
	  the connection is deleted, and an "inactivity_timeout" event
	  generated.

	get_matcher_stats(): matcher_stats

	  used for gathering statistics about the signature matcher

	rewriting_trace(): bool

	  returns T if -A was specified (anonymize/rewrite a trace),
	  F otherwise.

- New events:

	connection_state_remove(c: connection);

	  Invoked when the event engine has removed the connection from
	  its state.

	connection_SYN_packet(c: connection, pkt: SYN_packet);

	  Invoked for each SYN/SYN-ACK packet.

	connection_timeout(c: connection);

	  Invoked when the event engine times out a connection - for
	  example, because the originator sent a SYN that was never
	  answered, so the connection was never established.

	connection_reused: event(c: connection);

	  Invoked when the event engine decides that a new SYN for
	  an existing connection reflects a reuse of the connection
	  four-tuple, rather than belonging to the existing connection.

- New globals:

	const ignore_checksums = F &redef;

	  If true, then the event engine does not verify checksums (and
	  hence will not discard packets with bad checksums).

	const tcp_deliver_undelivered = F &redef;

	  If true, then when the event engine closes a connection, if
	  that connection has a chunk of data not yet delivered to its
	  analyzer (which only happens if the data is above a sequence
	  hole, indicating either a packet filter glitch or a protocol
	  botch), then the undelivered data will at that point be delivered
	  to the connection's analyzer.

	const tcp_reassembler_ports_orig: set[port] = {} &redef;
	const tcp_reassembler_ports_resp: set[port] = {} &redef;

	  Sets of ports for which, if a connection has the corresponding
	  originator/responder port, then the event engine will reassemble
	  the byte stream of the connection.

	  Normally, the event engine reassembles byte streams for any
	  connection for which there's an analyzer, and otherwise doesn't.
	  These variables can be used to force reassembly for the originator
	  or responder side (respectively) of connections for which there
	  isn't otherwise an analyzer.  This is useful when doing signature
	  matching on reassembled byte streams, for protocols that are
	  not otherwise analyzed by the event engine.

	const table_expire_interval = 1 min &redef;

	  How often to check table entries to see whether they've expired
	  (see &read_expire, etc., below).

	const requires_trace_commitment = F;

	  If true, then when rewriting/anonymizing traces, nothing will
	  actually be written to the edited trace file unless you call:

		rewrite_commit_trace(c: connection, commit: bool, future: bool)

	  If "future" is true, then future rewritten packets will be
	  automatically commited; otherwise, writing them to the trace
	  file requires another explicit rewrite_commit_trace() call.

	const inactivity_timeout = 0 secs &redef;

	  As noted above, when a connection becomes inactive, time it out
	  after this interval.  If 0 secs, then don't time it out.

- An SSH analyzer extracts SSH client/server version information.  See
  ssh.bro for the related policy script.

- There's now a (very) simple TFTP analyzer available in tftp.bro.

- You can now set the global "frag_timeout" to an interval which controls
  how long fragments are kept before discarding them (contributed by Ashley
  Thomas).  If you don't set the global, or set it to 0.0 sec, then fragments
  are kept around indefinitely.

- An implementation of an experimental anti-evasion technique, "active
  mapping", has been written by Umesh Shankar.  It is not yet ready for
  general use, and isn't compiled in unless -DACTIVE_MAPPING.

- Four new attributes can now be associated with tables (implemented
  by Robin Sommer): &read_expire, &write_expire, and &create_expire
  will delete table entries after a given interval has elapsed since
  the table entry was last read, written, or created.  For example:

	global a: table[addr] of count &default=0 &create_expire = 5 sec;

  will delete each entry added to it 5 seconds after the entry was added,
  regardless of subsequent read/write activity to the element.

  &expire_func allows you to associate a function with the table such that
  whenever an entry expires, the function is invoked.  It's passed the
  value of the table entry (not the index - perhaps this should be changed),
  and returns an "interval" value.  If the interval is <= 0.0 seconds, then
  the table entry is immediately deleted.  Otherwise, it is deleted after
  the given interval has elapsed.

- When listing multiple attributes, you no longer separate them with
  commas.  For example, if you used to have:

	global a: table[string] of count &default=0, &redef;

  you now need to use:

	global a: table[string] of count &default=0 &redef;

- You can now construct records using

	[$field1 = <expression>, $field2 = <expression>, ...]

  Such record values can be assigned to other records providing that the
  target value's type includes all of the fields (same name and type)
  present in the record value, and that any missing fields have the
  &optional or &default attribute (see next item).

  You can also include a record value inside the record constructor, and
  all of its fields will be included in the constructed record value.

- Record fields can now be annotated with &optional, to indicate
  that the field needn't be present, or &default, which indicates
  a default value to provide if the field is missing.

- You can query as to whether a record has a value for a given field
  using the new "?$" operator.  So for example:

	type my_rec: record {
		num: count &default = 0;
		msg: string;	# mandatory, since no &optional/&default
	};

	global r: my_rec;

	r = [$msg = "hello"];

	print r?$num, r?$msg, r$num;

  will print "F, T, 0," because even though 'r' has a default value
  for $num (which shows up when printing r$num), that field is missing,
  hence r?$num is F.

- An experimental scheme has been added (by Umesh Shankar) for managing
  general attributes associated either with all values ("global attributes")
  or particular particular values.  This scheme is likely to change in
  the near future, and hence isn't explained here further.

- The DNS analysis now includes ADDL and AUTH records, and much richer
  policy script analysis (per policy/dns.bro).

- You can now "redef" a function or event handler to override its
  previous definition.  For a function, this looks like:

	redef log_hook = function(msg: string): bool
		{
		...
		}

  For an event handler, it's just the usual definition preceded by "redef.
  For example,

	redef event ack_above_hole(c: connection) { }

  would replace the default ack_above_hole handler with one that does nothing.

- HTTP server and HTTP proxy backdoor detectors have been added,
  generating http_signature_found and http_proxy_signature_found,
  respectively (contributed by Ruoming Pang).

- A KaZaA backdoor detector has been added, which generates
  kazaa_signature_found for likely KaZaA connections.

- The new policy scripts flag-irc.bro and flag-warez.bro provide
  hooks for defining site policies for detecting IRC and access
  to warez.

- portmapper.bro now tracks the services it sees, and the names are
  used in connection summaries rather than generic services like port-656.

- bro -C (or redef'ing the "ignore_checksums" global to T) instructs
  Bro to ignore any checksum errors and go ahead and analyze such packets.

- The (trivial) policy script print-globals.bro dumps out all of the policy
  script global variables and the amount of memory they consume.

- The policy script code-red.bro has been renamed worm.bro and generalized
  to detect Nimda as well as Code Red 1 & 2.

- A bunch of additional default sensitive URIs have been added to http.bro.
  http.bro also now doesn't report worm-related URIs.

- A bunch of less common portnames were removed from port-names.bro.

- Empty regular expressions are now allowed.

- The finger_request event now has a third parameter, the additional
  text after the username.

- More systematic handling of NULs and CRLF by the event engine.

- Hex escape sequences now must have exactly two hexadecimal characters.

- FYI - work has begun on significantly altering the way that policy
  scripts generate alerts.

- Work has begun (by Robin Sommer) on a general framework for tracking
  client/server versions.  See software.bro.

- Work has begun on a NETBIOS analyzer (see NetbiosSSN.cc).  Contributions
  (e.g., finishing it :-) welcome.

- Work has begun on migrating the internals to process IPv6 in addition
  to IPv4.

- A number of bug fixes, leaks, and memory allocation lint tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a90 Thu Sep 06 00:50:43 PDT 2001

- Better state management, especially for use in high-volume sites.  There
  are now a number of ways to track the resources consumed by Bro while it
  runs, and to reduce the amount of memory Bro consumes.  Adding

	@load print-resources

  will log a summary of the resources consumed by Bro when it exits, and

	@load reduce-memory

  will change a number of default values in order to significantly diminish
  the amount of memory Bro requires.

  Other ways to lower the required memory discussed in the next three items.

- The global "maintain_http_sessions" now controls whether http.bro tracks
  multiple HTTP connections between the same hosts as a single HTTP session.
  Doing so costs a considerable amount of state (so set to F for reducing
  the memory impact of HTTP analysis).

- The global "suppress_scan_checks", if true, turns off address and port
  scan detection.  (You can't achieve this by simply not @load'ing scan.bro,
  because it's loaded by some of the default policy scripts.)  Turning it
  off can save a lot of memory.

- Note, the ident.bro is also expensive in terms of state-holding.

- New library functions:

	- resource_usage() returns a record detailing real time,
	  CPU time, memory, other getrusage info, and the current
	  and maximum number of TCP/UDP/ICMP connections, and
	  timers and fragments.

	- val_size() returns the size in bytes needed to represent
	  a given value (which can be a record, a table, or a
	  simple constant, for example).  It's not fully accurate
	  but is in the right ballpark.

	- global_sizes() return a table mapping every global
	  variable to its size (per val_size()).  Useful for
	  tracking which ones are growing large over time.

- You can now control a number of timers related to Bro's connection
  management.  Setting them to lower values generally decreases the
  amount of state Bro has to keep (see reduce-memory.bro), though
  this can also make it easier for an attacker to evade detection:

	tcp_SYN_timeout: interval
		Controls how long Bro waits after seeing the
		beginning of a connection (whether due to a SYN
		or not; the timer is misnamed) before checking
		whether it elicited any reply.

	tcp_session_timer: interval
		After a connection has closed, wait this long for
		further activity before checking whether to time
		out its state.

	tcp_connection_linger: interval
		When checking a closed connection for further
		activity, Bro should consider it inactive if there
		hasn't been any for this long.  It also complains
		if the connection is reused before this much time
		has elapsed.

	tcp_attempt_delay: interval
		Bro waits this long upon seeing an initial SYN
		before timing out the connection attempt.

	tcp_close_delay: interval
		Upon seeing a normal connection close, Bro flushes
		state after this much time.

	tcp_reset_delay: interval
		Upon seeing a RST, Bro flushes state after this
		much time.

	tcp_partial_close_delay: interval
		Bro generates a connection_partial_close event this
		much time after one half of a partial connection
		closes, assuming there has been no subsequent
		activity.

	non_analyzed_lifetime: interval
		If a connection belongs to an application that you
		aren't analyzing, Bro times it out after this
		interval.  If 0 secs (default), then it doesn't
		time it out.

	dns_session_timeout: interval
	ntp_session_timeout: interval
	rpc_timeout: interval
		Bro waits this long before timing out a DNS/NTP/RPC
		request.

	max_timer_expires: count
		The maximum number of timers to expire after
		processing each new packet.  The value trades off
		spreading out the timer expiration load with
		possibly having to hold state longer.  A value of 0
		means "process all expired timers with each new
		packet".

- Two new flags control other facets of Bro's connection management,
  and thus state-holding:

	partial_connection_ok: bool
		Whether Bro should analyze connections for which
		it doesn't see the beginning, only the middle.
		This can be very expensive to do in the face of
		stealth-scanning, which looks like a bunch of
		partial connections.

		Note, the HTTP analyzer has been modified to
		now always skip partial connections.  This should
		instead be user controllable.

	tcp_SYN_ack_ok: bool
		If true, Bro instantiates connection state when
		it sees a SYN ack but not the initial SYN (even
		if partial_connection_ok is false).  The intent
		behind this knob (which is not well tested) is
		to allow you to filter out initial SYNs and only
		react to SYN acks.  This keeps Bro from holding
		state during SYN scans and SYN flooding, except
		for when the destination responds.

- Some other miscellaneous thresholds that you can now modify from your
  policy script:

	heartbeat_interval: count
		How often to generate net_stats_update() events.
		This timer really isn't needed any more, since
		you can use "schedule" to achieve the same effect.

	tcp_storm_thresh: count
		If Bro sees this many FINs/RSTs in a row, it
		flags them as a "storm".

	tcp_storm_interarrival_thresh: interval
		The FINs/RSTs must come with this much time or less
		between them.

- The state management for line-oriented applications like HTTP requests
  has been improved.

- The HTTP analyzer now expands %hex sequences.  If anyone has a Unicode
  expander to contribute, that'd be terrific.

- The Code Red detection is more robust (fewer false positives).

- A new redefinable variable, skip_services, lists applications that should
  not be analyzed for purposes of detecting scans.  (Default:  ident)
  The point of having this (poorly named) hook is so that code-red.bro
  can add "http" to it, to avoid reporting the zillions of Code Red scans
  that a site can see.

- Bro now matches regular expressions using lazy DFA evaluation.  The upshot
  of this is (1) it no longer maintains .bro-RE-cache.v1, (2) it starts
  up fast regardless of whether you've added new regular expressions, (3)
  you can afford to add lots of regular expressions, and (4) it's actually
  a bit faster.

- The list of "hot_ids" has been expanded with a number of other common
  root-privileged accounts.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a61 Fri Apr 06 14:51:47 PDT 2001

- An NTP analyzer has been added.  See policy/ntp.bro for a corresponding
  policy file that detects the newly discovered NTP remote buffer overflow
  attack.

- example-attacks/ is a new directory that contains trace files of attacks.
  Currently, there are just two to play with:

	bro -r example-attacks/ftp-site-exec.trace mt

  will run on a trace of a "site exec" overflow attack, and

	bro -r example-attacks/ntp-attack.trace mt ntp

  will run on an example of the NTP overflow.

- The doc/ directory includes the postscript and HTML versions of the
  first draft of the Bro manual.

- A new policy file, icmp.bro, has preliminary (and only partially developed)
  policy for analyzing ICMP.

- The file libpcap.bufsize.patch includes the patch necessary on some systems
  to increase the maximum libpcap buffer size.

- You can now use anonymous functions in &default expressions, so for
  example you can do:

	global foo: table[count] of string = {
		[1] = "1st", [2] = "2nd", [3] = "3rd",
	} &default = function(n: count): string { return fmt("%dth", n); };

  and then referring to foo[5] will yield "5th".

- There's now a "for" statement to iterate over the indices of a table
  or the members of a set:

	for ( i in foo )

  for the above "foo" will iterate with i assigned to 1, 2, and 3; *but
  not in general in that order*.

- The function contains_string() has been removed, and now you can instead
  use an expression like

	"bar" in "foobar"

  which will yield T.

- The scan detection now has a mechanism for attempting to detect SYN flooding
  backscatter and flagging it as different from a stealth scan.

- New event handlers:

	new_connection_contents()
		like new_connection(), but reassembles the
		stream so you can use set_content_files() to
		write it to a file

	udp_session_done()
		invoked when a UDP session (which is defined on
		a per-protocol basis; currently only for NTP)
		finishes.

	ntp_message()
		invoked for each NTP message

- UDP processing now does accounting for detecting scans.

- UDP processing now tracks numbers of requests/replies for sessions that
  support that notion.  The connections are annotated by udp_session_done()
  with "[m,n]" for "m" requests and "n" replies, providing either m or n > 1.

- New variable accessible from policy:

	watchdog_interval
		how often the watchdog should check for whether
		Bro is making internal progress

- A bunch of functions no longer have a first argument of the current time;
  get it instead from network_time() if you need it:

	authentication_accepted
	authentication_rejected
	conn_weird
	conn_weird_addl
	flow_weird
	net_weird

- A bunch of functions now return bool rather than int values:

	set_contents_file
	set_login_state
	set_record_packets
	skip_further_processing

- The variable "hot_dests" has been renamed to "hot_dsts".

- 111/tcp is now identified as "portmap" rather than "rpc".

- Connections flagged as hot for some types of characteristics are now
  annotated with the reason associated with the decision.  (I think a lot
  more of this is needed.)

- Portmapper dumps are annotated with the results of the mapping.  This
  will be streamlined in the future.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a48 Wed Sep 13 14:37:30 PDT 2000

- Changes between this release and v0.6 missing :-(

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.6 Wed Jul 21 17:02:50 PDT 1999

- Support for regular expressions added.  You specify lex-style regular
  expressions between '/'s, for example "/\/etc\/(passwd|shadow)/" has
  the type "pattern" and matches /etc/passwd or /etc/shadow (the slashes
  in the pattern need to be escaped or else they'd delimit the end of the
  pattern).  Pattern-matching is via the "in" operator, so for example:

	if ( filename in /\/etc\/(passwd|shadow)/ )
		sensitive_file_access(filename);

  or

	const sensitive_files = /\/etc\/(passwd|shadow)/;

	...

	if ( filename in sensitive_files )
		sensitive_file_access(filename);

  Presently the "in" operator requires that the entire left-hand side
  be matched by the pattern.  So, for example, if you want to find the
  string "eggdrop" anywhere inside the string "line", you would use

	if ( line in /.*eggdrop.*/ )

  If you leave off either of the .*'s, then eggdrop will only be matched
  at the beginning or end of the line.

  In the future, there will be mechanisms for specifying whether you
  want to match anywhere in a line, or anchored; accordingly, *the above
  syntax is subject to change*.

  Bro compiles regular expressions into DFAs for fast matching.  This can take
  quite a bit of time for complicated patterns.  Consequently, it maintains a
  cache of compiled regular expressions in $HOME/.bro-RE-cache-v1.  You can
  always safely remove this file; Bro will recreate/repopulate it as needed.
  It does not clean up unused entries from it, so if you change your patterns
  frequently, you will accumulated lots of old ones and should delete the
  file to garbage collect them.

- An rlogin analysis module has been added and the telnet analysis
  generalized to generic "login" analysis, with the following events:

	login_failure(c: connection, user: string, client_user: string,
			password: string, line: string)
		Generated on a failed attempt to log in.  client_user is
		the remote user name, if the login is via the rlogin
		protocol.
	login_success(c: connection, user: string, client_user: string,
					password: string, line: string)
		Generated on a successful attempt to log in.

	login_input_line(c: connection, line: string)
		Generated per line of input typed by the user.
	login_output_line(c: connection, line: string)
		Generated per line of output generated by the server.

	login_confused(c: connection, msg: string, line: string)
		Generated when a login dialog confuses the heuristic
		analyzer.  msg is a tag for the state mismatch that
		was unexpected, line is the corresponding dialog text.

	login_confused_text(c: connection, line: string)
		Once a connection is in the confused state, then this
		is generated for each subsequent line.

	login_terminal(c: connection, terminal: string)
		Generated if the terminal type associated with the
		connection is seen.

	login_display(c: connection, display: string)
		Generated if the display associated with the connection
		is seen.

	excessive_line(c: connection)
		Generated when the connection has produced an excessively
		long line.

  login_input_line() and login_output_line() are very powerful for
  detecting intrusions, when coupled with regular-expression matching.

  login_terminal() is used to detect backdoors that are triggered
  by the terminal environment variable.

- An ident analysis module has been added (port 113).  It generates
  ident_request, ident_reply, and ident_error events.  Port 113 used
  to be referred to as "auth"; now it's referred to as "ident".

- A new type of scan detection has been added, which is triggered
  by a remote host trying a large number of username/password
  combinations.  See the account_tried() function in scan.bro.

- The default search path for .bro files is now

	  .:priv-policy:policy:pub-policy:/usr/local/lib/bro

  where priv-policy/ is intended for private policy and pub-policy/
  for public policy.  The Bro alpha distribution ships with a
  sample set of pub-policy scripts.

- New built-ins:

	system(s: string): int
		executes the given shell command using system()
		and returns its status.

	set_contents_file(c: conn_id, direction: count, f: file)
		copies connection c's reassembled byte stream in
		either the originator-to-responder direction (if
		direction is CONTENTS_ORIG) or the responder-to-
		originator direction (CONTENTS_RESP) to the file f.

	reading_live_traffic(): bool
		returns true if Bro is running on live traffic (read
		from a network interface), false if it's reading from
		a save file.

	mkdir(f: string): bool
		creates the given directory, returning true if it
		was able to, false if not.

	get_orig_seq(c: conn_id): count;
		returns the highest sequence number sent by the
		originator of connection c.
	get_resp_seq(c: conn_id): count;
		same for c's responder.

- Additional new events (other than those related to the new analyzers):

	new_connection(c: connection)
		is generated whenever a new connection is seen.

	partial_connection(c: connection)
		is generated whenever a new partial connection (one
		that doesn't begin with a SYN handshake) is seen.

	pm_bad_port(r: connection, bad_p: count)
		is generated when a portmapper response contains
		a bad port number.

- Functions, tables and sets can now be assigned.  Assignment is
  made by reference to the underlying object.

- Bro no longer looks up identifiers using getservbyname() to see if they
  should be interpreted as port numbers, since this led to portability
  problems.  Instead, a number of constants are defined in bro.init:
  bgp, domain, finger, ftp, gopher, http, ident, rlogin, smtp, ssh and telnet,

- Bro now supports an arbitrary number of open files (not bound by
  the system's limit on file descriptors).

- There's now a finger_reply event to go with finger_request.

- A bunch more RPC service names have been added, thanks to Job de Haas
  and others.

- A bug has been fixed in the watchdog handling that caused it to
  sometimes expire after a period of network inactivity.

- The Bro paper in doc/ has been revised (it isn't quite up-to-date,
  but considerably closer than the USENIX version).

- There has been a large amount of reworking of the internals, both
  to Bro itself and in the policy scripts.  If you find something you're
  wondering about, feel free to send me mail asking about it.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.5 Sun Oct  4 00:19:35 PDT 1998

- Added Linux support.

- Major autoconf changes.

- Some tweaks to suppress g++ warnings.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.4 Fri Sep 11 00:08:04 PDT 1998

- The new aux/ directory has some utility programs and scripts.
  See aux/README for details.

- Documentation (though exceedingly limited) describing the connection
  log summaries generated by policy/tcp.bro now available in doc/conn-logs.

- The Telnet username recognition code has been heavily reworked and is
  now significantly more robust.

- A new policy file, policy/scan.bro, detects port and address scanning.

- Bro now detects the "Land" attack (a trivial addition to its spoof
  detection).

- The distribution now comes with BIND 8.1.2.

- A new flavor of "weird" event has been added, flow_weird, for unusual
  events associated with <src IP addr, dst IP addr> flows (i.e., a coarser
  granularity than TCP or UDP connections).

- Bro now reassembles fragments, checking for overlaps and consistency.
  Associated with fragments are the following flow_weird's:

	excessively_large_fragment
	excessively_small_fragment
	fragment_inconsistency
	fragment_overlap
	fragment_protocol_inconsistency
	fragment_size_inconsistency
	fragment_with_DF
	incompletely_captured_fragment

- The TCP stream reassembly code was rewritten to share functionality
  with the new fragment reassembly code.

- If a handler for it is present, then Bro will generate "packet_contents"
  events containing the contents of each packet it receives.  This is just
  for experimenting with, and, in particular, this event does *not* reflect
  TCP stream reassembly.

- The handling of "conn_weird" events in tcp.bro now has more options.
  In the weird_action table, WEIRD_LOG_ALWAYS means that the weird event
  should always be logged; WEIRD_LOG_PER_CONN means that it should be
  logged once per connection; and WEIRD_LOG_PER_ORIG that it should be
  launched once per originator IP address.

- The example hot.bro now includes provisions for flagging sensitive
  inbound and outbound services.

- Bro now limits the number of events processed when draining the event
  queue after processing a packet.  This makes Bro less prone to packet
  loss during high-speed scans.

- The line-oriented TCP endpoint class was split into TCP_EndpointLine,
  which supports the line-oriented functionality, and TCP_NVT (derived
  from TCP_EndpointLine), which implements the Network Virtual Terminal
  used by Telnet and FTP.

- The TCP_NVT class now understands the Telnet Environment option.

- Escape sequences are now '\' followed by 1 or more octal digits,
  instead of excatly three octal digits (which is error prone).

- If the watchdog timer expires, it now reports the number of events
  processed in the current batch of packets, as well as other timing
  information.

- Bro now should not report packet drops that occur after it has begun
  to exit (these can occur when draining the pending event queue takes
  a while).

- Bro now detects TCP acknowledgements that occur above a sequence
  hole, generating an ack_above_hole event.  Nominally, this indicates
  packet filter drops, but in fact some buggy TCPs manage to do this :-(.

- Fledgling HTTP support added.  An http_request event is generated when
  a new HTTP request is seen, and http_stats is generated when an HTTP
  connection terminates, giving (uninteresting to most people) statistics
  concerning the request(s).  A lot more is needed: parsing persistent
  connections and HTTP replies, for one.  This is just a start.  A stub
  for http_reply exists but these events are not presently generated.

- Ported to Linux (thanks to Pascal Bouchareine).

- A bug in to_lower() and to_upper() was fixed.

- The reporting for unexpected FTP connections now more directly
  identifies the corresponding FTP session.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.3 Fri Feb 13 19:59:39 PST 1998

- Timers are now implemented using calendar queues rather than priority
  queues (with thanks to Craig Leres).

- The new byte_len() built-in returns the length of a string interpreted
  as a set of bytes (including any NUL bytes, especially the final one
  for a typical string).

- The new sub_bytes() built-in extracts a subset of a string interpreted
  as a set of bytes (i.e., immune to any embedded NULs).

- Fixed bad interaction with the latest version of libpcap that on some
  BPF systems would cause Bro to exit any time a little bit of time went by
  without any traffic matching its filter.

- A bug with constant-folding of the ?: operator has been fixed.

- A new "conn_stats" event delivers statistical analysis (number of packets
  transmitted, retransmitted, out-of-order, replicated) of each connection.
  If you define a handler for it, then *no* other TCP processing is done.
  This was added for off-line analysis of traces with large numbers of
  connections in them.

- Some minor portability tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.2 Sat Oct 25 11:22:37 PDT 1997

- Added autoconf support, thanks to Scott Denton.

- Ported to FreeBSD, Solaris.

- Fixed a bug in which partial connections were ignored even for
  protocols (e.g., FTP) that can make use of partial dialogs.

- Included BIND version 8 sources directly in the distribution.

- Better usage() information (again thanks to Scott), -h and -v flags.

- README, CHANGES files created.
