@(#) $Id: CHANGES,v 1.58 2004/03/25 17:09:35 vern Exp $ (LBL) -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0.8a79 Wed Mar 24 22:02:53 PST 2004 - Bro now has an SSL analyzer, written by Michael Kuhn and Benedikt Ostermaier, with further development by Scott Campbell. It generates the following events: event process_X509_extensions(c: connection, ex: X509_extension) event ssl_X509_error(c: connection, err: int, err_string: string) event ssl_certificate(c: connection, cert: X509, is_server: bool) event ssl_certificate_seen(c: connection, is_server: bool) event ssl_conn_alert(c: connection, version: count, level: count, description: count) event ssl_conn_attempt(c: connection, version: int) event ssl_conn_established(c: connection, version: int, cipher_suite: count) event ssl_conn_reused(c: connection, session_id: sessionID) event ssl_conn_server_reply(c: connection, version: int) event ssl_conn_weak(name: string, c: connection) event ssl_session_insertion(c: connection, id: sessionID) Note, it still has a lot of rough edges; particularly, handling non-conformant input. It also generates unnecessary ContentGap alerts due to the way it runs multiple analyzers (SSLv2 and SSLv3) on a single connection. This will be fixed in the fairly near-term future. - The manual has been updated with chapters on signatures (Robin Sommer) and using the interactive debugger (Umesh Shankar), along with a partial description of the new SSL analyzer (Michael Kuhn and Benedikt Ostermaier) and a number of updates to the documentation of built-in functions (Umesh Shankar), though this latter is still not complete since Umesh actually contributed this quite a while ago. - Ruoming Pang has contributed a crude analyzer for DCE/RPC (used for Windows). It generates simple dce_rpc_request and dce_rpc_reply events. It should not be considered stable. - The traditional connection logging format (traditional_conn_format) is no longer the default. The 0.8a70 release notes stated that this was the case but this time it really is :-). - An experimental "vector" type has been added (Umesh Shankar). A vector is an aggregate type. For example: local suspect_hosts: vector of addr; You can randomly access elements starting with the first as 1, e.g., suspect_hosts[1] = 10.0.0.8; and can also add elements at later postions even if there are gaps: suspect_hosts[31337] = 10.0.0.9; *The semantics and capabilities of vectors will be changing considerably.* - Umesh Shankar has developed a framework for generating IDMEF messages. Currently it needs a modified version of libidmef, which is not included in this distribution. Contact me or Umesh if you want a copy. - A new attribute &synchronized causes the given global variable to be *synchronized* across concurrent instances of Bro (which are intercommunicating via remote.bro). Any change made by one of them to the variable will be reflected (soon after) in the copy at the others. A new event remote_state_inconsistency is generated if two instances both change the value before they're synchronized. (Contributed by Robin Sommer.) - trw.bro implements a new scan detection algorithm, Threshold Random Walk (Jaeyeon Jung). It's described in an upcoming IEEE S&P symposium paper. The analyzer generates two events: TRWAddressScan, # source flagged as scanner by TRW algorithm TRWScanSummary, # summary of scanning activities reported by TRW TRW is generally much more sensitive than Bro's regular detection algorithm. - vlan.bro provides support for VLAN encapsulation. More generally, Bro now has support for any sort of constant-offset encapsulation (Vinod Yegneswaran). You specify the header size by redef'ing encap_hdr_size. You can also redef tunnel_port to be a UDP port which Bro treats as being the encapsulation (in the packet stream - not addressed to it) rather than all traffic. - If you turn on statistics (i.e., via @load statistics) and also redef segment_profiling to T, then Bro will generate to the statistics file a trace of its "segment" processing. A segment is a unit of internal execution. Profiles look like: 1058517499.615430 segment-processing-packet dt=0.000013 dmem=0 1058517499.615430 segment-draining-events dt=0.000012 dmem=0 1058517499.615671 segment-expiring-timers dt=0.000010 dmem=0 1058517499.615671 segment-processing-packet dt=0.000010 dmem=0 1058517499.615671 segment-draining-events dt=0.000012 dmem=0 1058517499.615671 segment-policy/conn.bro:282 dt=0.000011 dmem=0 1058517499.615671 segment-policy/conn.bro:253 dt=0.000012 dmem=0 The first line states that at the given (packet) timestamp, the event engine processed a packet, taking 13 usec of CPU time to do so, and not consuming any memory (from the kernel's perspective; this is *not* fine-grained memory consumption). The next lines indicate 12 usec were spent draining events and 10 usec expiring timers. The last two lines indicate that the functions at lines 282 and 253 in conn.bro were executed, requiring 11 usec and 12 usec, respectively. Note #1: timings are just what rusage() reports, so not necessarily very accurate for small times. Note #2: there's a bug in tracking function line numbers that hasn't been ferreted out yet, so they're only roughly correct. - The inactivity_timeout global has been split into tcp_inactivity_timeout/ udp_inactivity_timeout/icmp_inactivity_timeout (Robin Sommer). Using this, the default inactivity timeouts for UDP and ICMP have been changed from "no timeout" to 10 seconds. This is needed because otherwise analyzing a stream of UDP or ICMP traffic generally gobbles up memory quickly and never recovers it; and there's seems little point in trying to consolidate long-lived-but-often-inactive UDP/ICMP streams. - The new policy script cpu-adapt.bro is an extension to load-levels.bro (see CHANGES for 0.8a37 below) to adapt the packet filter based on the current CPU load. If the load is below cpu_lower_limit (default 40%), the load-level is decreased. If it's above cpu_upper_limit (default 90%), it's increased. (Robin Sommer) - The new policy script hand-over.bro can be used for a new running instance of Bro to smoothly take over operation from an old instance, i.e., it implements hand-over of state between two Bro instances when checkpointing (Robin Sommer). First, all persistent data (i.e. variables declared &persistent and connections for which make_connection_persistent() has been called) is transferred from the old instance to the new instance. Then the old instance terminates itself and the new one starts processing. The host from which we want to take over the state has to be added to remote_peers_clear (or remote_peers_ssl), setting hand_over to T. The host which is allowed to perform a hand-over with us has to be added with a port of 0/tcp and hand_over=T. An example for a handover between two instances on the same machine: @load hand-over redef remote_peers_clear += { [127.0.0.1, 47756/tcp] = [$hand_over = T], [127.0.0.1, 0/tcp] = [$hand_over = T] }; (This interface may be subject to change in the future.) - New script functions (Robin Sommer): function terminate() Terminates Bro via SIGTERM. function dump_config() Dumps Bro's full configuration into state_dir (one file per variable/type/function, etc.) function send_state(ip: addr, p: port) Send all of persistent state to the remote host. function set_accept_state(ip: addr, p: port, accept: bool) If accept is true, state sent by the given host will be accepted (default: false) function make_connection_persistent(c: connection) Declare the given connection state to be persistent (i.e. to be saved upon termination and exchanged by send_state). checkpoint.bro uses this to declare some services to be persistent by default. function is_local_interface(ip: addr): bool Returns true if the given address is assigned to a local interface. - Printing of sets and tables now includes timestamps indicating when the element was added (Robin Sommer): ID did_ssh_version = { [129.187.20.9, F] = 1 @11/01-15:55, [212.144.77.26, T] = 2 @11/01-15:55, [141.84.116.26, T] = 10 @11/01-15:55, [217.232.245.249, T] = 1 @11/01-15:55, [217.235.217.149, T] = 1 @11/01-15:55, [129.187.39.13, F] = 2 @11/01-15:55, [129.187.208.139, F] = 1 @11/01-15:55, } The format may change in the future, and will probably be made an option. - Similarly, you can print functions to get both a timestamp of the last time the given block was executed and a count of how often (Robin Sommer): ID record_connection = record_connection (@11/01-16:03 #6549) { id = c$id; local_init = is_local_addr(id$orig_h); local_addr = local_init ? id$orig_h : id$resp_h; remote_addr = local_init ? id$resp_h : id$orig_h; flags = local_init ? "L" : ""; if (remote_addr in neighbor_nets) (@ #0) flags = cat(flags, "U"); if ("" == flags) (@11/01-16:03 #2110) flags = "X"; is_tcp = is_tcp_port(id$orig_p); ; if (is_tcp) (@11/01-16:03 #6549) { if (c$orig$state in conn_closed || c$resp$state in conn_closed ) (@11/01-16:03 #4739) duration = fmt("%.6g", c$duration); else (@11/01-16:03 #1810) duration = "?"; [...] - You can now specify numbers using hex constants, e.g., 0xabcd = 43981 (Michael Kuhn and Benedikt Ostermaier). - A new function, join_string_array(sep: string, a: string_array) concatenates strings in 'a' and inserts 'sep' between every two adjacent elements (Ruoming Pang). E.g., join_string_array("", {"a", "b", "c"}) returns "a b c", and join_string_array("", a) is the same as cat_string_array(a). - checkpoint.bro now makes some services persistent by default (Robin Sommer). - The new_packet event now includes both the associated connection and a pkt_hdr describing the packet (Robin Sommer). - The functions functions connect_ssl() and connect_clear() have been replaced by a single connect() function taking an additional parameter to differentiate the types (Robin Sommer). - The new function stop_listening() unbinds the listen port (Robin Sommer). - A new flag packet_filter_default says whether the Bro-level packet-filter will by default accept all or reject everything (Robin Sommer). - Function calls can now be sent to remote Bro's, though there isn't yet an interface for accessing this from the script level (Robin Sommer). - Bro now has an generalized internal framework for serializing objects and monitoring access to state (Robin Sommer). - Better memory allocation accounting (Robin Sommer). - A minor tweak to the output generated by statistics.bro. - Improved localization of source code for functions in messages (but there are still some bug). - Autoconf looks for -ltermcap (Robin Sommer). - Fixes for bugs in the management of table expiration values (Chema Gonzalez). - A bug in printing "void" values has been fixed (Chema Gonzalez). - -t bug fixed (Chema Gonzalez). - A bug has been fixed in which sometimes "expression value ignored" was erroneously generated. - A bug with packet_contents and UDP packets with checksum errors has been fixed (Ruoming Pang). - A memory leak in packet timestamp sorting via packet_sort_window has been fixed (Ruoming Pang). - A bug has been fixed in expanding NULs when printing strings (Ruoming Pang). - Bug fixes for extracting connection contents via contents.bro (Ruoming Pang). - Bogus error message "Can't install default pcap filter" when using -F removed. 0.8a70 Sun Feb 8 14:19:45 PST 2004 - Bro has a new home page at http://www-nrg.ee.lbl.gov/bro.html It includes a "wish list" of Bro development projects: http://www-nrg.ee.lbl.gov/bro-wishlist.html - The "match" expression has been completely overhauled (Umesh Shankar). It now has the syntax: match EXPR1 using EXPR2 Its semantics are complicated, but it's very powerful (see its use for alert filtering below). EXPR1 can have any type T. EXPR2 must be of type "set[R]", where R is a record type. R must have the following fields: $pred - type is "function(T): bool". This is the predicate associated with the record. It is passed in EXPR1's value and returns true or false. $result - can have any type T'. This is the value to use when if $pred returns true for EXPR1. $priority - type must be arithmetic (count, int, double). This is the priority associated with the match of EXPR1 if $pred returns true. The way the expression works is that EXPR1 is evaluated yielding a value V. EXPR2 is then evaluated yielding a set of records whose type includes the above fields. Bro then spins through each of the records in the set and tests whether its $pred predicate holds for V. If so, it records the given $result and the associated $priority. It then returns for the value of the entire expression the $result with the highest $priority. Here's an example. The following script: global match_stuff = { [$pred = function(a: count): bool { return a > 5; }, $result = "it's big", $priority = 2], [$pred = function(a: count): bool { return a > 15; }, $result = "it's really big", $priority = 3], [$pred = function(a: count): bool { return T; }, $result = "default", $priority = 0], }; print match 0 using match_stuff; print match 10 using match_stuff; print match 20 using match_stuff; when executed will print: default it's big it's really big (Note that Bro actually will first evalute $priority before evaluating $pred, and if it already has a better (higher) priority result, it will not bother calling $pred.) - There's a new syntax for designating function values (Umesh Shankar). It currently only works when associating a function value with a record field in a record constructor: [$foo(bad_guy: addr) = { launch_counter_DDOS(bad_guy); return 3.14; }] is equivalent to: [$foo = function(bad_guy: addr): double = { launch_counter_DDOS(bad_guy); return 3.14; }] The goal is to make such functions easier on the eye to express. The changes are (1) no "function" keywork necessary, (2) no function return type necessary (note, it is inferred from the "return" statement in the function body; eventually this will work for all functions, not just those in record constructors), (3) the '=' sign comes after the ')' rather than before the keyword "function". Given this syntax, we can rewrite the initialization of match_stuff in the example above as: global match_stuff = { [$pred(a: count) = { return a > 5; }, $result = "it's big", $priority = 2], [$pred(a: count) = { return a > 15; }, $result = "it's really big", $priority = 3], [$pred(a: count) = { return T; }, $result = "default", $priority = 0], }; - The motivation behind these elaborate new mechanisms is to provide a powerful and streamlined way to filter alerts. According, alert.bro now processes any alerts generated via ALERT() through a new global, alert_policy. alert_policy's type is set[alert_policy_item], where alert_policy_item is: type alert_policy_item: record { result: AlertAction; pred: function(a: alert_info): bool; priority: count; }; The idea is that you specify your alert post-filtering by redef'ing new elements into alert_policy. For example, here are two post-filtering rules used at LBL to weed out uninteresting alerts: # Ignore connections marked as sensitive because they're # NTP to otherwise-sensitive hosts (first clause) or they happen # to involve 2766/tcp (Solaris Listen), which happens frequently # to ftp.ee.lbl.gov if Bro misses the PORT negotiation. [$pred(a: alert_info) = { return a$alert == SensitiveConnection && (a$conn$id$resp_p == 123/udp || # NTP a$msg == /Solaris listen service/); }, $result = ALERT_FILE, $priority = 1], # Ignore sensitive URIs if the request was unsuccessful (code 404, # or not answered.) [$pred(a: alert_info) = { return a$alert == HTTP::HTTP_SensitiveURI && a$msg == /.*((\(404)|(no reply)).*/; }, $result = ALERT_FILE, $priority = 1], These rules are part of: redef alert_policy += { ... these records and others ... }; The idea behind them is to demote certain alerts that would ordinarily be syslog'd (i.e., the associated action is ALERT_LOG_ALWAYS) to instead just be recorded in the alert.$BRO_ID file. Naturally, there are many other possibilities. For example: [$pred(a: alert_info) = { if ( a$alert == FTP::FTP_Sensitive && a$msg == /.*crown_jewels.*/ ) { system("page_the_duty_officer \"crown jewels theft!\""); return T; } else return F; }, $result = ALERT_LOG_ALWAYS, $priority = 1000], would run the program page_the_duty_officer with the argument "crown jewels theft!" if an FTP_Sensitive alert was generated and the log message included the text "crown_jewels". More generally, post-filtering needn't just be about deciding on how the alert is logged; the processing can run programs, update tables, etc., just like any other function call might. - You can use the new function tally_alert_type in an alert_action_filters initialization in order to suppress immediate logging of an alert and instead have Bro generate a summary of how many times the given alert was seen when it exits. You can use another new function, file_alert, to specify an alert_action_filters initialization that causes the alerts to just be written to the alert.$BRO_ID file but not otherwise logged. For example: redef alert_action_filters += { # Just summarize various packet capture glitches. [[RetransmissionInconsistency, ContentGap, DroppedPackets, AckAboveHole]] = tally_alert_type, [RemoteWorm] = file_alert, }; would specify that RetransmissionInconsistency (etc.) alerts should just be reported in the log file (log.$BRO_ID) as a total count, and RemoteWorm should only be put in the alert.$BRO_ID file, but not otherwise logged or counted. You could get the same functionality by writing alert_policy records, but they're quite a bit bulkier than the above. Note that alert_action_filters entries take precedence over alert_policy records, but are evaluated *after* the "match" on alert_policy, so if you have predicates in alert_policy with side-effects (like the invocation of page_the_duty_officer in the example above), those will still happen. - The alert_info record (which is used in calls to ALERT) now has slots for some more additional information: user: string; # can hold an assocaited username filename: string; # an associated filename method: string; # associated HTTP method URL: string; # associated URL n: count; # any associated count/number/status code (These are all &optional, so you don't need to specify them if they're not appropriate.) A number of ALERT's in the default policy scripts have been changed to include these. The intent is to add more such information in the future. Ideally, alert_policy records shouldn't be doing checks like "a$msg == /.*((\(404)|(no reply)).*/" but instead extracting the status code directly from a field of 'a' (which is an alert_info record). - ALERT now fills in the '$id' field in the alert_info record with the $id of the associated connection, if the caller didn't suppy a $id but did supply a $conn. Likewise, it will fill in $src with the $orig_h from $id (after first filling in $id). The net result is that you can rely on $id and $src being set for any alert that has an associated connection. - The HTTP analysis scripts (policy/http*.bro) have been converted to use the "module" facility, similar to how ftp.bro was converted for 0.8a48. This may require changing some of your own scripts, generally just to add "HTTP::" qualifiers. - Now that the variables associated with FTP analysis are part of an "FTP" module, the "ftp_" prefix associated with: ftp_excessive_filename_len ftp_excessive_filename_trunc_len ftp_guest_ids ftp_hot_cmds ftp_hot_files ftp_hot_guest_files ftp_ignore_invalid_PORT ftp_ignore_privileged_PASVs ftp_log ftp_skip_hot has been removed, and these are now called: excessive_filename_len excessive_filename_trunc_len guest_ids hot_cmds hot_files hot_guest_files ignore_invalid_PORT ignore_privileged_PASVs log_file skip_hot To get to them from other scripts, you specify, for example, redef FTP::guest_ids = { .... }; whereas before you had to use: redef FTP::ftp_guest_ids = { .... }; - The new connection logging format introduced in 0.8a57 is now the default, unless you redef the new variable "traditional_conn_format" to be T (Robin Sommer). Connections using unidentified ephemeral ports now have a service of simply "other" rather than other-XXXX. The 'U' connection status flag has been removed (unless you're using traditional_conn_format). - Tables can now be directly indexed by records, and indexing using records is no longer interchangeable with using a corresponding list of indices (Umesh Shankar). This may require adjustments to existing policy scripts. - Hostnames such as www.google.com now have type set[addr] rather than a funky internal list type. - The new function dump_current_packet(file_name: string) dumps a copy of the current packet to the file with the given name, appending it if the file already exists (Robin Sommer). The file is in tcpdump format. A handy use for this is in an event handler for signature_match(), to record packets that match given signatures. - The event new_packet() is invoked for each new packet (Robin Sommer). It currently doesn't provide the packet contents but soon will in a fashion similar to secondary-filter.bro. - "cf -f fmt" specifies a strtime() format. -u specifics UTC time rather than local time (Mark Delow and Craig Leres). cf now has a man page (Craig Leres). - Two new variables, backdoor_ignore_local and backdoor_ignore_remote, can be used to specify backdoor signatures that should be ignored if the server is local/remote. - A bug has been fixed in which a "next" executed in the final iteration of a for loop would mangle the subsequent processing of the outer statements (Chema Gonzalez). - Bug fixes for MIME and Base64 processing (Ruoming Pang). - pcap.bro now builds its filter in the opposite order (restrict_filters first), which can improve packet filtering performance (Robin Sommer). - A bug in &default has been fixed. - More SSL autoconf tweaks (Robin Sommer). - Portability for different pcap_compile_nopcap() calling sequences (Chema Gonzalez). - Some tweaks for a minor reduction in memory consumption. - A memory leak for secondary packet filters has been fixed. - The localization of error messages (what script line they correspond to) has been improved. 0.8a58 Tue Dec 16 08:55:47 PST 2003 - Compatibility with older versions of libpcap (Chema Gonzalez). 0.8a57 Tue Dec 9 10:14:30 PST 2003 - The format of Bro's connection summaries is changing. The new format looks like 1069437569.904605 0.230644 1.2.3.4 5.6.7.8 http 59377 80 tcp 610 275 S3 L That is, , , , , , , , , , , . (Robin Sommer) The script variable traditional_conn_format=T specifies to use the old format rather than this new one. This is *currently* the default, but will change soon to default to F instead. If you have comments on this new format, we'd like to hear them. - The SigAction's available in signatures.bro have been extended (Robin Sommer). SIG_FILE_BUT_NO_SCAN is like SIG_FILE but without any horizontal/vertical processing; SIG_LOG_ONCE logs only an alert only the first time it occurs; SIG_LOG_PER_ORIG logs only the first instance of an alert generated by a particular originator; SIG_COUNT has been renamed SIG_COUNT_PER_RESP; and SIG_SUMMARY suppresses logging of individual alerts but generates a per-originator summary. - A new -p option for snort2bro tells it to only process signatures that include matching on payload (Robin Sommer). - You can now explicitly include or exclude particular SIDs when running snort2bro by specifying a configuration file via -c (Robin Sommer). The format is simple, just "include" or "ignore" followed by the SID number: # sid-526 BAD TRAFFIC data in TCP SYN packet ignore 526 # sid-623 matches a null-flags stealth scan. Include it even # if we build with -p, since it doesn't tend to generate any # false positives. include 623 The new "snort2bro.cfg" file gives examples (i.e., the above). - Bro can now serialize functions and event handlers, meaning that these can be passed as values between Bro's and dumped using -g (Robin Sommer). One of the main goals in supporting this is to allow in situ alteration of the Bro's configuration (e.g., you can edit a function and change its functioning and have a running Bro pick up the change without having to stop and be restarted). Such dynamic reconfiguration is experimentally supported via -g (see below). - &persistent state is now stored in the *directory* given by state_dir (default: "./.state"), one file per variable, rather than a single file (Robin Sommer). - Storing &persistent state to disk is now done incrementally: after writing each file, there's a delay of state_write_delay (default: 0.1 secs) before working on the next file (Robin Sommer). This may introduce small inconsistencies, but prevents load spikes that can lead to packet drops. Currently, there is no mechanism to incrementally store a single variable (like a large table), although there is already some framework in place to eventually support this. - The *experimental* new -g option dumps the script-level configuration (excluding things defined in internal default scripts like bro.init) into the directory . These files may be printed with "bro -x ", or copied into the state_dir of a running Bro, which will then pick up the change if it has loaded checkpoint.bro. (When picking up changes, event handlers are always added, while functions, types, and variables replace the current ones). - Table values are now incrementally expired rather than all at once (Robin Sommer). That is, if the expiration timer goes off and hundreds of values should now be expired, the work of doing so is spread over chunks of table_expire_size (default: 50) at a time, separated by a delay of table_expire_delay (default: 0.1 secs). This change aims to prevent large processing spikes that can lead to packet drops. - New built-ins sub() and gsub() act like awk's functions of the same name, changing substrings (either first, or all) that match a given regular expression to a given target string. (Note, the calling sequence differs from the order used by awk.) - The new auxiliary script aux/scripts/mvlog is a handy way to manage checkpointed logs. See the script for documentation. - The &expire_func function now takes two arguments. The second is of type "any" and corresponds to the index(es) of the element being expired. To access the individual indices, you use a new assignment form: [a, b, c] = index_val; (where index_val is the second argument of type "any"). This assigns a to the first index, b to the second, and c to the third. NOTE: the use of "any" types here is *temporary* and will be changing in the future to a general "tuple" notion. (Robin Sommer) - scan.bro and conn.bro have been reworked to consume less memory and to support more flexible state expiration (Robin Sommer). - The new builtin rescan_state() causes Bro to re-read any persistent data values (Robin Sommer). - snort2bro now supports continued lines ("\") (Robin Sommer). - The calling sequences of the software_version_found() and software_parse_error() events has changed, and a new event, software_unparsed_version_found(), is invoked for raw version strings (i.e., the version string prior to the event engine attempting to parse it into version/major/minor) (Robin Sommer). - Software version tracking for clients now tracks all versions, not just the latest version (Robin Sommer). - alert_info records now include an optional field event_src, which is the source of the event if it was received from an external Bro (Robin Sommer). - Regular expressions now support {} iteration values of 0, and generate better error messages. - Output generated by icmp.bro is now redirected into an "icmp" log file (Robin Sommer). - autoconf tweaks for configuring OpenSSL on Linux (Ruoming Pang, Robin Sommer). Tested on RedHat (thanks to Anton Chuvakin), Debian, FreeBSD, Solaris. - You can now turn off using OpenSSL even if the OS supports it, via configuring with --disable-openssl (Robin Sommer). - Variable size computations (per global_sizes()) are now more accurate (Robin Sommer). - A bug with combining file encryption and log rotation has been fixed (Robin Sommer). - A problem tracking directionality in signatures fixed (Robin Sommer). - Bro now continues running if DNS is not functioning (Robin Sommer). - Rewriter memory use has been significantly reduced (Ruoming Pang). - Some bugs with -A/-w interaction have been fixed (Ruoming Pang). 0.8a48 Tue Oct 21 15:56:13 PDT 2003 - There is now a mechanism in place for multiple Bro's to communicate with one another via sockets (Robin Sommer). *This is quite experimental at this point* and may have significant bugs and/or need significant enhancements. By loading listen-clear.bro or listen-ssl.bro, an instance of Bro starts to listen on a TCP port. The first of these listens for unencrypted connections and the second for connections encrypted via OpenSSL. To connect to a listening Bro, you load remote-clear.bro or remote-ssl.bro. For this connection, you specify which events you want to receive by giving a regular expression (e.g. "/http_*/" for all HTTP events), although only those events for which you have defined a local handler are actually requested. Communication is uni-directional in the sense that for a certain connection the events go only from Bro-A to Bro-B but not from B to A (you could set up a second connection for the other direction, though). The OpenSSL certificate used to authorize remote Bro's is specified in the script variable "ssl_ca_certificate" and the private key and certificate for the local Bro via "ssl_private_key". If Bro A connects to Bro B, by default it sends over its capture_filter. But Bro B uses it only if it has loaded remote-pcap.bro. This is the beginning of structuring inter-Bro trust mechanisms. It is done via two new events, remote_capture_filter(ip: addr, p: port, filter: string) and send_capture_filter(ip: addr, p: port, s: string) : bool. The implementation forks a second process which does the socket communication, so that the main process should not be affected too much. The two processes communicate via a pipe. You can call is_remote_event() to determine whether the event currently being handled was originated by a remote Bro (if T) or the local Bro (if F). If a connection with a remote Bro terminates (for whatever reason), Bro may try to reconnect automatically. A new function, get_event_source(), returns a record event_source describing the source that raised the last event. See doc/ssl.txt for an explanation of how to create the keys/certificates. - A fledgling Gnutella analyzer has been contributed (Mark Allman). It generates the following events: event gnutella_text_msg(c: connection, orig: bool, headers: string) event gnutella_binary_msg(c: connection, orig: bool, msg_type: count, ttl: count, hops: count, msg_len: count, payload: string, payload_len: count, trunc: bool, complete: bool) event gnutella_partial_binary_msg(c: connection, orig: bool, msg: string, len: count) event gnutella_establish(c: connection) event gnutella_not_establish(c: connection) event gnutella_http_notify(c: connection) - Bro now supports a secondary channel for acquiring packets (Chema Gonzalez). You access it by redef'ing the new global "secondary_filters", adding table[string] of event(filter: string, pkt: pkt_hdr). The string specifies a tcpdump filter; anytime a packet matches the filter (including packets that would *not* otherwise be matched by capture_filter), then the given event handler is invoked. For example, redef secondary_filters += { ["tcp[13] & 7 != 0"] = rst_syn_fin_flag, } will invoke rst_syn_fin_flag() anytime a TCP packet is seen for which the SYN/FIN/RST bits are non-zero. The event handler will be passed the string "tcp[13] & 7 != 0" (so it can tell which of possibly multiple filters matched) and a pkt_hdr value, which includes the IP header and, if present, the TCP, UDP or ICMP header. Another example, and what motivated the addition, is: redef secondary_filters += { ["ip[10:2] & 0xffc == 0x398"] = sampled_1_in_1024_packet, } which will invoke sampled_1_in_1024_packet() any time the given 10 bits in the IP checksum match the pattern 0x398. If the checksum field is uniformly distributed then this roughly corresponds to 1-in-1024 random sampling. (Chema has also developed BPF additions to support true random sampling.) See policy/secondary-filter.bro for examples. - Bro now does a much better job of keeping track of how much memory has been allocated for different structures (Robin Sommer). This includes more accurate computations for global_size(). In addition, if you redef "statistics_interval" to be a non-zero time interval, then with that periodicity a summary of memory usage (including memory used by event engine components) is dumped to the file "statistics_file". In addition, at this point a "do_statistics" event is generated. You can also call the new built-in statistics_update() to generate memory statistics on demand. The above structure is likely to change in the future. statistics_interval will probably go away, to be replaced by either explicit calls to statistics_update() (which you can do on a timer if you like by using "schedule"), or by a built-in function that returns a record of all the statistics, that you can then format however you want. - A major memory leak in HTTP analysis has been fixed (Ruoming Pang). - New attributes &rotate = and &postprocessor = can be associated with a file variable in order to specify how often the file should be rotated to a new filename on disk, and, when rotation occurs, the name of a shell program to run on the now-older version as a postprocessor (Robin Sommer). - Similarly, log_postprocessor and log_rotate_interval specify the default values for files. Unless redef'd, these themselves default to the empty string (= no postprocessing) and 0.0 seconds (no rotation). (Robin Sommer) - A new attribute, &encrypt, applies to variables of "file" type. It specifies that the version on disk should be encrypted, using either the key specified as the value of the attribute, or, if no value is specified, using the value of the new script variable log_encryption_key. The key is an OpenSSL public key; it's used to then embed a Blowfish session key. (Robin Sommer) A new utility, aux/bdcat/bdcat ("Bro decrypt cat") can be used to decrypt the files. - The internal structure of TCP analysis has been significantly altered. Previously, TCP_Endpoint tracked endpoint state and TCP_EndpointContents (derived from it) did stream reassembly. These have now been separated; TCP_Endpoint still tracks endpoint state, but TCP_EndointContents has become TCP_Contents, and is only loosely coupled with TCP_Endpoint. The reason for this change is to lay the groundwork for (1) applying an application analyzer to a connection after several packets for the connection have already been read, and (2) applying *multiple* application analyzers to a single connection. - Bro now supports the universal hashing if USE_UHASH is defined (Ruoming Pang). Universal hashing is a lighter-weight alternative to MD5 hashing that retains the property of being very difficult for an attacker to guess. It comes in two flavors, a 32-bit version (which you get with USE_UHASH) and a faster 16-bit version (which you get if you also define USE_NH). Bro will likely switch to using these by default in the near future, as their performance gain over MD5 is significant. - New built-in functions srand() and rand() provide access to the usual C-style random number seeding & generation (Chema Gonzalez). - You can now specify server/client addresses to leave in the clear in IP address anonymization (via the built-in variables preserve_orig_addr and preserve_resp_addr). Correspondingly, the built-in function for IP anonymization now takes a parameter to specify the type of the address (orig, resp, or other), instead of the method of anonymization (seq, random, prefix-preserving). (Ruoming Pang) - Trace anonymization now has prelminary support for handling TCP options via the new event "tcp_option" (Ruoming Pang). It is only invoked if skip_event_tcp_option is F (it defaults to T). - A new event, tcp_segment, is similar to the event "packet_content" but provides more information: is_orig (directionality), sequence number, and acknowledgement number (Ruoming Pang). - ./configure finds OpenSSL if it's in some standard location. Otherwise, you may specify it --with-openssl=. If OpenSSL is not available, Bro compiles cleanly without and gives warnings if a script tries use SSL. (Robin Sommer) - The internal links in manual/entire.html have been fixed so they now work (Chema Gonzalez). - A new policy script, blaster.bro, detects instances of the W32.Blaster worm (Robin Sommer). - Signature files (for input to the signature engine) now reside in policy/sigs/*.sig. This directory is now on the default $BROPATH. - sig.ex.ssl-worm.bro and sig.ex.web-rules.bro have been updated to reflect changes in keywords (Robin Sommer). They've been renamed to ex.ssl-worm.sig and ex.web-rules.sig and reside in policy/sigs/, per the above. - The module facility has been changed to have its scope limited to the current file plus any additional files that are automatically processed based on its name plus $BRO_PREFIXES. - As an experiment, ftp.bro has been modified to a style that includes using "module FTP". Likely other policy scripts will be converted in the near future, and their variable names changed accordingly (e.g., "ftp_hot_files" will become "FTP::hot_files"). - The new "match" expression has been modified to allow any yield type rather than just strings. It is likely to change significantly again soon. - Iterating over multi-dimensional tables/sets now works (Robin Sommer). For example: const remote_peers_ssl : table[addr, port] of Peer &redef; [...] for ( [ip, p] in remote_peers_ssl ) connect_ssl(ip, p, remote_peers_ssl[ip, p]$retry); - Checkpointing of persistent state on SIGHUP now happens via bro.init (Robin Sommer). Not tested. - fmt() now supports %x for hexadecimal formatting (Chema Gonzalez). - alert.bro logs the source for remote alerts; by redefining the new "event_source_description: string" locally, you can specify how a Bro is to refered to on the remote side. (Robin Sommer) - software.bro now tracks HTTP clients, too (Robin Sommer). This will be extended in the near future. - Some FreeBSD 5.X porting tweaks (Sergey Osokin). 0.8a37 Wed Sep 3 23:20:21 PDT 2003 - A new attribute, "&persistent", indicates that a global variable's state should persist on disk (Robin Sommer). Currently, they reside in the file "state.bst". For example, given the following script: global a = 0 &persistent; event bro_init() { print ++a; } then every time you run it, a increases by one. You can dump the state file using "bro -x state.bst ". (There's also a partially-implemented XML dumper which you invoke via "bro -X state.bst ".) If you send Bro a HUP signal, then it will synchronize all persistent state to disk (checkpoint) at that point. More generally, the policy script can do so at any time by calling the new built-in checkpoint_state(). By including the new policy script "checkpoint", you can have Bro automatically checkpoint every time checkpoint_interval elapses (default 15 minutes). - You can also record events to disk by loading the "capture-events" policy script, or calling the new capture_events() built-in (Robin Sommer). The events are written to the file events.bst. You can dump these using bro -x (or -X), as above. You can also replay them using "bro -R ". By replaying them using the same policy script as generated them, you should get the same output. For example: bro -r tracefile myscript capture-events ... bro -R events.log myscript - An experimental module facility has been added (Umesh Shankar). The module facility implements namespaces. Everything is in some namespace or other. The default namespace is called "GLOBAL" and is searched by default when doing name resolution. The scoping operator is "::" as in C++. You can only access things in the current namespace, things in the GLOBAL namespace, or things that have been explicitly exported from a different namespace. Exported variables and functions still require fully-qualified names. The syntax is as follows: module foo; # Sets the current namespace to "foo" export { int i; int j; } int k; module bar; int i; foo::i = 1; bar::i = 2; print i; # bar::i (since we're currently in module bar) j = 3; # ERROR: j is exported, but the fully qualified name # foo::j is required foo::k = 4; # ERROR: k is not exported The same goes for calling functions. One restriction currently in place is that variables not in the "GLOBAL" namespace can't shadow those in GLOBAL, so you can't have module GLOBAL; global i: int; module other_module; global i: int; It is a little confusing that the "global" declaration really only means that the variable i is global to the current module, not that it is truly global and thus visible everywhere (that would require that it be in GLOBAL, or if using the full name is ok, that it be exported). Perhaps there will be a change to the syntax in the future to address this. The "module" statement cuts across @load commands, so that if you say: module foo; @load other_script; then other_script will be in module foo. Likewise if other_script changes to module bar, then the current module will be module bar even after other_script is done. However, this functionality may change in the future if it proves problematic. The policy scripts in the Bro distribution have not yet been updated to use it, but there is a backward-compatibility feature so that existing scripts should work without modification. In particular, everything is put in GLOBAL by default. - The hooks are now in place for communicating events between running Bro's. An experimental implementation of doing so (written by Robin Sommer) will be integrated shortly. - A side-effect of getting those hooks in place is that event handlers must now be directly specified (by naming them) rather than indirectly (for example, by indexing a table whose value yields an event handler). This may be fixed soon. - An experimental "match" expression scans a list of predicates to find the first one that's both true and has the highest priority (Umesh Shankar). The general form is: match some_record on { [ priority, class_name, predicate ], ... } where "predicate" is evaluated in the context of the value (and type) of some_record. For example, if some_record has a field "num_alerts" then predicate could be "$num_alerts > 5". "priority" is a non-negative integer (i.e., of type "count"), and, for now, "class_name" is a string. For example, global c: conn_id; c = [ $orig_h = 0.0.0.0, $orig_p = 0/tcp, $resp_h = 1.1.1.1, $resp_p = 1/tcp ]; print match c on { [ 2, "emptyweb", $orig_h == 0.0.0.0 && $resp_p == 80/tcp ], [ 1, "emptyhost", $orig_h == 0.0.0.0 ], [ 0, "should not match", 1 == 0 ] }; will print "emptyhost". The initial intent behind this is to eventually provide more flexible customization of alert processing, though it clearly has broader applicable. *It is very likely that the semantics and perhaps the syntax of "match" will change in the near future.* - Bro's packet filter is now computed via pcap.bro (which is automatically included via bro.init). It uses two new built-ins: precompile_pcap_filter(id: PcapFilterID, s: string): bool install_pcap_filter(id: PcapFilterID): bool The first is for precompiling a pcap filter so it can be installed or turned off dynamically. Associating an ID with the filter, you can then activate the filter by calling the second function (installing a new filter replaces the current one). (Robin Sommer) Be default, pcap.bro is responsible for building a pcap string based on the capture/restrict_filters defined by the various analyzers. It compiles and installs this filter, so there is no observable difference in usage to the old implementation, except capture/restrict_filter are now *tables* rather than strings, and are written as plural rather than singular. So the analyzers need to define something like this: [finger.bro] redef capture_filters += { ["finger"] = "port finger" }; This then allows "finger" to be used as the name for the corresponding filter element (see the next item). - load-level.bro is an experimental policy script for allowing Bro to shed or add on load (in terms of which protocols it analyzes). It provides three interface functions for setting the current loadlevel: # level is LoadLevel1, ..., LoadLevel10 function set_load_level(level: PcapFilterID): bool function increase_load_level() function decrease_load_level() load-levels.bro defines ten different load levels (from 1 to 10, with 10 being the one with the most load imposed on the system), which are configured by defining capture_load_level/restrict_load_levels: Example: redef capture_load_levels += { ["dns"] = LoadLevel1, ["smtp"] = LoadLevel2, ["http-reply"] = LoadLevel3, ["http-request"] = LoadLevel8, }; This means for example: "include the capture_filter associated with 'http-reply' if the current load level is 3 or below". There's a similar mechanism for restrict_filters: redef restrict_filters += { ["cs-only"] = "net 131.159.0.0/16", }; redef restrict_load_levels += { ["cs-only"] = LoadLevel7, }; This applies the given restrict_filter if the current load level is 7 or *above*. The pcap filters belonging to the ten load levels are built and pre-compiled on startup. The three functions shown above just call install_pcap_filter() then. (Robin Sommer) - drop-adapt.bro tries to adjust the load level based on the current packet drop rate (Robin Sommer). - synflood.bro is an experimental policy script for detecting SYN floods. It is not yet documented, other than brief comments in the script. (Robin Sommer) - Subnet lookups now use Patricia trees instead of brute-force, which should make them significantly more efficient for large lists, e.g., local nets. (Robin Sommer) - Due to the addition of modules, which use a C++-style "::" scope designator, you now need to use "0x" to introduce IPv6 address constants that use "::" to specify a series of empty octets. For example, you used to be able to specify "deadbeef::cafe", but now this needs to be "0xdeadbeef::cafe". Note that "1::2" can still be written without needing a "0x" prefix; it's just hex constants that start with letters that need the prefix. - A new built-in, escape_string(), takes a string and returns a copy of it that uses escape sequences for any problematic characters (Robin Sommer). - A number of low-level bug fixes and portability tweaks (Robin Sommer, Ruoming Pang, Christian Kreibich, Chema Gonzalez). - A new timer, status_update_timer, fires for each ongoing connection every connection_status_update_interval seconds (default = 0 seconds, which means "don't fire). (Robin Sommer) - An additional Bro-level packet filter can filter/sample packets based on their src/dest ip/subnet (using a Patricia tree for efficiency; Robin Sommer). install_src_addr_filter(ip, flags, p) drops packets originating from ip with probability p (0..100) if none of the given TCP flags is set. install_src_net_filter, install_dst_addr_filter and install_dst_net_filter" work similarly. The corresponding "uninstall_*" functions remove the filters again. - The @if/@else/@endif mechanisms have been made more powerful (Robin Sommer). - New configure option --enable-debug to compile without optimization (Robin Sommer). - Small tweaks to the mpatrol support (Robin Sommer). - SMTP is now one of the services which can use a small inactivity timeout in inactivity.bro (Robin Sommer). - Alerts for signatures which are triggered by worms may be suppressed if we already know that the triggering host is indeed infected by a worm (Robin Sommer). - Matches of a signature can now be counted per destination host (Robin Sommer). - snort2bro now ignores sid-526 ("BAD TRAFFIC data in TCP SYN packet"; Robin Sommer). Due to different semantics of Bro (matching stream-wise) and Snort (matching packet-wise) this signature generates a lot of false positives. 0.8a34 Sun Jul 13 09:11:32 PDT 2003 - The new "subnet" type corresponds to a CIDR prefix (Robin Sommer). You can use the '/' operator to convert an address to a subnet. The "in" operator tests whether an address matches a subnet (e.g., "1.2.3.4 in 1.2.255.255/16" yields T). You can index tables with index type "subnet" using addresses to retrieve the table entry with the longest-matching prefix for that address, and a number of tables/sets in the default scripts have been converted to this form. So for example the local_16_nets and local_24_nets variables have been replaced by local_nets, which has type "set[subnet]", is_local_addr() now refers to it, and skip_scan_nets_{16,24} have likewise been consolidated into skip_scan_nets. One present deficiency is that subnets can't be used in tables/sets with multiple indices. Fixing this is going to take some time ... Another deficiency is that the longest-match lookup is not very efficient. Caution: this feature has been only briefly tested with IPv6. - ALERT now generates an event "alert_action" which includes the alert_info and AlertAction associated with an alert. You can define your own handler to further customize alert processing. - The "snort2bro" conversion utility has been extended to recognize some of the new Snort options (depth, distance, within; Robin Sommer). For example: alert .... ( ... content:"|00 01 86 B8|"; content:"|00 00 00 02|"; distance:4; within:4; content:"%x %x"; distance:16; within:256; ... ) is converted to: signature sid-1891 { ... payload /.*\x00\x01\x86\xB8.{4}\x00\x00\x00\x02.{16}.{0,251}%x %x/ } Note that not all of the new Snort options are supported yet. - You can refer to script variables within signatures (Robin Sommer). For example, the following is now possible: signature sid-547 { ip-proto == tcp src-ip != local_nets dst-ip == local_nets dst-port == 21 event "FTP \"MKD \" possible warez site" tcp-state established payload /[mM][kK][dD] / } This makes the signatures independent of the environment ("local_nets" in the above examle). snort2bro now converts some well-known Snort variables into corresponding Bro variables, a number of which are listed in policy/site.bro. - The default action for signature matches is now logging insted of only writing it to a file (Robin Sommer). - You can now use the '^' and '$' regular expression operators inside subpatterns. For example, /foo$/ | /^bar/ works now. - You can now use "msec" for milliseconds (in addition to "usec" for microseconds, "sec" for seconds, etc). - The log_file_name and open_log_file functions are now redef'able if you want to change their behavior. - Bro now exits more cleanly upon exhaustion of memory (Robin Sommer). - A bug was fixed for the case of getrlimit() returning RLIM_INFINITY for the maximum number of open files. - Numerous additions of std:: scoping to address porting issues (Robin Sommer). - gcc 3.X and Solaris portability fixes. - A new event RemoteWorm is the complement of LocalWorm (Robin Sommer). - A bug in which the FTP analyzer would complain about failing to be able to look up a connection has been fixed (Robin Sommer). - You can now configure Bro using "--with-mpatrol" to activate MPatrol debugging hooks. When built in this mode, -m dumps the leak table and -M shows unfreed parts of the heap. These dumps can also be triggered using SIGUSR1 / SIGUSR2, respectively. (Robin Sommmer) - A script function get_contents_type() which returns the type of reassembling being done for a connection (none/orig-side/resp-side/both; Robin Sommer). - A minor bugfix for the regular expression matcher (Robin Sommer). 0.8a32 Thu Jun 12 23:33:21 PDT 2003 - The low-level hash functions have been reimplemented to use HMAC-MD5 to counter the "algorithmic complexity attacks" discussed in the USENIX Security 2003 paper by Scott Crosby & Dan Wallach (http://www.cs.rice.edu/~scrosby/hash/) (Ruoming Pang and Vern Paxson) - Ruoming Pang has made extensive changes to the anonymization/transformation framework. A paper describing it will appear in SIGCOMM 2003, and will be included with the Bro distribution once the final copy is ready. - Internal hash tables now resize incrementally, and more quickly. This makes a big difference in avoiding lengthy processing pauses when processing large volumes of traffic. (Craig Leres) - gcc 3.1 and Linux portability tweaks. - The calling sequence of http_request() has changed. It now includes both the original URI and the escaped URI. This was made explicit because we found that if only the original was passed in, it was too easy to forget to expand the escapes in it; and if only the escaped was passed in, some attacks could be missed. (Ruoming Pang) - Signature rules can now refer to "finger" payload as well as HTTP and FTP (Robin Sommer). - The signature engine now includes keywords "dst-ip", "dst-port", "ip-proto", "src-ip", "src-port". (Robin Sommer) - Packet sorting now defaults to off. - The FTP analysis now attempts to track the current directory (Ruoming Pang). - A number of scan detection additions have been added (Scott Campbell): activate_landmine_check = F if T, then access to more than landmine_thresh_trigger addresses in landmine_address (a set of addresses) constitutes a scan activate_priv_port_check = T if T, then inbound access to more than priv_scan_trigger privileged ports (i.e., ports < 1024) is considered a port scan. You can exclude particular services from the count via troll_skip_service, which defaults to { smtp, ftp, ssh, 20/tcp, http }. - The SMTP analysis now includes a new alert, HotEmailRecipient, which is triggered by mail sent to any of the addresses specified in the pattern hot_recipients (which defaults to a bare /@/). - The new built-in cat_string_array_n() is like cat_string_array() except it takes starting and ending indices as additional arguments (Ruoming Pang). - The new built-in sort_string_array() takes a string array and returns a sorted version (Ruoming Pang). It currently uses an N^2 algorithm so shouldn't be used on large arrays. - The new built-in subst() substitutes all instances of a given pattern in a given string with another string (Ruoming Pang). For example, subst("fooboo", /oo/, "xxx") returns "fxxxbxxx". - The new built-in cut_tail() trims characters from the end of a string (Ruoming Pang). For example, cut_tail("fooboo", 2) returns "foob". - sub_bytes() can now take a negative argument, which is with respect to the end of the string rather than the beginning (Ruoming Pang). - The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string (Ruoming Pang). The HMAC secret key is generated from available entropy when Bro starts up, or it can be specified for repeatability using the new -K flag. - The new built-in split_all() is like split() except that the returned array also includes the parts of the string that match the pattern used for the splitting (Ruoming Pang). - The new built-in split_n() splits up to a given number of instances, optionally returning the parts matching the split pattern (Ruoming Pang). - The new built-in split_complete() is the most general of the split functions (the others are written in terms of it). It splits a given string with separators that appear in either a given pattern or a given set of strings, optionally including the text matching the separators in its return value, and optionally limiting the number of matches to a specified maximum. (Ruoming Pang) - The new built-in to_string_literal() returns an escaped version of a string suitable to feeding into Bro's parser. For example, calling it on "foo\nbar" (where '\n' is an embedded newline) returns "foo\x0abar". (Ruoming Pang) - subst_substring() has been renamed subst_string (Ruoming Pang). - unescape_URI() no longer takes the connection as an argument (Ruoming Pang). - config.guess and config.sub updated - String escape expansion has been refined (Ruoming Pang) and some bugs fixed. It now supports a format that's consistent with Bro string literals. - Scanning of octal and hex escape sequences now stops after 3 or 2 characters, respectively. For example, "\0007" now expands to a NUL followed by the character '7'. (Ruoming Pang) - Bug fixes for handling of bare and 's when expected (Ruoming Pang), and associated "weird" events. - A bug in signature matching reassembly has been fixed (Robin Sommer). - A bug in reporting "bad connection size" for connection sizes > 2GB has been fixed. - A bug in computing sizes for large partial connections has been fixed. - A bug in delayed generation of connection closed events has been fixed. - A framework has been added for compression of some "weird" events to only be generated once-per-connection (Ruoming Pang). - Some of the "weird"'s generated by the HTTP analyzer have been regularized. - Some memory management fixes. - A performance problem in allocating List objects was fixed. - The copyright dates have been updated (Craig Leres). 0.8a22 Wed Jan 15 16:47:18 PST 2003 - There is now a "conditional compilation" feature similar to C's preprocessor (contributed by Robin Sommer). "@if (expr)" will include the remaining input lines up to "@endif" only if the given expression evaluates to true. "@ifdef (id)" will do so only if the given identifier has been defined, and "@ifndef (id)" only if it has not been defined. There's currently no "@else" directive, and conditionals cannot be nested. Both of these will be added in the future. - New built-in functions (contributed by Ruoming Pang): parse_dotted_addr takes a string in A1.A2.A3.A4 form an returns a corresponding "addr" value. unescape_URI takes a URI that possibly has embedded escape sequences ("%61" for 'a') and expands the sequences. - The URIs in HTTP requests are no longer automaticaly escaped. Instead, they need to be manually escaped using unescape_URI. As this is likely error-prone (users forgetting to do so), the plan is to change the calling sequence of http_request in the near future so that *both* the escaped and the unescaped URI are passed in. - A number of g++ 3.0 porting tweaks have been added (thanks to Sean Irvine). - The term "rule" has been systematically changed to "signature" (Robin Sommer). - The functionality of monitoring packet drops has been moved into its own policy script, netstats.bro (Robin Sommer). - A number of rewriter and location bugs have been fixed. - Some missing HTTP "weird"'s have now been included in weird.bro. 0.8a21 Thu Nov 28 23:31:38 PST 2002 - A new mechanism will reorder packets within a window of packet_sort_window if due to their sequence numbers they appear to be out of order. This can help a great deal when reading from multiple NICs. (Contributed by Ruoming Pang.) - A bug in regular-expression matching in which for example "2a1" == /[0-9]+/ would evaluate as true was fixed (Ruoming Pang). - There's now a rewriter/anonymizer for FTP (Ruoming Pang). - The rewriter/transformation framework now supports a notion of delaying transformation decisions until later (Ruoming Pang). - An incompatibility with bison 1.75 has been identified. The problem is that bison 1.75 generates incomplete location information for empty grammar productions (those for which the RHS is empty). This will be fixed soon. - Some bugs in the signature engine have been fixed (Robin Sommer). - The sources no longer use the not-fully-portable hash_map template (Umesh Shankar). - Some bugs with the debugger getting confused about source line number locations, and also with the -t option, have been fixed (Umesh Shankar). - If a content gap occurs wholly inside an HTTP entity, then the analyzer will skip over the entity and continue processing any subseqeuent HTTP items, rather than giving up on processing the connection (Ruoming Pang). - The following new built-in functions have been contributed by Ruoming Pang: function cat_string_array(a: string_array): string function split_all(str: string, re: pattern): string_array function strstr(big: string, little: string): count function subst_substring(s: string, from: string, to: string): string function int_to_count(n: int): count function fmt_ftp_port(a: addr, p: port): string 0.8a20 Sun Nov 17 20:09:31 PST 2002 - This is the first "public" Bro release (one accessible directly via the Web). The README has been correspondingly updated. - The user manual has been significantly expanded, and source for it is now included in the release. - Some "active mapping" bug fixes (contributed by Umesh Shankar). - The configuration script now checks for the presence of the necessary BIND libraries/headers, and also for bogus --enable options (contributed by Craig Leres). - backdoor.bro now includes a tcpdump filter for detecting (some) KaZaA traffic. - http-reply.bro now tracks the sizes of request/response entities. - http-request.bro now treats an URL accessing variants of /etc/netconfig as sensitive. 0.8a18 Sun Oct 27 15:28:23 PST 2002 - Improvements to the performance of the SMTP analyzer (Ruoming Pang). A new function, skip_smtp_data, skips over the content of an SMTP message. - If you're doing trace rewriting and specify -w but don't specify -A, then the rewritten traffic goes to the -w save file, along with the usual non-transformed traffic (Ruoming Pang). If a connection is being transformed (which is specified by the new rewriting_smtp_trace and rewriting_http_trace globals), then only its transformed packets are written to the -w file; not the original packets. This allows a potentially large degree of trace compression. There's also a new variable omit_rewrite_place_holder, which directs that when rewriting a trace, packets that are placeholders for now-missing packets (because you've excised content) are *not* generated. This diminishes the timing fidelity of the transformed trace, but also can save a large amount of space. - SMTP relay analysis is now standalone from regular SMTP analysis (Ruoming Pang). - Some memory management and error propagation fixes (Ruoming Pang and Vern Paxson). 0.8a16 Wed Oct 23 23:48:40 PDT 2002 - "--enable-brov6" was broken by some recent changes. It now works again. - Some "make distclean" tweaks. - Error checking for "redef enum". 0.8a15 Tue Oct 22 00:02:51 PDT 2002 - Fixed Makefile bug which left out wide-spread dependencies on event.bif.netvar_h. 0.8a14 Mon Oct 21 01:16:46 PDT 2002 - The "add" statement has been implemented. This allows you to add elements to sets. A bunch of policy scripts that used to use "table [] of bool"'s for this purpose have been modified to instead use sets. - You can now extend the elements of an enum using "redef enum foo += { ... }" where the additional names are listed in the braces (contributed by Umesh Shankar). A number of policy scripts have been tweaked to use this for the (fairly) new Alert enum. This allows you to create new Alert's without alert.bro having to know about them. - Some bugs in identifying error locations have been fixed. - -A now supports anonymizing IP addresses (contributed by Ruoming Pang). This includes four new functions: preserve_prefix, preserve_subnet, and preserve_net (which direct that the corresponding prefix/net/subnet not be anonymized) and anonymize_addr (which returns the anonymized version of the given address. - Some bugs in HTTP -A have been fixed (thanks to Ruoming Pang). - The beginnings of support for CIDR prefixes has been added by Ruoming Pang. - Porting tweaks (use of map's rather than hash_map's, courtesy of Umesh Shankar; libedit comments fixed). - http-detail.bro has gone away. - Some more copyright and $Header -> $Id fixes. - There is now a function string_cat() which concatenates two strings and returns the result (contributed by Ruoming Pang). This function is useful in place of the regular cat() because the latter will expand escape sequences etc. It probably shouldn't, and that may change in the future. 0.8a11 Sun Oct 13 10:53:07 PDT 2002 - The framework for defining built-in functions has been extended so it's also now the way to specify the interface to event handlers. See event.bif. (Contributed by Ruoming Pang) - A new policy script, http-abstract.bro, can be loaded when doing HTTP transformation (via http-rewriter.bro), contributed by Ruoming Pang. It trims HTTP text responses to the first "http_abstract_max_length" bytes. - A new built-in, skip_http_entity_data, specifies that the entity associated with the current HTTP request/response should be skipped (Ruoming Pang). - More changes have been made to the mechanisms for tracking source file locations in order to pinpoint error messages. If you encounter problems, please let me know. - If you try to configure but your resolve library lacks res_mkquery(), this is now a fatal error (Craig Leres). Ideally, someone will modify DNS_Mgr.cc to work even if only blocking DNS is available ... - In most (not all, yet) of the sources, the copyright notices have been updated, $Header has been changed to $Id, and config.h is included (Craig Leres). 0.8a10 Tue Oct 8 16:05:42 PDT 2002 - The way that Bro tracks the source-file location associated with different objects has been tweaked, with a resulting savings of about 10% in Bro's memory use and a smidgen of CPU time, too. - Built-in functions now are better about identifying source-file locations associated with errors. - The http.$BRO_ID log file format has changed to no longer track individual request streams. (So it is now closer to how it used to operate in 0.7.) - The autoconf setup has been tweaked so that you use: --enable-brov6 enable IPV6 processing --enable-activemapping enable active mapping processing --enable-expire-dfa-states enable DFA state expiration rather than --with-XXX as was the case in 0.8a9. 0.8a9 Mon Oct 7 10:15:12 PDT 2002 - A bunch of configuration/autoconf/portability tweaks (Craig Leres). These include compiling under Solaris. When running ../configure, you can now specify: --with-brov6 enable IPV6 processing --with-activemapping enable active mapping processing --with-expire-dfa-states enable DFA state expiration You no longer set these via editing the Makefile. - Some bugs with HTTP 1.0 keep-alive connections fixed (Ruoming Pang). - The "hf" suite of utilities is no longer distributed in aux/. Instead, get it directly from ftp://ftp.ee.lbl.gov/hf.tar.gz. - bro_logchk has been renamed bro-logchk.pl and is now distributed in aux/ (it was supposed to be before, but was inadvertantly left out of the list of distribution files). 0.8a7 Fri Oct 4 22:24:30 PDT 2002 - HTTP reply analysis has been split into a number of policy script files: http-body.bro http-detail.bro http-entity.bro http-event.bro http-header.bro http-reply.bro http-rewriter.bro so you can readily trade off how detailed the HTTP processing is vs. the CPU/memory it consumes (contributed by Ruoming Pang). - Bro now generates login_prompt events when the $TTYPROMPT environment variable is passed during a login session, in order to detect the recently publicized Solaris remote /bin/login exploit. - Ruoming Pang has extended the framework for defining "rewriter" functions to now also serve as the way to define general built-in functions. - bro.init has been rearranged to have a more logical structure (courtesy of Ruoming Pang). - Craig Leres contributed a number of portability & autoconf tweaks. - Craig Leres has extended nb_dns.{h,c} to support IPv6 DNS lookups. Bro does not yet take advantage of these extensions. - The beginnings of portability to gcc 3.2 were added. There unfortunately is more work to do here! - The README has finally been updated to have more correspondence with the 0.8 release. 0.8a6 Wed Oct 2 18:58:12 PDT 2002 - Upgrade to autoconf 2.53 (Craig Leres). 0.8a5 Tue Oct 1 19:04:53 PDT 2002 - The regular expression matcher how has a mechanism to stop scanning when no further match is possible (Robin Sommer). If you find problems with regular expression matching, especially if you're using EXPIRE_DFA_STATES, please let us know! - Rule/signature files are now searched for using $BROPATH (Robin Sommer). In addition, you can define a list of signature files to incorporate using the new global "signature_files". For example, redef signature_files += "web-stuff" will look for web-stuff and web-stuff.sig along $BROPATH. - The tcp_deliver_undelivered variable is now tcp_match_undelivered and only applies to delivering otherwise-undelivered data to the signature matcher; such data is *not* delivered to any associated analyzer. (Robin Sommer) - The framework for tracking version numbers now allows negative as well as positive versions (Robin Sommer). 0.8a4 Tue Oct 1 15:54:58 PDT 2002 - Support for extracting the contents of HTTP replies (and POST's), and for transforming/anonymizing HTTP traffic, contributed by Ruoming Pang. - Some minor internal tweaks to the timer management to help track patterns of timer expiration. 0.8a3 Mon Sep 23 22:48:07 PDT 2002 - HTTP reply handling refined in policy scripts. - New built-in functions to_int(), to_count() convert strings to integers/counts. - Bug fixes for DNS_Mgr and IPv6. - AckAboveHole alerts now ignored (just written to alert.$BRO_ID) by default. - Packets belong to ignored connections (for example, partial connections if partial_connection_ok is false) are no longer recorded to the save file. - Some minor formatting/naming tweaks. 0.8a1 Sat Sep 21 22:09:23 PDT 2002 - IPv6 support enabled if you build using -DBROv6. Deficiencies: Bro doesn't yet look up hostnames for AAAA records; no handling of extension headers (if you have traces of these, please send them to me!); no handling of FTP PORT/PASV w/ IPv6 addresses (again, if you have traces, please send them!); DNS analyzer doesn't understand AAAA yet (again, please send me traces!); you have to change the capture_filter line in tcp.bro (as indicated in the script) in order to process TCP traffic, due to deficiencies in libpcap's support for IPv6. - Bro is migrating towards a more structured way of handling log messages / alerts. Analyzers now @load alert.bro, which has a function ALERT() for processing alerts. Soon this function will provide a variety of filtering/processing hooks; expect changes. - Bro now has an HTTP response analyzer (contributed by Ruoming Pang). The HTTP policy scripts have been split up into http.bro (just general definitions), http-request.bro (handles requests; loaded by http.bro), http-reply.bro (handles replies; you need to explicitly load this), and http-detail.bro (handles individual headers). http-reply.bro will be undergoing some significant reworking in the near future; probably the scripts will be merged back into a single http.bro plus http-detail.bro. - ssl-worm.bro contains a prototype policy script for detecting the Scalper SSL worm (contributed by Robin Sommer). It uses the signature file sig.ex.ssl-worm.bro. If someone has traces of Scalper in action to send us, that would be great. - A new policy script, contents.bro, extracts the contents of each Bro connection into its own pair of files (one file for each direction). Use in conjunction with -f or discarder_XXX() to extract specific connections. - A new built-in function, strcmp(), returns the usual comparison between two strings (contributed by Robin Sommer). - A new event, content_gap(), is generated when Bro detects that it is forced to skip over data in a reconstructed TCP stream because it is missing from the packet input. - BIND8 is no longer included with the distribution. If this causes you problems, let me know. - aux/scripts/bro_logchk is a Perl script for parsing Bro HTTP & FTP logs (contributed by Jim Barlow). - You can now compare addresses to see which is larger. a < b means that in network order, the octets making up 'a' are ordered before those for 'b'. E.g., 128.2.3.4 < 128.2.3.5 < 129.0.0.1. Note that IPv4 addresses are all < IPv6 addresses (other than IPv4 addresses that are embedded in IPv6 addresses, e.g., ::128.2.3.4 < 128.2.3.5). - Serious bug in TCP option handling fixed. - Some bugs in CRLF handling fixed (courtesy Ruoming Pang). - Bug in the implementation of &optional fixed. - Bug in computing memory statistics when not reading packets (from an interface or the trace file) fixed. - You can now include a trailing comma after the last item in an "enum" enumeration list. - port-name.bro now maps 389/tcp to "ldap". - A bug has been fixed in loading files multiple times v0.7a175 Thu Aug 29 21:14:34 PDT 2002 - bro -s reads in a *signature* file to search for regular expressions in packet payloads or TCP byte streams (written by Robin Sommer). See policy/rules.bro for an example of a policy script for processing the matches. Note that this feature is experimental, and will be evolving in the near future. - The python script "snort2bro" reads in Snort signatures and translates them into Bro signature rules, suitable for processing using -s. An example of its operation is seen by running python snort2bro < sig.ex.web-rules.snort which, after reading in sig.ex.classification.config and sig.ex.reference.config, generates the output given in sig.ex.web-rules.bro, which is suitable to use as input to bro -s. - bro -d invokes a gdb-like debugger (written by Umesh Shankar). You can set breakpoints and watchpoints, examine tracebacks, print Bro expressions, and the like. Type "help" for on-line help. - bro -t turns on tracing of the policy script execution, written to the given file. - Bro now includes an SMTP analyzer, which includes processing MIME message bodies (written by Ruoming Pang). See smtp.bro and mime.bro for related policy scripts. smtp.bro includes several experimental techniques for detecting mail relaying. - You can now define enumerated types, such as type scanner_type: enum { SCANNER_STEALTH, SCANNER_HIGH_SPEED, SCANNER_AMBIGUOUS, }; Enumerated types can be compared for equality with one another, and used as table indices, but cannot be converted to/from integers. - bro -A invokes an experimental, general trace transformation/ anonymization framework (written by Ruoming Pang) which writes a modified tcpdump trace file from the input (which can be the network or another trace file) with potentially extensive modifications to the recorded packets. Transformers are built from .rw files (currently, {finger,ftp,ident,smtp}.rw), which are processed by the utility "rwcl" to generate both event engine analyzer components and rewriter policy scripts (for example, after configuring and building Bro, you'll find the scripts policy/{finger,ftp,ident,smtp}.rw.bro). See policy/smtp-rewriter.bro for an example of a policy script that performs transformation/anonymization. - New built-ins: split(s: string, p: pattern): string_array; takes a string and splits it into pieces at each occurrence of the regular expression pattern p. (The functionality is like that in awk.) It returns a string_array, which is a table[count] of string that is indexed starting at 1, giving the different pieces. For example, split("foobar", /o/) returns a 3-element table, for which [1] is the string "f", [2] is the string "" (empty), and [3] is the string "bar". split1(s: string, p: pattern): string_array; split1() does the same thing as split(), but only performs splitting at the first occurrence, so it returns either a one-element table (if the pattern doesn't appear in the string) or a two-element table. split1("foobar", /o/) returns a 2-element table for which [1] is "f" and [2] is "obar". md5_hash(s: string): string returns (in human-readable form) the MD5 hash of a given string. So, for example, md5_hash("foobar") yields "3858f62230ac3c915f300c664312c63f". to_addr(s: string): addr takes a string representing an address in "dotted quad" format and returns the correponding "addr" value. set_buf(f: file, buffered: bool) sets the given file to have its writes buffered or unbuffered depending on the value of "buffered". It does not return a value. connection_exists: function(c: conn_id): bool returns T if the given connection identifier corresponds to a currently instantiated connection (one for which the event engine has state), F otherwise. lookup_connection(c: conn_id): connection returns the "connection" record associated with the given connection identifier, or a fatal run-time error if there isn't one. set_inactivity_timeout(c: conn_id, timeout: interval): interval sets the inactivity timeout for the given connection to the given interval, returning the old interval. If the interval is non-zero, then when no packets have been processed for a connection after that much time has elapsed, the connection is deleted, and an "inactivity_timeout" event generated. get_matcher_stats(): matcher_stats used for gathering statistics about the signature matcher rewriting_trace(): bool returns T if -A was specified (anonymize/rewrite a trace), F otherwise. - New events: connection_state_remove(c: connection); Invoked when the event engine has removed the connection from its state. connection_SYN_packet(c: connection, pkt: SYN_packet); Invoked for each SYN/SYN-ACK packet. connection_timeout(c: connection); Invoked when the event engine times out a connection - for example, because the originator sent a SYN that was never answered, so the connection was never established. connection_reused: event(c: connection); Invoked when the event engine decides that a new SYN for an existing connection reflects a reuse of the connection four-tuple, rather than belonging to the existing connection. - New globals: const ignore_checksums = F &redef; If true, then the event engine does not verify checksums (and hence will not discard packets with bad checksums). const tcp_deliver_undelivered = F &redef; If true, then when the event engine closes a connection, if that connection has a chunk of data not yet delivered to its analyzer (which only happens if the data is above a sequence hole, indicating either a packet filter glitch or a protocol botch), then the undelivered data will at that point be delivered to the connection's analyzer. const tcp_reassembler_ports_orig: set[port] = {} &redef; const tcp_reassembler_ports_resp: set[port] = {} &redef; Sets of ports for which, if a connection has the corresponding originator/responder port, then the event engine will reassemble the byte stream of the connection. Normally, the event engine reassembles byte streams for any connection for which there's an analyzer, and otherwise doesn't. These variables can be used to force reassembly for the originator or responder side (respectively) of connections for which there isn't otherwise an analyzer. This is useful when doing signature matching on reassembled byte streams, for protocols that are not otherwise analyzed by the event engine. const table_expire_interval = 1 min &redef; How often to check table entries to see whether they've expired (see &read_expire, etc., below). const requires_trace_commitment = F; If true, then when rewriting/anonymizing traces, nothing will actually be written to the edited trace file unless you call: rewrite_commit_trace(c: connection, commit: bool, future: bool) If "future" is true, then future rewritten packets will be automatically commited; otherwise, writing them to the trace file requires another explicit rewrite_commit_trace() call. const inactivity_timeout = 0 secs &redef; As noted above, when a connection becomes inactive, time it out after this interval. If 0 secs, then don't time it out. - An SSH analyzer extracts SSH client/server version information. See ssh.bro for the related policy script. - There's now a (very) simple TFTP analyzer available in tftp.bro. - You can now set the global "frag_timeout" to an interval which controls how long fragments are kept before discarding them (contributed by Ashley Thomas). If you don't set the global, or set it to 0.0 sec, then fragments are kept around indefinitely. - An implementation of an experimental anti-evasion technique, "active mapping", has been written by Umesh Shankar. It is not yet ready for general use, and isn't compiled in unless -DACTIVE_MAPPING. - Four new attributes can now be associated with tables (implemented by Robin Sommer): &read_expire, &write_expire, and &create_expire will delete table entries after a given interval has elapsed since the table entry was last read, written, or created. For example: global a: table[addr] of count &default=0 &create_expire = 5 sec; will delete each entry added to it 5 seconds after the entry was added, regardless of subsequent read/write activity to the element. &expire_func allows you to associate a function with the table such that whenever an entry expires, the function is invoked. It's passed the value of the table entry (not the index - perhaps this should be changed), and returns an "interval" value. If the interval is <= 0.0 seconds, then the table entry is immediately deleted. Otherwise, it is deleted after the given interval has elapsed. - When listing multiple attributes, you no longer separate them with commas. For example, if you used to have: global a: table[string] of count &default=0, &redef; you now need to use: global a: table[string] of count &default=0 &redef; - You can now construct records using [$field1 = , $field2 = , ...] Such record values can be assigned to other records providing that the target value's type includes all of the fields (same name and type) present in the record value, and that any missing fields have the &optional or &default attribute (see next item). You can also include a record value inside the record constructor, and all of its fields will be included in the constructed record value. - Record fields can now be annotated with &optional, to indicate that the field needn't be present, or &default, which indicates a default value to provide if the field is missing. - You can query as to whether a record has a value for a given field using the new "?$" operator. So for example: type my_rec: record { num: count &default = 0; msg: string; # mandatory, since no &optional/&default }; global r: my_rec; r = [$msg = "hello"]; print r?$num, r?$msg, r$num; will print "F, T, 0," because even though 'r' has a default value for $num (which shows up when printing r$num), that field is missing, hence r?$num is F. - An experimental scheme has been added (by Umesh Shankar) for managing general attributes associated either with all values ("global attributes") or particular particular values. This scheme is likely to change in the near future, and hence isn't explained here further. - The DNS analysis now includes ADDL and AUTH records, and much richer policy script analysis (per policy/dns.bro). - You can now "redef" a function or event handler to override its previous definition. For a function, this looks like: redef log_hook = function(msg: string): bool { ... } For an event handler, it's just the usual definition preceded by "redef. For example, redef event ack_above_hole(c: connection) { } would replace the default ack_above_hole handler with one that does nothing. - HTTP server and HTTP proxy backdoor detectors have been added, generating http_signature_found and http_proxy_signature_found, respectively (contributed by Ruoming Pang). - A KaZaA backdoor detector has been added, which generates kazaa_signature_found for likely KaZaA connections. - The new policy scripts flag-irc.bro and flag-warez.bro provide hooks for defining site policies for detecting IRC and access to warez. - portmapper.bro now tracks the services it sees, and the names are used in connection summaries rather than generic services like port-656. - bro -C (or redef'ing the "ignore_checksums" global to T) instructs Bro to ignore any checksum errors and go ahead and analyze such packets. - The (trivial) policy script print-globals.bro dumps out all of the policy script global variables and the amount of memory they consume. - The policy script code-red.bro has been renamed worm.bro and generalized to detect Nimda as well as Code Red 1 & 2. - A bunch of additional default sensitive URIs have been added to http.bro. http.bro also now doesn't report worm-related URIs. - A bunch of less common portnames were removed from port-names.bro. - Empty regular expressions are now allowed. - The finger_request event now has a third parameter, the additional text after the username. - More systematic handling of NULs and CRLF by the event engine. - Hex escape sequences now must have exactly two hexadecimal characters. - FYI - work has begun on significantly altering the way that policy scripts generate alerts. - Work has begun (by Robin Sommer) on a general framework for tracking client/server versions. See software.bro. - Work has begun on a NETBIOS analyzer (see NetbiosSSN.cc). Contributions (e.g., finishing it :-) welcome. - Work has begun on migrating the internals to process IPv6 in addition to IPv4. - A number of bug fixes, leaks, and memory allocation lint tweaks. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.7a90 Thu Sep 06 00:50:43 PDT 2001 - Better state management, especially for use in high-volume sites. There are now a number of ways to track the resources consumed by Bro while it runs, and to reduce the amount of memory Bro consumes. Adding @load print-resources will log a summary of the resources consumed by Bro when it exits, and @load reduce-memory will change a number of default values in order to significantly diminish the amount of memory Bro requires. Other ways to lower the required memory discussed in the next three items. - The global "maintain_http_sessions" now controls whether http.bro tracks multiple HTTP connections between the same hosts as a single HTTP session. Doing so costs a considerable amount of state (so set to F for reducing the memory impact of HTTP analysis). - The global "suppress_scan_checks", if true, turns off address and port scan detection. (You can't achieve this by simply not @load'ing scan.bro, because it's loaded by some of the default policy scripts.) Turning it off can save a lot of memory. - Note, the ident.bro is also expensive in terms of state-holding. - New library functions: - resource_usage() returns a record detailing real time, CPU time, memory, other getrusage info, and the current and maximum number of TCP/UDP/ICMP connections, and timers and fragments. - val_size() returns the size in bytes needed to represent a given value (which can be a record, a table, or a simple constant, for example). It's not fully accurate but is in the right ballpark. - global_sizes() return a table mapping every global variable to its size (per val_size()). Useful for tracking which ones are growing large over time. - You can now control a number of timers related to Bro's connection management. Setting them to lower values generally decreases the amount of state Bro has to keep (see reduce-memory.bro), though this can also make it easier for an attacker to evade detection: tcp_SYN_timeout: interval Controls how long Bro waits after seeing the beginning of a connection (whether due to a SYN or not; the timer is misnamed) before checking whether it elicited any reply. tcp_session_timer: interval After a connection has closed, wait this long for further activity before checking whether to time out its state. tcp_connection_linger: interval When checking a closed connection for further activity, Bro should consider it inactive if there hasn't been any for this long. It also complains if the connection is reused before this much time has elapsed. tcp_attempt_delay: interval Bro waits this long upon seeing an initial SYN before timing out the connection attempt. tcp_close_delay: interval Upon seeing a normal connection close, Bro flushes state after this much time. tcp_reset_delay: interval Upon seeing a RST, Bro flushes state after this much time. tcp_partial_close_delay: interval Bro generates a connection_partial_close event this much time after one half of a partial connection closes, assuming there has been no subsequent activity. non_analyzed_lifetime: interval If a connection belongs to an application that you aren't analyzing, Bro times it out after this interval. If 0 secs (default), then it doesn't time it out. dns_session_timeout: interval ntp_session_timeout: interval rpc_timeout: interval Bro waits this long before timing out a DNS/NTP/RPC request. max_timer_expires: count The maximum number of timers to expire after processing each new packet. The value trades off spreading out the timer expiration load with possibly having to hold state longer. A value of 0 means "process all expired timers with each new packet". - Two new flags control other facets of Bro's connection management, and thus state-holding: partial_connection_ok: bool Whether Bro should analyze connections for which it doesn't see the beginning, only the middle. This can be very expensive to do in the face of stealth-scanning, which looks like a bunch of partial connections. Note, the HTTP analyzer has been modified to now always skip partial connections. This should instead be user controllable. tcp_SYN_ack_ok: bool If true, Bro instantiates connection state when it sees a SYN ack but not the initial SYN (even if partial_connection_ok is false). The intent behind this knob (which is not well tested) is to allow you to filter out initial SYNs and only react to SYN acks. This keeps Bro from holding state during SYN scans and SYN flooding, except for when the destination responds. - Some other miscellaneous thresholds that you can now modify from your policy script: heartbeat_interval: count How often to generate net_stats_update() events. This timer really isn't needed any more, since you can use "schedule" to achieve the same effect. tcp_storm_thresh: count If Bro sees this many FINs/RSTs in a row, it flags them as a "storm". tcp_storm_interarrival_thresh: interval The FINs/RSTs must come with this much time or less between them. - The state management for line-oriented applications like HTTP requests has been improved. - The HTTP analyzer now expands %hex sequences. If anyone has a Unicode expander to contribute, that'd be terrific. - The Code Red detection is more robust (fewer false positives). - A new redefinable variable, skip_services, lists applications that should not be analyzed for purposes of detecting scans. (Default: ident) The point of having this (poorly named) hook is so that code-red.bro can add "http" to it, to avoid reporting the zillions of Code Red scans that a site can see. - Bro now matches regular expressions using lazy DFA evaluation. The upshot of this is (1) it no longer maintains .bro-RE-cache.v1, (2) it starts up fast regardless of whether you've added new regular expressions, (3) you can afford to add lots of regular expressions, and (4) it's actually a bit faster. - The list of "hot_ids" has been expanded with a number of other common root-privileged accounts. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.7a61 Fri Apr 06 14:51:47 PDT 2001 - An NTP analyzer has been added. See policy/ntp.bro for a corresponding policy file that detects the newly discovered NTP remote buffer overflow attack. - example-attacks/ is a new directory that contains trace files of attacks. Currently, there are just two to play with: bro -r example-attacks/ftp-site-exec.trace mt will run on a trace of a "site exec" overflow attack, and bro -r example-attacks/ntp-attack.trace mt ntp will run on an example of the NTP overflow. - The doc/ directory includes the postscript and HTML versions of the first draft of the Bro manual. - A new policy file, icmp.bro, has preliminary (and only partially developed) policy for analyzing ICMP. - The file libpcap.bufsize.patch includes the patch necessary on some systems to increase the maximum libpcap buffer size. - You can now use anonymous functions in &default expressions, so for example you can do: global foo: table[count] of string = { [1] = "1st", [2] = "2nd", [3] = "3rd", } &default = function(n: count): string { return fmt("%dth", n); }; and then referring to foo[5] will yield "5th". - There's now a "for" statement to iterate over the indices of a table or the members of a set: for ( i in foo ) for the above "foo" will iterate with i assigned to 1, 2, and 3; *but not in general in that order*. - The function contains_string() has been removed, and now you can instead use an expression like "bar" in "foobar" which will yield T. - The scan detection now has a mechanism for attempting to detect SYN flooding backscatter and flagging it as different from a stealth scan. - New event handlers: new_connection_contents() like new_connection(), but reassembles the stream so you can use set_content_files() to write it to a file udp_session_done() invoked when a UDP session (which is defined on a per-protocol basis; currently only for NTP) finishes. ntp_message() invoked for each NTP message - UDP processing now does accounting for detecting scans. - UDP processing now tracks numbers of requests/replies for sessions that support that notion. The connections are annotated by udp_session_done() with "[m,n]" for "m" requests and "n" replies, providing either m or n > 1. - New variable accessible from policy: watchdog_interval how often the watchdog should check for whether Bro is making internal progress - A bunch of functions no longer have a first argument of the current time; get it instead from network_time() if you need it: authentication_accepted authentication_rejected conn_weird conn_weird_addl flow_weird net_weird - A bunch of functions now return bool rather than int values: set_contents_file set_login_state set_record_packets skip_further_processing - The variable "hot_dests" has been renamed to "hot_dsts". - 111/tcp is now identified as "portmap" rather than "rpc". - Connections flagged as hot for some types of characteristics are now annotated with the reason associated with the decision. (I think a lot more of this is needed.) - Portmapper dumps are annotated with the results of the mapping. This will be streamlined in the future. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.7a48 Wed Sep 13 14:37:30 PDT 2000 - Changes between this release and v0.6 missing :-( -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.6 Wed Jul 21 17:02:50 PDT 1999 - Support for regular expressions added. You specify lex-style regular expressions between '/'s, for example "/\/etc\/(passwd|shadow)/" has the type "pattern" and matches /etc/passwd or /etc/shadow (the slashes in the pattern need to be escaped or else they'd delimit the end of the pattern). Pattern-matching is via the "in" operator, so for example: if ( filename in /\/etc\/(passwd|shadow)/ ) sensitive_file_access(filename); or const sensitive_files = /\/etc\/(passwd|shadow)/; ... if ( filename in sensitive_files ) sensitive_file_access(filename); Presently the "in" operator requires that the entire left-hand side be matched by the pattern. So, for example, if you want to find the string "eggdrop" anywhere inside the string "line", you would use if ( line in /.*eggdrop.*/ ) If you leave off either of the .*'s, then eggdrop will only be matched at the beginning or end of the line. In the future, there will be mechanisms for specifying whether you want to match anywhere in a line, or anchored; accordingly, *the above syntax is subject to change*. Bro compiles regular expressions into DFAs for fast matching. This can take quite a bit of time for complicated patterns. Consequently, it maintains a cache of compiled regular expressions in $HOME/.bro-RE-cache-v1. You can always safely remove this file; Bro will recreate/repopulate it as needed. It does not clean up unused entries from it, so if you change your patterns frequently, you will accumulated lots of old ones and should delete the file to garbage collect them. - An rlogin analysis module has been added and the telnet analysis generalized to generic "login" analysis, with the following events: login_failure(c: connection, user: string, client_user: string, password: string, line: string) Generated on a failed attempt to log in. client_user is the remote user name, if the login is via the rlogin protocol. login_success(c: connection, user: string, client_user: string, password: string, line: string) Generated on a successful attempt to log in. login_input_line(c: connection, line: string) Generated per line of input typed by the user. login_output_line(c: connection, line: string) Generated per line of output generated by the server. login_confused(c: connection, msg: string, line: string) Generated when a login dialog confuses the heuristic analyzer. msg is a tag for the state mismatch that was unexpected, line is the corresponding dialog text. login_confused_text(c: connection, line: string) Once a connection is in the confused state, then this is generated for each subsequent line. login_terminal(c: connection, terminal: string) Generated if the terminal type associated with the connection is seen. login_display(c: connection, display: string) Generated if the display associated with the connection is seen. excessive_line(c: connection) Generated when the connection has produced an excessively long line. login_input_line() and login_output_line() are very powerful for detecting intrusions, when coupled with regular-expression matching. login_terminal() is used to detect backdoors that are triggered by the terminal environment variable. - An ident analysis module has been added (port 113). It generates ident_request, ident_reply, and ident_error events. Port 113 used to be referred to as "auth"; now it's referred to as "ident". - A new type of scan detection has been added, which is triggered by a remote host trying a large number of username/password combinations. See the account_tried() function in scan.bro. - The default search path for .bro files is now .:priv-policy:policy:pub-policy:/usr/local/lib/bro where priv-policy/ is intended for private policy and pub-policy/ for public policy. The Bro alpha distribution ships with a sample set of pub-policy scripts. - New built-ins: system(s: string): int executes the given shell command using system() and returns its status. set_contents_file(c: conn_id, direction: count, f: file) copies connection c's reassembled byte stream in either the originator-to-responder direction (if direction is CONTENTS_ORIG) or the responder-to- originator direction (CONTENTS_RESP) to the file f. reading_live_traffic(): bool returns true if Bro is running on live traffic (read from a network interface), false if it's reading from a save file. mkdir(f: string): bool creates the given directory, returning true if it was able to, false if not. get_orig_seq(c: conn_id): count; returns the highest sequence number sent by the originator of connection c. get_resp_seq(c: conn_id): count; same for c's responder. - Additional new events (other than those related to the new analyzers): new_connection(c: connection) is generated whenever a new connection is seen. partial_connection(c: connection) is generated whenever a new partial connection (one that doesn't begin with a SYN handshake) is seen. pm_bad_port(r: connection, bad_p: count) is generated when a portmapper response contains a bad port number. - Functions, tables and sets can now be assigned. Assignment is made by reference to the underlying object. - Bro no longer looks up identifiers using getservbyname() to see if they should be interpreted as port numbers, since this led to portability problems. Instead, a number of constants are defined in bro.init: bgp, domain, finger, ftp, gopher, http, ident, rlogin, smtp, ssh and telnet, - Bro now supports an arbitrary number of open files (not bound by the system's limit on file descriptors). - There's now a finger_reply event to go with finger_request. - A bunch more RPC service names have been added, thanks to Job de Haas and others. - A bug has been fixed in the watchdog handling that caused it to sometimes expire after a period of network inactivity. - The Bro paper in doc/ has been revised (it isn't quite up-to-date, but considerably closer than the USENIX version). - There has been a large amount of reworking of the internals, both to Bro itself and in the policy scripts. If you find something you're wondering about, feel free to send me mail asking about it. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.5 Sun Oct 4 00:19:35 PDT 1998 - Added Linux support. - Major autoconf changes. - Some tweaks to suppress g++ warnings. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.4 Fri Sep 11 00:08:04 PDT 1998 - The new aux/ directory has some utility programs and scripts. See aux/README for details. - Documentation (though exceedingly limited) describing the connection log summaries generated by policy/tcp.bro now available in doc/conn-logs. - The Telnet username recognition code has been heavily reworked and is now significantly more robust. - A new policy file, policy/scan.bro, detects port and address scanning. - Bro now detects the "Land" attack (a trivial addition to its spoof detection). - The distribution now comes with BIND 8.1.2. - A new flavor of "weird" event has been added, flow_weird, for unusual events associated with flows (i.e., a coarser granularity than TCP or UDP connections). - Bro now reassembles fragments, checking for overlaps and consistency. Associated with fragments are the following flow_weird's: excessively_large_fragment excessively_small_fragment fragment_inconsistency fragment_overlap fragment_protocol_inconsistency fragment_size_inconsistency fragment_with_DF incompletely_captured_fragment - The TCP stream reassembly code was rewritten to share functionality with the new fragment reassembly code. - If a handler for it is present, then Bro will generate "packet_contents" events containing the contents of each packet it receives. This is just for experimenting with, and, in particular, this event does *not* reflect TCP stream reassembly. - The handling of "conn_weird" events in tcp.bro now has more options. In the weird_action table, WEIRD_LOG_ALWAYS means that the weird event should always be logged; WEIRD_LOG_PER_CONN means that it should be logged once per connection; and WEIRD_LOG_PER_ORIG that it should be launched once per originator IP address. - The example hot.bro now includes provisions for flagging sensitive inbound and outbound services. - Bro now limits the number of events processed when draining the event queue after processing a packet. This makes Bro less prone to packet loss during high-speed scans. - The line-oriented TCP endpoint class was split into TCP_EndpointLine, which supports the line-oriented functionality, and TCP_NVT (derived from TCP_EndpointLine), which implements the Network Virtual Terminal used by Telnet and FTP. - The TCP_NVT class now understands the Telnet Environment option. - Escape sequences are now '\' followed by 1 or more octal digits, instead of excatly three octal digits (which is error prone). - If the watchdog timer expires, it now reports the number of events processed in the current batch of packets, as well as other timing information. - Bro now should not report packet drops that occur after it has begun to exit (these can occur when draining the pending event queue takes a while). - Bro now detects TCP acknowledgements that occur above a sequence hole, generating an ack_above_hole event. Nominally, this indicates packet filter drops, but in fact some buggy TCPs manage to do this :-(. - Fledgling HTTP support added. An http_request event is generated when a new HTTP request is seen, and http_stats is generated when an HTTP connection terminates, giving (uninteresting to most people) statistics concerning the request(s). A lot more is needed: parsing persistent connections and HTTP replies, for one. This is just a start. A stub for http_reply exists but these events are not presently generated. - Ported to Linux (thanks to Pascal Bouchareine). - A bug in to_lower() and to_upper() was fixed. - The reporting for unexpected FTP connections now more directly identifies the corresponding FTP session. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.3 Fri Feb 13 19:59:39 PST 1998 - Timers are now implemented using calendar queues rather than priority queues (with thanks to Craig Leres). - The new byte_len() built-in returns the length of a string interpreted as a set of bytes (including any NUL bytes, especially the final one for a typical string). - The new sub_bytes() built-in extracts a subset of a string interpreted as a set of bytes (i.e., immune to any embedded NULs). - Fixed bad interaction with the latest version of libpcap that on some BPF systems would cause Bro to exit any time a little bit of time went by without any traffic matching its filter. - A bug with constant-folding of the ?: operator has been fixed. - A new "conn_stats" event delivers statistical analysis (number of packets transmitted, retransmitted, out-of-order, replicated) of each connection. If you define a handler for it, then *no* other TCP processing is done. This was added for off-line analysis of traces with large numbers of connections in them. - Some minor portability tweaks. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v0.2 Sat Oct 25 11:22:37 PDT 1997 - Added autoconf support, thanks to Scott Denton. - Ported to FreeBSD, Solaris. - Fixed a bug in which partial connections were ignored even for protocols (e.g., FTP) that can make use of partial dialogs. - Included BIND version 8 sources directly in the distribution. - Better usage() information (again thanks to Scott), -h and -v flags. - README, CHANGES files created.