SPARTA ISSO

Finished Projects

Security Architecture and Modeling

Intrusion Detection Intercomponent Adaptive Negotiation

April 10, 1999

Prepared by:
Richard Feiertag, Principal Investigator
Lee Benzinger, Sue Rho
TIS Labs at Network Associates Incorporated
3965 Freedom Circle
Santa Clara, CA 95054
(408) 346 - 3789

Stuart Staniford-Chen,
Silicon Defense,
791 Shirley Blvd,
Arcata, CA 95521

Karl Levitt, Mark Heckman, Dave Peticolas
Department of Computer Science,
UC Davis,
Davis, CA 95616

1. Introduction
Intrusion detection systems (IDS) are becoming large and distributed software systems that watch protected hosts and networks around the clock. The IDS technology community is actively working to develop better techniques for collecting and analyzing data in order to handle intrusions. To take advantage of this on-going work, IDS systems should ideally be able to dynamically incorporate new and improved components. The goal of the Intrusion Detection Inter-component Adaptive Negotiation (IDIAN) project is to develop a negotiation protocol that permits intrusion detection components to come to an agreement on each other?s capabilities and needs. The protocol will provide negotiating components with an orderly and efficient means for reaching a common agreement on what information should be generated and processed. Moreover, the protocol will allow the negotiation process to be dynamic, so that the information that is generated and processed can evolve as the IDS evolves or the situation changes.

This report presents requirements for demonstrating how IDS systems can use the negotiation protocol to adapt to a dynamic environment. The negotiation protocol will be based on the Common Intrusion Detection Framework (CIDF) -- a still-evolving, DARPA-initiated effort to define a common set of APIs and protocols for IDS component interoperability. While the CIDF specification is focused on an API for communicating data between ID components, and a common language for expressing that data, these aspects represent only a first step. Another essential aspect is the ability to adapt both within individual components and among different components. Adaptivity is necessary for several reasons:

  • Intruders adapt, so an ID system must be able to change in response,
  • When an ID system detects suspicious behavior, it may need to modify its collection, analysis, and response behavior to focus in on the intrusion,
  • The ID system must adapt to changes in the monitored system,
  • The ID system must be able to adapt to new knowledge about intrusions, such as the discovery of new types of intrusions, and
  • The ID system itself must be able to evolve by addition of new components.

As a first step toward developing the negotiation protocol, we have determined the needs of three important classes of IDS problems in which an adaptive negotiation protocol would be most useful. We have used these classes of problems to drive our analysis of what the requirements should be for IDIAN. We will demonstrate the effectiveness of the protocol in dealing with each class of intrusion detection problem by exhibiting an implemention of the protocol handling representative scenarios. We will have three such scenarios, one for each of the problem classes.

The first problem class deals with the dynamic acquisition of new capabilities by an IDS. An IDS can acquire new capabilities when a new IDS component is activated or when an existing IDS component acquires a new capability. Our scenario shows how an IDS component that acquires a new capability negotiates with existing IDS components to enhance the functioning of the composite system. The second problem class deals with situations where the IDS becomes overloaded, either due to faults within the IDS, or due to a flooding attack from outside. The representative scenario demonstrates how the transmission throughput between an IDS?s sensors and analyzers can be negotiated. The last problem class deals with cost-effective installation of new attack signatures. Our scenario for this class shows how an IDS can dynamically determine if its components are capable of detecting and responding to a new attack signature and, if so, how it can use that information to later respond to an attack while considering tradeoffs between security and cost.

The rest of the report is organized as follows. Section 2 presents background information about the Common Intrusion Detection Framework (CIDF). Section 3 presents general background information on the three classes of IDS adaptation problems. Section 4 describes the representative scenarios. Section 5 presents design requirements on the negotiation protocol for the three scenarios. Section 6 describes modifications necessary to IDS components in order to use the negotiation protocol. Finally, section 7 presents our plans for developing the negotiation protocol and demonstrating its viability using the three scenarios.

2. Background on CIDF

2.1 Overview
The Common Intrusion Detection Framework (CIDF) is an effort by DARPA to develop a common language, protocols, and APIs that would allow DARPA funded intrusion detection prototypes to inter-operate and share information. While more information about CIDF is available elsewhere [Refs], and a thorough exposition would be too lengthy, we will attempt here to give the reader enough of an explanation to understand the rest of the document.

CIDF began in January 1997 as a consensus-based working group of all DARPA IDS researchers who were willing to participate. Non DARPA personnel have also participated at various points. The working group is still in existence and the specifications continue to evolve. This is a difficulty for the IDIAN project in that we do not have a stable target platform to work with. It is also a benefit in that it is possible for us to influence the CIDF specification to meet our needs.

CIDF is defined in several documents that are in various stages of completeness at this time. One document is the architectural overview [ref] that attempts to give an overview of the rest of the specifications and the conceptual model of CIDF. This document is in an early stage of revision and is also somewhat out of date. A second document is the definition of the Common Intrusion Specification Language (CISL) [ref], which is what CIDF entities use to communicate about intrusions. This specification is the longest, has received the most intellectual effort, and is being updated very regularly. A third document defines APIs [ref]. It is incomplete and has not been recently updated. A fourth document defines message protocols and directory infrastructure for CIDF [ref]. The section of this document that defines message protocols is complete, stable, and has been implemented. The section that defines directory infrastructure is incomplete and has been orphaned by the closing of the Open Group Research Institute, where the document was being authored.

Present plans for CIDF involve the preliminary completion of specifications to a level suited for demonstration by June 1999.

2.2 Architectural Assumptions
CIDF makes a number of assumptions about how an IDS is structured. Since the IDIAN project inherits those assumptions, we explain them here.

First, CIDF assumes that the ID system software is divided into components. These components have a persistent identity. New components might be started from time to time, or components might be terminated. It is assumed, however, that the lifetime of a component is generally long compared to the time required to deploy one, and typically longer than the duration of intrusion incidents. Components consist mainly of code, together with some configuration information (with the exception of database components; those potentially store extensive amounts of data).

CIDF components exchange data in the form of GIDOs (generalized intrusion detection objects). GIDOs consist solely of data (not code), and may be ephemeral - existing only as long as it takes to pass from one component to the next. They carry information about possible intrusions and responses to them.

CIDF assumes that components interact in a real-time dataflow model. One type of component is an event generator. Event generators are assumed to passively monitor sources of information about the computational universe and turn it into GIDOs. For example, one kind of event generator might monitor an IP network, turn the packet level information into GIDOs, and then pass those to other components. Another kind of event generator might monitor host audit trails. Event generators pass their output GIDOs to event analyzers. These components examine the incoming GIDOs and draw conclusions about what intrusive activity might be occurring, and create new GIDOs encapsulating these conclusions and possibly prescribing responses.

Response components take in GIDOs that order a response of some kind and carry it out. Typically, event analyzers will generate such GIDOs (for example, instructing a response component to reset a TCP connection). Finally, event databases consume GIDOs and store them in order to be able to answer GIDO queries at a later time.

2.3 The GIDO Format
In this section we give a high-level description of GIDOs. (See [ref] for a more complete description.) A GIDO consists of two components - a fixed format header, and a variable body. (There may possibly be GIDO addenda and signatures that are beyond the scope of this discussion.) The format of the header is as follows:

  • The Version ID denotes which version of CIDF is in use.
  • The Class ID indicates what kind of GIDO this is (a response instruction, a low-level event, a high-priority alarm, etc). This field is intended to support efficient queuing of GIDOs prior to full processing in an application.
  • The Length field gives the length in bytes of the GIDO.
  • The Timestamp is the last time of the events described in the GIDO.
  • The Originator ID is a 16 byte UUID that is unique to each CIDF component.
  • Flags indicate whether signatures, etc., are present.

Following the header is a GIDO body, which is an extensible object. We explain the structure of the body with an example (taken from [ref]).

(InSequence

(Login

(Location

(Time '14:57:36 24 Feb 1998')

)

(Initiator

(HostName 'big.evil.com')

)

(Account

(UserName 'joe')

(RealName 'Joe Cool')

(HostName 'ten.ada.net')

(ReferAs 0x12345678)

)

)

(Delete

(World Unix)

(Location

(HostName 'ten.ada.net')

(Time '14:58:12 24 Feb 1998')

)

(Initiator

(ReferTo 0x12345678)

)

(Source

(AbsoluteFileName '/etc/passwd')

)

)

(Login

(World Unix)

(Outcome

(CIDFReturnCode Failed)

(Comment '/etc/passwd missing')

)

(Location

(Time '15:02:48 24 Feb 1998')

)

(Initiator

(HostName 'small.world.com')

)

(Account

(UserName 'mworth')

(RealName 'Mary Worth')

(HostName 'ten.ada.net')

)

)

)

This GIDO basically says that a user joe logged in from big.evil.com to ten.ada.net. There he deleted the password file, and later a user mworth was unable to login because the password file was missing.

The structure of the GIDO is that of a Lisp-like S-expression; the parse tree of the expression is explicitly delineated with parentheses. Within the GIDO, two kinds of things occur. There is actual data (all of which is shown italicized in the above example). An example is the string ?mworth? near the bottom of the GIDO. Associated with each piece of data is a semantic identifier (SID). In the case of ?mworth?, the SID is UserName. This tells us that the meaning of the string ?mworth? is that it is a user account name. Every piece of data in the GIDO is immediately associated with a SID like this. These SIDS, which directly give the meaning of data are called atom SIDS in CIDF.

The atom SIDS alone, however, are not enough to tell us what is going on. Above this structure in the parse tree are additional SIDS that tell us how the various atoms of data fit together into a larger picture of the action. For example, we can look at the following piece of the GIDO:

(Location

(Time '14:57:36 24 Feb 1998')

)

(Initiator

(HostName 'big.evil.com')

)

(Account

(UserName 'joe')

(RealName 'Joe Cool')

(HostName 'ten.ada.net')

(ReferAs 0x12345678)

)

)

The SID at the head of this expression is Login. This is an example of a verb SID. Verbs are used to describe events that occur in the computational world, in this case the event of someone logging in. Every verb has some role SIDS that may appear beneath it. These provide additional information about the event (beyond the mere fact that a login occurred). For example, the Location role provides information about the time and place of the login (in this case just the time is given). Similarly, the Initiator role gives information about who began the login process, and the Account role provides information about what account was logged into.

Roles are fleshed out by the atoms beneath them; the atoms tie the actual data into the structure. For example, the string ?joe? is the user name associated with the account that was being logged into in the above event. Similarly, that account exists on a host named ten.ada.net.

Finally, multiple events, each with their own verb-headed S-expression, can be tied together with conjunction SIDS. In the example given at the start of this explanation, the three actions (Login, Delete, Login) are tied together into an InSequence conjunction.

GIDOs are flexible in the sense that the GIDO producer can leave out any unavailable information or may choose what information to include. All of the legal SIDs are defined in an appendix to the CISL specification [ref].

We should also point out that the ASCII representation of the GIDO body shown above is not what is actually transmitted across the network. The CISL specification defines a compact opaque representation for GIDOs for purposes of transmission and storage.

For complete details of CIDF, the reader should consult the specifications.

3. Three Classes of IDS Adaptation Problems
This section presents general background discussion for the three classes of IDS adaptation problems that our demonstration will address.

3.1 New IDS Components
In an operational environment, an IDS must be functioning at all times. Any gap in the operation of the IDS leaves the supposedly protected system vulnerable to penetration and compromise. The configuration of the IDS, however, may not be static. It is desirable to be able to incorporate new resources and technology, in the form of new IDS components, into the IDS as they become available while still maintaining continuous operation. Similarly, IDS components can fail, the resources they require can become unavailable, or they can become obsolete or ineffective. It may be necessary, therefore, to remove IDS components while keeping the system operational.

Consider first the case where a new component is added to the IDS. When a component is added, the rest of the IDS should adapt to take advantage of it. In order to do this, other components in the system must be aware of the capabilities of the new component. Because all components will have limited resources, these other components may negotiate to use only a subset of the capabilities offered by the new component. For example, a network sniffer component may be able to provide data on every packet traversing the segment of the network to which it is connected. A specific analysis component, however, may only be able to detect intrusions based on the file transfer protocol (FTP). Therefore, the analysis component could negotiate with the sniffer component to obtain only FTP packets. This would allow both to operate more efficiently; the sniffer only needs to format and transmit FTP related packets and the analyzer will only receive and process packets that it can analyze.

To support the addition of new components, the negotiation protocol must provide two classes of interaction:

  • Publishing the capabilities of a new component. This requires the ability to describe the capabilities of a new component and the ability to disseminate this description to other components.
  • The ability for collections of components to interact to determine a specific set of capabilities that they will utilize. This requires that the protocol be able to describe the data to be exchanged and to provide some type of quantitative measure of the resources used to provide that data. Other information may also be required, such as the latency for providing the data or a negotiation progress metric to assure that the negotiation will terminate. Resource metrics are discussed below as part of the description of the new attack signature and response scenario.

These classes of interaction are useful not only when a new component is added, but also when new capabilities are added to an existing component. For example, new response capabilities may be added to an operating system, such as the ability to maintain an audit trail of a user's actions. The operating system would announce these capabilities and possibly negotiate with an analysis component to accept requests for these new events.

Similar actions must occur when a component is removed or degraded. If possible, a component to be removed could announce its removal to allow other components to adapt. Similarly, a component could announce a degradation in its capabilities due to component failure or a loss of resources. Each of these situations requires the component to announce its change in capability and possibly to negotiate with other components for utilization of specific capabilities.

3.2 Overloading and Flooding of ID Systems
The second problem class where we believe adaptation and negotiation are important is in the case of an overloaded IDS. Overloading can occur due to faults on the network, on hosts, in applications, or within the IDS itself. Alternatively, overloading might be due to deliberate attempts to flood the IDS by an attacker. In this section we categorize the kinds of overloading and flooding situations that could take place in an IDS.

Categorization of Overload Situations

Our general assumptions are that the IDS is handling a certain flow of data in the course of its activities and it is using certain resources in order to handle that flow. Resources that it is using probably include:

  • CPU resources on some computers
  • Bandwidth on some networks
  • Disk space on some hard-disks
  • Memory on some computers
  • Screen space on some monitors

It can only use a finite amount of these resources. The limits on its resource use may arise because:

  • All the resources are inherently finite (CPUs are only so fast and disks so big)
  • Some of the resources are already being used by other entities and the IDS does not have the scope to preempt that usage (e.g. it can't delete another applications? files in order to create more disk space).
  • The IDS has some explicit limit or contract on what resources it can use (e.g. an IDS might guarantee to use only 10% of the CPU on a machine except in emergencies).

So these are limits on the supply of resources that the IDS needs in order to do its work. Just as there is a supply of resources that the IDS needs, so is there a demand for the services of the IDS. There are two ways of viewing this.

One is that the level of demand for IDS services is dictated by the amount of log data that must be processed by the IDS. If more audit log data comes in, that is a larger demand that the system might or might not be able to meet.

A more sophisticated view might be that the demand for the IDS is dictated by the security guarantees that the user asks the IDS to fulfill (e.g. "detect and respond to all snork attacks on the 3rd floor ethernet"). Even in this view, however, we must recognize that the demand on IDS resources involved in fulfilling a particular security guarantee varies depending on the amount of data that must be processed to do so (e.g. how much it costs to detect all the snork attacks on the 3rd floor ethernet probably depends on how many packets traverse that network).

An overload condition occurs because the demand for IDS services exceeds the supply of resources that the IDS has available to use to deliver those services. There are basically two classes of situations in which this overload can occur:

Demand-driven overloads occur when the demand for IDS services increases beyond the available supply. This could be because

  • The IDS is asked to provide additional detection capabilities beyond what it previously was asked to do.
  • The amount of data that must be processed by the IDS to do its existing detection tasks increases. This in turn may be due to natural fluctuations in the amount of data, or it could be due to malicious flooding of the system, decoy attacks, etc.

Supply-driven overloads occur when the supply of IDS services decreases below what is necessary to fulfill the extant demand; e.g., resources that were previously available become unavailable

  • Computers or networks might go down
  • IDS components might be crippled or compromised making them unavailable
  • Competing users of resources might increase their utilization (e.g. large background number-crunching jobs are competing with the IDS for the CPU).

Adapting to overload situations

Given that an overload situation has occurred, there are three classes of adaptation that can be used.

  1. The supply of available resources or components can be increased to the point where demand is met.
  2. The demand can be reduced to a manageable level.
  3. The demand may not be met, but the system may adapt to ensure that certain (presumably more important) demands are met.

Class 1 adaptations include such strategies as:

  • Starting new components, (and possibly thereby accessing previously unused resources ? e.g.,starting a new analyzer on a host that was previously not used by the IDS).
  • Making more resources available to existing components
    • by negotiating new limits on how much system resources can be utilized
    • by preempting resources from other tasks (e.g., killing processes, deleting files, resetting connections that are competing with the IDS for resources)
    • by seeking human assistance

Class 2 adaptations include such strategies as

  • Identifying the source of excessive data and eliminating it (e.g., modifying packet filtering rules in order to eliminate bogus data flooding the system from outside, killing processes that were generating massive floods of OS audit records)

Class 3 adaptations involve such things as

  • Identifying detection goals that are less critical and stopping work on them in order to reduce load.
    • Reduce the number of kinds of attack detected (e.g., decrease false negative rates)
    • Reduce the number of systems/networks that are covered by the IDS
  • Identifying data that is less critical than other data for efficient detection, and suppressing it.
  • Allowing a processing backlog to develop (this is the default).

3.3 New Attack Signatures and Responses
Our third IDS problem class where adaptation and negotiation is crucial is in the case where an IDS attempts to detect a new attack signature and respond appropriately. Current commercial IDSs operate by searching for any of a list of attack signatures in the data stream they are examining. New signatures can be added manually at any time, either because they are supplied by the IDS vendor in response to newly documented attacks or because they are developed by the IDS administrator to reflect site-specific policy.

Installation of new signatures, however, carries a cost; each signature adds to the computational burden shouldered by the IDS. There is a tradeoff between the security benefit of searching for the events described by the signature and the resources used to do so (or possibly the processing of other signatures that must be foregone in order to search for this one).

The reason we examine this class of problems is to show how the negotiation protocol could enable an IDS to handle a new attack signature in a cost-effective manner. It should enable an IDS to 1) determine if the capability exists within the system to respond to the intrusion described by the attack signature; and 2) evaluate dynamically the cost of response, such as degradation in performance and loss of functionality, against the damage caused by the intrusion. For example, an installation may have two different types of firewalls: fast but less secure packet filtering-based firewalls and slower but more secure proxy-based firewalls. During benign conditions, the installation uses both types of firewalls. In the event of an intrusion, however, the installation may want to use only the more secure proxy-based firewalls. But before doing so, the installation may want to consider the potential performance degradation caused by directing all network traffic to the proxy-based firewalls. The negotiation protocol allows the ID system to perform this assessment and to decide on an appropriate response.

Strategies for Handling New Signatures

When an IDS is protecting a large number of hosts that may be running different versions of some type of vulnerable software, the system may need to take different actions when responding to the same attack signature. Furthermore, the IDS may not initially have all of the information that it requires to be able to handle a new attack signature, such as the configuration and version of the software each host is running and factors that affect the cost of responding to the attack.. For example, on UNIX systems an attacker can fool some versions of the /bin/mail program into writing to an arbitrary file by guessing the name of the temporary file that /bin/mail creates. When this vulnerability is discovered, a system administrator might command the IDS to protect against attacks of this type. The IDS, however, must first find out what versions of the mail program each host is running. If one of the host systems is running a vulnerable version of the program, IDS may take one of several actions:

  1. Cut off mail to that host,
  2. Monitor the mail system on that host, attempting to detect attacks against it,
  3. Raise a flag for the administrator that a potential problem exists, or
  4. Do nothing.

The IDS may choose action 1, for example, if there is little or no mail traffic sent to the host, so that shutting down mail would cause little or no service loss. It may choose action 2 if it is infeasible to shut down mail to the host, but the risk of attack is high enough that monitoring the mail system is worth the bandwidth and cycles that it will cost. When an attack is detected, which results in an increased calculation of risk, the IDS may at that time decide to shut down mail to the host. Action 3 might be chosen if it is both unfeasible to shut down mail service and impossible or too costly to monitor the host. For example, the host may be unable to generate audit records for its mail program so the IDS cannot detect attacks made against it. Or the host may be able to generate audit records, but the IDS as a whole is already heavily loaded and cannot handle the new records. If the risk is so high that being unable to take action 1 or 2 is unacceptable, the IDS has no choice but to notify the system administrator that it cannot handle the new attack signature. Action 4, do nothing, might be chosen if the risk is deemed low relative to the costs of any of the other actions.

4. Selected Scenarios
In this section, we describe representative scenarios from each of the classes of IDS adaptation problems described in the last section. We intend to use the scenarios for two purposes:

  • To drive the process of developing requirements for our IDIAN mechanism
  • As the basis for a demonstration that will show how our negotiation protocol meets the needs of a dynamically changing IDS.

The first scenario is actually a set of scenarios that demonstrate the negotiation protocol when a newly activated IDS component joins an already running IDS, or when the capability of an already running data producer is enhanced or diminished. The second scenario demonstrates the behavior of an IDS as it uses the negotiation protocol to handle a flooding attack. The third scenario shows how an IDS dynamically determines whether or not it can handle a new attack signature and later handles the attack.

4.1 New IDS Component Scenarios
These simple scenarios demonstrate the dynamic ability of an IDS that uses the negotiation protocol to adapt to new IDS components.

New Producer

In this scenario, a new IDS event generator (E-box) publishes its capabilities to a pre-existing event analyzer (A-box). The A-box examines the new capabilities to determine if it can make use of them, decides that it can, and sends a proposal message to the new component. The proposal message contains a description of the desired data and an upper bound on the acceptable costs associated with supplying the capability.

For example, the E-box might be able to supply all the system-call audit-trail from a particular Unix host. The A-box might be interested in being able to correlate all inbound TCP/IP connections to that host with all outbound connections (perhaps in order to search for connection laundering). It might ask to receive the results of any fork, exec, and socket related system calls, which would allow it to reconstruct dataflow between connections and processes on the host. (Since it would know which processes received inbound connections, which processes created which other processes, and which processes initiated outbound connections).

The E-box determines if it can supply the capabilities within the cost limits. If so, it sends an acceptance message to the A-box. If not, it either sends a rejection message to the A-box or, if the minimum cost is relatively close to the upper bound set by the A-box, it sends a counterproposal to the A-box. For example, the E-box might not be able to fine tune the audit output to only what was requested, and would need to ship a larger volume of audit records at a greater cost (perhaps all process related system calls, rather than just fork and exec). The counterproposal contains the same type of information as the original proposal, i.e., description of capabilities and cost limits. The A-box can accept or reject a counterproposal, or it can create yet another proposal. A progress metric ensures that the negotiation eventually terminates.

New Consumer

The new consumer scenario is very similar to the new producer scenario, but the new component is a response component (R-box) rather than an E-box. The new R-box broadcasts its capabilities to a pre-existing A-box . The negotiation transpires in the same manner as for the new producer.

Enhanced Capability

In this scenario an upgraded E-box has new capabilities, which it broadcasts to an A-box. The A-box then renegotiates its utilization of the capabilities of the E-box. There are no restrictions on when the capabilities of the E-box can change - they can change even during an ongoing negotiation, and the IDS components must dynamically adjust.

Diminished Capability

Instead of acquiring a new capability as in the scenario above, in this scenario an E-box loses some capability. The E-box broadcasts its reduced capability, which requires an A-box that had a previous agreement with the E-box to renegotiate for a reduced level of service.

Missing Component

In this scenario an E-box is suddenly incapacitated and is essentially removed from the IDS. For example, the component may have been compromised, or its network connection may have been severed. An A-box detects this change and negotiates with a second E-box to assume some of the responsibilities of the first.

4.2 Overloading and Flooding Scenario
In this scenario, a centralized IDS protects a large number of hosts in a corporation. The hosts transmit audit data across the company network to a central analyzer. One of the hosts is a laptop that was taken to a conference where a company employee left it unattended. An opportunistic hacker has stolen it, and from his cheap motel room, he and his friends have connected via the corporate VPN to the network. As a result, the laptop's local E-box has begun shipping audit data to the central A-box.

The hackers try to leverage their access on the laptop to gain access to other machines. In order to deflect suspicion, they run a tool that causes the laptop to generate lots of spurious audit data that gets fed to the A-box. The A-box is able to handle this load. After a short interval, the hackers compromise a second machine inside the company, and cause it to also generate copious spurious audit data.

At this point the load on the central audit processor becomes greater than the A-box can handle, and absent any adaptive response it would begin to fall badly behind. The hackers next begin heavy scanning of a large set of other machines in the company, causing them to also generate significant numbers of audit records. The central A-box would become completely unresponsive and uninformative due to the excess load if it had no mechanism for handling the situation.

The A-box, however, is coded so that any time an E-box sharply increases its audit output the A-box marks it as "potentially compromised", and sends it a message requesting reduced audit output (e.g., using a TCP-like window size reduction). The hacker program on the laptop responds affirmatively that it will reduce its output, and does in fact do so. (The hackers used the tool with this option because they were aware that if their program did not negotiate correctly, the A-box would mark the laptop as definitely compromised and tell the firewall to cut off the VPN connection.) After a period of time where the laptop behaves itself, the A-box renegotiates a higher quota for it. The laptop again begins flooding the A-box. The A-box again negotiates it to a small quota and the laptop complies.

In the meantime, the second compromised machine also begins to flood the A-box. When the A-box sends it a smaller quota instruction, the host acknowledges the instruction but does not reduce its audit output. The A-box notes this host as definitely compromised. As there is no firewall or filtering router between the host and the A-box, the A-box's only option is to drop all received audit from this host unparsed. It does so. This reduces, but does not eliminate, the processing load from this spurious audit data.

In the meantime, excessive audit load begins to arise from the deep scans being perpetrated by the laptop and the second compromised host against the corporate network. The A-box is not able to handle all of them simultaneously. We assume in this scenario that the A-box has a priori knowledge of which machines in its domain are important in some appropriate way. Then it negotiates smaller quotas or directly filters and drops unimportant machines in order to focus its efforts on determining if the important machines are being compromised. This also allows the console to stay usable during the incident.

The A-box also rapidly determines that the laptop is responsible for some of the scanning activity. This causes it to tell the firewall to drop the laptop from the VPN. The second compromised machine, however, continues to scan until human administrators take it off-line. At that point the flood of new audit stops, and the A-box is able to catch up on the delayed audit records in order to determine what hosts might have been compromised during the attack and now need to be cleaned up.

4.3 New Attack Signature Scenario
In this scenario a government installation has two different types of firewalls: fast and cheap packet filtering firewalls and more-secure but slower proxy-based firewalls. Under benign conditions the installation uses both types of firewalls to handle the network traffic. The IDS recognizes an attack signature for the packet filtering firewalls, but shutting down the packet filtering firewalls and diverting all network traffic to the proxy-based firewalls is considered by the system administrators to be too costly (i.e., in terms of a network throughput penalty) compared to the risk level. The IDS, therefore, does not monitor possible attacks against the packet filtering firewalls.

The system administrators discover, via a CERT advisory, that the mail program used on some or all of the hosts contains a flaw that allows intruders to assume root privilege in UNIX systems. This new vulnerability causes great consternation among the system administrators. They believe that this new flaw, combined with the previously identified vulnerability in the packet filtering firewalls, could compromise the entire facility. The IDS is instructed to detect the new attack signature. If the IDS detects just one of the two attacks, it is to take no action. If, however, the IDS detects simultaneous attacks on both vulnerabilities it is to immediately divert all network traffic to the more-secure proxy-based firewalls.

This scenario has two parts: the first part consists of the IDS determining whether or not it is able to detect and respond to the intrusion described by the new attack signature, while the second part consists of detecting and responding to an attack.

Part 1

An A-box in the IDS begins the negotiation by trying to find out the following information:

  • Which of its hosts are running the vulnerable version of the mail program.
  • The location of E-boxes so that the A-box can request them to send audit records for detection the exploitation of the both vulnerabilities.

Next, the A-box sends messages to the E-boxes to send audit records for detecting the exploitation of the two vulnerabilities. Finally, the A-box must negotiate with the proxy-based firewalls to agree on a strategy for filtering messages to reduce the network traffic in case the firewalls are being overloaded. The A-box must also negotiate with the packet-filtering firewalls to find out if they can shut down gracefully when asked to do so.

Part 2

Some time later, the E-boxes send audit records to the A-box that indicate an exploitation of the packet filtering firewall vulnerability. The IDS takes no action. Still later, the A-box detects an attack against the mail program. At this point, the A-box responds to the new attack signature by instructing the packet-filtering firewalls to shut down and by telling all hosts to direct their network traffic to the proxy-based firewalls.

5. Design Requirements
In this section we discuss design requirements on our negotiation mechanisms and on the CISL that are required to support the kinds of interactions that occur in the scenarios. We derived these requirements by analyzing the three scenarios to identify the steps the ID components need to carry out to reach a common agreement and will use them as the success criteria for the project.

The requirements are described in two sections. Section 5.1 describes the requirements for GIDO filters, which are meta-GODOs that allow IDS components to specify the type of information that they can produce or consume. Working manually through the negotiation steps gave us a valuable insight that the success of the negotiation depends on IDS components? ability to describe precisely the type of information they can produce or consume. Section 5.2 describes the rest of the requirements.

5.1 GIDO Filtering Requirements
GIDO filters are meta-GIDOs that IDS components can use to specify the type of information that they produce or consume. They have identified the following requirements:

  1. Filters should be expressive enough that parties can specify all the sets of GIDOs that are useful to them. In particular, it must be possible to specify only part of the GIDO required to match the filter, allowing the GIDO to contain additional information that is not necessarily of interest to a particular party. It must be possible to filter on any data field in the GIDO, and on any combination of data fields.
  2. Filters should be precise. It should be deterministic what GIDOs satisfy a filter and what do not. In particular, the filter language should be purely syntactic. It should be possible to tell whether a GIDO matches a filter without knowing anything about the meaning of the SIDS involved, but only the data types of the SIDS in question (except for the needs of requirement 4, below). The rationale for this requirement is that filtering is distinct from analysis; filtering is supposed to be dumb and to not require intelligent understanding of the GIDO. Hence we can and should demand a high degree of precision.
  3. Filters should allow a way to extract particular data values out of the incoming GIDOs, so that instead of returning the whole GIDO, only the data of interest need be returned.
  4. Filters should support the possibility of use in a simple signature mechanism -- when an incoming GIDO matches a filter, a response GIDO is sent to some appropriate R-box.
  5. The filter language should allow for efficient implementations. In particular, it should allow for implementations that are tightly integrated with encoding and decoding, or implementations in which filters are sent to the producing side in order to minimize what data was sent. Also, the process of matching a GIDO to a filter should have a polynomial time implementation. The rationale for this requirement is that it would help overall efficiency greatly if it were possible to only partially decode or encode messages that were not going to satisfy the filter. The filter design, therefore, should not preclude this possibility.
  6. There should be an explicit way to define the geographic scope of filters. It should be possible to tell what hosts and networks could generate and/or consume GIDOs that match filters like this. This requirement is to support the use of filters in a plug/play implementation.
  7. Given two filters, F1 and F2, which match sets of GIDOs G1 and G2 respectively, it should be possible to define unambiguously a new filter Fu that matches the union G1 and G2, and a new filter Fi that matches the intersection of G1 and G2. These new filters should be computable in polynomial time. We need this requirement for negotiation between components as to what GIDOs should be sent where.
  8. Filters must have conceptual clarity. For people who understand GIDOs, it should be easy to write or understand a GIDO filter that does what they need. In particular, as much of the existing GIDO structure as possible should be used in filters, rather than defining wholly new constructs.

5.2 Protocol and Component Requirements

Automated Component Location and Association

IDS components need a mechanism to locate and identify each other. For example, in the new component scenario a new E-box must be able to specify its identity and location so that A-boxes that want to take advantage of the E-box?s capabilities can request its services. The mechanism could be similar to the component directory that the Open Group was working on for CIDF under the MTNS project [ref] in order for components to find one another.

This mechanism must also have the concept of an association between two components that are going to have a producer/consumer relationship. The negotiation protocol must be able to establish this association.

Termination of Negotiation

IDS components need an algorithm or mechanism to ensure that a negotiation will terminate. For example, a simple algorithm might use a counter to represent the number of exchanges between ID components. The initiator of the negotiation can then decide to terminate the negotiation when the counter reaches some pre-determined limit. The negotiation protocol must be able to support a termination mechanism.

Automated Establishment of New ID System Capability

As the capabilities of components in an ID system change (due to the introduction of new producers or consumers), the potential for a change in the ID system capability exists. The change can be either new detection capability or new response capability. The producers and consumers need to specify partial and standard capabilities. They also need to adapt to satisfy requests for new capabilities. A suitable protocol should have the ability to be used for the process of producer/consumer capability negotiation. The purpose of this negotiation is to establish which potential capabilities will actually be used. In effect, this is the process of establishing a new ID system detection capability.

Specification and Negotiation of Quotas

Every consumer/producer association needs a quota for how much information can be shipped per unit time. A consumer must monitor its producers' quotas so the consumer is not overwhelmed. A consumer must be able to request a reduction in a producer's quota when the consumer is about to be overwhelmed. Producers must acknowledge and comply with such requests. Producers must be able to request a larger quota, but consumers may or may not comply.

Thus, the protocol requires a mechanism to specify and negotiate quota parameters during association setup and during operations. Note that this mechanism may effectively duplicate the windowing mechanism in the transport layer (assumed here to be TCP). This duplication is necessary because there is no application layer control of the transport layer mechanisms.

Action Specification

The protocol must be expressive enough to specify actions that ID components should perform. For example, the flooding scenario requires the ability to instruct appropriate firewalls to drop particular connections. Similarly, the new attack signature needs to instruct R-boxes to perform whatever action specified by the attack signature.

Querying Component Capabilities

A-boxes must be able to query both E-boxes and R-boxes about whether they are capable of performing some particular action. For example, in the new attack signature scenario, the A-box queries E-boxes to determine if the new attack can be detected and queries R-boxes to determine if the appropriate response can be initiated. Querying is different than simply requesting or commanding some action and, thus, this notion must be incorporated into CISL.

Cost Model

ID components need to specify the cost of implementing their capabilities. For example, E-boxes need to specify the cost of generating sensor data and R-boxes need to specify the cost of executing requested actions. A-boxes need to assess the cost of, for example, deploying a new attack signature. Assessment might include determining the stress on the system if the chosen firewalls cannot handle all network traffic directed to them.

Specifying and assessing cost requires a cost model which provides a common framework for communicating cost information. Such a model could have many components such as network usage, CPU usage, etc.

6. Component Modifications
This section describes how IDS components must be modified to implement the three scenarios described above. Developing an adaptive IDS requires two areas of research. The first area of research concerns figuring out how IDS components can adapt to changes in their environment, such as detection of suspicious behaviors. This area of research concerns problems such as figuring out how IDS can dynamically modify its audit data collection, analysis, and response behaviors. The second area concerns the means by which ID components can negotiate with each other to enhance their own capabilities. The aim of the IDIAN project is to focus on the second area. Consequently, the modifications we discuss below are changes we need to make to the ID components so that they can negotiate with each other.

6.1 Requirements on Components for new Capability Scenario
Every capability that can be broadcast must have a corresponding CISL specification. An E-box or R-box uses this specification to describe the capability to other components. Similarly, an A-box must be able to process CISL capability specifications that it receives and to generate appropriate negotiation responses, such as proposals and counterproposals, refusals and acceptances.

The simplest type of component is one that can publish its capabilities and can accept or reject a proposal. This type of component never generates a proposal; it relies on other components to generate the proposals. Examples of this type of component would be a simple sensor or packet filter.. You could not build an IDS entirely out of this type of component because no proposals could be generated and no agreements could be made.

A more sophisticated component would be one that, in addition to the functions of the simple type above, could accept published capability descriptions and generate proposals to use all of the published capabilities. An IDS consisting of only components of this type could function given sufficient resources, but might be very inefficient.

An additional degree of sophistication is added when a component can construct a proposal that requests the use of a subset of the published capabilities. A component must have algorithms for selecting an appropriate subset. As part of this project we expect to construct such algorithms for some components and to begin to formulate general algorithms for this purpose.

These same algorithms can be used for simple negotiations. For example, suppose that an E-box E publishes a list of capabilities L0. An A-box A requests the use of a subset of L0, L1. After it broadcast its capabilities, meanwhile, E has been overwhelmed with requests and cannot provide A with the complete L1, so E makes a counter-proposal to A to provide a subset of L1, L2. This process repeats until both A and E are satisfied or until one of them rejects the latest proposal. As long as each proposal contains a proper subset of the capabilities of the immediately preceding proposal, eventual termination of the negotiation is guaranteed.

The most sophisticated components will be able to generate new proposals that contain more arbitrary lists of capabilities, unconstrained by a preceding proposal. For example, suppose that an R-box R announces a list of capabilities L0. An A-box A requests a list L1 that is a subset of L0. R comes back with a list L2 that is a subset of L1. Unsatisfied, A proposes an entirely new list M that is a subset of L0 but that may share only some capabilities with L1. The more sophisticated the negotiation, the more sophisticated the algorithms that are needed for guaranteeing termination of the negotiation.

6.2 Requirements on Components for Flooding Scenario
The flooding scenario requires several E-boxes, which could all be of the same type. The nature of the GIDOs they produce is unimportant, but it is necessary that the rate of data production be adjustable. Each E-box needs to be wrapped with appropriate code to ship data in the right format, and to implement the appropriate controls on the data production rate.

The flooding scenario also requires an A-box that can do some analysis on the E-box GIDOs. The nature of this analysis is again unimportant, but we need to modify the A-box by adding extra code to look at the data rates that are coming in to it. The A-box must keep track of machine importance and quotas, and must manage the level of input in the way described in the scenario.

The A-box must also be able to send an appropriate message to a firewall to cut off a connection.

6.3 Requirements on Components for New Attack Signature Scenario
The new attack signature scenario requires two R-boxes to represent the packet filtering and proxy-based firewalls, plus at least two E-boxes to represent hosts. An additional E-box reports on attacks on the packet filtering firewall. An A-box is required to perform some analysis on the E-box GIDOs to determine when attacks against the packet filtering firewall and the hosts are occurring. The A-box must also be able to send a message to the packet filtering firewall to cut off connections.

7. Project Plan
The project will proceed on two tracks:

The track on the left is the protocol design and implementation. The result of this track is a specification for inclusion in the CIDF specification together with library routines that provide an API for using the specification. Both the protocol design and implementation will build on existing CIDF design and implementation. The track on the left is the application track and implements the scenarios that use the protocol. The scenarios will be designed and implemented separately, but a great deal of overlap in the design and implementation of the different scenarios is anticipated. The application will use existing ID components, modified to use the negotiation protocol. The application track will provide a vehicle for trying out the design and implementation of the protocol and provide a means for demonstrating the protocol.

The high-level and low-level protocol design tasks serve to define and specific the negotiation protocol. The high-level design focuses on identifying the messages in the protocol, the protocol states, and the fields of the messages. The low-level design focuses on specific representations of fields and what constitutes proper messages and proper message sequences. The protocol implementation will develop a set of libraries that implement an API that ID components can call to easily utilize the negotiation protocol.

This library will be based on existing CIDF library software.

The ID component selection task identifies the existing ID components to be used for each of the scenarios. ID components may be used in more than one scenario. The component modification specification task will describe the modifications that need to be made to each of the selected components to implement the scenarios.