Method of improving security performance in stateful inspection of TCP ...

METHOD OF IMPROVING SECURITY PERFORMANCE IN STATEFUL INSPECTION OF TCP CONNECTIONS

BACKGROUND OF THE INVENTION [0001] 1. Field of the Invention

[0002] The present invention relates, in general, to a method of improving security performance in a stateful inspection of transmission control protocol connections and, more particularly, to a method of improving security performance in a stateful inspection, which sets an optimal timeout to be sufficiently long not to influence the normal operation of legitimate flows in the stateful inspection of transmission control protocol connections, and sufficiently short to minimize the number of session entries generated by abnormal flows, such as attacks, so that stateful inspection continues even in the face of network attacks, thus improving the security performance of a stateful inspection computer.

[0003] 2. Description of the Related Art

[0004] Recently, with the development of the Internet, various types of computers specified for packet processing have been used. Representative of these computers may be a firewall [1], a Virtual Private Network (VPN), a network intrusion detection system, traffic monitoring equipment [2, 3], an accounting and charging system [4] or load balancing equipment [5], in addition to equipment such as a router or switch. As the rate of Internet traffic increases to exceed the rate of Moore's law [6], the load of a packet processing task increases in such a computer, so that the optimization of packet processing is required to improve performance. Therefore, various research into the improvement of the efficiency of functions required for packet processing, such as routing table lookup and packet classification, have been conducted [7, 8 and 9]. However, research into the configuration and management of dynamically allocated memory to execute packet processing is relatively insufficient. Therefore, the present invention handles the issue of configuring and managing dynamically allocated memory in packet processing.

[0005] Packet processing in a stateful packet inspection is influenced by previous packets in the same flow, in addition to individual data values of a corresponding packet. Therefore, it is required to maintain information about the states of previous packets in the same flow. For this operation, as a flow is generated or deleted, a corresponding entry is created in or purged from a packet inspection computer. Currently, all of a firewall, a VPN, a network intrusion detection system, traffic monitoring equipment and a usagebased charging system require stateful inspection in different degrees.

[0006] Generally, a stateful inspection computer purges invalid entries using a timeout mechanism to improve space utilization and lookup efficiency. However, such a computer only allows a developer to arbitrarily designate a timeout value (typically, a considerably high value, such as 60 seconds or 120 seconds) or allows a user to configure a timeout value, but does not present a systematic guideline for timeout, that is, a guideline based on protocol and traffic analysis [10]. However, the setting of a suitable timeout is necessary for efficient packet processing. First, if a timeout

is excessively short, the excessive creation and deletion of entries occurs, thus causing undesirable results. For example, if an entry corresponding to a permitted flow is deleted, a firewall may block a packet even though the packet is legitimate. In contrast, if a timeout is lengthened, an entry in an expired flow is maintained for an unnecessarily long time, thus increasing the amount of memory required [11]. Furthermore, even if a packet inspection computer itself is not a target of network attacks, memory overflow may be caused by the attacks. This is because an IP address or port number continuously changes with respect to each packet in the case of an attack traffic stream, so that packets are recognized to be in different flows from the standpoint of the definition of typical flows. In this case, since each attack packet corresponds to a single flow entry, the amount of memory required to create flow entries rapidly increases in a computer performing a stateful inspection on the traffic.

[0007] As described above, conventional research has been concentrated on the reduction of a static table size and the minimization of lookup time for packet classification, not on the management of dynamic memory for a stateful inspection [7, 8 and 9]. In a table used for a stateful inspection employing a session or flow table, only one thesis [2] has mentioned the probability of overflow caused by attacks. However, even this thesis merely mentions that overflow is an element disturbing packet monitoring in high speed links, but research into a method of setting a timeout value is not mentioned. It is possible that a dearth of such research exists because it is difficult to obtain a great number of "typical" Internet traces. That is, in order to set a guideline, a large amount of actual network traffic must be analyzed, and the time for which most TCP connections are set up must be clarified. Therefore, actual systems, such as Cisco, Netscreen or Checkpoint, set a default to a value of at least 60 seconds due to the lack of a guideline [10]. The present invention addresses such a problem first through the analysis of Internet backbone trace of about 1 terabyte capacity, so that it is determined that preceding research addressing the problem scarcely exists.

[0008] Dynamic State Management

[0009] A stateful packet inspection computer has a list of information about currently tracked flows at an observation location in a network, which is generally designated as a session table. Typically, information about a single flow is composed of a protocol, an origin IP address, an origin port number, a destination IP address, and a destination port number. According to the application, additional information may be required. For example, in the case of a stateful inspection firewall, a TCP sequence number is recorded [1]. A packet inspection computer extracts flow information for each observed packet and compares the flow information with an entry in a session table. If an entry having matching information exists, an action defined in the corresponding entry is performed on the packet. For example, a firewall admits therethrough or blocks a packet, or a usage-based charging system increases a packet or byte count. In contrast, if an entry having matching information does not exist, that is, if a current packet is a start packet of a new flow, a new entry for the flow is created in a session table. Further, if the termination of the flow is observed, a corresponding entry is purged from the session table.

[0010] The determination of the start and end of a flow differs according to the protocol used. In a connectionless protocol, such as a User Datagram Protocol (UDP), the end of a flow is determined by means of presumption, strictly speaking. Typically, if a packet for a corresponding flow has not been observed for a predetermined period of time, it is considered that the flow is terminated.

[0011] FIG. 1 is a view showing a process of setting up a TCP connection, observed in a packet inspection computer.

[0012] A TCP of FIG. 1 is a representative of a connection-oriented protocol. As shown in FIG. 1, TCP connection setup is designated as a 3-way handshake because three packets are exchanged between two hosts [12]. First, in order to initiate connection, host A transmits a SYN packet to the other host B. When receiving the SYN packet from host A, host B transmits a SYN/ACK packet to host A to establish a reverse data channel while transmitting an acknowledgement of the SYN packet. The TCP is a fullduplex protocol, which requires a single data channel in each direction with respect to each connection, so bidirectional synchronization packets are required. Finally, host A transmits an ACK packet that is an acknowledgement of the SYN packet to host B, thus completing the setup of a TCP connection.

[0013] It is assumed that a stateful inspection computer is placed at a location on a network through which the connection, formed between hosts A and B, passes (FIG. 1). In this case, a TCP connection setup event can be detected and, additionally, the progress of the connection setup can also be monitored during a 3-way handshake. Further, if a connection setup delay is Dc, DC'=DC is observed, so that the connection setup delay can be measured.

[0014] In accordance with the Request For Comments (RFC) 2988 standard [13], if a TCP SYN packet is lost, retransmission is attempted. At this time, a k( = l)-th retransmission of the SYN packet must be performed within 3x2 seconds after a (k-l)-th retransmission of the SYN packet (according to the definition, the 0-th retransmission is the first transmission of the SYN packet). This is called exponential backoff, which is a kind of congestion control mechanism. If the transmission of the SYN packet successively fails, the time interval between the retransmissions of the SYN packet gradually increases, for example, to 3,6, 12 and 24 seconds, during the 3-way handshake, so that Dc, that is, Dc', increases.

[0015] The TCP allows a FIN packet and an ACK packet of the FIN packet to be exchanged to terminate the connection in a manner similar to that of the connection setup. If the

exchange of the FIN and ACK packets is performed with respect to both channels, a packet inspection computer purges a corresponding entry from a session table. Further, if a connection is interrupted by a RST packet, a corresponding entry is purged from the session table.

[0016] In the stateful inspection computer, the total number of entries in the session table depends on the number of concurrent active flows. In the core part of the. Internet in 2003, it can be observed that at least hundreds of thousands of flows typically and simultaneously pass through a single link. For example, a maximum of 2-37,000 flows were simultaneously observed in a certain OC-48 (2.4 Gbps) link corresponding to an Internet backbone in April, 2003 [11]. Recently, if the fact that an OC-192 (10 Gbps) link is starting to be used in a backbone network is taken into consideration, it can be predicted that several million flows will simultaneously exist in a high speed link in the future.

[0017] Analysis of the Influence of Network Attacks

[0018] The size of a session table is the multiplication of the number of entries by the size of the entries. If the size of each entry (including two IP addresses, two port numbers, a protocol number and additional overhead for table maintenance) is 40 bytes, the size of a session table in a packet inspection computer having a million entries is 40 Mbytes. Considering the memory capacity of the current computer, the session table having such a size can be sufficiently supported. However, as network attacks are conducted, the number of entries in the session table may explosively increase.

[0019] The present invention is focused, among network attacks, on Denial of Service (DoS) attacks and scanning that can influence a stateful inspection, and describes the features thereof.

[0020] Table 1(a) shows part of the packet flow (trace) information of a DoS attack observed in an actual backbone [14].

[0021] In this case, the host IP of a victim is expressed as "y.y.y.y" to protect the privacy of the victim. Typically, an attacker fixes the host IP address Id of the victim in the case of the DoS attack, while the attacker fills Is, ps and pd with randomly generated numbers. The attacker not only does not attempt to connect to the victim, but also randomly selects Is to avoid the tracing of the attacker's IP [15]. For example, because an origin address of "87.231.152.166" shown in Table 1(a) is an address that is not assigned to anyone in the Internet Assigned Numbers Authority (IANA) [16], it can be known that the origin address is an invalid address.

TABLE 1

(a) DoS attack

[0022] In a host scan, the host IP address Id of a victim varies according to packet. Typically, a hacker attempts a host scan to detect vulnerability prior to initiating an attack, and conducts a host scan to detect a target host to be infected in the case of a worm. An attacker randomly conducts a scan with respect to an arbitrary range of IP addresses to detect a vulnerable host address. For example, it can be seen that an address of "58.80.253.118", which is not currently assigned by IANA, appears on part of an actual trace of Code Red II worm shown in Table 1(b).

[0023] The packet inspection computer creates a single session entry with respect to each flow, so that separate entries are created even though any one value of flow identifiers Is, ps, Id and pd, differs. There is a difference in that Is, Id and pd are changed in a DoS attack, a host scan and a port scan, respectively. That is, because all packets belonging to the same attack do not share the same flow identifier, different session entries are created with respect to individual packets. A more serious problem is that these attacks have a very high probability of creating packets. If several attacks among large-scale attacks having occurred on the Internet are described as examples, this problem is clarified (an extreme example is taken for emphasis). In the case of a DoS attack, as the probability of packet generation increases, attack power can increase. Therefore, referring to a DoS attack on a root Domain Name System (DNS) server occurring in October, 2002, about one hundred thousand to two hundred thousand attack packets per second on a single server were recorded [17]. This example means that, if a certain packet inspection computer is placed near the root DNS server, attack-related entries will be created in a number which is much greater than that of flows that can be typically simultaneously observed in an OC-48 link within several seconds. Even in the case of a host scan, as the rate at which packets are created increases, the infection rate of

a worm increases, or a vulnerable host can be detected fast. In the case of a host scan based on the Structured Query Language (SQL) Slammer worm, there was the case in which a single infected host transmitted a maximum of 26,000 packets per second [18]. For example, if the stateful inspection computer is placed at the boundary of an enterprise network including 10 infected hosts, the number of attack-related entries will exceed a million within 4 seconds after the initiation of the attack.

[0024] The fact that entries created by attack packets may exist in a session table for a maximum allowable time period further worsens the situation. In a normal TCP flow, a FIN or RST packet is exchanged at the time of termination and is observed, so a corresponding entry can be purged. However, in the case of a DoS attack, since a FIN or RST packet does not exist, an attack-related entry still remains in the session table until it is purged by a timeout. In the case of a host scan, the lifespan of an entry differs according to protocol. If a scanner uses TCP, a scanned host reacts variously according to the scanning technique [19]. For example, if Code Red II succeeds in finding an infectable host, normal connection setup and termination (after the worm is transmitted) are performed, so a corresponding entry is purged from the session table. However, most scan packets are transmitted to an unused IP address, and then a router causes a destination unreachable Internet Control Message Protocol (ICMP) error. A flow entry created by packets causing the ICMP error will not be purged until the stateful inspection computer separately processes an ICMP message for the purpose of purging the entry.

[0025] In summary, an entry caused by the attack is created with respect to each attack packet at high speed, and remains in the session table for a long period of time. Since the stateful inspection computer performs session table lookup with respect to each packet, this lookup performance