Estimated Reading Time: 15 minutes

Introduction to the Basics of IPSec

IPSec (Internet Protocol Security) is a suite of protocols designed to secure communications over IP networks. It offers encryption and authentication for both IPv4 and IPv6, ensuring the confidentiality, integrity, and authenticity of data during transmission. The aim of IPSec is to protect data from unauthorized access, eavesdropping, tampering, and other forms of attacks.
Originally developed to address security issues in internet communications, IPSec's evolution traces back to the 1990s. As the internet became more widespread, the accompanying security challenges grew increasingly prominent. To tackle these challenges, the IPSec standards were established, becoming one of the key technologies for securing internet communications.
Today, IPSec is widely used in virtual private networks (VPNs) for enterprises, government agencies, and individuals, effectively securing network layer data transmission. By establishing encrypted tunnels through IPSec, organizations or individuals can ensure the security of their data as it traverses the internet, protecting against cyber attacks and unauthorized access.

Structure of IPSec

IPSec consists of the following protocols:

• AH (Authentication Header) provides connectionless data integrity and authentication of data origin, as well as protection against IP header modification attacks and replay attacks.
• ESP (Encapsulating Security Payload) offers confidentiality, connectionless data integrity, data origin authentication, anti-replay services (a form of partial sequence integrity), and limited traffic flow confidentiality.
• ISAKMP (Internet Security Association and Key Management Protocol) offers a framework for authentication and key exchange. The actual authentication key material is supplied by manually configured pre-shared keys, Internet Key Exchange (IKE and IKEv2), Kerberized Internet Negotiation of Keys (KINK), or IPSECKEY DNS records. Its purpose is to generate Security Associations (SAs), including the set of algorithms and parameters required to perform AH and/or ESP operations.

IPSec Messages

IPSec messages are divided into two types: AH and ESP.

Authentication Header

AH operates directly over IP, using IP protocol number 51.

The format of the AH message is as follows:

• Next Header (8 bits) indicates the type of the upper-layer protocol being protected, drawn from the IP Protocol Numbers list.
  • IANA Protocol Numbers
  • Wikipedia List of IP Protocol Numbers
• Payload Length (8 bits) measures the length of AH in multiples of 4 bytes minus 2. Under IPv6, the AH header is in multiples of 8 bytes.
• Reserved (16 bits) is reserved for future use (currently set to zero).
• SPI (Security Parameter Index, 32 bits) is a negotiated arbitrary value used by the receiver to identify the associated Security Association (SA).
• Sequence Number (32 bits) is an incrementing sequence number (increasing by 1 for each packet sent) to prevent replay attacks. When replay detection is enabled, the sequence number is never reused. A new SA must be negotiated before reaching the maximum value of the sequence number.
• ICV (Integrity Check Value, multiple of 32 bits) is a variable-length check value. It's used for padding to accommodate the 8-byte increments for IPv6 or 4-byte increments for IPv4.

Encapsulating Security Payload

ESP operates directly over IP, using IP protocol number 50.

The format of the ESP message is as follows:

• SPI (Security Parameter Index, 32 bits) is a negotiated arbitrary value used by the receiver to identify the Security Association (SA).
• Sequence Number (32 bits) is an incrementing sequence number (increasing by 1 for each packet sent) to prevent replay attacks. Each SA has its own separate counter.
• Payload data (variable length) contains the protected contents of the original IP packet, with the type of protected content indicated by the Next Header field.
• Padding (0-255 bytes) is used for encryption, to extend the payload data to a size that fits the encryption's cipher block size, and to align the next field.
• Pad Length (8 bits) indicates the size of the padding.
• Next Header (8 bits) specifies the type of the next header. Its value is taken from the IP Protocol Numbers list.
• ICV (Integrity Check Value, multiple of 32 bits) is a variable-length check value. It aligns to the 8 bytes for IPv6 or 4 bytes for IPv4 and may include padding.

IPSec Operating Modes

IPSec supports two operating modes: Transport Mode and Tunnel Mode.

• Transport Mode typically encrypts or authenticates only the data of the IP packet. Since the IP header is not modified or encrypted, the routing path remains unchanged; when using AH, IP addresses cannot undergo Network Address Translation (NAT) as it would invalidate the hash value. The transport and application layers are always protected by hashing, thus they cannot be modified, including port numbers and more.
• Tunnel Mode encrypts and authenticates the entire IP packet, which is then encapsulated into a new IP packet with a new outer IP header. Tunnel mode is used to create communication between VPN sites, including host-to-network and host-to-host communications.

IPSec Encryption Algorithms

Common encryption algorithms used in IPSec include:
Symmetric Encryption Algorithms:

• AES (Advanced Encryption Standard): A widely used symmetric encryption algorithm that supports key lengths of 128 bits (AES128), 192 bits (AES192), and 256 bits (AES256).
• DES (Data Encryption Standard): An older symmetric encryption algorithm that is now considered less secure due to its short key length (56 bits) and is rarely used.
• 3DES (Triple Data Encryption Standard): An enhanced version of DES that improves security by applying DES three times.

Key Exchange Algorithms:

• Diffie-Hellman (DH): A key exchange protocol based on the discrete logarithm problem, used to establish a shared secret between two parties.

Authentication Algorithms:

• RSA (Rivest–Shamir–Adleman): An asymmetric encryption algorithm that uses a pair of keys (public and private) for encryption and decryption.
• PSK (Pre-Shared Key): An authentication method based on a pre-shared key that does not require the use of public and private keys; instead, a shared secret is pre-configured on both ends.
• ECDSA (Elliptic Curve Digital Signature Algorithm): A public-key encryption technique based on elliptic curve theory.

IPSec primarily uses symmetric encryption for encrypting data during transmission, while asymmetric encryption is mainly used during the key exchange (IKE) phase.

Security Associations (SA)

IPSec uses Security Associations (SAs) to establish and manage encryption and authentication parameters. An SA is a set of security parameters between two communicating entities, including encryption algorithms, keys, hash algorithms, and more.
IPSec SAs are established through the Internet Security Association and Key Management Protocol (ISAKMP). ISAKMP implements SAs via manual configuration of pre-shared keys, Internet Key Exchange (IKE and IKEv2), among other methods.
To determine how packets should be protected, IPSec uses the Security Parameter Index (SPI), which acts as an index to the Security Association Database (SADB), combined with the destination address in the packet header, to uniquely identify the SA for a packet.
A similar process is executed for both outbound and inbound packets, thus IPSec communication requires at least two SAs (one for inbound traffic and one for outbound traffic). IPSec commonly uses the Internet Key Exchange (IKE) protocol to establish, negotiate, and manage SAs.

Internet Key Exchange (IKE)

In IPSec, the IKE protocol operates over UDP port 500 for initiating and responding to negotiations. In NAT scenarios, it communicates over UDP port 4500.

IKEv1 and IKEv2

IKEv1 includes two phases: IKE phase 1 and IKE phase 2.

1. IKE Phase 1
   The goal of IKE Phase 1 is to establish a secure, encrypted channel for secure communication between two entities. This phase primarily involves authenticating the parties and negotiating a shared session key for encrypting communications in IKE Phase 2. There are two modes in Phase 1: Main Mode and Aggressive Mode, where Main Mode offers better security, but Aggressive Mode is faster.
   After completing Phase 1, the parties establish an IKE SA, a secure channel dedicated to IKE messaging itself, not for protecting IPsec data transmission.

2. IKE Phase 2
   The purpose of IKE Phase 2 is to negotiate IPsec SA parameters for protecting IP packets, using the secure channel established in Phase 1. This phase typically employs Quick Mode exchanges.
   Upon completion of Phase 2, an IPsec SA for actual data transmission is established. This SA is unidirectional, thus two SAs are usually set up, one for inbound data and another for outbound data.

IKEv2 improves upon IKEv1 by offering enhanced security and a more efficient SA negotiation mechanism. It streamlines the protocol into a single phase that establishes the IKE SA and negotiates the IPsec SA simultaneously. IKEv2 also introduces several enhancements, such as support for NAT traversal, EAP authentication, and automatic reconnection after disconnections.

IPSec Workflow

Communication Process

1. Identifying "Interested Traffic". Upon receiving a packet, network devices typically match the packet's quintuple (source IP, destination IP, source port, destination port, and protocol) with the IPSec policy to determine if the packet should be transmitted through the IPSec tunnel. Traffic that needs to be transmitted through an IPSec tunnel is often referred to as "interested traffic".

2. Negotiating Security Associations (SA). An SA represents an agreement between two communication parties on certain negotiated elements, such as the security protocols used, the encapsulation mode for data transmission, the encryption and authentication algorithms adopted, and the keys for data transmission. Secure data transmission can only proceed once an SA is established between the two parties.
   After identifying interested traffic, the local network device initiates SA negotiation with the remote network device. During this phase, the parties negotiate and establish an IKE SA (for identity verification and key exchange) through the IKE protocol, and then, based on the IKE SA, they negotiate and establish an IPsec SA (for secure data transmission).

3. Data Transmission. Once the IPsec SA is successfully established, both parties can transmit data through the IPsec tunnel.
   To ensure the security of data transmission, IPSec encrypts and authenticates the data using the AH or ESP protocol during this phase. The encryption mechanism ensures data confidentiality, preventing data from being intercepted during transmission; the authentication mechanism ensures data integrity and authenticity, preventing data from being spoofed or altered.
   
   The IPSec sender encrypts the message using the encryption algorithm and key, encapsulating the original data. Then, both the sender and receiver process the encrypted message with the same authentication algorithm and key to obtain the Integrity Check Value (ICV). If the ICVs calculated by both ends match, it indicates that the message has not been altered during transmission, and the receiver decrypts the authenticated message; if the ICVs do not match, the message is discarded.

4. Tunnel Teardown. Typically, session aging between communication parties signifies the completion of data exchange. Therefore, to conserve system resources, the tunnel between the parties is automatically deleted after a certain period of inactivity.

IPsec Traffic Matching

In configuring IPsec tunnels, ensuring that the configurations for interested traffic on both ends match is crucial. For example, if Site A's addresses are 10.0.1.0/24 and 10.0.2.0/24, and Site B's address range is 10.10.10.0/24, then to ensure compatibility and the correct establishment of the tunnel:

• The policy at Site A (local) should define traffic from 10.0.1.0/24 and 10.0.2.0/24 to Site B's destination subnet 10.10.10.0/24.
• Conversely, Site B’s (remote) policy should reflect this configuration in the opposite direction. Hence, Site B needs to define 10.10.10.0/24 to Site A's corresponding 10.0.1.0/24 and 10.0.2.0/24, preferably not using 10.10.10.0/24 to Site A's 10.0.0.0/22.

Aggregating multiple Class C addresses (e.g., 10.0.0.0/24 - 10.0.3.0/24) into a larger subnet might lead to SA negotiation failures due to configuration inconsistencies, so it's crucial to align policies on both sides.

For example, Router-1 configures two detailed ACLs, establishing two outbound SAs (since no active traffic, no IPsec SA has been established yet).

Router-2 configures an aggregated subnet, thus establishing only one outbound SA.

When VPC10 (10.10.10.10) -> VPC2 (10.0.2.10),
Router-2 initiates an outbound SA negotiation based on its interested traffic configuration:

• Source address: 10.10.10.0/24
• Destination address: 10.0.0.0/22
• SPI: TBD

Router-1 finds this doesn't match its interested traffic, leading to unsuccessful negotiation and inaccessible communication.

Conversely, when VPC2 (10.0.2.10) -> VPC10 (10.10.10.10), Router-1 initiates an outbound SA based on the detailed 10.0.2.0/24 to 10.10.10.0/24:

• Source address: 10.0.2.0/24
• Destination address: 10.10.10.0/24
• SPI: 0x77716905

Router-2 matches this subnet with its interested traffic and successfully negotiates, establishing the corresponding SA.

Checking Router-1 shows that the corresponding IPsec SA is active, with both outbound and inbound successfully negotiated.

Similarly, Router-2 shows that after matching the interested traffic, an IPsec SA is created with negotiated inbound and outbound SPIs matching those of Router-1.

Subsequently, attempting access from VPC10 (10.10.10.10) -> VPC2 (10.0.2.10) again, communication is successful through the newly established IPsec SA.

Therefore, when configuring IPsec in production environments, it's vital to maintain consistent configurations on both ends to avoid issues that could render IPsec unusable.

Summary

While IPSec might seem complex, involving numerous protocols and a variety of encryption algorithms, in essence, you don’t need to delve too deeply into how it conducts secure negotiations or how the encryption algorithms encrypt data. It's more important to understand the process of tunnel establishment and the procedures necessary for internal data to communicate effectively.
With this knowledge, you should be better equipped to understand what is required to successfully establish an IPSec site, and when maintenance issues arise, how to narrow down and pinpoint the problem.
Hopefully, the explanation provided in this article is clear enough to deepen your understanding of IPSec.

Useful Tools

For calculating the overhead and effective payload size in an IPSec packet, you might find the following online tools helpful:

• IPsec Overhead Calculator
• Cisco Community's IPSec Calculator for NAT and GRE

References

• Cisco's Guide on IP Security: Cisco IPsec Provisioning Guide
• The Internet Key Exchange (IKE) protocol (RFC 2409): RFC 2409
• Wikipedia article on IPsec: IPsec on Wikipedia
• Huawei Support on IPsec: Huawei IPsec Support Document