What is more exciting than creating a virtual tunnel able to carry out confidential data securely across the world? IPSec, by its flexibility and its multiple protocols, modes and purposes enable devices separated by an insecure network to communicate between each others, by ensuring that Authentication, Privacy and Integrity are respected.

IPSec components are defined by several RFCs, that make it an inter-operable and widely supported solution. It can be configured in two modes, depending on your needs and supports several authentication and encryption mechanisms.

Unlike HTTPs or SSH, IPSec is a layer-3 security solution: it does not rely on applications, and is directly integrated into a router, firewall or OS’ kernel.

In this article we will take a look at the following points:

Advantages and limitations of using IPSec;

  • ESP and AH;
  • Tunnel vs Transport mode;
  • IKE Phase 1 and Phase 2;
  • IKE Main vs Aggressive mode.


I- IPSec Advantages and Limitations

Before we go deeper into IPSec, let’s consider the following advantages and drawbacks of using this suite.

  • Pro: Operating at layer 3 makes IPSec transparent to higher layer applications. Also, it is appropriate for real-time traffic transmission, such as voice / video data.
  • Pro: IPSec is flexible, widely supported and can be used in plenty of scenarios.
    Con: Implementation can become tricky; be sure to know the technology and have a clear understanding of your needs.
  • Pro: IPSec provides security by encrypting, authenticating and optionally preventing replay attacks.
    Con: Performances are impacted because of the resulting overhead. CPU and bandwidth usage may also increase, hence the use of hardware appliances or VPN concentrators in critical environments.


II- Encapsulating Security Payload and Authentication Header

ESP (IP protocol number 50) and AH (IP protocol number 51) are part of the IPSec suite used to encrypt and ensure connectionless data integrity, respectively. Due to their roles, ESP wraps the original IP packet with its header and trailer, while AH adds its header only (Cf encapsulation schemes below).

ESP and AH can either be used alone or together in order to combine their benefits, but it can burden the network because of the encapsulation header’s size.

AH’s main role is to protect the entire packet. Notice that the new IP header may change as the packet passes through devices and some fields like ECN, TTL, ToS or Fragment Offset cannot be authenticated by AH.

Warning: AH is designed to authenticate immutable fields in an IP. Although fields that may change as the packet is transmitted from its source to its destination are excluded from authentication, this does not apply to IP addresses. As a result, AH is entirely incompatible with NAT, whether in Tunnel or Transport mode, since an address translation is performed.

View AH

Notice that new IP header is not shown for simplicity reasons as it may differ depending on the mode in use. For a complete scheme, please refer to the ‘Tunnel vs Transport’ section.

ESP encrypts and provides mechanisms to authenticate data. In addition to its header, ESP also appends a trailer that is encrypted along with the original IP packet, as well as an authentication portion that contains and Integrity Check Value, used to verify the sender’s identity and data integrity.

View ESP

Notice that new IP header is not shown for simplicity reasons as it may differ depending on the mode in use. For a complete scheme, please refer to the ‘Tunnel vs Transport’ section.

III- Tunnel vs Transport Modes

Tunnel Mode

This is the default IPSec mode, usually used between two secure gateways. When using tunnel mode, a gateway simply wraps the original IP packet into a new one, eventually encrypts it, and sends it to the other tunnel endpoint. Since the aim of Tunnel mode is to protect the original packet, ESP is commonly used along with it.

When using AH with Tunnel mode, you get the following encapsulation layers:

Authentication Header in IPSec Tunnel Mode

Similarly, ESP with Tunnel mode results in the following diagram:

Encapsulating Security Payload in IPSec Tunnel Mode

Transport Mode

IPSec transport mode is used for host-to-host communications, for example, for RDP sessions between a workstation and a remote server. When using transport mode, the original IP header encapsulated into ESP / AH is duplicated to the front (with minor modifications to the protocol field) an used as the resulting header that will be used for transit over the network.

When using AH with Transport mode, you get the following encapsulation layers:

Authentication Header in IPSec Transport Mode


Similarly, ESP with Transport mode results in the following diagram:

Encapsulating Security Payload in IPSec Transport Mode

IV- IKE Phase 1 and Phase 2

Internet Key Exchange was originally defined by the RFC 2409. It is used to establish a shared security policy and authenticated keys between devices that use services that require such keys. When dealing with IPSec, IKE sets up a Security Association (SA).

IKE relies on part of the Oakley protocol and ISAKMP. Typically, ISAKMP is used to authenticate peers and establish a shared session secret.

Notice: Although the Cisco IOS’ implementation of IKE and ISAKMP refers to the same thing, it is important to notice that in reality, IKE and ISAKMP are different.


IKE Phase 1

This phase is used to negotiate a secret key based on one of the 3 methods provided by IKE. The resulting key will serve as a basis for generating 3 other keys: 1 for authentication, 1 for encryption and 1 for the phase 2.

  • Pre-Shared Key: This method implies that an administrator configured a pre-shared key on the peers.
  • Asymmetric Signing: The method relies on PKI.
  • Signature Mode: The asymmetric algorithms are used to sign traffic and authenticate the peers while the Diffie-Hellman algorithm is used to exchange the secret key.

After IKE Phase 1 is established, a new ISAKMP SA is set up between the peers.

IKE phase 1 can be performed in 2 modes: Main or Aggressive. The aggressive mode decreases the exchange overhead since the number of packets is reduced by 2, but it is a bit less secure as several settings are exchange in clear.


IKE Phase 2

The first secure channel set up in phase 1 is used to exchange IPSec SA option for effective exchanges. The resulting SA will be stored in a database called the Security Association Database (SAD).

Peers will exchange their preferences regarding tunnel establishment and encryption settings. Remember the 3rd key previously generated in the first phase: it is used to generate sessions keys, unless Perfect Forward Secrecy is enabled. This feature ensures that a same key cannot be generated in both phases, by forcing a new Diffie-Hellman exchange to be performed. Notice that PFS must be supported on both sides.

After the negotiation phase is finished, 2 SA by hosts will be generated (1 per direction) an communication can occur between peers.

This article is intended to share knowledge. If you find something missing, or that should be improved, I would be pleased to add such information.



Submit a Comment

Your email address will not be published. Required fields are marked *

Let’s connect today

© 2019 Pierre Collard. All rights reserved. Solutions Architect Consultant at EVA GROUP.

Privacy Policy