IPSEC
IPSec’s protocol objective is to provide security services for IP packets such as encrypting sensitive data, authentication, protection against replay and data confidentiality. As outlined in our IPSec protocol article, Encapsulating Security Payload (ESP) and Authentication Header (AH) are the two IPSec security protocols used to provide these security services. Analyzing the ESP and AH protocols is out of this article’s scope, however you can turn to our IPSec article where you’ll find an in-depth analysis and packet diagrams to help make the concept clear.
TUNNEL MODE & TRANSPORT MODE
IPSec can be configured to operate in two different modes, Tunnel and Transport mode. Use of each mode depends on the requirements and implementation of IPSec.
IPSEC TUNNEL MODE
IPSec tunnel mode is the default mode. With tunnel mode, the entire original IP packet is protected by IPSec. This means IPSec wraps the original packet, encrypts it, adds a new IP header and sends it to the other side of the VPN tunnel (IPSec peer).
Tunnel mode is most commonly used between gateways (Cisco routers or ASA firewalls), or at an end-station to a gateway, the gateway acting as a proxy for the hosts behind it.
Tunnel mode is used to encrypt traffic between secure IPSec Gateways, for example two Cisco routers connected over the Internet via IPSec VPN. Configuration and setup of this topology is extensively covered in our Site-to-Site IPSec VPN article. In this example, each router acts as an IPSec Gateway for their LAN, providing secure connectivity to the remote network:
Another example of tunnel mode is an IPSec tunnel between a Cisco VPN Client and an IPSec Gateway (e.g ASA5510 or PIX Firewall). The client connects to the IPSec Gateway. Traffic from the client is encrypted, encapsulated inside a new IP packet and sent to the other end. Once decrypted by the firewall appliance, the client’s original IP packet is sent to the local network.
In tunnel mode, an IPSec header (AH or ESP header) is inserted between the IP header and the upper layer protocol. Between AH and ESP, ESP is most commonly used in IPSec VPN Tunnel configuration.
The packet diagram below illustrates IPSec Tunnel mode with ESP header:
ESP is identified in the New IP header with an IP protocol ID of 50. The packet diagram below illustrates IPSec Tunnel mode with AH header:
The AH can be applied alone or together with the ESP, when IPSec is in tunnel mode. AH’s job is to protect the entire packet. The AH does not protect all of the fields in the New IP Header because some change in transit, and the sender cannot predict how they might change. The AH protects everything that does not change in transit. AH is identified in the New IP header with an IP protocol ID of 51.
Cryptographic algorithms
Cryptographic algorithms defined for use with IPsec include:
- HMAC-SHA1 for integrity protection and authenticity.
- TripleDES-CBC for confidentiality
- AES-CBC for confidentiality.
AH versus ESP
"Authentication Header" (AH) and "Encapsulating Security Payload" (ESP) are the two main wire-level protocols used by IPsec, and they authenticate (AH) and encrypt+authenticate (ESP) the data flowing over that connection. They are typically used independently, though it's possible (but uncommon) to use them both together.
Tunnel mode versus Transport mode
Transport Mode provides a secure connection between two endpoints as it encapsulates IP's payload, while Tunnel Mode encapsulates the entire IP packet to provide a virtual "secure hop" between two gateways. The latter is used to form a traditional VPN, where the tunnel generally creates a secure tunnel across an untrusted Internet.
MD5 versus SHA-1 versus DES versus 3DES versus AES versus blah blah blah
Setting up an IPsec connection involves all kinds of crypto choices, but this is simplified substantially by the fact that any given connection can use at most two or (rarely) three at a time.
Authentication calculates an Integrity Check Value (ICV) over the packet's contents, and it's usually built on top of a cryptographic hash such as MD5 or SHA-1. It incorporates a secret key known to both ends, and this allows the recipient to compute the ICV in the same way. If the recipient gets the same value, the sender has effectively authenticated itself (relying on the property that cryptographic hashes can't practically be reversed). AH always provides authentication, and ESP does so optionally.
Encryption uses a secret key to encrypt the data before transmission, and this hides the actual contents of the packet from eavesdroppers. There are quite a few choices for algorithms here, with DES, 3DES, Blowfish and AES being common. Others are possible too.
IKE versus manual keys
Since both sides of the conversation need to know the secret values used in hashing or encryption, there is the question of just how this data is exchanged. Manual keys require manual entry of the secret values on both ends, presumably conveyed by some out-of-band mechanism, and IKE (Internet Key Exchange) is a sophisticated mechanism for doing this online.
Main mode versus aggressive mode
These modes control an efficiency-versus-security tradeoff during initial IKE key exchange. "Main mode" requires six packets back and forth, but affords complete security during the establishment of an IPsec connection, while Aggressive mode uses half the exchanges providing a bit less security because some information is transmitted in cleartext.
References
