Share this page : facebooktwitterlinkedinmailfacebooktwitterlinkedinmail

IPsec VPNs offer flexible and scalable connectivity. Site-to-site connections can provide a secure, fast, and reliable remote connection. With an IPsec VPN, the information from a private network is securely transported over a public network. This forms a virtual network instead of using a dedicated Layer 2 connection. To remain private, the traffic is encrypted to keep the data confidential.

IPsec is a framework of open standards that spells out the rules for secure communications. IPsec is not bound to any specific encryption, authentication, security algorithms, or keying technology. Rather, IPsec relies on existing algorithms to implement secure communications. IPsec allows newer and better algorithms to be implemented without amending the existing IPsec standards.

IPsec works at the network layer, protecting and authenticating IP packets between participating IPsec devices, also known as peers. IPsec secures a path between a pair of gateways, a pair of hosts, or a gateway and host. As a result, IPsec can protect virtually all application traffic because the protection can be implemented from Layer 4 to Layer 7.

There are no issues with routing. IPsec functions over all Layer 2 protocols, such as Ethernet, ATM, or Frame Relay.

IPsec characteristics can be summarized as follows:

  • IPsec is a framework of open standards that is algorithm-independent.
  • IPsec provides data confidentiality, data integrity, and origin authentication.
  • IPsec acts at the network layer, protecting and authenticating IP packets.

IPsec security services provide four critical functions:

  • Confidentiality (encryption) – In a VPN implementation, private data travels over a public network. For this reason, data confidentiality is vital. It can be attained by encrypting the data before transmitting it across the network. This is the process of taking all the data that one computer is sending to another and encoding it into a form that only the other computer will be able to decode. If the communication is intercepted, it cannot be read by a hacker. IPsec provides enhanced security features, such as strong encryption algorithms.
  • Data Integrity – The receiver can verify that the data was transmitted through the Internet without being changed or altered in any way. While it is important that data is encrypted over a public network, it is just as important to verify that it has not been changed while in transit. IPsec has a mechanism to ensure that the encrypted portion of the packet, or the entire header and data portion of the packet, has not been changed. IPsec ensures data integrity by using checksums, which is a simple redundancy check. If tampering is detected, the packet is dropped.
  • Authentication – Verify the identity of the source of the data that is sent. This is necessary to guard against a number of attacks that depend on spoofing the identity of the sender. Authentication ensures that the connection is made with the desired communication partner. The receiver can authenticate the source of the packet by certifying the source of the information. IPsec uses Internet Key Exchange (IKE) to authenticate users and devices that can carry out communication independently. IKE uses several types of authentication, including username and password, one-time password, biometrics, pre-shared key (PSK), and digital certificates.
  • Anti-Replay Protection – This is the ability to detect and reject replayed packets and helps prevent spoofing. Anti-replay protection verifies that each packet is unique and not duplicated. IPsec packets are protected by comparing the sequence number of the received packets with a sliding window on the destination host or security gateway. A packet that has a sequence number that is before the sliding window is considered to be late or a duplicate packet. Late and duplicate packets are dropped.Anti-Replay Protection
IPsec framework


VPN traffic is kept confidential with encryption. Plaintext data that is transported over the Internet can be intercepted and read. Encrypt the date to keep it private. Digitally encrypting the data renders it unreadable until it is unencrypted by the authorized receiver.

For encrypted communication to work, both the sender and the receiver must know the rules that are used to transform the original message into its coded form. Rules are based on algorithms and associated keys. In the context of encryption, an algorithm is a mathematical sequence of steps that combines a message, text, digits, or all three with a string of digits that are called a key. The output is an unreadable cipher string. The encryption algorithm also specifies how an encrypted message is decrypted. Decryption is extremely difficult or impossible without the correct key.

Confidentiality is achieved through the encryption of traffic as it travels through a VPN. The degree of security depends on the key length of the encryption algorithm and the sophistication of the algorithm. If a hacker tries to hack the key through a brute-force attack, the number of possibilities to try is a function of the key length. The time to process all of the possibilities is a function of the computer power of the attacking device.

Encryption algorithm

DES and 3DES are no longer considered secure; therefore, it is recommended that AES be used for IPsec encryption. The greatest security for IPsec encryption of VPNs between Cisco devices is provided by the 256-bit option of AES. In addition, 512-bit and 768-bit Rivest-Shamir-Adleman (RSA) keys have been cracked and Cisco recommends using 2048-bit keys with the RSA option, if used during the authentication phase of IKE.

Symmetric Encryption

Encryption algorithms, such as AES, require a shared secret key to perform encryption and decryption. Each of the two networking devices must know the key to decode the information. With symmetric key encryption, also called secret-key encryption, each device encrypts the information before sending it over the network to the other device. Symmetric key encryption requires knowledge of which devices talk to each other so that the same key can be configured on each device.

Here is a synopsis for symmetric algorithms:

  • Uses symmetric key cryptography
  • Encryption and decryption use the same key
  • Typically used to encrypt the content of the message
  • Examples: DES, 3DES, and AES

How do the encrypting and decrypting devices both have a shared secret key? One could use email, courier, or overnight express to send the shared secret keys to the administrators of the devices. Another, more secure method is asymmetric encryption.

Asymmetric Encryption

Asymmetric encryption uses different keys for encryption and decryption. Knowing one of the keys does not allow a hacker to deduce the second key and decode the information. One key encrypts the message, while a second key decrypts the message. It is not possible to encrypt and decrypt with the same key.

Public key encryption is a variant of asymmetric encryption that uses a combination of a private key and a public key. The recipient gives a public key to any sender with whom the recipient wants to communicate. The sender uses a private key that is combined with the public key of the recipient to encrypt the message. Also, the sender must share its public key with the recipient. To decrypt a message, the recipient will use the public key of the sender with its own private key.

Here is a synopsis for asymmetric algorithms:

  • Uses public key cryptography
  • Encryption and decryption use a different key
  • Typically used in digital certification and key management
  • Examples: RSA

Data Integrity


Diffie-Hellman (DH) is not an encryption mechanism and is not typically used to encrypt data. Instead, it is a method to securely exchange the keys that encrypt data. (DH) algorithms allow two parties to establish a shared secret key that is used by encryption and hash algorithms.

Introduced by Whitfield Diffie and Martin Hellman in 1976, DH was the first system to utilize public key or asymmetric cryptographic keys. Today, DH is part of the IPsec standard. Also, a protocol known as OAKLEY uses a DH algorithm. OAKLEY is used by the IKE protocol, which is part of the overall framework called Internet Security Association and Key Management Protocol.

Like all cryptographic algorithms, DH key exchange is based on a mathematical sequence of steps.

Hash Algorithms

A hash, also called a message digest, is a number that is generated from a string of text. The hash is smaller than the text itself. It is generated by using a formula in such a way that it is extremely unlikely that some other text will produce the same hash value.

The integrity and authentication of VPN traffic is handled by hash algorithms. Hashes provide data integrity and authentication by ensuring that unauthorized persons do not tamper with transmitted messages.

The original sender generates a hash of the message and sends it with the message itself. The recipient parses the message and the hash, produces another hash from the received message, and compares the two hashes. If they are the same, the recipient can be reasonably sure of the integrity of the original message.

VPNs use a message authentication code to verify the integrity and the authenticity of a message, without using any additional mechanisms.

Hash-based Message Authentication Code (HMAC) is a mechanism for message authentication using hash functions. A keyed HMAC is a data integrity algorithm that guarantees the integrity of a message.

An HMAC has two parameters, a message input and a secret key that is known only to the message originator and intended receivers. The message sender uses an HMAC function to produce a value (the message authentication code) that is formed by condensing the secret key and the message input. The message authentication code is sent along with the message. The receiver computes the message authentication code on the received message using the same key and HMAC function as the sender used. Then the receiver compares the result that is computed with the received message authentication code. If the two values match, the message has been correctly received and the receiver is assured that the sender is a member of the community of users that share the key. The cryptographic strength of the HMAC depends upon the cryptographic strength of the underlying hash function, on the size and quality of the key, and on the size of the hash output length in bits.

There are two common HMAC algorithms:

  • MD5 – Uses a 128-bit shared secret key. The variable-length message and 128-bit shared secret key are combined and run through the HMAC-MD5 hash algorithm. The output is a 128-bit hash. The hash is appended to the original message and forwarded to the remote end.
  • SHA – SHA-1 uses a 160-bit secret key. The variable-length message and the 160-bit shared secret key are combined and run through the HMAC-SHA1 hash algorithm. The output is a 160-bit hash. The hash is appended to the original message and forwarded to the remote end.

Note: Cisco IOS also supports, 256-bit, 384-bit, and 512-bit SHA implementations.

There are two peer authentication methods:

  • PSK – A secret key that is shared between the two parties using a secure channel before it needs to be used. Pre-shared keys (PSKs) use symmetric key cryptographic algorithms. A PSK is entered into each peer manually and is used to authenticate the peer. At each end, the PSK is combined with other information to form the authentication key.
  • RSA signatures – RSA is made of the initial letters of the surnames of Ron Rivest, Adi Shamir, and Leonard Adleman, who first publicly described the algorithm in 1977. Digital certificates are exchanged to authenticate peers. The local device derives a hash and encrypts it with its private key. The encrypted hash, or digital signature, is attached to the message and forwarded to the remote end. At the remote end, the encrypted hash is decrypted using the public key of the local end. If the decrypted hash matches the recomputed hash, the signature is genuine.

IPsec uses RSA (public-key cryptosystem) for authentication in the context of IKE. The RSA signature method uses a digital signature setup in which each device digitally signs a set of data and sends it to the other party. RSA signatures use a certificate authority (CA) to generate a unique-identity digital certificate that is assigned to each peer for authentication. The identity digital certificate is similar in function to a PSK, but provides much stronger security. Each initiator and responder to an IKE session using RSA signatures sends its own ID value, its identity digital certificate, and an RSA signature value consisting of a variety of IKE values, all encrypted by the negotiated IKE encryption method (such as AES).

The Digital Signature Algorithm (DSA) is another option for authentication.


There are two main IPsec protocols depicted in Figure above:

  • Authentication Header (AH) – AH is the appropriate protocol to use when confidentiality is not required or permitted. It provides data authentication and integrity for IP packets that are passed between two systems. However, AH does not provide data confidentiality (encryption) of packets. All text is transported in plaintext. Used alone, the AH protocol provides weak protection.
  • Encapsulating Security Payload (ESP) – A security protocol that provides confidentiality and authentication by encrypting the IP packet. IP packet encryption conceals the data and the identities of the source and destination. ESP authenticates the inner IP packet and ESP header. Authentication provides data origin authentication and data integrity. Although both encryption and authentication are optional in ESP, at a minimum, one of them must be selected.


Figure above illustrates the components of IPsec configuration. There are four basic building blocks of the IPsec framework that must be selected.

  • IPsec framework protocol – When configuring an IPsec gateway to provide security services, an IPsec protocol must be selected. The choices are some combination of ESP and AH. Realistically, the ESP or ESP+AH options are almost always selected because AH itself does not provide encryption, as shown in Figure 3.
  • Confidentiality (If IPsec is implemented with ESP) – The encryption algorithm chosen should best meet the desired level of security: DES, 3DES, or AES. AES is strongly recommended, with AES-GCM providing the greatest security.
  • Integrity – Guarantees that the content has not been altered in transit. Implemented through the use of hash algorithms. Choices include MD5 and SHA.
  • Authentication – Represents how devices on either end of the VPN tunnel are authenticated. The two methods are PSK or RSA.
  • DH algorithm group – Represents how a shared secret key is established between peers. There are several options, but DH24 provides the greatest security.

It is the combination of these building blocks that provides the confidentiality, integrity, and authentication options for IPsec VPNs.

Note: This section introduced IPsec to provide an understanding of how IPsec secures VPN tunnels. Configuring IPsec VPNs are beyond the scope of this course.

Tunnel and Transport Modes

IPSec can be run in either tunnel mode or transport mode. Each of these modes has its own particular uses and care should be taken to ensure that the correct one is selected for the solution:

  • Tunnel mode is most commonly used between gateways, or at an end-station to a gateway, the gateway acting as a proxy for the hosts behind it.
  • Transport mode is used between end-stations or between an end-station and a gateway, if the gateway is being treated as a host—for example, an encrypted Telnet session from a workstation to a router, in which the router is the actual destination.

For more check here: