Setting up IPSEC VPN gateway with Xauth and PSK

Many people map ports on their home or small office routers, so their webserver or any other service is reachable from the internet. It is ok, but can be potentially a security flaw. It would be much more secure if instead of mapping ports you installed a VPN server.

With VPN in place first you have to connect (and authorize) on the VPN, and only then you can access your local LAN services.

Since the processing power required to run a VPN gateway isn’t very high, this is a perfect use case for a Raspberry PI.

Raspberry PI default OS is Raspbian (which is basically a Debian distribution specifically compiled for the RasPI CPU), so this tutorial will be based on Debian. Please know the same service would also work on Red Hat or CentOS, but directory paths to configuration files will be different.

We assume Raspberry PI is behind NAT, so it will have a local IP address, and on the router there will be two ports mapped to the RasPI to allow incoming VPN connectivity.

As sample road-warrior client we can use Cisco VPN Client for Windows or Apple iOS built in Cisco VPN.

Here’s our settings - Raspberry PI local IP address - our local LAN subnet - Raspberry PI VPN client address - VPN clients subnet
vpn-clients - pre-shared group username
vpnsecret321 - pre-shared group password
using /etc/shadow for client authentication

First install the racoon package

sudo apt-get update && apt-get install racoon

Once installed, we will have to set up racoon config files.

The main config file is located in /etc/racoon/racoon.conf and should look like this:

path pre_shared_key "/etc/racoon/psk.txt";

log info;

    natt_keepalive 10sec;

remote anonymous {
    exchange_mode aggressive;
    doi ipsec_doi;
    generate_policy on;
    lifetime time 28800 sec;
    passive on;
    initial_contact off;
    nat_traversal on;
    proposal_check obey;
    proposal {
        encryption_algorithm 3des;
        hash_algorithm sha1;
        authentication_method xauth_psk_server;
        dh_group 2;

mode_cfg {
    auth_source system;           # using system passwords from /etc/shadow
    network4;        # virtual IP adress of VPN clients; won't be visible in ipconfig
    pool_size 254;                     # client pool size
    dns4;              # local lan DNS
    default_domain "";  # our local domain
    split_network include;   # our LAN network. Can be more networks sperated by ,
         # this will be injected into routing table of our VPN clients
    split_dns "";             # our local domain
    banner "/etc/racoon/motd";            # message of the day file, 
    save_passwd on;                           # can client save passwords?
    pfs_group 2;                               # required by cisco vpn client

sainfo anonymous {
    pfs_group 2;
    lifetime time 3600 sec;
    encryption_algorithm aes,3des ;
    authentication_algorithm hmac_sha1,hmac_md5 ;
    compression_algorithm deflate;

Now in /etc/racoon/psk.txt we have to define pre shared keys:

    vpn-clients vpnsecret321

vpn-clients is group username and vpnsecret123 is group password

We can set an optional Message of the Day in /etc/racoon/motd file:

    You have successfully connected to the VPN service.

To allow network traffic we will have to update the iptables rules

#vpn rules - incoming
    /sbin/iptables -A INPUT -p udp --dport 500 -j ACCEPT
    /sbin/iptables -A INPUT -p udp --dport 4500 -j ACCEPT
    /sbin/iptables -A INPUT -p esp -j ACCEPT
    /sbin/iptables -A INPUT -p ah -j ACCEPT
    /sbin/iptables -A INPUT -p ipcomp -j ACCEPT

#vpn rules - forward

    /sbin/iptables -A FORWARD --src -j ACCEPT

#vpn rules - masquerade

    /sbin/iptables -t nat -A POSTROUTING -p esp -j ACCEPT
    /sbin/iptables -t nat -A POSTROUTING -p ah -j ACCEPT
    /sbin/iptables -t nat -A POSTROUTING -p ipcomp -j ACCEPT
    /sbin/iptables -t nat -A POSTROUTING -s -d -j ACCEPT
    /sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

The last step is routing

Racoon is not creating a pseudo interface, so the default routes to VPN client network won’t be created either.

To enable the traffic back to VPN clients we have to add:

    route add -net netmask gw

It’s best to add the above rule to /etc/network/interface so it will be persistent across reboots:

# The loopback network interface
  auto lo
  iface lo inet loopback

# The primary network interface
  auto eth0

iface eth0 inet static
  up route add -net netmask gw dev eth0

The very last step is to enable routing on the linux kernel.

Open /etc/sysctl.conf, find the line:


Remove the comment (‘#’ sign), save the file. It will work after reboot.
Now on your router map udp ports 500 and 4500 ports to your VPN server ( in this example)

2 thoughts on “Setting up IPSEC VPN gateway with Xauth and PSK

  1. I can change my ip to any of 100+ locations with arcvpn. I’m having good service with them so far.

    • I’ll check it out. This post is however about a different thing: accessing your home PC or NAS or server securely from anywhere in the world.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.