Proxying SSH connections

Learn how to use SSH to traverse network boundaries and proxy SSH connections through other hosts.

Early in my career, I was fortunate enough to be introduced to SSH and all the extra things you can do with it besides opening a terminal on a remote host.

One of the features I find myself using frequently is the ProxyCommand. The man page states:

Specifies the command to use to connect to the server

Using this directive in your ~/.ssh/config file you can route SSH connections to remote hosts through other machines on the network. To say this slightly differently, you can force SSH connections to a remote host to go through another host first instead of taking a direct network path.

Why is this important?

Looking at the diagram at the top of this page we can see a user, let's call him Bob, who has SSH access to an AWS instance in a public subnet. Now Bob can SSH into the Bastion host and then make another SSH connection to the backend instance he wants access to, however, this would be tedious and not very productive for a few reasons:

• No one wants to make 2 SSH connections every time to access a server.
• You can’t ssh into a group of hosts in a Bash script to automate tasks.
• If Bob needs to transfer files with SCP he would have to copy the files to and from the Bastion host every time.

Now by adding a Host section in Bob’s ~/.ssh/config file for the remote subnet and providing the ProxyCommand directive he can SSH directly into the backend AWS instances. Bob doesn’t have to worry about manually connecting to the Bastion host first.

To help demonstrate how the ProxyCommand works in SSH I have created a simple example in Docker that is available on GitHub: https://github.com/mgale/examples.git

Docker Example

Prerequisites:

• Docker — Tested on version 19.3 and 20.10.2
• Docker-compose — Tested on version 1.24.1 and 1.25.0
• Git

Use a User with docker permissions or run sudo docker-compose up

To get the example going, in a terminal window execute the code below:

git clone https://github.com/mgale/examples.git
cd examples/ssh-proxycommand
docker-compose up

Once the containers are ready, in a second terminal window run:

ssh-test.sh

If everything worked as expected you should see the output of the docker ps command and two ssh connections:

###########################################################
## SSH into proxyServer
 03:49:46 up 3 days,  1:08,  0 users,  load average: 0.05, 0.13, 0.10
Hostname: proxyServer
###########################################################
## SSH into backendServer
 03:49:46 up 3 days,  1:08,  0 users,  load average: 0.05, 0.13, 0.10
Hostname: backendServer

The above example creates two containers based on Ubuntu 20.4’s image with the following additions:

• Installs OpenSSH server.
• Adds SSH keys for the Ubuntu user.

You can see the alterations that are made to the containers by examining the docker-compose.yml file and bin/startup-script.sh. I have specified bin/startup-script.sh as the command that docker runs when the Ubuntu image is started. That way I do not have to manage a custom image, you can see that the image used is from Ubuntu and are free to examine the changes I am making at startup.

One container is labelled proxyServer which represents the Bastion host and exposes port 2222 on the network that maps to the SSH process in the container. A second container labelled backendServer is on the same internal docker network as proxyServer, however, it is not directly accessible from the host machine because no ports are exposed.

This setup can be very similar to a lot of real-world situations, there is a host on a network that you need access to, however direct network connections are not possible.

All the magic happens in the ssh-config file which represents your ~/.ssh/config file.

Let's quickly go over the important parts of the file:

• Line 3: We override the default SSH port because the proxyServer container is exposing port 2222.
• Lines 6,16: Show the private keys being used, each host in our example has its own private key. The intermediate host and the backend host don’t require the same private key.
• Line 21: This is the critical line, it tells SSH that when we want to connect to the Host backendServer that it should connect using the command “ssh -F ssh-config -W %h:%p ubuntu@proxyServer”. The -W argument to SSH states that

Requests that standard input and output on the client be forwarded to host on port over the secure channel.

That means that SSH will first make a secure connection to proxyServer and request that our client's STDIN and STDOUT be forwarded to host %h, which is our intended backend. It is important to understand that the SSH argument “-F ssh-config” is normally not part of the ProxyCommand. It is only used in this example because I want to override the default (~/.ssh/config) configuration file.

As you can see using the ProxyCommand directive can be very powerful, however, it doesn’t always have to be used with Bastion hosts. Sometimes there is a routing boundary in the way …

VSCode, AWS Workspaces and SSH ProxyCommand

Currently, I am testing out AWS Workspaces as an alternative to running local VM’s, there are a few reasons for this but I won’t get into them in this article.

By default access to AWS Workspaces is done through a client app or web browser using PCoIP (PC over IP) or WorkSpaces Streaming Protocol (WSP) and so far this seems to work really well.

However, I want to keep the current workflow of using Visual Studio Code (VS CODE) with the Remote SSH extension (https://code.visualstudio.com/docs/remote/ssh). From the website

This lets VS Code provide a local-quality development experience — including full IntelliSense (completions), code navigation, and debugging — regardless of where your code is hosted.

Normally VS Code would use the local SSH client and connect to my local VM providing access to the source code, plugins, etc.

Now I need VS Code to connect to the AWS Workspace instance instead which is only accessible over a VPN connection. To facilitate this I could run the required VPN software directly on my Macbook but …

• I prefer not to install anything on my Mac unless necessary.
• I already have the VPN client setup and configured inside my local VM along with other required utilities.

SSH ProxyCommand to the rescue!!!

So at this point, we have:

• My Macbook can SSH into the local VM
• The local VM can SSH into the AWS Workspace instance

All I needed to do was add a Host entry for the AWS Workspace instance with the ProxyCommand routing the connection through the local VM in my ~/.ssh/config on my Macbook.

VS Code is now able to access the workspace instance over SSH allowing me to continue with the same workflow while validating AWS Workspaces.

Last Example

The below SSH config file (~/.ssh/config) is from a Linux VM I use regularly. I have changed some names and removed content that was not required.

The important pieces are still there:

• I have configured a Host labelled awsBastionBox
• I have a Host for subnet 10.183.*.* which uses the ProxyCommand allowing me to access 10.183.*.* through the AWS Bastion Host (awsBastionBox)
• I have a lot of different private keys, I do that to compartmentalize work. For example, GitHub might be for material that is publicly accessible and GitLab could be internal private material.

michaelgale@michaelgale-pvp:~$ cat ~/.ssh/config 
Host github.com
 Hostname github.com
 IdentityFile /home/michaelgale/.ssh/gale_github
 ServerAliveCountMax 100
 ServerAliveInterval 1
 TCPKeepAlive yes
Host gitlab.com
 Hostname gitlab.com
 IdentityFile /home/michaelgale/.ssh/gale_gitlab
 ServerAliveCountMax 100
 ServerAliveInterval 1
 TCPKeepAlive yes
Host gitlab.*
 IdentityFile /home/michaelgale/.ssh/gale_gitlab
 ServerAliveCountMax 100
 ServerAliveInterval 1
 TCPKeepAlive yes
Host git-codecommit.*.amazonaws.com
  User <key-here>
  IdentityFile ~/.ssh/gale.michael.aws.codecommit
Host awsBastionBox
  Hostname <aws-nlb-dns-name>
  User michaelgale
  IdentityFile /home/michaelgale/.ssh/michaelgale-aws.prod
  TCPKeepAlive yes
Host 10.183.*.*
  User michaelgale
  ProxyCommand ssh -W %h:%p michaelgale@awsBastionBox
  IdentityFile /home/michaelgale/.ssh/michaelgale-aws-internal.key
Host *
    IdentitiesOnly yes
    StrictHostKeyChecking=yes
    CheckHostIP=yes
    ServerAliveInterval=10
    TCPKeepAlive=yes

If you are looking for more SSH information checkout

• sshuttle: where transparent proxy meets VPN meets ssh: https://github.com/sshuttle/sshuttle
• https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/