AWS SSM Port Forwarding: Persistent Tunnel Failure & Reconnections

Select Language:

If you’re managing long sessions for port forwarding between an EC2 instance and an on-premise hybrid server, you might run into a problem where your tunnel stops working after several hours, even though everything seems fine at first. Here’s a clear guide to explain what’s happening and how you can keep your connection alive longer.

Your setup involves an Amazon Linux 2023 EC2 instance with the latest session-manager-plugin and an on-premise agent. You’re using the AWS-StartPortForwardingSession document to create a tunnel that stays open for a long time, typically between 12 to 24 hours. To manage idle time, you use the ResumeSession option, which helps keep the session alive during idle periods.

However, after a while, the tunnel suddenly stops working. When checking the logs, you’ll see that the SSM service is closing the WebSocket connection every 60 minutes. The on-premise side successfully reconnects most of the time, but occasionally, the connection fails on the EC2 side. When this happens, the on-premise agent gets a message that says, “Session is already terminated,” indicating that the EC2 instance has given up and ended the session.

Here’s what the logs show:

Every hour, the on-prem agent receives a message indicating the WebSocket was closed normally.
After this, the agent attempts to reconnect. Usually, it manages to reconnect successfully, and the tunnel continues working.
But sometimes, the EC2 side fails to reconnect, and the session is terminated unexpectedly.

The key issue seems to be that WebSocket connections are being closed after about an hour, and if reconnection doesn’t happen swiftly enough, the session gets dropped. Once dropped, the session cannot be restored without starting a new one.

To keep your session working longer without failures, consider the following solutions:

Adjust the WebSocket timeout settings: Investigate if there are parameters in your network or AWS configuration that control WebSocket keep-alive intervals. Extending these intervals can help prevent disconnections.
Implement keep-alive messages: Use scripts or tools to send small packets at regular intervals to maintain the connection active. This can be configured in your networking equipment or within the software managing the session.
Configure the session timeout settings: If the AWS-StartPortForwardingSession allows, increase the session or idle timeout limits to prevent sessions from closing automatically.
Monitor and alert: Keep an eye on your session logs so you can intervene quickly when a disconnect occurs frequently.

Remember, the main challenge is the WebSocket connection’s lifecycle. Managing these and ensuring continuous activity can help your tunnels stay open as long as you need. This way, your long-lived port forwarding sessions remain stable and reliable over extended periods.