AWS ALB: How to Troubleshoot Intermittent Long Response Times

Select Language:

If you’re running an application on ECS with an Application Load Balancer (ALB) in the region ap-southeast-1, and you’ve noticed some intermittent delays in your API responses, here’s a simple way to understand and troubleshoot the issue.

You’re using IP-based target groups with ECS services on Fargate, and your ALB has a keep-alive timeout of 40 seconds. Your Express.js server has a keep-alive setting of 60 seconds, so ideally, the ALB should timeout first.

However, sometimes, your API calls take around 21 seconds to complete, even though your app’s logs indicate responses are quick, usually less than a second. During testing, when you send an API request every 5 seconds, you see this delay happening about 4 or 5 times within a couple of minutes. Your target groups are healthy, your ECS tasks have been running for hours without recycling, and increasing the number of tasks didn’t fix the issue. Additionally, your ALB connection logs show an error called “Failed UnmappedConnectionError” when these delays happen.

This kind of problem could be caused by network connection issues between the ALB and your ECS tasks, especially when the ALB is waiting for a response that isn’t arriving in time. The 21-second delay suggests that the ALB is timing out while waiting for the server to respond, which can happen if there’s a temporary network glitch, resource contention on your ECS tasks, or some intermittent server processing delay.

To fix this, consider the following steps:

– First, check your ECS tasks’ CPU and memory usage to see if they’re getting overwhelmed during those delays.
– Review your network settings, especially security groups and subnet configurations, to ensure there’s no intermittent connectivity problem.
– Increase the ALB’s idle timeout beyond 40 seconds if your API sometimes takes longer to respond.
– Verify if any other network security measures could be causing temporary blocks or delays.
– Log detailed metrics and traces to find patterns just before the delay occurs, which can help identify whether the issue is with the app, the network, or the load balancer.

By systematically checking these components, you can identify the root cause of the intermittent 21-second delays and ensure your application responds quickly and reliably.