How to Send ECS Task Logs to Two Locations Using AWS

Select Language:

If you want to send your ECS task logs to both CloudWatch Logs and an external endpoint like New Relic, there’s a straightforward way to do it without creating complex configurations that require frequent maintenance.

A common setup involves using the awsfirelens driver with a sidecar container, typically Fluent Bit. While this method works well, it does require creating and maintaining a custom Fluent Bit configuration file, which can become a hassle over time.

Instead, consider a simpler approach: configure your ECS tasks so that their logs are sent directly to CloudWatch Logs and then set up an additional process to forward those logs to your external endpoint.

Here’s a step-by-step solution:

Configure ECS Task to Send Logs to CloudWatch:
When defining your task, specify the log driver as awslogs. This setup makes sure your task’s logs go straight into CloudWatch without extra configuration. This method is easy to manage and reduces the need for custom configurations.
Create a Log Group and Stream in CloudWatch:
Set up your CloudWatch Log Group and Log Stream. When configuring your ECS task, point your logs to this group and stream.
Set Up a Log Forwarding Process:
Since your goal is also to send logs to an external service like New Relic, you can put a small, dedicated process that reads logs from CloudWatch and forwards them. This can be a Lambda function triggered by new log entries or a lightweight server running on your infrastructure.
- Lambda Approach:
  Create a Lambda function that listens for new log events in CloudWatch Logs, then forwards those logs to New Relic’s API. This gives you real-time forwarding without managing complex configurations inside ECS.
- Server-based Approach:
  Alternatively, run a small container or script that periodically fetches logs from CloudWatch and sends them to External endpoints.

This setup allows your ECS tasks to send logs directly to CloudWatch, and then your log forwarding process handles external endpoint delivery. It simplifies maintenance and provides flexibility, especially valuable when testing different storage options like Infrequent Access Logs in CloudWatch.

By adopting this approach, you avoid managing extra Fluent Bit configurations and streamline your logging process, making your proof-of-concept more manageable and reliable.