How to Avoid Unexpected AWS SageMaker Costs

Select Language:

If you’re noticing unexpected charges from SageMaker even after stopping your project, it usually means some resources are still running in the background. Here’s how you can identify and clean up those hidden components:

First, check for any MLflow tracking servers. These servers sometimes stay active and show up as charges labeled “RunTrackingServer.” Use the AWS CLI or API to list and delete these servers if they’re still running.

Next, look for inference endpoints. These endpoints keep running and can lead to ongoing charges even if you’re not actively using them. Make sure to delete any endpoints you no longer need.

Then, examine Canvas workspaces. Even if you log out of Canvas, there may be active “Workspace Instance Sessions” that aren’t visible in the console but still cost money. Confirm and shut down any open sessions.

Also, remember that stopping a project doesn’t automatically remove associated storage. Check for EBS volumes and other storage tied to your SageMaker projects. Storage costs around $0.112 per GB per month, so deleting unused volumes can save money.

It’s important to review all regions. SageMaker resources are region-specific, so resources in other regions might still be active and adding to your bill. Verify each region to ensure nothing is missed.

Be aware that some components might not shut down automatically. These include HyperPod clusters, pipelines, EMR clusters, RStudio environments, and TensorBoard apps. Review these and terminate them if they are no longer needed.

Also, check for any lingering VPC resources related to SageMaker. These can include subnets, security groups, or other network components that might still incur charges.

To find out exactly what’s costing you, use the AWS Cost Explorer. Filter by SageMaker under the service menu, and look at API operations to see which actions are generating charges. This helps you identify which resources need to be shut down.

If after all these checks you still can’t pinpoint the charges, reach out to AWS Support. They can provide a detailed breakdown and help you locate and terminate any lingering resources.

For ongoing management, consider setting up AWS Budgets. They can alert you before costs exceed your planned budget, helping you prevent surprises in your bill.

These steps should help you find and stop unnecessary expenses related to SageMaker.