AWS Best Practice for Stable Docker Images in SageMaker BYOC Async Inference

Select Language:

When working with container images, it’s better to pin by image digest instead of just using the tag. Doing so can help you avoid issues caused by updates to images without changing their tags.

Here’s what can happen: Even if you specify a specific tag like “2.6.0-transformers4.49.0-gpu-py312-cu124-ubuntu22.04,” the cloud provider may update the image behind that tag. For example, AWS can replace the image content linked to that tag with a newer version that has incompatible changes, such as updated dependencies like NCCL. Your build process might assume it’s working with the same image, but in reality, the image has changed, leading to unexpected problems.

The best way to prevent this is to pin the specific image digest in your Dockerfile. This means referencing the exact image version by its SHA256 digest. This way, every time you build or deploy, you’re using the same image, ensuring consistency. Your Dockerfile’s FROM line should look something like this:

FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference@sha256:

To find the correct digest, you can use commands like describe-images on your container registry to get the exact image version you’re referencing. When you build your image, record this digest so you know exactly which base image is used.

Controlling when you update your base image gives you stability and confidence. When you’re ready to update, pull the latest image, test it thoroughly, then update your Dockerfile with the new digest.

For your own custom images in services like Amazon ECR, consider enabling tag immutability. This setting prevents tags from being changed, making sure your deployments are always consistent.

Using image digest pinning helps keep your development process predictable and reduces surprises from automatic updates. You get the peace of mind that your builds and deployments happen exactly as planned, with updates happening on your schedule.