How to Set Up AWS EKS & EFS for Batch Computing

Select Language:

Here’s a simple way to use persistent storage with AWS Batch on EKS, especially if you’re aiming to mount an existing Kubernetes Persistent Volume Claim (PVC).

First, understand that AWS Batch on EKS now supports mounting an existing PVC directly in the job definition. This means you don’t need to rely on HostPath makeshift storage. However, be aware that the CDK’s EksVolume might not currently support PVCs through the high-level APIs. Instead, you can use a lower-level approach with CfnJobDefinition to specify your PVC until CDK updates its support.

Start by creating your persistent storage in the Kubernetes cluster. The easiest method is to define a Persistent Volume (PV) and a Persistent Volume Claim (PVC) pointing to an Elastic File System (EFS). Here’s an example using a static PV:

yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-pv
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:

ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: efs.csi.aws.com
volumeHandle: fs-12345678

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-pvc
spec:
accessModes:
ReadWriteMany
resources:
requests:
storage: 100Gi
volumeName: efs-pv

Apply this configuration with:

bash
kubectl apply -f efs-pv-pvc.yaml

Once your PVC is ready, you can reference it in your Batch job definition. For CLI users, define your job like this:

json
{
“jobDefinitionName”: “eks-efs-job”,
“type”: “container”,
“platformCapabilities”: [“EKS”],
“eksProperties”: {
“podProperties”: {
“containers”: [{
“name”: “app”,
“image”: “public.ecr.aws/amazonlinux/amazonlinux:2”,
“command”: [“bash”,”-lc”,”df -h; ls -la /mnt/efs; sleep 30″],
“volumeMounts”: [{ “name”: “efsvol”, “mountPath”: “/mnt/efs” }]
}],
“volumes”: [{
“name”: “efsvol”,
“persistentVolumeClaim”: { “claimName”: “efs-pvc”, “readOnly”: false }
}]
}
}
}

bash
aws batch register-job-definition –cli-input-json file://jobdef.json

And then submit your job:

bash
aws batch submit-job –job-name test-efs –job-queue YOUR_QUEUE –job-definition eks-efs-job

If you’re using CDK, note that support for PVCs might not be fully integrated yet. You can work around this by using a low-level CloudFormation resource (CfnJobDefinition) to specify the PVC in your job’s volume configuration, allowing you to mount existing storage.

This approach gives you a reliable way to connect your Batch jobs with persistent storage, making data sharing and storage management much simpler on AWS EKS.