How To Handle AWS Callbacks Without Using Task.Token

Select Language:

If you’re working with AWS Step Functions to process a large number of audio files stored in an S3 bucket, you might want to control how many transcription jobs run at once. For example, when dealing with around 18,000 files, it’s helpful to make the workflow wait until each transcription is complete before submitting more jobs. This helps prevent exceeding service limits and keeps your process manageable.

A good way to do this is to use the “Wait for Callback” feature available in the StartTranscriptionJob API. This feature allows your workflow to pause after starting a transcription, and then continue only after receiving a callback indicating completion. However, passing the task token—used as a callback identifier—can be tricky.

In your setup, you’re attempting to pass the task token using a tag called “TaskCallbackToken,” pulling its value from $states.context.Task.Token. Unfortunately, this approach doesn’t work because $states.context.Task.Token isn’t available at that point; it’s undefined. Without this token, the callback can’t be properly associated, and the workflow won’t wait for the job to finish.

To fix this, you should pass the $taskToken directly into the startTranscriptionJob.waitForTaskToken call, instead of trying to set it through tags. Here’s what the adjusted step should look like:

json
“StartTranscriptionJob”: {
“Type”: “Task”,
“Resource”: “arn:aws:states:::aws-sdk:transcribe:startTranscriptionJob.waitForTaskToken”,
“Parameters”: {
“TranscriptionJobName”: “{% $join([‘BatchTranscribe’, $states.context.Execution.Name],’-‘) %}”,
“Media”: {
“MediaFileUri”: “{% $join([‘s3://’, $Bucket,’/’, $states.input.Key]) %}”
},
“LanguageCode”: “en-US”,
“OutputBucketName”: “{% $Bucket %}”,
“OutputKey”: “{% $replace($states.input.Key, /\/source\/(.+)\.(wav|mp4)/, ‘/output/$1.json’) %}”,
“Settings”: {
“ShowSpeakerLabels”: true,
“MaxSpeakerLabels”: 3
},
“TaskToken”: “$$.Task.Token” // Pass the task token directly here
},
“End”: true
}

Note the key change: instead of trying to fetch the token from $states.context.Task.Token, use $$.Task.Token directly, as this is how Step Functions exposes the current task token within tasks that wait for callbacks.

This way, your workflow will pass the correct token to the transcription service, and once the job completes, your callback handler (which watches for transcription status changes) can notify the execution to proceed. Make sure your callback URLs and permission policies are set correctly to enable this communication.

By making this change, you ensure your process waits for each transcription to finish, controlling resource use effectively during large batch processing.