How to Fix AWS OpenSearch Blue/Green Deployment Stuck at Cleanup

Select Language:

If your OpenSearch Service domain is stuck in a blue/green deployment with six nodes running instead of the intended two, and the shard migration seems finished while the cluster remains healthy, don’t worry. This is a common issue that often occurs during configuration updates.

During blue/green deployments, OpenSearch creates a new environment with new nodes and moves data from old nodes to the new ones. Sometimes, the process can get stuck in a “Processing” or “Modifying” state even when everything looks fine on the surface.

This usually happens because of a few reasons:
– The shard migration process faces problems, even if the shards look healthy.
– Validation checks fail, showing errors.
– The system cannot properly shut down the old nodes after moving data.

Some common causes for these issues include:
– Very large shards, over 50 GB, which take longer to copy.
– High activity levels like many searches or updates, which increase resource use.
– Not enough disk space on the new nodes.
– Reaching the limit on the number of shards allowed.

To troubleshoot and hopefully resolve the issue, start by checking some key metrics on Amazon CloudWatch:
– Free Storage Space
– Cluster Status
– Whether index writes are blocked
– CPU Utilization
– JVM Memory Pressure

Next, review your domain’s status in the console or through API calls. Look for any error messages or failed activities during the transition.

You can also check the progress of shard migration by using the API command that shows ongoing recovery operations. If all shards are marked as started and no tasks are pending, it indicates the migration finished but cleanup didn’t complete properly.

At this point, it’s important to understand that this situation often requires support from AWS. Because the cleanup process involves internal system actions, you can’t manually fix this alone. It’s best to contact AWS Support, providing details such as your domain name, region, account ID, and Change ID. The support team can investigate the issue thoroughly and safely finish the deployment process.

In the meantime, avoid making any more changes to your domain. Adding new configurations could complicate the problem further.

Remember, expert assistance is available to help you resolve this smoothly. Following these steps will help clarify the situation and lead to a proper fix.