Select Language:
Running AWS Database Migration Service (DMS) serverless in a way that reduces out-of-memory (OOM) errors during full load can be challenging, especially when you need to keep costs under control. Here’s a straightforward guide to help you optimize your migrations to prevent these issues while staying budget-friendly.
Start by breaking down your migration tasks into smaller parts. Instead of running one big task, split it into multiple smaller ones. This reduces the amount of memory each task uses and can help prevent OOM errors during the initial full load.
When loading tables, control how many tables are processed at the same time. Use the setting that limits the number of tables loaded in parallel—starting with lower values like 4 or 6 instead of the default 8. Also, adjust the number of load threads carefully, because more threads mean more memory use. Keep in mind that the total memory footprint depends on both the number of threads and the number of tables loaded simultaneously. Increasing the commit rate so that more records are transferred in each batch can significantly cut down on memory consumption.
If your tables do not contain large objects like LOBs (Large Objects), disable support for them to improve performance and reduce memory usage. If LOBs are necessary, set a size limit for handling these large objects, such as 32 KB, to avoid large data chunks consuming too much memory. Also, avoid full LOB mode if possible, and opt for limited size LOB mode.
To speed up the load process, consider delaying the creation of primary keys and indexes until the full load completes. This approach reduces the load on memory during the initial load. You can do this by setting the task to create primary keys and indexes only after the full load is finished. Additionally, configuring your target tables to be dropped and recreated during the load can make the process smoother and more efficient.
Proper memory management is key. Allocate between 50% and 70% of your available memory to the total memory limit setting. Also, control how long in-memory changes are kept before being written to disk by adjusting the keep time (usually around 60 seconds). Monitoring your system closely with native metrics like available memory and swap usage helps you catch issues early. If you notice high swap usage, it indicates that your system is running out of memory. In such cases, temporarily increasing your DMS’s allocated resources during the full load phase can help. Once the full load is finished and change data capture (CDC) begins, you can lower the resources back to save costs.
Since your system handles CDC better, you might find it more efficient to split your migration into two phases. First, run just the full load task with settings that don’t process changes. Once the initial load is done, start a CDC-only task to keep your data synchronized efficiently. This approach not only manages resources better but can also reduce costs because you only use a high compute unit (DCU) level during the initial load.
Make sure your target system can handle the write throughput without slowing down — backpressure can cause increased memory usage. Also, prioritize smaller tables for initial loads or exclude very large ones to prevent memory issues. If some tables are especially troublesome, consider loading them separately or customizing load settings per table.
By following these tips, you should notice fewer OOM errors and better overall performance, all while keeping your costs in check. Proper task segmentation, careful setting adjustments, and system monitoring are the keys to a successful migration with AWS DMS.
ChatGPT
Perplexity
Gemini AI
Grok AI
