AWS CLI: Fixing 'aws s3 cp' Directory Exclusions on Windows & VS Code

Select Language:

If you’re trying to copy only specific HTML files from your local folder to an Amazon S3 bucket while excluding the node_modules directory, it can be frustrating when the usual commands don’t work as expected. Here’s a straightforward way to get it done, without including the unwanted directories like node_modules.

First, it’s important to understand that including or excluding directories with the aws s3 cp command using wildcards and patterns isn’t always reliable, especially with complex folder structures. Instead of relying solely on exclude and include options, a more precise approach is to perform the copy operation in smaller, controlled steps.

Here’s what you can do:

Use the aws s3 sync command but with specific filters. Although you’ve mentioned that using sync isn’t helpful, this method can give you more control. If you prefer to stay with the cp command, you’ll need to manually specify the files or use a script to target only the files you want.

ADVERTISEMENT
One effective way is to create a list of all the HTML files that are directly under your main directory, excluding subfolders like node_modules. You can do this using a command line or a simple script in PowerShell.

For example, in PowerShell, run something like:

powershell
Get-ChildItem -Path “C:\Users\15102\Documents\aws\marcab3\s3” -Filter *.html -File | Where-Object { $_.FullName -notmatch “nodemodules” } | ForEach-Object {
$dest = “s3://marcab3/” + $.Name
aws s3 cp $_.FullName $dest
}

This script searches for all HTML files in your main directory, filters out any files inside the node_modules directory, and uploads only those files.

Make sure your directory structure and file paths are formatted correctly for your operating system and that you’re using the proper quotes and escape characters if needed.
If you’re working with nested directories and only want the HTML files from the main directory (not subfolders), this approach ensures you only upload the files you intend to.

Remember, the key is to avoid trying to exclude directories with complex patterns in the aws s3 cp command because it can become unreliable. Instead, filter your files upfront and upload only those. This method gives you clean control over what gets uploaded.

Hopefully, this solution helps you upload only the desired HTML files while skipping the node_modules folder. If you keep encountering issues, reviewing your directory structure and carefully scripting the file selection will often solve the problem.