![]() ![]() There is a sweet spot for parallel listing as well. Without some control over the concurrency, the poor NodeJS instance ran out of memory and crashed. What seemed like a cludge at first ended up being a performance win since it allowed us to leverage multiple CPU cores.Īll of this aggressive parallel copying needed limits however. We overcame copy-object's lack of support for objects larger than 5gigabytes by shelling out to aws s3 cp for these larger files. We just needed to fire off simultaneous calls to copy-object. It’s blazingly fast, performing 100 or more S3 list-object requests in parallel.Ĭopying S3 items was more straightforward. Doing efficient bisection over an unknown key distribution can be tricky, but we made it work. This was the hardest part of the project. We solved the listing problem with a cleaver use of divide-and-conquer and AWS S3 SDK's startAfter option introduced with listObjectsV2.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |