Combine all CSV Files within S3 Bucket into Single File!

[AmericanY]

I do have a list of ~ 500 CSV files within S3 Bucket which am looking to concatenate all of them into a single CSV file.

Note that each file is around 1 GB (6 Milions of lines)

Is that possible without downloading all files ?

[aBurmeseDev]

Hi @AmericanY, thanks for reaching out. Could you elaborate more on what you’re trying to accomplish?
For reading/combining multiple CSV files in S3, I found these Stack Overflow posts here and here that might be helpful.

[AmericanY]

@aBurmeseDev Thanks for taking time checking my issue. Actually the posts which you mentiond is not helpful within my case. The reason behind that, I'm dealing with 520 files (each file around 1GIB) containing ~ 6 Million of lines. so reading each file and append it to DF in order to combine all of them into single file, Will eat up all the memory + it's all based on connection speed as well.

My question is, Is there a way where i can combine the files into single one within S3 locally without needing of read each file and store it in memory ?

[aBurmeseDev]

@AmericanY Sorry for the delay. Unfortunately, S3 is just a simple key-value storage and there’s no such feature like it as of today. Several users have requested something similar to this, which is to zip everything in an S3 bucket as discussed here. You could try downloading everything using one of the third-party services mentioned there and then combine the CSV files using other third-party services. Or if there are other files in your S3 bucket that you want to ignore, then you could use our include/exclude filters with an aws cp command to only download files with a .csv extension.