Bug Fix: Nil pointer exception when `WriteHeaders` is set to fal

If writeHeaders set to false, dont write to csv buffer.

Signed-off-by: thatInfrastructureGuy <[email protected]>

Return Row Count.

• Return number of rows processed
• Use env var LOG_LEVEL. Defaults to "INFO"

Upgrade Dependencies

• Upgrade pgzip and aws-sdk-go

Use 4 Upload Threads by Default

• Updated documentation.
• Use static upload threads for consistent memory usage.

Remove Sql Batch buffer.

• Remove sql batch buffer.
• Preset initial starting value for csvBuffer. See #8

Memory Optimization

Reduce fmt.Sprintf calls whereever possible #7

Abort S3 upload when SIGTERM/SIGINT is received.

• Takes care of cleaning up unfinished uploads.
• S3 costs money even if upload is unsuccessful. Hence, we need to abort the uploads explicitly if program encounters error.

Introduced UploadConfig() and WriteConfig()

• UploadConfig() replaces DefaultConfig()
• WriteConfig() replace New()

Major bug fix: Corrupted Gzip on S3 upload

• Fixed a bug where s3 upload would corrupt gzip.
• Reduced memory footprint.
• Disabled some debug logs.
• Adjusted Defaults.
• Added documentation and comments.

Upload directly from sql to s3

Features

• Consistent memory, cpu and network usage whether your database has 1 Million or 1 Trillion records.
• Multipart upload if file size > 5 Mb
• Buffer size is set to 5 Mb by default.
• No need for writable disk as we directly upload to s3.
• Log levels set: Error, Warn, Info,Debug, Verbose
• Various bug fixes.
• Breaking API changes except for WriteFile(file, rows) and Write(w io.writer) methods.
• New methods: UploadToS3(rows) and Upload() to upload to s3

---

Caveats

• Maximum of 10000 part uploads are allowed by AWS. Hence, (5Mb x 10000) 50Gb of gzipped data is supported by default settings.
• Increase buffer size if you want to reduce parts or have more than 50Gb of gzipped data.
• Currently only supports upload to AWS S3 API compatible storage.