Thinknear is delighted to announce that our
tn_s3_file_uploader is released as open-source under APLv2.
tn_s3_file_uploader is a Ruby gem that we use internally to upload log files to Amazon S3 where they can be stored until we need to retrieve them for further analysis or processing.
Thinknear's Real Time Bidding service produces billions of log entries per day and we have set up a simple but robust pipeline around rotating those logs and storing them on Amazon S3. The two key parts of this pipeline have been released to the public as open-source software.
Second the aforementioned ruby gem uploads the rotated logs on S3 with a year/month/day/hour partitioned destination path that makes it easy to retrieve and analyze the logs for a specific period of time.
Thinknear Engineering values the openness of the open source software model and we would like to invite anyone that finds
tn_s3_file_uploader useful to contribute. Visit the project's issue tracker page to submit questions, bugs or new features.