Tn_s3_file_uploader Released

Announcement

Thinknear is delighted to announce that our tn_s3_file_uploader is released as open-source under APLv2.tn_s3_file_uploader is a Ruby gem that we use internally to upload log files to Amazon S3 where they can be stored until we need to retrieve them for further analysis or processing.

You can visit the gem's page on Ruby Gems. Please, visit the project's getting-started page for more information on how to install and use the gem.

Motivation

Thinknear's Real Time Bidding service produces billions of log entries per day and we have set up a simple but robust pipeline around rotating those logs and storing them on Amazon S3. The two key parts of this pipeline have been released to the public as open-source software.

First our AWS Elastic Beanstalk logrotate and config scripts rotates the logs produced by our service.

Second the aforementioned ruby gem uploads the rotated logs on S3 with a year/month/day/hour partitioned destination path that makes it easy to retrieve and analyze the logs for a specific period of time.

Contribute

Thinknear Engineering values the openness of the open source software model and we would like to invite anyone that finds tn_s3_file_uploader useful to contribute. Visit the project's issue tracker page to submit questions, bugs or new features.

If you want to hack around the code, fork the project on GitHub and see the how to contribute page if you want to submit your code to the project.

Contact us. Let's create magic together.

Our Newsletter is good. Sign up so we can deliver the goods. (Not bad, huh?)

Request a call