Combine all s3 archive data for a single device

Feature Request
1

We have been using the built-in feature to archive data to an Amazon S3 bucket which is working fine and gives us the ability to retain data beyond the 6month limit of the Losant platform. Data in the bucket is separated into a new csv file each day for each device. We need to be able to work with this data in a format similar to the ‘data export’ feature the platform provides, meaning we get a single file with all of the archived data for a single device. Doing this within S3 is cumbersome because you have to download all of the daily files and then merge them together externally. Do you know of a simple tool or lambda function that has this functionality? We would like to use something readily available rather than writing our own tool and re-creating the wheel.

Thanks!

3

Hello @Edward_Cline,

This is a great question.

You mentioned Lamdba Functions as a way to combine the CSVs, and that’s immediately where my brain went as well. I’m not sure of a pre-baked Lambda Function that does this already, but I’ll dig around and see what I can find for you.

There are some other options that integrate well with data already stored in S3. AWS Athena and AWS Redshift treat your S3 storage as a Data Lake, or as a big dataset.

Please let me know if any of this works for you or if this is what you were looking for.

Thank you,
Heath