Using archives for processing using external tools by periodically (every day) make a copy of all device data from the application into an Amazon S3 Storage buckets.
Is there a way to change the file format of the archived file from CSV to JSON?
Is there a way to separate the device archiving in a way that will send each device to a different bucket directory? Currently I can only choose to to archive all devices into one specific bucket directory.
Is there a way to parse the device information that is going to be archived in a way that will represent averaging of the data (i.e 30 min average)?
For all of your questions, the answer is no. Unfortunately, Archiving can’t do that as it is today. However, a lot of these questions sound very close to batch processing use cases.
Using Losant Notebooks (combined with a Workflow), you’re able to run a Juypter Notebook every day. This would give you the ability to manipulate format, pre process the data, or change the storage destination.
Building off of @anaptfox response above, using Notebooks and workflows will allow you to change the file type from CSV to JSON (question 1), with this process being performed by your Notebook. The output from that process can then trigger a workflow that places the data into a specific AWS bucket using the AWS Put Node (question 2).
For your third question, are you asking if you can preprocess the data to show a 30-day average prior to sending it to a Notebook or to AWS? Notebooks would be the preferred method of performing any batch processing/analytics. Let us know if we’ve misinterpreted your question.
To be clear, the process we’ve described would only work for data less than 5MB, which is the maximum size of the payload.