Send device data to AWS-S3

I need to send some device data from multiple devices to AWS-S3.
Would the AWS S3 Put Node be my best option?
How do you suggest I generate the file for upload?

Let’s say I want to upload past weeks raw time series data.

Or is it better using the Archiving?

Hi @Lars_Andersson,

You have multiple options for how to accomplish this, but the path you choose will depend on your use case.

API Node

The API Node includes a Data Export method for devices and their attributes. Within the workflow, you could utilize the Export method to query your devices and send the state data to your S3 bucket.

Time Series Node

The Time Series Node gives you the option of exporting easily with just two nodes, but when querying a large time range, you’ll have to set a resolution, which could cause you to lose some data.

Note, if you are working with a very large dataset, it’s possible you may into trouble with these nodes or within the workflow.

Notebook Export

If you wanted to move this data just a time or two, you also have the option of exporting to a CSV using Notebooks. You can select the devices you would like as well as configure which attributes (if you’d like) and the length of state data to return (30 days, 90 days, etc.). Then you can request a data export from the page of this data.

This question did get some discussion started internally, and I will be making a feature request for this functionality to be wrapped up together in one place. I’ll get back to you with more information once I hear more.

Thanks so much!
Julia

Is there a reason why I couldn’t/shouldn’t use the archiving feature?

Hi Lars,

There’s no reason you can’t use the archiving feature; I just suggested the previous options based on your use case. You wrote you want to upload the “past week’s raw time series data,” but keep in mind data will be only archived once it becomes older than 30 days.

Thanks so much,
Julia

Just curious to what happened to the “internal discussion” that got started?

@Lars_Andersson,

Well, it turned out great! We now have a whole new set of Bulk Actions for devices:

Including Bulk Export:

It’s a much more configurable way to export data. :slight_smile:

I just tried “bulk device data export” using a filter based on a tag value, but I get data from devices with ID I can’t find in my application. Isn’t the first column in the CSV the device ID?

@Lars_Andersson,

I’m going to open up a DM with you to work through this and share more information.

Actually, I misspoke. The file I attached was actually from a data export in the notebook input section, but never the less, I used the same tag selection to filter devices so there is still something not right.