Bulk csv export of device data?

It’s super easy to export individual device data via the Devices-> Data -> Request data export option.

Is there an equivalent export request for a group of devices?

I’ve tried the Data Explorer by selecting a group of devices by a Tag but it seems to freeze the browser.

Thanks

Paul

Hi @paul_wareham,

It was recommended that this be done in the Notebook Input configuration. You can select the devices you would like as well as configure which attributes (if you’d like) and the length of state data to return (30 days, 90 days, etc.). Then you can request a data export from the page of this data.

Thanks!
Julia

@JuliaKempf

I’ve tried this but and it accepted the request, however its been about 3 hours and I have not received the email with the download link.

Hi @paul_wareham, I just tried this out and got one immediately. Are you still having issues?

@anaptfox

No I could not get it to work at all. I still haven’t gotten the email after a day in between. I made two separate attempts.

Maybe it went to quarantine but i do get the other export reports on the same email address via the Devices tab.

I just tried it again, but instead reduced it down to one device only and I got the email immediately.

However my tag query did not work. It is in the form expressed in the photo attached.

FYI I later tired a device tag resulting in a small group of devices and it worked.

Soon I think the platform may just overwhelmed by the number of devices chosen (400 devices) and all its attributes for 90 days.

Hi @paul_wareham,

I still haven’t been able to reproduce ( been trying ). However, for now, I’m tossing to the Losant Eng team to see if anything is wrong.

Could you send me your application ID?

I will PM that to you

@anaptfox sorry I could PM you. Is there another way to get that to you?

@paul_wareham,

You should try this again now to see if you still have issues. We pushed out a fix that should solve your issue :slight_smile:

that’s working now.

thank much

1 Like

Hello,

I have a similar question. I am trying to set up a regular data export to S3 Bucket. All is clear except which Losant API call to use and how to configure it. Is there an easy configuration allowing to get CSV files in the same way as manual export from Device area works? I mean all attributes and already nicely packed in CSV format.

That’s what I have already tried:
Losant API Node. Data: Export. Works Ok, but requires manual listing of attributes. How can I list all of them? Tired , "", not including attributes field. The output result needs to be processed to CSV format field by fied afterwards that’s not convinient.

image

Thank you in advance for your advice.

Hi @Alexander_Kondrov,

Just a quick clarification question, what data are you hoping to export?

In addition, have you tried out the application export feature? This does allow for optional resources to include device configurations.

Thanks!
Julia

Julia,

Thank you for your reply. I look for a way to export device historical data. Basically the same thing as functionality of GUI form in devices called “Request Data Export”. I attached a cut for your reference
image

I’ve took a brief look at Optional Resources link of Application export you included. It does not seem to include historical data, but only " your device configuration"

@Alexander_Kondrov,

I did some testing for you. If you remove the attributes field entirely, you’ll get them all. :slight_smile:

Also, that API endpoint requires an email. This API queues an export request and sends the export to the email provided. This means you wouldn’t be able to feed that export into the CSV Encode Node. Is this just a one time export? Or, are you trying to automate the export?

1 Like

Taron,

Thank you for the trick with deleting attributes. It worked as well for me.

I would like to set up regular monthly device data export. One time manual export with a link in the email will not work for my goal since I don’t want spend time downloading, renaming and organizing exporting manually data for N devices.

I decided to go ahead and build a workflow for monthly export in a required format using Losant API and a call
image
The problem with this approach that “Raw” Data Export (like Mean aggregation for 1 second) for a period of a month takes longer than 1 min and causing a workflow to crash.

It would be great if you can share an idea of a workaround.

@Alexander_Kondrov,

Today, this is a tough one. I have two ideas for you.

Option 1: You can use Notebooks as an exporter. So, instead of the “Request Data Export”, you can configure the inputs of the notebooks as the export you’d want every month. Then, your notebook wouldn’t do anything (or if you want to edit/filter/clean in the notebook you can). The notebook can simply save the input as a Custom File Output.

After that, you can easily execute the notebook once a month, and trigger a workflow when done and do something with the CSV FIle.

Option 2: You can make the “Data: Time Series Query” in a Lamda Function and trigger it from within a workflow using the AWS Lamda Node. We support all the major vendor’s cloud functions, any of them would apply here. This route would remove you from the 60-sec time limit.

1 Like

Hi Taron,

Thank you for good advices. I am new to Notebooks, may be it’s a good time to start learning.
I will try to do a loop for now splitting data into pieces of 3 days. And I will do one of two methods you adviced later.

@Alexander_Kondrov,

My pleasure. We are always here to help. For now, here is the best way to get started with Notebooks:

Hi @anaptfox,

Just wanted to let you know that I’ve solved my goal of regular data archiving by using Data Archive feature recently added to Losant. I will try not to miss a feature release presentation next time :slight_smile: