In a workflow, I’m currently getting time series data from multiple devices and then trying to transform data found in each device into data to be used in other places. I’m having trouble figuring out how to decode all of the signed URL blobs in these devices.
The data comes in as something like:
{
working: {
target: {
devices: {
device1: {
points: [{num: 80, data: storage.googleapis.com/…}, …]
}
device2: {
points: [{num: 42, data: storage.googleapis.com/…}, …]
}
}
}
}
}
In the function node I can grab all these blobs, but I’m not sure how to decode them into useable JSON for the function. I know there is a json decoder node, but I’m not sure how it would be used in a workflow like this where I’m working with many blobs. Is there some JavaScript I can use in the function node to fix this or do I need to go about this another way?
It looks as though you’re reporting blob data. Since blob data is arbitrary binary data, and can sometimes be quite large, it is not automatically provided when querying time-series data.
To get the contents of a blob attribute, you’ll have to use an HTTP Node to download it using that URL. This can definitely be challenging in a workflow since you may timeout before retrieving every blob.
Can you describe what you’re storing? We might be able to recommend alternative approaches.
Data is coming in from the device in the form of something like:
[{name: “John”, num: 12, phone: “1234567”, …}, …] and the array can have many objects in it. I do some crunching and transform the data into a different array of objects that can still be quite large. I need to store this in losant for later use. It’s too long to store as a string so I think a blob is the only option I have for storing.
Is this actual time-series data? In other words, are these fields being reported over and over again with different timestamps? This looks more like user/profile data.
Sorry I should have clarified those are just placeholders. Yes it is time-series data. The actual data coming in is an array of objects that represents all the interactions a user had with the device during a session (touching a certain button, going to a certain screen, etc.). The device can have many sessions throughout the day so I am storing a lot of those payloads as blobs. I need to grab all of the sessions from all of the devices I have every hour or so and use all that data for calculations.
Makes sense. A normal workflow is unlikely to work for this volume of data. There are two options:
Resource Jobs
A resource job can iterate over every device and triggers a workflow for each one. In each workflow you’d query the time-series data for that device, download the blobs, and perform the required calculations. This only works if you don’t require data from multiple devices at the same time.
Notebooks
My guess is this is where you’ll eventually end up. Notebooks can query all the time-series data and will automatically download all blob data. If you require data from multiple devices to perform your calculations, this is the only real option. Notebooks are entirely designed to process large volumes of data.
Alternatively, is it possible to break apart your large object into separate fields that can be reported on individual device attributes?
I don’t think there’s a way to break it up unfortunately. I will look into Notebooks though. Thanks!