Time Series Filtering

Help
,
1

Time Series Filtering

I am attempting to filter time-series data for a device time series graph. In this scenario I would like to graph my ‘predictedValue’ column where the ‘device_id’={{ctx.device_id}} and ‘measureValue’==‘airPressure’.

The device I am attempting to graph is a representation of time series data from an anomaly detection model which detects/predicts anomalies across multiple devices and attributes (e.g Sensor 1 - airPressure, Sensor 2 - Humidity, etc). On every trigger this model returns device state with a predictedValue, measureValue (actual recorded value), an anomaly boolean (true when an anomaly has been detected), and metadata regarding the ‘measureName’ and ‘device_id’ for the returned data.

The attributes for this device are as follows:

  • timestamp: time anomaly detection model was triggered
  • device_id: name of the sensor (NOT UNIQUE, multiple device_id’s will populate this table)
  • measureName: name of the attribute (NOT UNIQUE, multiple measure types will populate this table)
  • measureValue: the input value from the anomaly detection algorithm (actual value)
  • predictedValue: numeric value returned from anomaly detection algorithm (expected value)
  • anomaly: boolean value returned from anomaly detection algorithm

Is it possible to filter my time series data when I come to visualize this data on a dashboard ? Ideally I would like a separate time series dashboard for each ‘device_id’ and ‘measureName’ (e.g Sensor 1 - airPressure, Sensor 2 - Humidity).

An easy solution would be to create separate devices that store anomaly detection data for each device_id and measureName, but I would like to avoid this if possible. Seems like a lot of devices to create.

Is there an alternate solution you would suggest ?

I would also consider hosting my time-series data for my anomaly detection algorithm elsewhere if that seems like a better option. I would then have to create a custom time-series dashboard that is able to query my external time-series db by ‘device_id’ and ‘measureName’.

I hope I have made things as clear as possible. Feel free to ask if anything needs further explanation.

Cheers!

Edouard

2

Hi @Edouard_Eltherington, and welcome to the Losant Forums!

Storing “measureName” as a telemetry attribute alongside “measureValue” and “predictedValue” is not how I would recommend approaching this. Data is stored in our time series database - and is thus queryable - keyed by the unique combination of a Losant device ID and an attribute.

The way I would approach this problem is to create a unique Losant device for every sensor you have (device_id in your model) and I would store that device_id value not as an attribute but as a device tag, since it is metadata describing the sensor that does not change over time.

Then, assuming the value types you are recording (temperature, humidity, etc.) are reasonably limited, I would create separate device attributes for each of them, as well as for their predictions and anomaly detections - as in …

  • actualTemperature (number)
  • predictedTemperature (number)
  • anomalyTemperature (boolean)
  • actualHumidity
  • predictedHumidity
  • anomalyHumidity
  • …and so on

I would also record this data with a custom timestamp matching the timestamp value you mentioned above, as opposed to recording that as a separate attribute. This will make it much easier to query and visualize your data.

Each device is limited to 256 attributes so assuming no sensor reports more than 85 data types, you should be good with this approach.

One thing you did not mention is how this data is getting from your model over to Losant. Depending on how that is happening, if the devices and their attributes are not known ahead of time, you could create devices / add attributes to existing devices on demand to fit the data that is coming in if you do not have matching devices/attributes in your application already. I can provide further details on how to do that if you can tell us more about how you are piping this data over to our platform.

Let us know if you have any other questions, and again, welcome to Losant!

1 Like
3

Hey @Dylan_Schuster,

I really appreciate this response!

I am going to provide a bit more context on what I have set-up to make sure I understand your solution.

I have 6 sensors on the platform that record state into the anomaly detection model. In your solution I would create 6 additional devices each responsible in retrieving anomaly data for each sensor device? For sensor_1, I would create another device maybe called anomaly_1 which would have actualTemp, predictedTemp, etc. In this case where would I want to include the device_id tagging piece you mentioned ?

More about the model I am implementing. The model is simple enough to create in a Losant Workflow, so I am not piping data from an external source. It’s an Exponential Moving Average Model that measures a moving average across an incoming signal value (airPressure, airTemp) on a device. This model uses some thresholding to detect if the incoming signal value is an anomaly or not. Knowing that this model is in a workflow, would it still want to create a custom timestamp ? My assumption is that timestamps are automatically created when a device reports state (or when state is reported from a workflow).

Lastly, it would be sweet to have a method of dynamically creating new devices / attributes for anomaly detection if I decided to include additional sensors / measure values in the future, although this is not a huge priority. If you had any suggestions on how this would work, I would be very interested to hear more.

4

Can you please PM me a link to the workflow(s) so I can take a closer look? I want to make sure I am fully understanding what you have in place now before making any more recommendations. Thanks.

5

No, I am saying your device that represents the sensor on Losant gets the additional attributes. Say in the real world this represents a temperature sensor … so the Losant device would have an attribute called “temperature” or “actualTemperature” or something to represent the raw data received from the sensor. But on that same device you would create “predictedTemperature” and “anomalyDetected” attributes.

Your anomaly detection workflow then fires whenever that sensor receives actual data on its “actualTemperature” attribute (i.e. through a Device: State Trigger) and records the calculated values of “predictedTemperature” and “anomalyDetected” at the same timestamp that “actualTemperature” was reported using a Device: State Node.

Knowing that this model is in a workflow, would it still want to create a custom timestamp ? My assumption is that timestamps are automatically created when a device reports state (or when state is reported from a workflow).

When reporting state, you may report a custom timestamp with the data (up to 30 days in the past or 1 day in the future). If you do not, the current time will be assumed. When using a Device: State Trigger, the value you pass for the state time is the value of the time property on the initial payload.

Lastly, it would be sweet to have a method of dynamically creating new devices / attributes for anomaly detection if I decided to include additional sensors / measure values in the future, although this is not a huge priority. If you had any suggestions on how this would work, I would be very interested to hear more.

The Device: State Trigger will not fire for any devices/attributes that have not already been defined in your application. You could create attributes on the fly by first reporting the state values to a custom MQTT topic, using a Device: Update Node to add the missing attributes, and then translating that into a state report in a workflow, but that would double your payload usage (one payload for the custom topic message, another for the state report).

For device registration, we have a Dynamic Device Registration template that you may find helpful for that.

1 Like
6

This is a very straight forward solution I can implement on my end. I will have the anomaly detection WF trigger each time a device reports state and then return the predicted values + anomalies back onto the device using the same timestamp from the Device State Node.

I am not used to adding additional attributes onto devices that don’t come directly from the device, but knowing that a device can support up to 256 attributes this seems like a fine approach.

I will also look into that Dynamic Device registration template you sent over. That seems something that would be pretty cool to implement.

Thank you for all of your help on this, I really appreciate it.