Im running into this issue where im trying to mass update a device attributes, to contain any new attributes they didnt already have that are coming in from this webhook, the update goes through successfully but then the state update fails and times out, the payload indicates that the attributes were added but checking the device shows it has none? im not sure if its possible another workflow to be overriding the attributes consistently in the time between the update and state update?
state update fails and times out
Can you drop a screenshot of the Timing tab? In most cases, it wasn’t this node that timed out, but rather this is where in the workflow it ran out of time. It would be good to what node used up the most time.
indicates that the attributes were added but checking the device shows it has none
I don’t quite understand. Are you saying if you open the device overview page, the device no longer has any attributes under the Attributes tab?
Yah the payload path for the device update node, suggest it succesfully added all the attributes, but going into the devices attributes page they are all gone
Im not sure why it would take so long to update, it only seems to be freezing out like this on this device i believe.
That is definitely unusual. We’ll see if we can reproduce the issue. Is it happening every time you attempt to execute this workflow?
seems to be, only for this device only a couple of devices
here is a device working fine ^^^
here is another example of it taking a different path but the same thing happening, same device vvvvv
There are other devices with the same issue I have found, I will try to catch their payloads at the top of the hour when they come in
(There should be alot more than that)
Thanks for the information. I am monitoring the debug log for that workflow and will hopefully catch the error.
1 Like
We’ve identified that the problem devices are having their attributes removed and recreated many times. This is causing performance issues during the update since your time-series data is comprised of the “same” attribute name spanning thousands of underlying identifiers.
Somewhere in your workflows is a Device: Update Node, probably using the payload path option, that is not providing the entire set of attributes. Any attribute not provided with this option are deleted. Somewhere else those attributes are then recreated. We have not yet found where in the workflow(s) this is happening.
Based on our inspection of the time-series database, looks like something changed around August 14th. We’ll continue to look, but maybe that will help you find the issue as well.
1 Like
I have 3 of those device updates that i know of off the top of my head all in the Black Box Parser workflow
these 3 are using payload path to update device state, autoset thresholds and zero counts, are to change the attribute tags, and the webhook one is to dynamically add attributes since the names are unknown till they come in, is there a better way to do any of these. and or could they be causing issues
The concept of what you’re doing is fine. You’ll just need to check your logic to ensure that existing attributes are maintained. For example, device ID 6880c4e9ea00561bcd2f94d2 completely swapped attributes roughly 3 minutes apart today. They went from:
- UsedCalFree
- WeighmentCount
- ZeroCalCounts
- …
to
- CalDateTime
- CalGeoZone
- CalUnits
- …
- IOError1
- IOError2
- …
I’m still looking through your workflows, but maybe this will point out the problem areas.
There are also potentially race conditions if you have two workflows executing in parallel attempting to update the same device. Both workflows may perform the Device: Get Node prior to either of them updating the device. This will cause the workflows to delete each other’s updates. It doesn’t look like this is a likely cause since I think everything is initially triggered from the webhook and subsequent updates are triggered by the resulting state reports from that.
1 Like
There could possibly be a race condition as I have two different device state triggers both updating attributes via payload path, the concern here is that i dont see why the attributes from the webhook would be getting deleted as those should already be updated and assigned before the device state triggers go off, so if there was an issue with that i would only expect the attributes from the device state triggers to be having issues
On top of that this has been running without issue for a while now so im not quite sure what change wouldve caused these things to start happening, unless ive been getting lucky on the timing of everything until now
Would be nice to have a device update method that allowed added individual attributes like the single line input method, but using json so that it would be easier to add a dynamic amount of them. Bascially not having to get all attributes, modify and resubmit.
I feel like that would completely fix my issue of over writing new attributes as i wouldnt be reassigning the whole list only adding new ones
Here’s a run I found that removed all the device attributes. The original device coming in has 12 attributes, but an empty array is passed to the Device: Update Node. This would not be a race condition, this is an error in your logic that is not correctly keeping the original attributes on the device.
1 Like
That said, the 2-second delay between Device: Get and Device: Update greatly increases your chances of a race condition. Even if that logic is fixed, if somewhere else adds another attribute during that 2-second window, the update will immediately remove it.
Yeah that would be it, probably assumed at the time an empty array would be ignored or something like that
This flow does different logic based on how many data points a device has, the issue probably always existed for new devices and we just havent made one in a while
Would be nice to have a device update method that allowed added individual attributes like the single line input method, but using json so that it would be easier to add a dynamic amount of them. Bascially not having to get all attributes, modify and resubmit.
I agree, this would be a nice improvement.
1 Like
Im gonna guess it was added there in the first place to prevent a race condition not sure why though cant remember. Thank you
Ive made the changes and hopefully that will be the end of this thank you for the help!
We did identify a performance improvement that can be made to how attributes generate their internal IDs. That will address any future timeouts caused by this specific scenario, but you would have still certainly noticed your attributes didn’t have any historical data (since they were being removed/recreated).
Yeah, unfortunate but thankfully only happened on test devices
@Brandon_Cannaday Actually seem to be having the same issue still for some devices. Im not sure at this point what it could be since ive address what we talked about (I think)
