**TooManyRequestsError**

Help
1

My workflow device state nodes are reporting:

TooManyRequestsError
Over rate limit, request throttled. Retry in 0 seconds

I have never noticed this error in the past and my workflow has not changed recently so I’m wondering if there is any new reason why I might be running into these rate limits now?

Thanks!

Paul

2

Hi @paul_wareham,

I took a look at your workflow and believe I have located the problem. The associated device log looks as though state is being reported more than twice per second sometimes, which is why the errors are arising in the workflow. This seems to only happen about every 7 seconds, so it’s possible three state reports are trying to occur in the same second and one is getting rejected, resulting in the error. Has anything changed recently with your Webhook? That would be my best guess for the errors arising without a change of the workflow.

Thanks so much!
Julia

3

no nothing has changed.

I think what is happening is that one device in our fleet may have been out of cellular coverage for some period of days but still powered up. It is set to “store and forward” where it buffers all the messages to memory until network coverage can be found. it will then proceed to send all the messages from its log at once.

I assume that each of those errors will result is the data being lost (state not being updated) correct?

Is there any way to buffer and store the messages more slowly to get around that rate limit or do I need to handle that on the device end?

4

Hi @paul_wareham,

Unfortunately that data was not reported since it did throttle device state. Our engineers came back with some great information about state reporting which may help solve this for the future. The Device State Node does accept an array of multiple separate state messages. Even though it is multiple states, we only count it as a single message in terms of rate limiting. This means two things:

  1. The workflow could be modified to buffer several messages using Workflow Storage before using the Device State node to report them. The buffering would be interesting to avoid race conditions, but it’s possible:
    a. Receive the Webhook body
    b. Put the body in a Workflow Storage value using the timestamp as the key
    c. On a timer (every few seconds) pull all keys, construct an array payload, send using Device State
    d. Remove the storage values you just reported
  2. The device can be updated to store/forward multiple state messages at once instead of individually. I’d recommend batching them into 10-20 state messages per Webhook request.

Let me know if I can explain further!
Thanks,
Julia

5

Thank you!

Is there any way to get a notification when such an error - or any system error - is triggered? I’d like to have a log whenever it happens so I can know that I’ve missed state updates so that it can be corrected.

6

Hi @paul_wareham,

I don’t know that there is an easy way to do this at the moment, but we do currently have a ticket for a trigger node that would fire whenever an error arises in the workflow. I will add this post as a +1 to the ticket. The way I would do this now would work if you had the workflow firing at a certain frequency for each device (one every five seconds for example), then you would be able to check if the messages were coming too frequently, which would then alert you of possible dropped states. Again, this is not a great way to do this, especially if the data comes in spurts or variable speeds, but it could help in the meantime.

Thanks!
Julia

7

An error node is a great idea.

In the meantime is there any where I could periodically check error logs manually?

Alternatively how about using the throttle node to detect messages generated to quickly and then send a notification?