Workflow race condition

Help
1

Hey Losanters,

I’m looking for suggestions on how to handle a race condition in one of my workflows. I have a workflow with a File Trigger.

The workflow reads the uploaded file, performs some calculations, and then sets the device state. One of the device state values is a total cycle counter.

I need this total cycle counter because I want to keep track of the total number of cycles over several years, while the normal time-series data is only stored for one year. So I cannot simply calculate the total by querying all historical cycle data.

The workflow currently does the following:

Gets the previous total cycle count from the device state. Adds the cycle count from the uploaded file. Writes the updated total cycle count back to the device state.

This works fine when only one file is uploaded at a time.

The problem occurs when multiple files are uploaded at nearly the same time. The workflow runs in parallel, so both workflow runs read the same previous total before either one has written the updated value.

For example:

Previous total cycle count = 100

File 1 added cycle count = 20

File 2 added cycle count = 30

Both workflow runs start at the same time and both read the previous total as 100.

So the results become:

File 1 writes: 100 + 20 = 120

File 2 writes: 100 + 30 = 130

The final total cycle count then becomes either 120 or 130, depending on which workflow finishes last. The correct value should be 150.

One approach I am considering is to stop updating the total counter directly from the File Trigger workflow. Instead, each File Trigger workflow would write the calculated cycle increment to a Data Table, for example:

deviceId | fileName | cycleIncrement | processed | timestamp

Then a separate workflow could process these table rows one by one, add each increment to the current total cycle counter, update the device state, and mark the row as processed.

Does this sound like a good approach, or is there a smarter/easier way to handle this kind of cumulative counter update safely?

Sorry for the long post, but I wanted to provide enough detail to clearly explain the problem.

Thanks in advance!

2

You’ve got the right idea, but instead of a data table I would use workflow storage.

The Storage: Set Value Node is an atomic operation - meaning that when it is executing, any other workflow runs trying to also set its value are locked until the write is complete. This solves your race condition problem. You can also use the Increment Value operation in the node to just add your additional runs to the current value.

You can set the key of the storage value using variables (e.g. deviceId) to keep track of cycles independently as well. And whenever you want to report as state, you can look up the current value using the Storage: Get Value Node (also an atomic operation). Just make sure that however you’re reporting state is done in the same workflow as where you are setting the values, as workflow storage values cannot be shared across workflows (they are scoped per workflow).

3

Hey Dylan,

Thanks for the quick reply!

I got some trouble implementing the solution you suggested.

I have multiple total counters, so I need to use multiple Storage Set Value nodes and include the device ID in the storage key, for example:

key: 6a387450475943_totalCycle
value: 223

key: 6a387450475943_totalCycleTime
value: 5689

After that, I use a Storage Get Value node to retrieve all storage values and a Function node to extract the values for the current device:

const deviceId = payload.data.info.deviceId;

const storage = payload.storage_res || {};

payload.data.output.push({

  time: new Date().toISOString(),

  data: {

    totalCycle: storage[`${deviceId}_totalCycle`] || 0,

    totalCycleTime: storage[`${deviceId}_totalCycleTime`] || 0

  }

});

return payload;

Finally, I use a Device State node to write the updated totals back to the device.

The problem is that this results in multiple device state entries being created, while all I really want is a single device state update containing the new total values.

image
image895×217 13.3 KB

It feels like a lot of additional logic is required to work around concurrent file uploads. To me, this could be avoided entirely if there were an option to process File Trigger workflows sequentially. In other words, when a file triggers a workflow, any subsequent files would wait until the current workflow execution has finished before starting a new one.

Is there a recommended way to achieve this behavior, or another pattern you would suggest for maintaining per-device totals without creating additional device state entries?

Thanks!

4

Sorry but you lost me here … can you PM me a link to the workflow?

When totalCycle is being written to a private file, has it already also been reported as device state? I’m failing to understand why this is resulting in multiple state reports per file write. I would think the workflow would be something like the following …

  1. File Trigger fires on change, and the file contains the number of cycles for a given period of time (not the forever total) for a given device
  2. Storage: Set Value Node that uses the Increment Value operation to add to the previous total for the given device (with the storage value keyed by device ID). Using the Result Path, you can the place the incremented value on the payload.
    1. Again, this is an atomic operation, so if there are dual file writes at the same time, the second time this node is hit it is guaranteed to have the value from the previous increment
  3. Device State Node recording the cycles for the period of time and the incremented total cycle count
5

For people wondering how this race condition problem was partly solved.
In rare condition a race condition still happens, but for my total counters this is easily solved, by just taking the MAX value instead of the last received datapoint.

image
image131×658 13.9 KB