Stumped - Workflow loop only iterating over the first 10 entities of the result set of a data table query

Hi

I am stumped and probably cannot see the wood for the trees.
I have a workflow that queries a data-table for a number of records.

20 are returned.

I loop over these records with the intent on processing each. The Work flow in principal works however I find it is only looping over the first 10 elements in the result set.

I thought I was doing something wrong in my logic, so pared the whole example down to

virtual button -> table query -> loop -> debug -> next

No matter what I do it seems to only iterate over the first 10 items of the table query result set. I do not get an error in the debug console.

  • I have checked the length of the result set array (it returns 20).
  • I have copied the array using slice function, only first 10 items are iterated over.
  • By creating array with the java script statement “new Array(20)” - I can loop over 20 items.
  • Choosing a different table I am able to loop over all of the items.
  • I can start a slice anywhere in the desired table results and will only see 10 items iterated over. ie start at 5 should see 14 results looped over, but only 10 are processed.
  • I have recreated a new workflow from scratch which only loops over the result set with a debug and the same situation holds true.

I am really stumped. It suggests to me that there is something inherent in this data table result set that is causing the looping problem. Oh and I really hope I am doing something stupid :wink:

T

Ok,

The plot thickens. I decided to remove the generic debug node from inside the loop.
(This workflow worked fine with the debug in place with other tables)

By removing the debug from inside the loop, it appears to reliably loop over all items.

It appears that with this specific data table, that a default debug node, is causing the loop to fail early.

Why do I believe this to be the case ? I have a final debug node after the loop in the top level workflow, and that is not outputting anything, so it appears that inner debug is terminating the loop and the workflow without an error.

@Tim_Hoffman,

Hi Tim, thanks for your detailed response. We will look into this today to see if we can reproduce the results. However, I think I have an inkling on what you might be experiencing. The Debug panel is throttled to the browser for performance reasons.

The question I have is, are you using the messages in the Debug Panel to determine if the row in the table was processed by the loop? Or the actual end result ( outside of the loop )? It could be the case that it was working the entire time, you just weren’t seeing the Debug message due to throttling.

Hi

Thats is what I thought, however when I run the same query on a different table with 40 results I get all of the debug statements plus the final one. (so 42 in debug outputs in total, 1 pre loop debug, 40 in loop debugs, and 1 post loop debug).

The same workflow on the offending datatable has only rows, and only 11 debug (pre loop debug) and 10 in loop debugs, and 1 post loop debug)

Can I send you via email the specific workflow, which is exhibiting it.

Thanks

Tim

Hi @Tim_Hoffman,

I will send you a private message to include your information. :smile:

Thanks!
Julia

Hi @Tim_Hoffman,

I ran the workflow with both of your data tables and was able to replicate the debug log as you described. I believe you are only seeing 10 entries as the debug is throttled, so it is likely that your loop is completing, you are just not seeing all of the logs in the debug panel! I added a .5 second delay in the loop and was able to get all 20 entries and the final debug.

41%20AM

The loop for this data table is running faster than your other data table, which is why the debug would be throttled for only one of the tables. I tested this by adding a Mutate Node at the beginning of the loop to get the current time, and at the end I added a Math Node to calculate the current time at the end of the loop. This showed that your larger data table loops took twice as long to complete (readings are in milliseconds):


When I removed the Function Node and replaced it with a Mutate Node, the loop time for both tables is the same:

Let me know if this does not solve it!
Julia

Hi Julia

Thanks for the explanation.

I had incorrectly assumed that throttling debug statements would not actually throttle the operation of the workflow. That they would be just binned. So was not expecting the workflow to fail.

Again thanks for the support.

2 Likes