Going through the docs and it looks like you are able to make requests for additional resource limits. I don’t think the docs mention increasing the 5mb limit for payload sizes, but are people able to make requests for this as well? I’m currently working on making the size limit less of an issue, but the client has indicated interest in having the size limit increased regardless if possible.
Sorry but no, that is not a limit we can increase on our public cloud installation. It may be adjustable if you were to purchase a private installation; if that is something you are interested in, I can verify with the engineering team.
If you can tell me a little more about what the end goal is - i.e. why you need the limit increased beyond 5MB - I may be able to suggest a workaround.
There are a lot of devices in the system and a lot of data is coming in. I’m querying out as much as I can based on the parameters, but, at the end of the day, there is gonna be a lot of data being passed to the function in the workflow which is where I run into the issue with the 5mb limit. The widget it’s for is a bar graph that I can’t just do simple pagination for. I need all the data at once to crunch the numbers. My first thought was to do multiple calls to the endpoint and return some of the data at a time until I have all of it and then do all the crunching in the widget javascript instead of the function, but that causes the payload I’m sending back to the widget to be too large and I get the error there instead.
So these are experience endpoint requests you are issuing replies to? If so, then yes, you’d run into the 256KB limit in the reply body. We do gzip the response so you typically can fit a lot more than 256KB in there, but nowhere close to 5MB.
A couple ideas come to mind …
- If this chart is being created in a Custom HTML Block, you could use the block’s data queries to return all the raw data you need and then do the manipulation in the block code.
- The Function Node allows for passing in a scope to limit what is actually serialized into the node’s sandbox environment. You may be able to run multiple Function Nodes each handling a different slice of the raw data, though bear in mind that this could lead to performance issues.
- I don’t know what kind of manipulation you are doing, but it may be possible to bypass the Function Node and use, for example, the Loop Node; Mutate Node; Object Node; Array Node; and others. Those, in conjunction with the Handlebars helpers we expose in the workflow engine, may be enough to do your manipulation.
Re: this suggestion, we just released a feature that allows for advanced queries in the Custom HTML Block’s Time Series and Gauge queries, which would give you for more control over the devices whose data you return. Maybe that would allow you to bypass the Function Node in the workflow and the experience endpoint request altogether?