I am trying to create a long-running AWS Lambda function that runs inside a resource job workflow to retrieve a large dataset. However, since the default Lambda node timeout is 30 seconds, the function often times out when processing larger requests. I attempted to increase the timeout, but encountered the following error:
ValidationError: timeoutTemplate cannot be greater than 30 seconds.
Is there any way to increase the Lambda function timeout beyond 30 seconds?
Set the timeout to something small - like 100ms. We aren’t ever expecting this to complete in time. But if the node times out, the Lambda function still keeps executing.
Ensure you are passing the value of data.iterationId through to the function. You will need it to call back to Losant when the function completes.
Select the “Store AWS Lambda error at payload path” option and enter a payload path for where to put the error. (You won’t ever use this but it will prevent the workflow from halting execution if you have anything running after the Lambda Node.)
In the resource job configuration, set its Iteration Timeout Length to something appropriate for how long you expect this function to take. For now let’s say the maximum 15 minutes, which is the same maximum timeout for a Lambda function to execute in AWS.
Create a webhook in your Losant application to receive a callback from your Lambda function. (More on this below.)
Create a new Losant workflow - or use the same workflow that you are invoking the Lambda function from - and add the following …
Connect a Job: Acknowledge Node to the Webhook Trigger. Set its Iteration ID Path to data.body.iterationId.
This will allow us to asynchronously acknowledge the resource job iteration through a callback sent from the Lambda function when it is complete, as described below.
Within AWS, make the following changes to your Lambda function …
Set the timeout length to something appropriate.
You can continue returning a result but it will never make its way to Losant. Instead, before ending the invocation, make an HTTP POST request to the URL of the webhook you created, making sure to pass a JSON body with iterationId as a top-level property, with the value received when invoking the function. Also include any other needed info for when the Webhook Trigger fires in response to the request - such as whether the iteration should be marked as successful.
Let me know if you have any questions about that approach!