Platform Error when running AWS Lambda Node

Hi,
We have been calling several AWS Lambda’s from the AWS Lambda Node, and we recently have seen this error: Platform Error - Error Code 0x101
Any idea what causes this and could be something we are doing? This only started today, and we do not think we have seen this error before.
Thanks,
Stephen

Stephen,

We took a look and it seems like starting yesterday around 5pm EDT we are getting occasional DNS errors resolving lambda.us-east-2.amazonaws.com - which results in the error you are seeing in the workflow node. We are currently digging in further to see what the cause is (as we don’t seem to be having DNS issues elsewhere, and it is only a fraction of a % of the usage of that node that is seeing the error). Thanks for bringing it to our attention!

Hi - as of this morning, we were still seeing this platform error - any insight on your side why this is happening?
Thanks in advance,
Stephen

We still don’t have a clear cause of the problem; we’ve restarted a couple of the workflow runners where the error occurred, but the fact that it’s not localized to a single runner means that’s unlikely to resolve the issue. We may be PM’ing you for more information as we try and hunt this down. Thanks for the information you’ve provided so far.

Hi, a quick update:
Not sure if you did anything on your side, but we replaced the certificates for both of our custom domains, and we have not seen the error for 24hrs now. Replacing the certs does not feel like that should have corrected this issue, but maybe that did something??.
Stephen

We did make some changes on our end to adjust our DNS lookup code this morning and have not seen the error since.

While we did see the error occur a couple times yesterday evening, neither of those were from your application. We haven’t seen the issue in your application since about 2pm EDT yesterday … Was that around the time that you made the changes on your end?

“Was that around the time that you made the changes on your end?”
Yes, that was about the time we replaced our certs.

Well, that didn’t last :frowning:
We just had the same error, so nothing we did fixed it.
Regards,
Stephen

Just letting you know we’re still looking into this. I will keep you posted. Thanks for your patience.

While we still haven’t found the underlying cause of the issue, we did push out a fix yesterday that will drastically mitigate it.

When running the Lambda Node, if we receive the DNS error you reported, we immediately try invoking the node again. So far this has held well; we’ve seen the DNS error come through but the second attempt has always worked. So the good news is, if the node was working 99.9% of the time previously, we believe it will work about 99.9999% of the time with this fix in place :crossed_fingers:.

We’re going to continue monitoring this and see if we can isolate the cause, but hopefully this mitigation step will all but resolve the issue.

Thanks again for your patience and your help in tracking this down.