MQTT Outages Today

I’ve observed several 5-10 minute outages the last 18-24 hours but don’t see any incidents reported on the status page. We have multiple devices in different locations that are showing disconnected within 5-10 seconds of each other and then recovering 5-10 minutes later, also within a few seconds of each other.

This is fairly trusted firmware that has been operating on the Losant platform for many months.

Are there currently ongoing maintenance or issues not showing up on the status page?

Thanks.

Hey @Zach_Jacobs,

There is not any ongoing maintenance or issues that are not showing up on the status page.

I do, though, just want to verify that you are seeing your devices did reconnect after the disconnection? Or would they try to reconnect and then immediately get disconnected again? How is your firmware set up to handle reconnections?

Some degree of MQTT connection cycling is to be expected, and can be caused for many reasons - Losant only considers it an outage if an issue on the Losant side causes devices to fail to reconnect entirely or to extremely frequently cycle their connections.

Thank you,
Heath

The devices did reconnect on their own and resumed nominal operation within 5-10 minutes.

We saw every device in our fleet go down within 5-10 seconds of each other 3-4 times in an 18 hour window starting at 8:54PM EST last night (15 Aug 2021).

We’ve had devices on the platform for quite a while and this isn’t behavior we’ve seen in the past.

@Zach_Jacobs,

As I sent that, our engineering team made me aware of a current issue with the Google Cloud Platform that could be causing an impact.

We are monitoring this issue now, and I will be sure to follow up with you immediately as I learn more information.

Thank you for sharing this with us.

Thank you,
Heath