[Solved] Particle Integration Down?

I’ve been running a Temp,Humdity, VOC dashboard that is receiving data from a Particle Photon and I’m pushing the data to Losant via the Particle Integration.

Looks like even though the Particle Publish is still working the dashboard is not updating anymore as of 15 hours ago for some reason.

My publish events are working but Losant is no longer seeing them coming in for some reason.

I’m just curious if the Particle integration is down because nothing has changed on my end and this has worked flawlessly for some time now.

The particle integration does not seem to be having any issues, as far as our metrics show - we are actually processing more particle messages than yesterday. We also have a couple applications using particle, and they are working correctly. If you go to the particle integration in your application, what is it’s connection status?

Thanks for checking on this for me.

I can confirm that the Photon is publishing properly via the Particle Console:

But nothing shows up on the Data screen:

I don’t remember changing anything to the dashboard so I’m not sure why this would all the sudden stopped updating data on the Dashboard.

The particle event is not triggering the workflow for some reason?

The Workflow is running and has not been touched.

Any ideas?

Could you check the status of your particle integration itself? In the integrations section in your application, you can see the connection status of the particular integration - like the screenshot here.

If it says that it is connected, could you give me your application ID or your integration ID (they will be in the url), and I can investigate more on our side.

Here is the screenshot your asking for:

Application ID = 5951600effe8e800072adf5e

Integration ID = 599c94bf2fc2a6000767f27e - I think this the integration ID.

I got the integration ID from here:

I dug into the health of your particular integration’s connection to Particle, and it looked fine, but no data was coming over the channel from Particle. I then caused the integration to disconnect and then reconnect to Particle, and now data seems to be streaming in again (as a note, you can always cause an integration to disconnect/reconnect just by clicking “save” on the integration’s page).

We have not seen this behavior before - we might have to add some additional health checks on our connections to Particle (i.e., if no messages come in over the channel over X time, disconnect and reconnect). I’m going to file a ticket on our side to investigate further.

Weird, but glad it wasn’t something that I caused by accident.

I didn’t touch that integration or dashboard so it must have been something in the background that happened.

It’s working now!

Thanks for your help!

As a note I had to do as you stated above and click the “Save” button on the integrations page to cause the data being pushed to Losant via the Particle Publish to start displaying on my Dashboard again. I was not accepting successfuly sent Publish events again for some reason.

@Brandon_Cannaday I’ll keep monitoring this and let you know if it keeps happening.

Did Particle have any insights on a cause of this when you talked with them?

It’s basically a Voc, Temp, Humidity sensor sending data every 5 mins 24/7.

I can say it’s been reliable other than these couple events.

We actually have some additional Particle integration health checks going out this week that should help - essentially auto reconnecting if no messages have been received in a while. We did talk to Particle, and they did not have any insight into the source of the issue, but were going to investigate on their side to see if they could find anything (have not heard anything back yet). It seems like it might be an issue with long-running connections - the time lengths between Dec 20th -> Jan 3rd (your first issue) and Jan 3rd -> Jan 16th are suspiciously similar.

Thanks for the info.

Your auto-reconnect feature should do the trick once up and running.

I’ll let you know if anything else stands out.

Thanks!

Very similar app to what RWB is running, fine for several days, no changes. 12 hour outage last night on the Losant dashboard. Integration with Losant indicates Particle re-connected at 12:30 local time, when the Losant dashboard came back to life. Is there a way to show the historical activity on the Particle integration for debugging purposes?

1 Like

We don’t have historical tracking on the particle integrations, we only keep track of the latest connection time (or latest connection error, if the connection is failing).

The reconnection today is probably because we rolled out new Particle integration code, with the newer health checks I was referring to above - so that caused every integration to reconnect. These new health checks will (hopefully) detect when Particle is no longer sending events over a currently open stream, and will cause us to close and establish a new connection.

OK thanks. Seems whatever you did already helped. Cheers

Is there a preset time frequency it checks for these connections?

It would be nice if this were adjustable to suit different application update frequencies but let’s see how this works as you have it.

If a Particle connection has previously received events, and has not received any new events in the past 5 minutes, we consider it to be unhealthy, and close it and open a new one.

I’m hoping that I can eventually find a more direct way to detect these “zombie” connections - it doesn’t appear like Particle sends any kind of “keepalive” messages, which would be optimal for detecting this.

5 mins should be fine for just about all applications so it think that should work perfectly.

Is there an issue conversation you desire the Particlers take back to Particle?

We are actually already talking to them - they are looking into possible sources of the issue on their side.

Has anyone else been experiencing Particle Integration issues recently? I’ve been running a Particle data feed into a Dashboard for many months without issue, until several days ago. The data will flow in just fine for a few hours (the Particle devices report every 15 minutes) and then for no apparent reason they just stop. Disabling the Integration, saving, and then Enabling it again causes the feed from Particle to start working again. It is highly repeatable and happens after just a couple hours of completely normal operation. Nothing is changing on the Particle side, so this seems to be some new issue in Losant.

and while we are on the subject of Particle Integration, is there a reason that Losant chose to ‘pull’ data from Particle, rather than using their Webhook which offers a ton of useful features. Allowing Particle to push data would seem to me to be a far better option than pulling. Losant is doing a lot of pulls when there are no new events to retrieve which seems highly inefficient.