[Solved] Particle Integration Down?

@Michael_Kuehl @Brandon_Cannaday I have over the last few days been required to stop and then restart the Particle Integration after it stopped working.

It’s been working great since December since you guys set up the auto restart feature so not sure what may be causing it but wanted you guys to be aware this is happening again.

I did confirm the Particle Webhook publish was happening as expected so it was not an issue on their end.

@David_Armitage The latest Losant Particle integration does use the Webhook to receive data from Particle.

It only runs a workflow when it receives a new webhook with the trigger name you set in the Particle Integration.

@RWB I am still using the ‘Access Token’ approach. Can I get rid of that and instead set up a Particle Webhook that points at Losant?

Look into this thread for more info on the new method.

Some background, there are two main ways to get data from Particle into Losant:

  1. Set up a Particle Integration in Losant, which you have done. This requires no configuration on the Particle side. Losant does not actually “pull” data from Particle in that case - it opens a single connection to Particle over their streaming API, and Particle sends the data to us over that open connection.

  2. Set up a Webhook in Losant, and then go to Particle and configure Particle to make a request to that Losant webhook whenever Particle receives data.

In terms of efficiency, the first option is actually quite a bit more efficient (a single open request from Losant to Particle, instead of many requests from Particle to Losant).

We will take another look at the health monitoring of Particle connections - @RWB when did you start having to enable/disable the integration again?

Hi, glad someone opened a discussion about particle integration.
Had some questions myself…

Why is it that devices set up through the integration show the disconnected icon in the devices window. (looks like the grey plug with line through it)

Also, we cant seem to get the device:connect and device:disconnect nodes in workflow to trigger a notification based on their data. Does that have something to do with the devices identifying as offline?

Does that also have something to do with the fact that particle devices are set up as gateways and not direct connect to losant?

We have no problem streaming data into losant, functional dashboards, data tables, etc. just cant do the above.

A device in Losant is only marked as “connected” or “disconnected” based on its connection to the MQTT broker, not based on any other properties - such as recent state reports via the REST API or an open connection to an integration.

As of now we do not have a first-class connection between an integration and a device, so the device in question is not even aware that its state data is being populated via a Particle integration. But that’s an interesting idea to consider: tying a device directly to an integration. We’ll kick it around internally and let you know if it is something we will add.

As for firing the “Connect” and “Disconnect” triggers, in light of my explanation here, could the Device: Inactive Trigger work for your purposes? It fires whenever devices of your choosing do not receive a new state report for a given length of time.

thanks dylan, will play with the inactive trigger, it may work fine for the time being. but unfort it does not address our need for connect/reconnect notifications.

Thanks…this is helpful. (Of course it would have been smart to study getEventStream before I speculated about how the Particle Integration was set up).

I have seen this behavior over the last 2 to possibly 3 days. Can’t remember exactly but it’s been rock solid since our last communication about this in Jan.

There are special events within Particle for when a device goes online or offline. According to the docs, it looks like the event name might be spark/status. You might be able to add this as an event name in the Losant Particle integration to receive these events. I haven’t tried it yet, so let me know if it works!

Just experienced another Integration outage this morning. 8:45aMT (don’t know if that’s helpful or not)

Stopping and restarting the Integration resolved the issue…but this is not going to work for large scale commercial deployment.

@David_Armitage 8:45am in what timezone? Also, do you know how long the you were not receiving data for, before you kicked the integration?

The way our health checks work for the Particle integration is that we monitor the connection for an integration, and if it appears to be connected but receives no messages for over 5 minutes, we consider it dead and close it and make a new connection.

MT (Denver -6)

It just happened again at 1:45pMT, btw…

My dashboard gets data from remote sensors every 15 minutes. I can see the payloads arriving at the Particle Console as expected, and for months Losant has preformed flawlessly, until just these past couple weeks.

I am going to manually stop/start the Integration again right now…

By “past couple weeks”, do you have a specific date where the issues started? Also, for this most recent issue at 1:45pm, you received the data for 1:30pm, but then did not receive the data for 1:45pm?

I got the 1:45pMT payloads, but not the 2:00p (and I confirmed that there are payloads in Particle at 2:00, 2:15, and 2:30.)

I can’t give you a precise date when it started, but it has been recent…

This health check has worked perfectly since you set it up until recently. Something has changed which is causing it to not work because both of us have not had to touch it since you set up the health check so something is going on.

If the problem only started occurring last week, unfortunately (or fortunately), it is unlikely a change on the Losant side, since the last Losant production release was late June.

We are going to spin up a couple particle devices in the office with different data report rates and try and reproduce this issue. We do already have a “canary” that makes a particle event api call once a minute (and checks that our integration receives it), and that canary has not had any issues, so we are going to try with actual hardware instead.

In addition, we are going to make the health check a bit more aggressive - at the moment, that “5 minute since last message received” check only kicks in if we have received a message on that connection since it has connected. We are going to change it to start counting from the moment the connection is established (which will catch the case that the connection never sends any messages at all).

Really, these checks are almost too aggressive - for instance, in your 15 minute data reporting case, we will almost certainly be disconnecting and connecting when we don’t need to - but there doesn’t seem to be any other way for us to detect when the channel is no longer correctly sending data.

Great work guys!

I am publishing every 10 to 15 seconds 24/7 most of the time.