How to extract just most recent data from this payload?


#1

Hi,

I receive the following response from a server (in resp field). It seems that it always sends the last hours worth of packets. How can I sort through the packets and add just the packets to my devices history that are new (ie not already in its history). The timestamp could be used to determine what is new and old but I dont know how to do this in code!

Sorry if I didnt explain this welll! I want my device on Losant to have an accurate history on a pretty graph but I cant go adding all these packets to the devices data each time they come in or it will be incorrect.

Note the packets below is just test data so no use right now!

Thanks

Ashley

{
  "time": "2017-02-03T08:28:07.278Z",
  "data": {
    "resp": {
      "body": [
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:29:01.633743895Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:30:08.739453686Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:31:15.438436849Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:32:21.042039361Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:33:28.220250963Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:34:34.967480877Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:35:42.027905498Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:36:47.746421722Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:37:55.005809675Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:39:01.662163164Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:40:07.082448786Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:41:13.711374281Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:42:20.050892947Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:43:26.084943908Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:44:31.622349726Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:45:38.246701734Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:46:43.773509036Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:47:50.624798601Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:48:57.358942085Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:50:04.298175065Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:51:11.363789731Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:52:17.756093121Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:53:23.892936971Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:54:30.244384878Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:55:36.013976936Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:56:42.074566296Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:57:48.386459835Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:58:54.831169482Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:00:01.99543312Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:01:07.794959181Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:02:14.311859726Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:03:19.827013029Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:04:26.157162786Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:05:33.267910549Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:06:40.559615577Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:07:47.578531448Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:08:53.343229379Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:09:59.189379697Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:11:05.274168536Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:12:11.115751344Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:13:18.217860846Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:14:24.685836488Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:15:31.897476982Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:16:38.687598674Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:17:45.532724868Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:18:52.732661041Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:19:58.92864656Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:21:04.746328341Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:22:11.935106468Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:23:18.253859217Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:24:24.516178184Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:25:30.492639369Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:26:35.962937332Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:27:43.108995416Z"
        }
      ],
      "headers": {
        "server": "nginx/1.11.9",
        "date": "Fri, 03 Feb 2017 08:28:07 GMT",
        "content-type": "application/json",
        "transfer-encoding": "chunked",
        "connection": "close"
      },
      "statusCode": 200
    }
  },
  "applicationId": "588ef1976b20be000123213a",
  "triggerId": "588ef25aa053970001303935-r1YqeF3ve",
  "triggerType": "virtualButton",
  "relayId": "588ef147a053970001303933",
  "relayType": "user",
  "flowId": "588ef25aa053970001303935",
  "flowName": "AshleysWorkflow",
  "applicationName": "Ashleys App",
  "globals": {}
}

#2

The first thing you’ll want to do is use the Gauge Query to find the most recently reported state for your device. You’ll want to set the Time Range to Last Received Data Point. The result you get back will include the time of when that value was reported. For this example, let’s assume you put the result on the data.lastValue field of the payload.

You’ll then want a function node to iterate over each incoming packet and build a new state payload that includes all new entries. Loops are not very easy in our workflow engine (we’re working on that), so it’s easier to just build a new array of all new states and push them using the Losant API Node.

payload.newStates = [];
payload.data.resp.body.forEach(function(packet) {
  if(new Date(packet.time) > new Date(data.lastValue.time)) {
    payoad.newStates.push({
      time: packet.time,
      data: {
        led: packet.led,
      }
    });
  }
});

You now have a state array that can be directly posted to your device using our API. The easiest way to do this would be to use the Losant API node.

  • Set the Resource and Action to “Devices: Send State”.
  • Set the deviceId to the ID of your device.
  • Set the deviceState radio Payload path.
  • Set the path to “newStates”.

This will push all new states into your device with just the single query.


#3

Awesome, thanks for your great help. I will give it a go as soon as I can find some spare time…


#4

Hi Brandon,

Just tried to test that code. What does “Unexpected strict mode reserved word” mean?

Also, not criticising or being a know all but you may want to edit your post above and add a closing } on the if statement just incase someone else copies it in the future.

Thanks again

Ashley


#5

LOL, spotted it. I think you meant “packet” not “package”! Now I get a “payload.data.resp.body.forEach is not a function” warning. I think it may be because body is currently empty (as I just started my gateway. I will wait for some data to accumulate and run it again when it doesn’t have an empty body.


#6

Ha, that’s what I get for trying to type code in a comment box! Thanks for finding the bugs. I have edited the post with fixes. Hopefully it has pointed you in the right direction. As for the forEach being undefined, I would simply add a quick check around the loop. If body is empty, the workflow will continue with an empty array.

payload.newStates = [];
if(payload.data.resp.body) {
  // the loop
}

#7

Yup, had added that. Thanks again. I am still waiting to test it properly as my gateway had died and still trying to sort it.

Cheers

Ashley


#8

Reviving my old topic as finally got back on to it.

With reference to my original post, I am trying to retrieve only new packets that I have not retrieved before. I have (with help) got this code so far. It fals wth an error “Cannot convert object to primitive value” on the lne that compares the two tmes. Note that payload.data.lastTime is loaded and stored in the worflow so persistent across workflow runs. Any ideas?

payload.newStates = [];

if (payload.data.lastTime == null)
{
  // First time persistent variable ever used, init
  payload.data.lastTime = new Date();
  payload.data.firstTimeInit = true;
}

payload.data.resp.body.forEach(function(packet)
{
    if(new Date(packet.time) > payload.data.lastTime)
    {
        payload.newStates.push(
        {
            time: packet.time,
            data:
            {
                led: packet.led,
                Text: packet.Text,
            }
        }
        );
        payload.data.lastTime = payload
    }
});

if (payload.newStates.length)
{
  payload.data.lastTime = new Date(payload.newStates[payload.newStates.length - 1].time);
}

I do have some worries with messagws that occur at the same time, but wlll get to that.


#9

Ashley,

Not sure that this is the source of the issue, but I think you have a typo in your code. At the end of your loop, you are setting

payload.data.lastTime = payload

I’m pretty sure you want

payload.data.lastTime = packet.time


#10

Haha, good spotting! Must have missed that. The code doesn’t even execute that far yet but it saves me looking for another bug later!