How to extract just most recent data from this payload?

Ashley_Duncan · February 3, 2017, 8:41am

Hi,

I receive the following response from a server (in resp field). It seems that it always sends the last hours worth of packets. How can I sort through the packets and add just the packets to my devices history that are new (ie not already in its history). The timestamp could be used to determine what is new and old but I dont know how to do this in code!

Sorry if I didnt explain this welll! I want my device on Losant to have an accurate history on a pretty graph but I cant go adding all these packets to the devices data each time they come in or it will be incorrect.

Note the packets below is just test data so no use right now!

Thanks

Ashley

{
  "time": "2017-02-03T08:28:07.278Z",
  "data": {
    "resp": {
      "body": [
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:29:01.633743895Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:30:08.739453686Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:31:15.438436849Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:32:21.042039361Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:33:28.220250963Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:34:34.967480877Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:35:42.027905498Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:36:47.746421722Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:37:55.005809675Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:39:01.662163164Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:40:07.082448786Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:41:13.711374281Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:42:20.050892947Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:43:26.084943908Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:44:31.622349726Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:45:38.246701734Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:46:43.773509036Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:47:50.624798601Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:48:57.358942085Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:50:04.298175065Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:51:11.363789731Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:52:17.756093121Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:53:23.892936971Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:54:30.244384878Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:55:36.013976936Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:56:42.074566296Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:57:48.386459835Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T07:58:54.831169482Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:00:01.99543312Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:01:07.794959181Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:02:14.311859726Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:03:19.827013029Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:04:26.157162786Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:05:33.267910549Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:06:40.559615577Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:07:47.578531448Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:08:53.343229379Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:09:59.189379697Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:11:05.274168536Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:12:11.115751344Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:13:18.217860846Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:14:24.685836488Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:15:31.897476982Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:16:38.687598674Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:17:45.532724868Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:18:52.732661041Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:19:58.92864656Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:21:04.746328341Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:22:11.935106468Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:23:18.253859217Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:24:24.516178184Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:25:30.492639369Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:26:35.962937332Z"
        },
        {
          "Text": "Hello, world!",
          "device_id": "03ff0001",
          "led": 72,
          "raw": "SGVsbG8sIHdvcmxkIQ==",
          "time": "2017-02-03T08:27:43.108995416Z"
        }
      ],
      "headers": {
        "server": "nginx/1.11.9",
        "date": "Fri, 03 Feb 2017 08:28:07 GMT",
        "content-type": "application/json",
        "transfer-encoding": "chunked",
        "connection": "close"
      },
      "statusCode": 200
    }
  },
  "applicationId": "588ef1976b20be000123213a",
  "triggerId": "588ef25aa053970001303935-r1YqeF3ve",
  "triggerType": "virtualButton",
  "relayId": "588ef147a053970001303933",
  "relayType": "user",
  "flowId": "588ef25aa053970001303935",
  "flowName": "AshleysWorkflow",
  "applicationName": "Ashleys App",
  "globals": {}
}

Brandon_Cannaday · February 3, 2017, 9:06pm

The first thing you’ll want to do is use the Gauge Query to find the most recently reported state for your device. You’ll want to set the Time Range to Last Received Data Point. The result you get back will include the time of when that value was reported. For this example, let’s assume you put the result on the data.lastValue field of the payload.

You’ll then want a function node to iterate over each incoming packet and build a new state payload that includes all new entries. Loops are not very easy in our workflow engine (we’re working on that), so it’s easier to just build a new array of all new states and push them using the Losant API Node.

payload.newStates = [];
payload.data.resp.body.forEach(function(packet) {
  if(new Date(packet.time) > new Date(data.lastValue.time)) {
    payoad.newStates.push({
      time: packet.time,
      data: {
        led: packet.led,
      }
    });
  }
});

You now have a state array that can be directly posted to your device using our API. The easiest way to do this would be to use the Losant API node.

Set the Resource and Action to “Devices: Send State”.
Set the deviceId to the ID of your device.
Set the deviceState radio Payload path.
Set the path to “newStates”.

This will push all new states into your device with just the single query.

Ashley_Duncan · February 4, 2017, 7:50pm

Awesome, thanks for your great help. I will give it a go as soon as I can find some spare time…

Ashley_Duncan · February 5, 2017, 3:20am

Hi Brandon,

Just tried to test that code. What does “Unexpected strict mode reserved word” mean?

Also, not criticising or being a know all but you may want to edit your post above and add a closing } on the if statement just incase someone else copies it in the future.

Thanks again

Ashley

Ashley_Duncan · February 5, 2017, 3:33am

LOL, spotted it. I think you meant “packet” not “package”! Now I get a “payload.data.resp.body.forEach is not a function” warning. I think it may be because body is currently empty (as I just started my gateway. I will wait for some data to accumulate and run it again when it doesn’t have an empty body.

Brandon_Cannaday · February 6, 2017, 4:10pm

Ha, that’s what I get for trying to type code in a comment box! Thanks for finding the bugs. I have edited the post with fixes. Hopefully it has pointed you in the right direction. As for the forEach being undefined, I would simply add a quick check around the loop. If body is empty, the workflow will continue with an empty array.

payload.newStates = [];
if(payload.data.resp.body) {
  // the loop
}

Ashley_Duncan · February 8, 2017, 8:34am

Yup, had added that. Thanks again. I am still waiting to test it properly as my gateway had died and still trying to sort it.

Cheers

Ashley

Ashley_Duncan · October 20, 2017, 9:33am

Reviving my old topic as finally got back on to it.

With reference to my original post, I am trying to retrieve only new packets that I have not retrieved before. I have (with help) got this code so far. It fals wth an error “Cannot convert object to primitive value” on the lne that compares the two tmes. Note that payload.data.lastTime is loaded and stored in the worflow so persistent across workflow runs. Any ideas?

payload.newStates = [];

if (payload.data.lastTime == null)
{
  // First time persistent variable ever used, init
  payload.data.lastTime = new Date();
  payload.data.firstTimeInit = true;
}

payload.data.resp.body.forEach(function(packet)
{
    if(new Date(packet.time) > payload.data.lastTime)
    {
        payload.newStates.push(
        {
            time: packet.time,
            data:
            {
                led: packet.led,
                Text: packet.Text,
            }
        }
        );
        payload.data.lastTime = payload
    }
});

if (payload.newStates.length)
{
  payload.data.lastTime = new Date(payload.newStates[payload.newStates.length - 1].time);
}

I do have some worries with messagws that occur at the same time, but wlll get to that.

Michael_Kuehl · October 21, 2017, 5:36pm

Ashley,

Not sure that this is the source of the issue, but I think you have a typo in your code. At the end of your loop, you are setting

payload.data.lastTime = payload

I’m pretty sure you want

payload.data.lastTime = packet.time

Tim_lamborn · October 22, 2017, 6:43am

Haha, good spotting! Must have missed that. The code doesn’t even execute that far yet but it saves me looking for another bug later!

Topic		Replies	Views
Insert old state device data Help	2	613	March 15, 2018
If a device obtained data at a certain time in the past Help workflow	3	251	June 2, 2023
Batch payload reporting is not updating device state Help mqtt , device	4	17	July 18, 2024
Extract specific 'string' patterns/values from inconsistent payloads Help workflow	2	498	January 26, 2021
Losant API Payload Breakdown Help workflow , losant-api	8	11	November 5, 2024

How to extract just most recent data from this payload?

Related topics