Timeseries query gives fewer results for a strictly-wider time range

Help
1

I’m using a Losant API node to perform a “Data: Timeseries” query. For the query, I am supplying a start-time and an end-time using a Virtual Button for testing:

{ 
 "searchstart": 0,
 "searchend": 1571436455
}

Using the above parameters, I get results which suggest that searchstart and searchend are being interpreted as seconds-since-epoch rather than milliseconds-since-epoch as stated here:

https://forums.losant.com/t/schema-for-the-body-of-a-time-series-query/1586/4

The time-series query returns the time-range used as:

"end":"2019-10-18T22:24:53.427Z"
"start":"2019-10-18T22:07:35.000Z"

If the value 1571436455 was being interpreted as milliseconds-since-epoch, this would correspond to just a few hours after Jan 1 1970, so we can’t possibly have any data for that time-range.

So, this is phenomena #1 which I don’t understand.

Secondly, if I change the Virtual Button to provide the following values, then I get no results from the time-series even though this query is strictly wider!

{ 
 "searchstart": 0,
 "searchend": 1571436455000
}

In this case, the query returns the search-range as:

"end":"2019-10-18T22:27:08.070Z"
"start":"2019-04-21T22:27:08.070Z"

It looks like there is range-dependent casting.

2

Hi @Alexander_Farley,

I am currently testing this and have not yet been able to replicate your first question. Is there anything I am missing to reproduce this, or any configuration I need to set? I included your same values and was given the date in April:

54%20AM
Screen Shot 2019-10-21 at 10.18.54 AM.png2102×1220 380 KB

I will look further into the “no result” Virtual Button case and let you know if I am able to reproduce the issue.

Thanks!
Julia

5

Hi Julia

I think your test does reproduce a couple of the phenomena that I don’t understand:

  1. Your start-date (1571436455) appears to be later than your end-date (0), but the query responds with a start-date (April 24, 2019) earlier than the end-date (October 21, 2019). How?

  2. If these values are in milliseconds-since-epoch, the timestamp you entered (1571436455) corresponds to Jan 1 1970 at 4:30AM. But your query-response indicates April 2019. How?

I’m making a screen-video so I can demonstrate exactly what I’m seeing. Thanks!

1 Like
7

Here’s a clip which demonstrates what I’m seeing:

Just let me know if you need more details.

1 Like
8

Hi @Alexander_Farley,

Wow, thank you so much for your awesome video! It was extremely helpful as this is one of the more difficult Nodes :smile:

I am curious, for your use case, was there a decision to use the API Node over the Time Series Node? And if so, why?

I think it may be helpful to clarify further the start and end values and how they work. Keep in mind, for both start and end, positive values are timestamps, negative values are amounts of time offset from now, while 0 is the timestamp now.

The start date is the beginning of the range in milliseconds. If you wanted to get data from the last week until now, you would set your start value to the timestamp of the first time you want to see data, which is the timestamp from a week earlier.

The end date is the end of the range in milliseconds since Epoch. 0 or blank means current time, and negative values are milliseconds into the past from now. Thus, when I use 0 in my end date, I am using the time now for the end of my query. If I were to set the end value to -60000, this would give me a timestamp of one minute before now.

To summarize, the start date should be the beginning of the time range you want to query, while end is the end of the time range.

Since 0 correlates to the time now, when you change your start time in your video, you are querying from now through the end timestamp, which is probably why you are seeing no data in the time period you queried.

For your second question, my converter is showing that, in seconds, the date is the one you are seeing:

32%20PM
Screen Shot 2019-10-21 at 3.35.32 PM.png1322×504 67 KB

The Workflow Engine, however, will swap your values for you if your end date is prior to your start date.

Please let me know if I can explain anything further!
Julia

9

Thank you very much. That answers most of my questions.

The reason I chose the Losant API node is because it allows me to specify the start and end-time explicitly; the time-series node only seems to give me the ability to query between some time in the past to time-now.

I should point out that neither the documentation, or the additional details here, say anything about the start being interpreted as current-time for value 0:

This sounds like a dangerous behavior, since I would assume that most users looking to search “all data” would choose 0 as the search-start point, unless they know of this behavior.

For my application, starting at 1 instead of 0 should suffice. I will test with this.

Regarding the milliseconds vs. seconds issue: the value you’ve pasted into that box appears to be a time-stamp in milliseconds since epoch. However, the start-time value in your example is a factor of 1000 lower.

So it looks like you’re putting a value of seconds-since-epoch into the Losant API node, but you’re putting a different value in that Javascript time-converter tool. If you paste your original value (1571436455) into a converter which does not perform any casting (and which treats the values as milliseconds), then you will see a date on Jan 1 1970.

The critical thing is to look at the “Supports Unix timestamps in seconds, milliseconds…” note under that input-box. This tells me that the input-box is interpreting the value differently depending on its size, so it’s tricky to use for comparison purposes.

I’ve been using this page, to ensure that the value is interpreted as milliseconds-since-epoch:
https://currentmillis.com

I don’t completely understand this wording:

End: End of range in milliseconds since Epoch. 0 or blank means current time, and negative values are milliseconds into the past from now. End can also be an Epoch timestamp.

This sounds like “Epoch timestamp” is different from “End of range in milliseconds since Epoch”. Does this mean that “end” can be interpreted as either seconds-since-Epoch or milliseconds-since-Epoch?

1 Like
10

Ok, I think I may understand what’s going on.

If you put time=1, the returned query indicates that it’s starting approximately 6 months in the past. I guess this is happening because of Losant’s data-retention period.

My expectation is that the “start” and “end” values returned from the query would exactly reflect the end and start timestamps, but it looks like they are being forced into the time-range of the last 6 months.

This behavior was confusing me, and I ended up thinking that Losant was guessing whether this value was interpreted as milliseconds or seconds depending on the size.

If you think about the converter you’re using, it must have some poorly-defined behavior. I can see that if I enter either of the following values, I get the same result:
1571689030
1571689030000

So my concern was that Losant was doing something similiar in the background. This raises the question - how is this tool deciding whether the input is in seconds or milliseconds?

11

Hi @Alexander_Farley,

You are correct, the six-month prior timestamp is due to the data retention period. I spoke with our engineers and was informed that should you include a value less than 1000000000000, Losant will interpret it as seconds since Epoch, since there is no data in Losant prior to 2015.

new Date(1000000000000)
2001-09-09T01:46:40.000Z

I will also be sure to pass along your feedback to our engineering team. Please let me know if there is anything you would like to add or additional questions you may have!

Thanks so much,
Julia

1 Like
13

Thank you very much! This is exactly what I what looking for. That’s what I meant by “range-dependant casting” in my original question; 1000000000000 is the value I was wondering about.

In other news, it seems that I have 2 user-accounts for the forum - I’ll have to figure that out seperately.

1 Like