User Experience above Retention limits

Help
1

Hi there,
I am trying to understand how can I achieve feature parity with our existing stack (Grafana+InfluxDB) where we can do things like:

  • Seasonality comparison: show me week 1 vs week 10
  • Aggregate comparison: show me average (or other statistical properties) from week 1 vs week 10
  • Window comparison: show me [date1,date2] vs [date3,date4]

We retain full timeseries data so all those queries are possible but of course with Losant there is a maximum retention for device time series.

So the question is what is the best practice to replicate this functionality in a way that I can easily integrate it in dashboard or user experience?

Cheers!

2

Hey @Paolo_Proxy,

As you’ve noticed, Losant dashboards are optimized for showing a moving window of relatively recent data. The platform, in general, is geared more towards real-time processing and decisions vs. historical data exploration and analysis.

Here are some options when it comes to exploring and comparing historical data:

  1. Even though the data retention for enterprise accounts defaults to 6 months, it’s quite common for customers to increase that to 1 year. When it comes to presenting data to customers, Losant dashboards rarely provide the flexibility required to present the data in a custom and domain-specific way. Therefore, our customers usually develop Experiences and bring in third-party charting libraries to present the data to their customers. With this option, Experience Workflows can query the required windows of data (e.g. week 1 vs. week 10) and Experience Pages, utilizing any chart library you prefer, present the data in a way that makes the most sense to your customers.

  2. Application Archiving automatically backs up all device data, as CSV files, to your own cloud storage bucket. There is no limit to the amount of data that can be stored, which makes it possible to explore and analyze years of device data. If you’d like to continue to use Grafana to explore the data, this option can be combined with Grafana’s CSV plugin.

  3. Device Data Export is similar to Application Archiving, but provides an on-demand way to export your device data as CSV files.

  4. Using workflows, you can store a copy of all device data in a “Warm Storage” data warehouse of your choosing. We have a how-to guide on using Google BigQuery for warm storage. This options makes it easier to use other Google services to analyze and explore the data.

1 Like
3

Hi Brandon,
thanks for the pointers, extending the retention to 1 year will be actually a good compromise for the time being.
With regards to application and data export, would it be also useful to add a popular column based format such as Avro or Parquet (I personally use the latter quite a lot), the CSV files are nice but there are a few quirks when working with data science solutions.
I have never used BigQuery so I cannot comment on that.
Thanks.

4

Also do you have any examples of using Experience with charting libraries loading data from warm/cold storage from AWS such as S3?

5

Supporting additional data serialization types is interesting - not something we ever considered. It does make sense.

In terms of examples, I don’t think there’s anything readily shareable. It would make a good how-to guide at some point. To implement it, you’ll have to become comfortable with JavaScript. Getting data from the payload into JavaScript is often the first challenge. Here’s the mechanism I’ve been using:

<div id="chartContainer" data-chart-data='{{jsonEncode pageData.chartData}}'></div>

I like to use Data Attributes since it keeps all the template code within the markup. You can then access that data from your code.

let container = document.getElementById('chartContainer');
let chartData = JSON.parse(container.dataset.chartData);

// Perform any required data transformations required by the charting library.

When this page is rendered, the Experience Workflow queries whatever warm storage option you’re using (S3 or BigQuery Nodes) and provides the result to the page to be rendered.

In terms of charting libraries, these are our favorites:

  1. Chart.js
  2. ECharts
  3. Vega and Vega Lite
1 Like
6

Yes makes sense, I am guessing one would really be careful about the size of the chartData otherwise it will affect the loading time of the page.