Send device data to AWS-S3

[A_G]

Followup questions to the above trail. Please elaborate on: “With a Lambda Function you can download the contents of the export anywhere in AWS”

1. Once we finish setting up the Policy with the Lambda ARN where will the export of 1000 devices go into in AWS? AWS CONFIG in the AWS LAMBDA node in Losant allow access to AWS but it does not seem to set a location within AWS to place the export at. Please elaborate on the final location of the export.
2. I defined the data method under the AWS LAMBDA node LAMBDA CONFIG as workflow payload. Will all 1000 device workflow payloads be exported? with what extension name?

[anaptfox]

@A_G,

I want to take a step back a bit to make sure we are leading you down the right path. Earlier in this thread, we mentioned that this route was a more advanced option.

@Heath

There is not currently an easy way to do this in Losant today, but there is a more advanced option

To accomplish this goal, we recommend learning and taking a look at AWS and learning a bit more about:

• AWS Policies
• AWS Lambda

Here is a great video and resoruce online that helps pull some of these pieces together:

Introduction to AWS Lambda & Serverless Applications

AWS Lambda – Getting Started

I recommend taking a deeper dive into AWS’s resources and going through some basic Lambda examples.

I’m recommending this route because of these questions:

@A_G

• Once we finish setting up the Policy with the Lambda ARN, where will the export of 1000 devices go into in AWS? AWS CONFIG in the AWS LAMBDA node in Losant allows access to AWS, but it does not seem to set a location within AWS to place the export at. Please elaborate on the final location of the export.
• I defined the data method under the AWS LAMBDA node LAMBDA CONFIG as workflow payload. Will all 1000 device workflow payloads be exported? With what extension name?

The answers here depend on how you want to configure your Lambda function and application behavior.

However, after you digging to Lambda a bit, here is the answer to your questions:

@Aidan_Zebertavage

One additional piece of information to consider when you are configuring both the Losant workflows and the AWS Lambda function - when you trigger the API for the data export, it generates a URL, not the actual data export file itself. The AWS Lambda function will need to be configured to reference that URL and pull the actual data export file

The export doesn’t generate raw data. It generates a link to the raw data. You can make an HTTP request within the Lambda Function. And then, within the function, you’ll have access to the data, and you can decide where to forward it from there within your AWS environment.

[A_G]

Thank you for this clarification.

I understand that the AWS Lambda function will need to be configured to reference a URL (the Webhook Trigger URL I assume?) and pull the actual data export file (generated by Losant API). The Daily exported file (triggered by Losant Timer Node) will be pulled by AWS from Losant with a Lambda Function. Only once the AWS Lambda Function is configured can the Policy ARN be completed.

My followup questions are the following:

1. How will this file look like? I think the format is set by Losant, please confirm and explain how a 1000 device file be formatted? How can I choose only the device ID and data to be exported for all data within a 24 hour period?
2. I will need to synchronize the Pull (by AWS Lambda Function) and the Devices:Export generated by LOSANT API and Timer nodes workflow. Is the below timeline guaranteed for 24 hours gathered data of 1000 device export file from 12:00(previous day)-12:00(current day)?

3. I will post the AWS Lambda function I eventually used from within AWS to pull the LOSANT API node exported file through the AWS LAMBDA node Webhook URL + Appropriate AWS CONFIG. I am looking for an AWS Lambda function that will pull it to an S3 Bucket now. However, if you have experience with similar AWS Lambda functions and wish to share it will be greatly appreciated.

[anaptfox]

@A_G,

1. You may find an example fo the formating in the documentation. For the second part of the question, I think this was answered well here:

@Heath

Instead of the Device: Export option in the Losant API Node, I would like to suggest using the Devices: Export.
Device: Export is for only a singular device, where Devices: Export is for all devices or devices that you select via a query. You can read more about the options we provide in our Rest API Documentation.

2. The configuration in your screenshot is for the Timer Node and will determine when the workflow is triggered. To customize the export time please refer to the API docs (linked in question 1)

@Heath

Another tool that you can use to test this is Postman

You can use POSTMAN to test the API, and it’s parameters to get the desired output you want.

3. Like I mentioned before, AWS docs on Lambda fills in a lot of these gaps. For example AWS Lambda function handler in Node.js - AWS Lambda

@A_G

I will post the AWS Lambda function I eventually used from…

Unfortunately, building and providing guidance on building a Lambda Function is getting pretty far outside of Losant itself and not something we’ll be able to help much with.

For help building your Lambda function, I recommend:

AWS Support
Stack Overflow.

[A_G]

This is good guidance which I intend to take.

With respect to the Losant aspect of connecting the AWS Lambda Node:

1. Please confirm the URL from which AWS Lambda Function pull the Exported File will be the webhook URL I generated in Webhook node in the workflow above.

2. From the documentation you refer to regarding my question about Devices:Export it seems the file that the AWS Lambda Function will pull through the URL will be with a CSV extension similar to the file generated by the import/export (Can you confirm that?)
   

   

Thank you for the detailed assistance

[A_G]

Hi Aidan,
I was able to establish an AWS Lambda function pull. Today was the first time the URL pull was conducted from the Devices:Export and the format of the pulled CSV file is the following:

This is not what I was expecting, as you can see this look like a general summary of the devices. I would like to export all devices data attributes collected for the last 24hours (from noon the day before to noon at the time of the Lambda pull) into the Device:Export. I tried to understand this through the Rest API Documentation but could not find how to fine-tune the Device:Export resource & action.

Do I need to place the Advanced Device Query Schema in the Losant API node? If yes, can you provide a version that query all device’s data attributes?

Your guidance was fruitful, looking forward to your answer on this step

[anaptfox]

@A_G

It looks like the API you linked to is a Schema.

I think the API you’re looking for his here:

Device Actions

This is the API to a device data export. If you take a look at the “Request Query Parameters” you can see that it contains a callbackUrl.

[A_G]

Followup question.

Now that the AWS Lambda function is able to pull the Devices:Export I received a payload with a “downloadUrl” that store the CSV file in a google cloud client library: https://storage.googleapis.com/
Can you confirm this storage in google cloud is temporary? What is the life span of my exported file at Google?

Thank you for your insights

[anaptfox]

@A_G,

That’s a great question, let me confirm with the engineering team.

[anaptfox]

@A_G,

Yes, those export links are temporary and expire after 7 days!

[A_G]

This is good to know

I do not think I have set the Losant API to be linked to a Schema.

Focusing on the callbackUrl,
I have set the Webhook trigger URL into the callbackUrl of the Losant API node

The webhook trigger URL is also linked to the AWS Lambda (as shown in previous treads of the above interaction).

1. Should I have used a different URL in the callbackUrl ?
2. This is the URL I have in my AWS Function as well. Shouldn’t these match?

[Aidan_Zebertavage]

Hi @A_G,

I’m going to open up a direct message with you to recommend some next steps.

Thanks,
@Aidan_Zebertavage