Send device data to AWS-S3

[A_G]

The following is how I set my Storage Configuration:

I have this Bucket

With the following “directory” (Prefix)

Data Archive Configuration in Losant

With yours it seems like you have two setups of Storage Configuration.

1. for the “directory” (prefix) called losant-device-archive to which you successfully archived your CSV files
2. for the “directory within a directory” called sub-folder to which you did not save any files.

Not sure why the second storage configuration is needed. Also, I noticed you did not use globals to set up the Bucket Template or the Directory inside the Bucket. I do not know if you used globals in Access Key Template or Secret Access Key Template please let me know if you did. If any or all of these can not accept globals please let me know.

Will appreciate your guidance on required changes on my setup and if possible answers on the previous tread.

[Heath]

@A_G,

The information you’ve provided is fantastic. Thank you so much!

Have you attempted to enable AWS S3 Data Archiving with the values you are using in Application Globals rather than global variables only? If you have not, can I suggest attempting this test with just 1 device? This would allow the Archive Backfill to run a bit quicker, as it is less data than the 11 devices you have listed.

Thank you,
Heath

[A_G]

Good advice, Archive initiated followup questions came up. I think the conclusion on this is that Globals should not be used in Bucket Template or in Directory inside the Bucket.

Moving on to the next question with your permission

Running the Test Archive and the Archive Backfill on 5 devices I was able to populate 60 objects in the prefix created in the Bucket. Date spanning from 2020-06-04 to 2020-08-02.

The 5 devices I archived have the following IDs:

All Objects files have only one device ID shown in them

1. When and how will all devices will be shown in the S3 Objects?
2. When will 2020-08-02 to 2020-09-02 data arrive?

[Heath]

@A_G,

Globals are able to be used, but a bug was found where the Directory Inside the Bucket field is not able to be templated. A ticket was filed for this, so I will be sure to update you when this is fixed.

1. When and how will all devices will be shown in the S3 Objects?

In S3, you will have a file that looks similar to 5ca0f8ead9882500089bb01b-5f3c70d39eafaa0006417c60-1583366400000-1583452799999.csv which is applicationID-deviceID-beginningTime-endingTime where beginningTime and endingTime are in milliseconds since epoch.

As far as “when” those devices will be available in S3 is dependent on the amount of data associated with each device.

2. When will 2020-08-02 to 2020-09-02 data arrive?

As our documentation states, data archiving will occur when the device data is older than 30 days. Therefore, data will be archived from the date the device state was reported + 31 days.

Thank you,
Heath

[A_G]

Thank you, but I do not follow the logic in the timeline. Will the data always be 30 days old in the bucket and will update each day by one object per device? (i.e. tomorrow I will see the object for Aug 3rd 2020)?
Also, if data will be archived from the date the device state was reported + 31 days why did I receive 60 days’ worth of data to my bucket (from 2020-06-04 to 2020-08-02)? Today I can see 60 objects, will tomorrow I see 61 objects?

Regarding the time to archive all devices: I would like to understand the dependency formula that relates the amount of data to the archive with the time it takes it to be shown in S3 (in Gb/Hour values). In the next few months, I will ramp up from 10 devices to 1000 devices and will need to know how much time it will take to accomplish that.

Thank you for your assistance

[Heath]

@A_G,

Regarding Data Archiving and the timeline, there is a forums post where we have discussed this. The same principles outlined in that thread for Azure also apply for Data Archiving with AWS.

I would like to understand the dependency formula that relates the amount of data to the archive with the time it takes it to be shown in S3 (in Gb/Hour values).

I cannot give an accurate value for this, as this process is carried out by a backend process and is subject to available resources when the processes are kicked off.

Thank you,
Heath

[A_G]

Thank you, this helps. I was hoping with S3 it will differ.

In this post a restriction of 5MB was placed on a using AWS Put Node (correct me if I did not understand this correctly).

1. What do you suggest as a method to archive/send data from all devices (up to 1000 devices) on a daily basis (not from 30 days in the past) to S3?
2. How would you recommend to calculate/evaluate the payload I intend to send to S3 AWS?

[Heath]

@A_G,

Would you be able to give some more detail into why you would like to send data to AWS S3 daily?

There is not currently an easy way to do this in Losant today, but there is a more advanced option. You could have a workflow set on a daily timer to make a request to the Device Export API Endpoint. In that request, the callbackUrl is set to a Losant Webhook which then triggers another workflow that utilizes the AWS Lambda Node. With a Lambda Function you can download the contents of the export anywhere in AWS.

Let me know if you have any other questions.

Thank you,
Heath

[A_G]

I was trying to enable a socket.io but understood from this post that it is not supported directly through Losant. I am confined by the Socket Server URL to use only Socket.IO event-based communication.

From AWS I can create this type of connection although I would rather do all these connections from Losant… If you think socket,io is possible from Losant please let me know. For now, I would like to try your suggestion.

Regarding your suggestion (if you wish to maintain it after reading the above): I would like to understand it in greater detail.

1. Are there any payload size restriction in your proposed method?
2. Please confirm the workflow setups
   2.1 Workflow one (Daily Timer to Device Export)
   
   2.2 Workflow two (Webhook to AWS Lambda)
   
   Please let me know if I am missing any nodes

Your assistance is steller. Thank you

[Aidan_Zebertavage]

Hi @A_G,

Your workflows are correct. One additional piece of information to consider when you are configuring both the Losant workflows and the AWS Lambda function - when you trigger the API for the data export, it generates a URL, not the actual data export file itself. The AWS Lambda function will need to be configured to reference that URL and pull the actual data export file. Because in your first workflow you are generating this URL output, you won’t run into any payload size restrictions.

Thanks,
@Aidan_Zebertavage

[A_G]

Thank you for this input. Diving into the Losant API configuration trying to establish the Losant API to export to the S3 Bucket Container/Prefix:

Questions about the below workflow:

selecting the request to make
Under the Losant API setup I picked the following resource:

configuring the parameters for that request

1. Under PARAMETERS I see I can choose the deviceId. What is the format of choosing more than one device? What is the format of choosing 1000 devices?
   all.Application ???
   APPLICATION_ID?? ID associated with the application ??

2. When selecting more than one device is there a meaning for the start and end? Can these be kept at “0” and “0”
   

3. Confirm I will need to place the https of the webhook in callbackUrl under this format: https://example.com/cburl

4. How will the downloadable links look like is includeBlobData is not set as true?

determining where to store the result

Once you confirm the above and answer these question I will move to the second workflow.
Your guidance is greatly appreciated.

[Heath]

@A_G,

Instead of the Device: Export option in the Losant API Node, I would like to suggest using the Devices: Export.

Device: Export is for only a singular device, where Devices: Export is for all devices, or devices that you select via a query. You can ready more about the options we provide in our Rest API Documentation.

1. Under PARAMETERS I see I can choose the deviceId. What is the format of choosing more than one device? What is the format of choosing 1000 devices?
   all.Application ???
   APPLICATION_ID?? ID associated with the application ??

If the switch is made to Devices: Export, the documentation I linked for the Rest API should guide you through this configuration, but please let me know if you need more assistance.

2. When selecting more than one device is there a meaning for the start and end? Can these be kept at “0” and “0”

If using Devices: Export these fields will not be available.

3. Confirm I will need to place the https of the webhook in callbackUrl under this format: https://example.com/cburl

Yes, but will be a Losant Webhook, which will trigger the second workflow.

4. How will the downloadable links look like is includeBlobData is not set as true?

The links will not differ in appearance. Including Blob Data for an export will simply include Blob data.

Storing the API result in the Payload is not required, but if the field is filled out (as you have it), a success or fail message will be placed at the payload path specified for you to handle as needed.

Another tool that you can use to test this is Postman. You can see below that I have requested the Devices: Export from the Losant API, and below is the response with

{
  "success": true
}

Please let me know if you have any other questions.

Thank you,
Heath

[A_G]

Thank you for this detailed explanation. I think Losant API is now set up correctly.

I got a similar result on the Webhook created for Losant API

Moving on to the following Workflow:

Here the webhook node is associated with the Webhook tested above in Postman (the webhook https referenced in the Losant API is issued to this Webhook Node).

The AWS Lamda questions are the following:

1. AWS Config is straightforward (I used Globals)
   
   Remembering that Data Archive had a restriction that did not allow me to use Globals I have to ask if I can use Globals here?

2. Lambda Config was set up the following way
   

   
   
   I received the following message in Debug
   
   Your guidance on how to amend it will be greatly appreciated.

[Aidan_Zebertavage]

Hi @A_G,

For your first question, yes you can use globals in your AWS config.

For your second question, the error you are receiving is due to the fact that your access key and secret likely do not have the right access level within your IAM policy.

You’ll want to confirm your IAM policy and ensure that the access keys and secret have the right access level. You’ll also want to double-check that the key and secret are properly set for your globals within Losant, to ensure spelling and casing are correct as well.

Thanks,
@Aidan_Zebertavage

[A_G]

Thank you for your reply,

Is there a different IAM policy access level required for AWS Lambda then the one set up for the Data Archive option from within Losant to AWS S3? I am using the same key and secret in the AWS Lambda.

The functional Data Archive setup in Losant had me set an IAM Policy that allow the CSV files to be archived in S3 Bucket/Prefix format.

Where will the above configuration be shown within AWS?

[Aidan_Zebertavage]

Hi @A_G,

You’ll need to use explicit access key and secret keys for allowing access to explicit resources within the AWS ecosystem, you won’t be able to use the same policy to provide access to all resources. The Lambda node documentation recommends creating an access key/secret pair to be used in your workflow that only has access to your Lambda function in AWS:

AWS Lambda Node

I would take a look at the below documentation from AWS regarding IAM guidelines and providing Lambda permissions.

https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

https://docs.aws.amazon.com/lambda/latest/dg/lambda-permissions.html

We are currently testing internally to be able to generate and provide an example policy output for your reference, and I will provide that to you when available.

Thanks,
@Aidan_Zebertavage

[Aidan_Zebertavage]

Hi @A_G,

Some more information on this topic for you after some investigation on our side.

You’ll want to follow the steps below to generate a new policy and providing it access to your Lambda resource:

1. Create a new policy - following with the AWS docs: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html

2. Create a user - following with the AWS docs: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html

3. Attach the Policy to the User, for example, see the policy output below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
        }
    ]
}

The policy above gives access to a single Lambda function. AWS provides additional documentation around this subject should you want to change that: https://docs.aws.amazon.com/lambda/latest/dg/lambda-permissions.html.

Hopefully, this gets you headed in the right direction - let us know how we can help further.

Thanks,
@Aidan_Zebertavage

[A_G]

Thank you for these instruction Aidan. I placed the above json in the json Tab (under create policy) and got 2 warnings:

I got the following summary when trying to set up the policy:

What should I adjust?

Thank you for the continued assistance

[Aidan_Zebertavage]

Hi @A_G,

My apologies, I should have clarified when sending the policy code over that this is an example and contains some placeholder text.

You will need to modify the policy JSON to reflect your specific Lambda function - specifically, the following fields will need adjustment:

<region>
<123456789012>
<example_function>

These need to be replaced with your specific region, account number, and function name.

The AWS wizard for creating a policy takes you through these specific steps for generating the policy without the need for any manual JSON editing on your part:

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Authorizing.IAM.LambdaCreatePolicy.html

Let me know if this clears up your issues.

Best,
@Aidan_Zebertavage

[A_G]

This clarifies it, thank you. I know my region (us-east-2) and account number but for our purpose how should I set up the function?

Resources and conditions for Lambda actions - AWS Lambda

Seems like the following set up:

Yield this result: