I need to send some device data from multiple devices to AWS-S3.
Would the AWS S3 Put Node be my best option?
How do you suggest I generate the file for upload?
Let’s say I want to upload past weeks raw time series data.
Or is it better using the Archiving?
Hi @Lars_Andersson,
You have multiple options for how to accomplish this, but the path you choose will depend on your use case.
API Node
The API Node includes a Data Export method for devices and their attributes. Within the workflow, you could utilize the Export method to query your devices and send the state data to your S3 bucket.
Time Series Node
The Time Series Node gives you the option of exporting easily with just two nodes, but when querying a large time range, you’ll have to set a resolution, which could cause you to lose some data.
Note, if you are working with a very large dataset, it’s possible you may into trouble with these nodes or within the workflow.
Notebook Export
If you wanted to move this data just a time or two, you also have the option of exporting to a CSV using Notebooks. You can select the devices you would like as well as configure which attributes (if you’d like) and the length of state data to return (30 days, 90 days, etc.). Then you can request a data export from the page of this data.
This question did get some discussion started internally, and I will be making a feature request for this functionality to be wrapped up together in one place. I’ll get back to you with more information once I hear more.
Thanks so much!
Julia
Is there a reason why I couldn’t/shouldn’t use the archiving feature?
Hi Lars,
There’s no reason you can’t use the archiving feature; I just suggested the previous options based on your use case. You wrote you want to upload the “past week’s raw time series data,” but keep in mind data will be only archived once it becomes older than 30 days.
Thanks so much,
Julia
Just curious to what happened to the “internal discussion” that got started?
Well, it turned out great! We now have a whole new set of Bulk Actions for devices:
Including Bulk Export:
It’s a much more configurable way to export data. 
I just tried “bulk device data export” using a filter based on a tag value, but I get data from devices with ID I can’t find in my application. Isn’t the first column in the CSV the device ID?
I’m going to open up a DM with you to work through this and share more information.
Actually, I misspoke. The file I attached was actually from a data export in the notebook input section, but never the less, I used the same tag selection to filter devices so there is still something not right.
Following the S3 Archiving Guidelines: https://docs.losant.com/applications/archiving/
Setting the permissions for Amazon S3 the advice is to set the user’s [IAM Policy] with the following access point ARN:
{
"Version": "YYYY-MM-DD",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:getBucketAcl"],
"Resource": ["arn:aws:s3:::bucketname"]
},
{
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": ["arn:aws:s3:::bucketname/*"]
}
]
}
Please confirm this should be done when the access point is configured in S3.
Also, please confirm the access key ID template will have the access point name
Could you provide more details about what you mean in the quote above?
I would love to know:
Are you attempting this feature, and somethings not working?
Are you looking for clarification as to how something works?
That would help me point you in the right place!
I am trying to archive all my devices to an S3 Bucket
In your instructions there are several configurations I need to bring from the S3 Bucket:
- Bucket Template (this is understood)
- Access Key ID Template (I am trying to find where this is at in the S3 bucket? I was able to generate an Access point ARN number, just not sure if this is the same as Access key ID)
- Secret Access Key Template (I am trying to find where this is at in the S3 bucket)
- Directory Inside the Bucket
In order to set the permission for Amazon S3 you advice adding “effect”:“allow”
I am trying to understand where to place these and I ask if these should be set up under the Access Point ARN.
Your help will be greatly appreciated
Hi @A_G,
The AWS Key and the AWS Secret come from an IAM Policy.
In AWS, you would create a new “user” that represents Losant. That user now has a key and secret to authenticate to AWS. The policy you define tells AWS, what this new user should or shouldn’t be able to do.
I can update the docs to make this a bit clearer.
Thank you for your fast reply.
- Please confirm the terminology Losant use for AWS Prefix is: “Directory”
- Please confirm that set the user’s IAM Policy is the same as placing the JSON string suggestion from your instructions in the Bucket policy editor ARN.
Hey @A_G,
Just so that I could have context to your problems, I just set up Data Archiving to AWS S3 in my own personal project. So hopefully I can help clear some things up for you.
- Please confirm the terminology Losant use for AWS Prefix is: “Directory”
Could you share with me a screenshot of where you’re seeing AWS Prefix being used?
In our Data Archiving configuration Directory Inside the Bucket is the folder in which you wish the data to be archived. So, if I have a bucket called Device-Data and inside that bucket I have a folder called Device-1, you would put Device-1 in the Directory Inside the Bucket field in the configuration.
- Please confirm that set the user’s IAM Policy is the same as placing the JSON string suggestion from your instructions in the Bucket policy editor ARN.
This is incorrect. You will need to create a new User for your AWS account, which you can do from the IAM Console. When you create this new user, ensure you select “Programmatic Access”, and don’t set permissions yet. Once the User is created, click on that User in the list of Users and in the Permissions tab, click (on the right side) Add Inline policy.
There you will find a tab that shows JSON, where you will use the JSON String provided (with bucketname updated).
This configuration allows you to obtain the Access Key and Access Key Secret needed for the Data Archiving configuration as well as ensure that the proper access is associated for the User.
Please let me know if this works for you, or if you have any other questions.
Thank you,
Heath
Thank you for your reply.
-
AWS does not have folders (even though AWS allows you to “+create folder”). Any “directory” in losant terms seems to be a “Prefix” in AWS terms. I just want to validate this. Also, what should be the value format of the “directory” in the losant application global? AWS differentiates the Prefix from the Bucket with a backslash. Does Losant do that in the back end? Should I just have the “directory” name in the application global? Following your example for S3 Data Archive under Directory Inside the Bucket should I associate this with {{globals,Directory}}? In global application this will be:
Key: Directory
Value: Device-1 -
I am trying section 2 which is very well explained. Much appreciated. The Access Key and Access Key Secret seem to appear before setting the JSON tab. The version string needs to be changed from yyyy-mm-dd to the following criteria.
Note I have device data that is older than 30 days,
Enabled Device Data Archiving.
Engaged the “test archive”
Engaged the archive backfill
What is the expected timeline to see the objects (losant csv files) in the S3 bucket/prefix?
Hey @A_G,
Before I go into your questions above, is Data Archiving working for you as it stands right now? Knowing this will help me better answer the rest of the questions you’ve included here.
What is the expected timeline to see the objects (losant csv files) in the S3 bucket/prefix?
For my work that I did this morning, I moved a small amount of data in about 15 minutes, but the time overall is dependent on the amount of data you are archiving, so I cannot give an accurate time frame.
Thank you,
Heath
So far no. Under the prefix I set in the bucket no objects have populated yet.
Your guidance is appreciated.
@A_G,
Would you be able to share with me a screenshot of your Data Archive configuration?
I have this bucket:
With the following inside:
Here’s my Data Archive configuration in Losant:
I also, have another folder inside losant-device-archive called sub-folder:
And with the following Data Archive config, I was able to backfill my device data:
Thank you,
Heath