# Send device data to AWS-S3 **URL:** \ **Category:** Help\ **Tags:** workflow\ **Created:** [June 4, 2020, 7:38pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006 "2020-06-04T19:38:03Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [June 4, 2020, 7:38pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/1 "2020-06-04T19:38:03Z")
I need to send some device data from multiple devices to AWS-S3. Would the AWS S3 Put Node be my best option? How do you suggest I generate the file for upload? Let’s say I want to upload past weeks raw time series data. ---
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [June 4, 2020, 8:05pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/2 "2020-06-04T20:05:22Z")
Or is it better using the Archiving? ---
**Author:** ![JuliaKempf](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/juliakempf/32/2446_2.png) [@JuliaKempf](https://forums.losant.com/u/JuliaKempf)\ **Post date:** [June 4, 2020, 9:34pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/6 "2020-06-04T21:34:05Z")
Hi @Lars_Andersson, You have multiple options for how to accomplish this, but the path you choose will depend on your use case. ## API Node The API Node includes a Data Export method for devices and their attributes. Within the workflow, you could utilize the [Export method](https://docs.losant.com/rest-api/data/#export) to query your devices and send the state data to your S3 bucket. ## Time Series Node The Time Series Node gives you the option of exporting easily with just two nodes, but when querying a large time range, you’ll have to set a resolution, which could cause you to lose some data. ### Note, if you are working with a very large dataset, it’s possible you may into trouble with these nodes or within the workflow. ## Notebook Export If you wanted to move this data just a time or two, you also have the option of exporting to a CSV using Notebooks. You can select the devices you would like as well as configure which attributes (if you’d like) and the length of state data to return (30 days, 90 days, etc.). Then you can request a data export from the page of this data. This question did get some discussion started internally, and I will be making a feature request for this functionality to be wrapped up together in one place. I’ll get back to you with more information once I hear more. Thanks so much! Julia ---
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [June 12, 2020, 7:13pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/7 "2020-06-12T19:13:56Z")
Is there a reason why I couldn’t/shouldn’t use the archiving feature? ---
**Author:** ![JuliaKempf](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/juliakempf/32/2446_2.png) [@JuliaKempf](https://forums.losant.com/u/JuliaKempf)\ **Post date:** [June 19, 2020, 4:51pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/11 "2020-06-19T16:51:04Z")
Hi Lars, There’s no reason you can’t use the archiving feature; I just suggested the previous options based on your use case. You wrote you want to upload the “past week’s raw time series data,” but keep in mind data will be only archived once it becomes older than 30 days. Thanks so much, Julia ---
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [July 7, 2020, 11:55am UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/13 "2020-07-07T11:55:58Z")
Just curious to what happened to the “internal discussion” that got started? ---
**Author:** ![anaptfox](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/anaptfox/32/873_2.png) [@anaptfox](https://forums.losant.com/u/anaptfox)\ **Post date:** [July 7, 2020, 1:43pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/16 "2020-07-07T13:43:24Z")
@Lars_Andersson, Well, it turned out great! We now have a whole new set of Bulk Actions for devices: > **[Platform Update - Bulk Device Actions](https://www.losant.com/blog/platform-update-bulk-device-actions)** > > This update makes it easy to perform several bulk actions against some or all of your Losant devices. Learn more. Including Bulk Export: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/e/e2795c88f5b4b7ce3ccf5c594ba70d17b831602a.jpeg) It’s a much more configurable way to export data. 🙂 ---
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [July 7, 2020, 2:29pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/17 "2020-07-07T14:29:42Z")
I just tried “bulk device data export” using a filter based on a tag value, but I get data from devices with ID I can’t find in my application. Isn’t the first column in the CSV the device ID? ---
**Author:** ![anaptfox](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/anaptfox/32/873_2.png) [@anaptfox](https://forums.losant.com/u/anaptfox)\ **Post date:** [July 7, 2020, 2:37pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/19 "2020-07-07T14:37:19Z")
@Lars_Andersson, I’m going to open up a DM with you to work through this and share more information. ---
**Author:** ![Lars\_Andersson](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lars\_Andersson](https://forums.losant.com/u/Lars_Andersson)\ **Post date:** [July 7, 2020, 2:59pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/20 "2020-07-07T14:59:03Z")
Actually, I misspoke. The file I attached was actually from a data export in the notebook input section, but never the less, I used the same tag selection to filter devices so there is still something not right. ---
**Author:** ![A\_G](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@A\_G](https://forums.losant.com/u/A_G)\ **Post date:** [August 28, 2020, 12:17am UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/22 "2020-08-28T00:17:46Z")
Following the S3 Archiving Guidelines: [https://docs.losant.com/applications/archiving/](https://docs.losant.com/applications/archiving/) Setting the permissions for Amazon S3 the advice is to set the user’s [IAM Policy] with the following access point ARN: ```auto { "Version": "YYYY-MM-DD", "Statement": [ { "Effect": "Allow", "Action": ["s3:getBucketAcl"], "Resource": ["arn:aws:s3:::bucketname"] }, { "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": ["arn:aws:s3:::bucketname/*"] } ] } ``` Please confirm this should be done when the **access point** is configured in S3. Also, please confirm the access key ID template will have the access point name ---
**Author:** ![anaptfox](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/anaptfox/32/873_2.png) [@anaptfox](https://forums.losant.com/u/anaptfox)\ **Post date:** [August 28, 2020, 6:46pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/24 "2020-08-28T18:46:25Z")
> [@A\_G](#): > > Please confirm this should be done when the **access point** is configured in S3. > > Also, please confirm the access key ID template will have the access point name Could you provide more details about what you mean in the quote above? I would love to know: Are you attempting this feature, and somethings not working? Are you looking for clarification as to how something works? That would help me point you in the right place! ---
**Author:** ![A\_G](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@A\_G](https://forums.losant.com/u/A_G)\ **Post date:** [August 28, 2020, 11:41pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/26 "2020-08-28T23:41:37Z")
I am trying to archive all my devices to an S3 Bucket In your [instructions](https://docs.losant.com/applications/archiving/) there are several configurations I need to bring from the S3 Bucket: 1. Bucket Template (this is understood) 2. Access Key ID Template (I am trying to find where this is at in the S3 bucket? I was able to generate an Access point ARN number, just not sure if this is the same as Access key ID) 3. Secret Access Key Template (I am trying to find where this is at in the S3 bucket) 4. Directory Inside the Bucket In order to set the permission for Amazon S3 you advice adding “effect”:“allow” I am trying to understand where to place these and I ask if these should be set up under the Access Point ARN. Your help will be greatly appreciated ---
**Author:** ![anaptfox](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/anaptfox/32/873_2.png) [@anaptfox](https://forums.losant.com/u/anaptfox)\ **Post date:** [August 31, 2020, 1:15pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/28 "2020-08-31T13:15:24Z")
Hi @A_G, The AWS Key and the AWS Secret come from an [IAM Policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policy-types). In AWS, you would create a new “user” that represents Losant. That user now has a key and secret to authenticate to AWS. The policy you define tells AWS, what this new user should or shouldn’t be able to do. I can update the docs to make this a bit clearer. ---
**Author:** ![A\_G](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@A\_G](https://forums.losant.com/u/A_G)\ **Post date:** [September 1, 2020, 12:16am UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/30 "2020-09-01T00:16:46Z")
Thank you for your fast reply. 1. Please confirm the terminology Losant use for **AWS Prefix** is: “Directory” 2. Please confirm that set the user’s [IAM Policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policy-types) is the same as placing the JSON string suggestion from your instructions in the Bucket policy editor ARN. ---
**Author:** ![Heath](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/heath/32/3036_2.png) [@Heath](https://forums.losant.com/u/Heath)\ **Post date:** [September 1, 2020, 3:02pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/32 "2020-09-01T15:02:41Z")
Hey @A_G, Just so that I could have context to your problems, I just set up Data Archiving to AWS S3 in my own personal project. So hopefully I can help clear some things up for you. > 1. Please confirm the terminology Losant use for **AWS Prefix** is: “Directory” Could you share with me a screenshot of where you’re seeing `AWS Prefix` being used? In our Data Archiving configuration `Directory Inside the Bucket` is the folder in which you wish the data to be archived. So, if I have a bucket called `Device-Data` and inside that bucket I have a folder called `Device-1`, you would put `Device-1` in the `Directory Inside the Bucket` field in the configuration. > 1. Please confirm that set the user’s [IAM Policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policy-types) is the same as placing the JSON string suggestion from your instructions in the Bucket policy editor ARN. This is incorrect. You will need to create a new User for your AWS account, which you can do from the [IAM Console](https://console.aws.amazon.com/iam/home?region=us-east-1#/home). When you create this new user, ensure you select “Programmatic Access”, and **don’t set permissions yet**. Once the User is created, click on that User in the list of Users and in the `Permissions` tab, click (on the right side) `Add Inline policy`. ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/2/21d97c65d11454719726bbe539085c3c42cfe947.png) There you will find a tab that shows `JSON`, where you will use the JSON String provided (with `bucketname` updated). ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/e/e866572d616f11ea457086f92b382f6369b8cd6e.png) This configuration allows you to obtain the Access Key and Access Key Secret needed for the Data Archiving configuration as well as ensure that the proper access is associated for the User. Please let me know if this works for you, or if you have any other questions. Thank you, Heath ---
**Author:** ![A\_G](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@A\_G](https://forums.losant.com/u/A_G)\ **Post date:** [September 1, 2020, 6:35pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/33 "2020-09-01T18:35:34Z")
Thank you for your reply. 1. AWS does not have folders (even though AWS allows you to “+create folder”). Any “directory” in losant terms seems to be a “Prefix” in AWS terms. I just want to validate this. Also, what should be the value format of the “directory” in the losant application global? AWS differentiates the Prefix from the Bucket with a backslash. Does Losant do that in the back end? Should I just have the “directory” name in the application global? Following your example for S3 Data Archive under _Directory Inside the Bucket_ should I associate this with {{globals,Directory}}? In global application this will be: Key: Directory Value: Device-1 2. I am trying section 2 which is very well explained. Much appreciated. The **Access Key and Access Key Secret** seem to appear before setting the JSON tab. The version string needs to be changed from yyyy-mm-dd to the [following](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_version.html) criteria. Note I have device data that is older than 30 days, Enabled Device Data Archiving. Engaged the “test archive” ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/d/d41aa944263016d9f2edd378a7b538ec91470a0d.png) Engaged the archive backfill ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/7/76ee7947423ddc471ffd34d45459f521bf79db50.png) What is the expected timeline to see the objects (losant csv files) in the S3 bucket/prefix? ---
**Author:** ![Heath](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/heath/32/3036_2.png) [@Heath](https://forums.losant.com/u/Heath)\ **Post date:** [September 1, 2020, 7:08pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/34 "2020-09-01T19:08:11Z")
Hey @A_G, Before I go into your questions above, is Data Archiving working for you as it stands right now? Knowing this will help me better answer the rest of the questions you’ve included here. > What is the expected timeline to see the objects (losant csv files) in the S3 bucket/prefix? For my work that I did this morning, I moved a small amount of data in about 15 minutes, but the time overall is dependent on the amount of data you are archiving, so I cannot give an accurate time frame. Thank you, Heath ---
**Author:** ![A\_G](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@A\_G](https://forums.losant.com/u/A_G)\ **Post date:** [September 1, 2020, 11:01pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/35 "2020-09-01T23:01:37Z")
So far no. Under the prefix I set in the bucket no objects have populated yet. Your guidance is appreciated. ---
**Author:** ![Heath](https://sea1.discourse-cdn.com/flex015/user_avatar/forums.losant.com/heath/32/3036_2.png) [@Heath](https://forums.losant.com/u/Heath)\ **Post date:** [September 1, 2020, 11:15pm UTC](https://forums.losant.com/t/send-device-data-to-aws-s3/3006/36 "2020-09-01T23:15:53Z")
@A_G, Would you be able to share with me a screenshot of your Data Archive configuration? I have this bucket: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/7/7875cb3241c3bcd2137e7b5c5c9433eeeae9f3de.png) With the following inside: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/a/a445cd45ea227c9a5260be53daf659c5903da471.png) Here’s my Data Archive configuration in Losant: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/7/709cbf90fb07172962ae7c4030f7af42e372d15f.png) I also, have another folder inside `losant-device-archive` called `sub-folder`: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/4/409e1e69c264627c4206eb37d403381392c33ed2.png) And with the following Data Archive config, I was able to backfill my device data: ![image](https://us1.discourse-cdn.com/flex015/uploads/getstructure/original/2X/d/d151ec970aaaeb5df9e253cfec6b8c9006db25ab.png) Thank you, Heath [Next page](https://forums.losant.com/t/send-device-data-to-aws-s3/3006.md?page=2)