Hi,
I created an APPLICATION ARCHIVING to the Azure Storage Type
STORAGE CONFIGURATION is successful as the container in Azure seem to populate a daily folder from the Device I want to archive with the correct headers (representation of the device attributes).
Check marks on:
Enable Device Data Archiving
Enable Events Archiving.
Data for the last 31 days have not arrived (csv table in Azure is Empty, only with headers)
Please assist, what in the configuration need to be adjusted?
Hi @Amit_Gitterman,
It sounds as if you’re describing Losant’s archiving feature correctly, Losant does not begin to archive device or event data until the data is at least 31 days old. Device state data can be overwritten up until it is 30 days old, which is why we wait until 31 days to begin archiving.
So the question for you is, do you have device data that is at least 30 days old?
I would take a look at the docs around archiving: https://docs.losant.com/applications/archiving/
You may also want to take a look at utilizing Notebooks within your application to collect and analyze device state data: https://docs.losant.com/notebooks/overview/
If you have any other questions just let us know!
Thanks,
@Aidan_Zebertavage
Thank you for getting back with me,
The device is older than 31 days. Data is not populated in the Container, just the headers are shown for some reason.
Any thing I can do from my side to make it happen?
The device is older than 31 days, but has the device been reporting state for longer than 30 days? A few ways are available for you to confirm this.
Two options for you to explore:
Once we confirm that you do have data that is at least 31 days old, we can explore options for troubleshooting any communication issues with Azure.
Thanks,
@Aidan_Zebertavage
I tried both options per your instructions. Both show that I have attributes in the device of interest reporting a state for longer than 30 days. State reporting from 11th May 2020 15:00 to date (17th June 2020).
You assistance is highly appreciated
Thanks for confirming on the history of the device state data.
Next question, did you configure archiving for your application in the past few days? Or did you have this configured from the onset of your application?
If you just recently configured archiving, you may need to initiate a backfill of the device data that is older than 31 days but still saved in Losant.
Is the backfill option available to you on your configuration page for application archiving?
Thanks,
@Aidan_Zebertavage
Archiving was not configured from the onset of the application.
The exact date of archiving configuration was within the first 31 days of data accumulation.
Thanks for the clarification on when archiving was set up.
Yes, pressing "Archive Backfill’ will start working on the backfill, and each day will still be its own file.
Pressing the Archive Backfill button today will archive all data stored that is 31 days old or older - each day moving forward from today that is 31 days or older will be archived.
It may take some time to fully populate the Azure Container depending on how much data is going to be archived, which, yes, depends on how often you are reporting device state data. The backfill process will start immediately once you click the button.
If for some reason the backfill cannot complete, you will receive an email alerting you to that fact.
Thanks,
@Aidan_Zebertavage
Thank you for your answers. I am confused about “2” and the last remark about the email alert
[A] “Pressing the Archive Back fill button today will archive all data stored that is 31 days old or older - each day moving forward from today that is 31 days or older will be archived.”
Each day from today that is 31 days or older be archived : Does this mean that only tomorrow I will see data in the container within Azure for the data that was accumulated today? Will the archive back fill retroactive fill the data for all the last 31 days?
[B] “email alerting you to that fact”
When should this email be generated? Is this an immediate email that check a daily content match of the Azure container data tables? Will this check be able to differentiate between a back fill of data and a back fill of headers?
[C] Just as a reminder I do believe the archive is working properly as the headers for the data stored in losant are shown in the daily container folders within Azure Storage. The only thing that is missing is the data. If the back fill email alert is associated with a check for connectivity functionality with Azure I do not think it will detect the lack of Data Content. Please describe what the email alert check for.
My apologies, sorry for the confusion.
To be clear, archiving only occurs on data that is 31 days old or older. So you will not see data in Azure tomorrow that was accumulated today. The data you see in Azure will always be data that is at least 31 days old. The reason for this 30 day delay prior to archiving is that up until that point data can be changed or overwritten, and thus by only archiving data 31 days we are ensuring that it is the final data in the archive.
The emails for archiving will be generated when an error occurs that is on Losant’s side or if some permissions were changed locking us out of the container.
After you’ve pressed the Archive Backfill button earlier this afternoon has any data been populated into your Azure environment? Are you still not seeing data?
Thanks,
@Aidan_Zebertavage
No email was generated.
Just checked the container folders for dates: 17 April 2020-17 June 2020 (each day have its own folder- going 60 days back)
All CSV files have the correct headers but the data is blank (no data has been filled back)
Please assist/ direct how to followup on this
I apologize you’re still having this issue.
We are going to set up a test internally here to ensure that Losant’s communication with Azure is working correctly. Since your headers are populating, I believe your configuration set up is correct.
I will follow up when I have more information.
Thanks,
@Aidan_Zebertavage
We’ve done some digging on our end and we are able to confirm that Losant is communicating with Azure appropriately.
My apologies if this seems redundant but let’s verify that your device state data is reaching Azure. I’m not sure if you’ve you’ve gone through and checked every individual container file but I’d like to ask you to go and look in Azure for data associated with May 19th, 2020, and device SMC LTE - and confirm with us if there is or is not data being logged.
A good secondary check on this is to look into the data explorer method I mentioned earlier, so you can see what your expected data being archived into Azure looks like.
If you do see a blank file, that actually doesn’t indicate that there is a problem, it just means that there is no device state data to report for that day. The blank files are entirely intentional on our part, its to indicate that there was no data to report on that day, otherwise it may be misinterpreted that we are skipping days in the archive process.
Thanks,
@Aidan_Zebertavage
Thank you for answering
I checked the date you refer to and for some reason it is missing from my container
What do you estimate happened? I am missing <10 days from the container
Following up from your response last week, is your Azure archive reflecting device data folders up through May 22nd? Could you confirm?
The gap in files you’re seeing is due to the difference in how device archives and event archives are handled.
Device data archiving is subjected to the 31 day requirement we’ve been discussing. Event data archiving, is handled on a daily basis.
The 2020-05-18 directory is your most recent device data archive.
Thanks,
@Aidan_Zebertavage
Thank you for answering
Azure Archive dates reflect the device data folders from Dec 4 2019 to June 17 2020 in the Azure Container. Seems like we are heading toward the desired outcome. When I checked today 18 May 2020 had data for that day. Following this find I checked the following folders in Azure:
May 1 to May 11 2020 does not have data in the respective folders (note that May 11 2020 had only partial data).
Data in CSV was populated for May 12 to May 18 2020.
June 1 to June 17 2020 does not have data in the CSV respective folders.
The only gap in the AZURE Container consecutive files is from 18 May 2020 to 1 June 2020
If the gap is consistent due to the difference in how device archives and event archives are handled why does it appear only between 18 May 2020 to 1 June 2020. Shouldn’t the gap be consistent throughout the back fill period?
When do you think the CVS will start populating 31 days of backfill data? Did I miss the last cycle? Will 18 May 2020 to 1 June 2020 folders appear in the next cycle?
Hi @Amit_Gitterman,
Checking in - If you check the status of your archive today, you should have device data through May 23rd - is that correct?
In response to your above questions:
By backfilling you should currently have device data up to May 22 because archiving happens daily. Only device data is available to backfill, event data is archived daily. The June data does not have any device data right now, it only has event data. The gap will go away as it continues to archive and add device data.
Once you initiate archiving, the process is run every 24 hours, and looks for data that is 31 days old, and pushes that to your Azure container.
Best
@Aidan_Zebertavage
