Here is some additional data
Here is the “Config” file that I used
C:\Users\v111405\Downloads>type config.iotgw
[logger]
out = "/data/losant-edge-agent-log.log’
level = ‘verbose’
[gateway]
id = ‘5d5ea5081a74840006eabffc’
key = ‘42530772-d9a7-44ca-8feb-fd87dXXXXXX’
secret = ‘6bf8896ce0502155959c3f92c040a12dc0becXXXXXXXXXXXX’
Here is the actual Docker command I used to start the “Losant Agent” container
C:\Users\v111405\Downloads>type dockerCommand
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.iotgw:/etc/losant/losant-edge-agent-config.iotgw losant/edge-agent
ed
Hi @Edward_Szajner,
I apologize for the delay, this is not an issue we have run into before, so we have been discussing the possible causes. I received a few suggestions (and have a couple of questions as well) so we can get further into debugging! I apologize if any of these questions and suggestions are redundant to your own debugging
- I am curious, is there a reason you are using a .iotgw file over a .toml file?
- Suggestion: Try removing the [logger] info (out and level as well) from toml (iotgw) file so it is not trying to write back to the read-only file system
- You said you were troubleshooting with Dell as well, were they able to provide any information?
- Suggestion: Try removing
-v /home/losant-edge-agent/data:/data
from your docker run command so Docker is not trying to write back to the read-only file system
- You said earlier that you are running Ubuntu, but the commands look as though they may be for a Windows system. I just want to confirm that it is Ubuntu
Thank you so much for your patience and diligence in finding the cause of this!
Julia
Julia,
Thanks for the update. Below you will find the answers to your questions
- I am curious, is there a reason you are using a .iotgw file over a .toml file?
[ECS] I was not under the impression that the .toml suffix was significant. I thought it might
be the name of the person preparing the video. I can certainly change that if you think
it plays a part in the issue.
- Suggestion: Try removing the [logger] info (out and level as well) from toml (iotgw) file so it is not trying to write back to the read-only file system
[ECS] The logging information was a suggestion from one of your colleagues, Taron Foxworth, as he
asked us to set “Logging Level” to “verbose”. Initially I did not have anything in there. Also, I
now have a “R/W” disk attached and this is where the log file should be getting written to. This
is also the volume that we are presenting to the container via the “-v” arguments in the docker
command. Initially we did not use a config file and only executed the CLI command to startup
the agent docker container with the same exact results. This implies that nothing in the config
file, contents, or .toml/.iotgw suffix are playing a role in this issue
- You said you were troubleshooting with Dell as well, were they able to provide any information?
[ECS] We only sent them an Email regarding the GW having a RO filesystem. They simply sent us
a user manual. We figured out on our own that we did in fact have a R/W partition in “/home”
which is where we create the “losant-edge-agent” and “losant-edge-agent/data” directories
that we present to the Losant edge agent docker container via the docker command. This
command is in a previous post for you review
- Suggestion: Try removing
-v /home/losant-edge-agent/data:/data
from your docker run command so Docker is not trying to write back to the read-only file system
[ECS] This is not an issue as the directory “/home/losant-edge-agent/data” is RO for everyone. With
this permission set Docker should not have any issues writing to this directory
- You said earlier that you are running Ubuntu, but the commands look as though they may be for a Windows system. I just want to confirm that it is Ubuntu
[ECS] Here is the data that you requested
Looks like the data got truncated for the Linux (Ubuntu) verification. Here is another try
Directory command showing the permissions for /home/losant-edge-agent and /home/losant-edge-agent directories.
As you can
see the permissions are “rwxrwxrwx (777)” for both
total 16
drwxr-xr-x 4 root root 4096 Aug 27 13:49 …
-rw-rw-r-- 1 admin admin 231 Aug 29 15:14 config.iotgw
drwxrwxrwx 2 admin admin 4096 Aug 29 15:29 data
-rw-rw-r-- 1 admin admin 0 Aug 30 16:58 data.txt
drwxrwxrwx 3 admin admin 4096 Aug 30 16:58 .
Output from “uname -a” indicating a Ubuntu OS
Linux B7Z8N42 4.4.0-159-generic #187-Ubuntu SMP Thu Aug 1 16:28:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Hello Edward,
Sorry for the back and forth while we try and figure out the issue you are experiencing. With the Ubuntu CORE OS, I think you are installing Docker with snap. If so, can you uninstall Docker and try and install it again with this command:
sudo snap install docker --devmode
Once Docker is installed, you can follow the steps that you were already doing to pull the Losant Edge Agent Docker image and start the container. Here are the links.
You will have to change your config file extension from .iotgw
to .toml
as well. Just a reminder to do this at the file and also both places in the docker run command that it is referenced.
Let us know how it goes!
Thank you,
Kevin
1 Like
Unfortunately there is no change in behavior. I issued the following commands to remove the current docker image
sudo snap stop docker
sudo snap remove docker
Then I installed per your suggestion
sudo snap install docker --devmode
Then I pulled over the latest Losant Edge Agent
sudo docker pull losant/edge-agent
I also renamed my config file from config.iotgw to config.toml
Lastly I used the following command to start the Losant Edge Agent
sudo docker run -d --restart always --name losant-agent
-v /home/losant-edge-agent/data:/data
-v /home/losant-edge-agent/config.tom;:/etc/losant/losant-edge-agent-config.toml
losant/edge-agent
Not sure if I should start a different thread for this but I am having (what seems to be) the same issue.
I’m running an Advantech UTX-3115 with a fresh install of Ubuntu Server 18.04. Docker installed with apt, not snap, for the record.
Container appears to start, but when I check the logs I get “Illegal Instruction (Core Dumped)”
I have tried with and without a data directory, same issue.
Tried a fresh install of Ubuntu Server 18.04, and this time tried installing docker with
sudo snap install docker --devmode
as suggested above.
Followed install instruction exactly, config.toml file as follows:
[logger]
out = '/data/losant-edge-agent-log.log'
level = 'verbose'
[gateway]
id = '5d6cb82################'
key = 'ec15554e-1d57-4956-################'
secret = '4725780baceabab2a7a####################################'
[store]
path = '/data/losant-edge-agent-store.db'
docker run called as follows:
sudo docker run -d --restart always --name docs-agent -v /home/nated0g/losant/data:/data -v /home/nated0g/losant/config.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent
Same issue, docker container reboots continuously, logs show only:
Illegal instruction (core dumped)
Illegal instruction (core dumped)
Illegal instruction (core dumped)
I tried an identical installation process on my Ubuntu 19.04 development machine and everything works properly and I can see the Edge Agent from Losant.
Not sure if relevant but I notice when I install docker, I get a warning:
aufs aufs_fill_super:912:mount[17219]: no arg
overlayfs: missing 'lowerdir'
@Edward_Szajner and @Nate_Usher,
I have been able to reproduce the Illegal instruction (core dumped)
issue on our Edge Agent version 1.11.0. I tested this on an Advantech UTX-3115 gateway that is running Ubuntu 16.04 (I am not using snap). The issue appears to be limited to gateways that have an Atom processor.
I am working with Engineering on this issue now and will reply back as soon as I know more. Thank you for reporting this issue and your patience while we investigate.
Thank you Edward and Nate,
Kevin
1 Like
@Edward_Szajner and @Nate_Usher,
The Illegal Instruction (core dumped)
error is limited to running the non-alpine version 1.10.0 or 1.11.0 of the Losant Edge Agent on an Atom device.
You can run the alpine version of the Losant Edge Agent. The alpine version does not come with the TensorFlow: Predict Node. The command to run the latest alpine version (1.11.0 at the time of this post) is sudo docker pull losant/edge-agent:1.11.0-alpine
. You will have to change losant/edge-agent
to losant/edge-agent:1.11.0-alpine
at the end of your docker run command.
I will update this post when we have a fix for the non-alpine version.
Thank you,
Kevin
Kevin,
That worked except it did not like passing in the “configuration” (.toml) file. I had to use the following docker command
sudo docker run -d --restart always --name losant-agent -e ‘DEVICE_ID=5d5ea5081###############’ -e ‘ACCESS_KEY=42530772-d9a7-44ca-8feb-##############’ -e ‘ACCESS_SECRET=6bf8896ce0502155959c3f92c040a12d#########################’ losant/edge-agent:1.11.0-alpine
Here is the output from the docker logs command
admin@B7Z8N42:/home/losant-edge-agent$ sudo docker logs 17e3357953c0
2019-09-03T19:45:52.949Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T19:45:54.626Z [info] Agent Starting…
2019-09-03T19:45:54.629Z [info] Agent Version: 1.11.0
2019-09-03T19:45:54.844Z [info] Webserver started on port: 8080
2019-09-03T19:45:54.853Z [info] Workflows initialized and running…
2019-09-03T19:45:54.856Z [info] Connecting to: mqtts://broker.losant.com …
2019-09-03T19:45:55.412Z [info] Connected to: mqtts://broker.losant.com
2019-09-03T19:45:55.637Z [info] Workflows received from Losant and applied.
Edward,
I am glad you got it working! Thank you for your patience as we worked through that issue.
In regards to the toml file issue: in your post from August 30th, the Docker run command that you were using was:
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.tom;:/etc/losant/losant-edge-agent-config.toml losant/edge-agent
In the Docker run command above, the l
in .toml
is missing. You had a ;
in place of the l
which might have been a typo? When you executed that Docker run command last, was the ;
still there instead of the l
? Your updated Docker run command should be:
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine
Thank you Edward,
Kevin
Kevin,
I believe it was a typo. Here is the command I used from the “history”
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine
ed
Edward,
The next thing to check would be to ensure the config.toml file is really a toml file instead of something like config.toml.iotgw since you previously renamed your file extension from .iotgw to .toml. I have tried renaming a file extension before and it actually added what I thought was the new extension to the file name instead of updating the actual file extension.
If you decide to try and run with the toml file again, please post the error(s) that you receive.
Thank you,
Kevin
Kevin,
I did try again just to see what the error initially was. Here is the current log output
2019-09-03T20:34:13.034Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:14.139Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:14.143Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:14.168Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required
2019-09-03T20:34:35.488Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:36.617Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:36.619Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:36.642Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required
2019-09-03T20:34:57.580Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:58.666Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:58.670Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:58.694Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required
Edward,
The error that you are seeing states Unable to read and parse agent config file at expected path...
means that either the toml file does not exist in the expected directory, that it is not a toml file (it has a different extension), or formatting is not correct in the file. I have listed a couple suggestions below - please try them in the order listed and continue to move on to the next suggestion if the current one does not work:
-
In your post from Aug 29, it looks like your toml file on your gateway is in this file location: C:\Users\v111405\Downloads
. Is it still in this location? This file should be located in /home/losant-edge-agent/
based off your current Docker run command.
-
Ensure that /home/losant-edge-agent/
is the absolute path to your config.toml file and that there aren’t other folders/directories before the home
folder.
-
Did you confirm that your toml file is definitely a toml file and not something like config.toml.iotgw?
-
Can you create a new toml file within the /home/losant-edge-agent/
directory (you can delete the old one and create a new config.toml file OR create a new file with a new name like config_test.toml - if you create a new file with a new name, just a reminder to update your docker run file to look like below:
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config_test.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine
Thank you Edward,
Kevin
Kevin,
Here are the answers below. Also I cut-n-pasted the content of the .toml file from your documentation page. This was the last issue you mentioned that the formatting was incorrect.
The error that you are seeing states Unable to read and parse agent config file at expected path...
means that either the toml file does not exist in the expected directory, that it is not a toml file (it has a different extension), or formatting is not correct in the file. I have listed a couple suggestions below - please try them in the order listed and continue to move on to the next suggestion if the current one does not work:
-
In your post from Aug 29, it looks like your toml file on your gateway is in this file location: C:\Users\v111405\Downloads
. Is it still in this location? This file should be located in /home/losant-edge-agent/
based off your current Docker run command.
[ECS] The path “C:\Users\v111405\Downloads” was just the path I copied the file to in order for me
to attach it to this thread. I was on my home computer but the GW is on the other side of a
VPN so I was not able to copy directly. Here is the actual location of the “config.toml” file
admin@B7Z8N42:/home/losant-edge-agent$ pwd
/home/losant-edge-agent
admin@B7Z8N42:/home/losant-edge-agent$ ls
config.toml data
-
Ensure that /home/losant-edge-agent/
is the absolute path to your config.toml file and that there aren’t other folders/directories before the home
folder.
[ECS] See “ls” and “pwd” command output above under #1. It show that the “config.toml” file is in
the “/home/losant-edge-agent/”
-
Did you confirm that your toml file is definitely a toml file and not something like config.toml.iotgw?
[ECS] I believe the file is a .toml file and not another extension. I will delete it and recreate it. My
experience with Linux that there are not hidden extensions display. What you see is what you
get.
-
Can you create a new toml file within the /home/losant-edge-agent/
directory (you can delete the old one and create a new config.toml file OR create a new file with a new name like config_test.toml - if you create a new file with a new name, just a reminder to update your docker run file to look like below:
[ECS] I will delete and re-create the .toml file as “config.toml”. Again, I will cut-n-paste the contents
from your documentation page
Kevin,
I deleted the config.toml file and re-created it from scratch and used the command that you provided in the previous post. It is now working as expected. Obviously there was something wrong with the previous version. My take is that the “content” was not correct. Thanks for the help. Please let us know when there is a “bug fix” for the “alpine” issue so we can upload and test the new image/container
ed
Edward,
Thank you for the follow up letting us know that you got it working, great work! I will be sure to update this post when we have fixed the issue that we are seeing in the non-alpine Losant Edge Agent v1.10.0 and v1.11.0 when running on an ATOM processor.
Thank you for initially reporting this bug!
Kevin
Kevin,
Do you have any status on when a new Losant Edge Agent might be available that addresses/resolves the issue of the agent crashing with the “Illegal Instruction” when running on the GW with the ATOM processor? Just curious as we want to upgrade our lab environment as soon as possible
thanks
ed