Illegal Instruction

Here is some additional data

Here is the “Config” file that I used

C:\Users\v111405\Downloads>type config.iotgw
[logger]
out = "/data/losant-edge-agent-log.log’
level = ‘verbose’

[gateway]
id = ‘5d5ea5081a74840006eabffc’
key = ‘42530772-d9a7-44ca-8feb-fd87dXXXXXX’
secret = ‘6bf8896ce0502155959c3f92c040a12dc0becXXXXXXXXXXXX’

Here is the actual Docker command I used to start the “Losant Agent” container

C:\Users\v111405\Downloads>type dockerCommand
sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.iotgw:/etc/losant/losant-edge-agent-config.iotgw losant/edge-agent

ed

Hi @Edward_Szajner,

I apologize for the delay, this is not an issue we have run into before, so we have been discussing the possible causes. I received a few suggestions (and have a couple of questions as well) so we can get further into debugging! I apologize if any of these questions and suggestions are redundant to your own debugging :smile:

  1. I am curious, is there a reason you are using a .iotgw file over a .toml file?
  2. Suggestion: Try removing the [logger] info (out and level as well) from toml (iotgw) file so it is not trying to write back to the read-only file system
  3. You said you were troubleshooting with Dell as well, were they able to provide any information?
  4. Suggestion: Try removing -v /home/losant-edge-agent/data:/data from your docker run command so Docker is not trying to write back to the read-only file system
  5. You said earlier that you are running Ubuntu, but the commands look as though they may be for a Windows system. I just want to confirm that it is Ubuntu :smile:

Thank you so much for your patience and diligence in finding the cause of this! :smiley:
Julia

Julia,

Thanks for the update. Below you will find the answers to your questions

  1. I am curious, is there a reason you are using a .iotgw file over a .toml file?
    [ECS] I was not under the impression that the .toml suffix was significant. I thought it might

be the name of the person preparing the video. I can certainly change that if you think

it plays a part in the issue.

  1. Suggestion: Try removing the [logger] info (out and level as well) from toml (iotgw) file so it is not trying to write back to the read-only file system
    [ECS] The logging information was a suggestion from one of your colleagues, Taron Foxworth, as he

asked us to set “Logging Level” to “verbose”. Initially I did not have anything in there. Also, I

now have a “R/W” disk attached and this is where the log file should be getting written to. This

is also the volume that we are presenting to the container via the “-v” arguments in the docker

command. Initially we did not use a config file and only executed the CLI command to startup

the agent docker container with the same exact results. This implies that nothing in the config

file, contents, or .toml/.iotgw suffix are playing a role in this issue

  1. You said you were troubleshooting with Dell as well, were they able to provide any information?
    [ECS] We only sent them an Email regarding the GW having a RO filesystem. They simply sent us

a user manual. We figured out on our own that we did in fact have a R/W partition in “/home”

which is where we create the “losant-edge-agent” and “losant-edge-agent/data” directories

that we present to the Losant edge agent docker container via the docker command. This

command is in a previous post for you review

  1. Suggestion: Try removing -v /home/losant-edge-agent/data:/data from your docker run command so Docker is not trying to write back to the read-only file system
    [ECS] This is not an issue as the directory “/home/losant-edge-agent/data” is RO for everyone. With

this permission set Docker should not have any issues writing to this directory

  1. You said earlier that you are running Ubuntu, but the commands look as though they may be for a Windows system. I just want to confirm that it is Ubuntu
    [ECS] Here is the data that you requested

Looks like the data got truncated for the Linux (Ubuntu) verification. Here is another try

Directory command showing the permissions for /home/losant-edge-agent and /home/losant-edge-agent directories.

As you can

see the permissions are “rwxrwxrwx (777)” for both

total 16

drwxr-xr-x 4 root root 4096 Aug 27 13:49 …

-rw-rw-r-- 1 admin admin 231 Aug 29 15:14 config.iotgw

drwxrwxrwx 2 admin admin 4096 Aug 29 15:29 data

-rw-rw-r-- 1 admin admin 0 Aug 30 16:58 data.txt

drwxrwxrwx 3 admin admin 4096 Aug 30 16:58 .

Output from “uname -a” indicating a Ubuntu OS

Linux B7Z8N42 4.4.0-159-generic #187-Ubuntu SMP Thu Aug 1 16:28:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hello Edward,

Sorry for the back and forth while we try and figure out the issue you are experiencing. With the Ubuntu CORE OS, I think you are installing Docker with snap. If so, can you uninstall Docker and try and install it again with this command:

sudo snap install docker --devmode

Once Docker is installed, you can follow the steps that you were already doing to pull the Losant Edge Agent Docker image and start the container. Here are the links.

You will have to change your config file extension from .iotgw to .toml as well. Just a reminder to do this at the file and also both places in the docker run command that it is referenced.

Let us know how it goes!

Thank you,
Kevin

1 Like

Unfortunately there is no change in behavior. I issued the following commands to remove the current docker image

sudo snap stop docker
sudo snap remove docker

Then I installed per your suggestion

sudo snap install docker --devmode

Then I pulled over the latest Losant Edge Agent

sudo docker pull losant/edge-agent

I also renamed my config file from config.iotgw to config.toml

Lastly I used the following command to start the Losant Edge Agent

sudo docker run -d --restart always --name losant-agent
-v /home/losant-edge-agent/data:/data
-v /home/losant-edge-agent/config.tom;:/etc/losant/losant-edge-agent-config.toml
losant/edge-agent

Not sure if I should start a different thread for this but I am having (what seems to be) the same issue.

I’m running an Advantech UTX-3115 with a fresh install of Ubuntu Server 18.04. Docker installed with apt, not snap, for the record.

Container appears to start, but when I check the logs I get “Illegal Instruction (Core Dumped)”

I have tried with and without a data directory, same issue.

Tried a fresh install of Ubuntu Server 18.04, and this time tried installing docker with

sudo snap install docker --devmode

as suggested above.

Followed install instruction exactly, config.toml file as follows:

[logger]
out = '/data/losant-edge-agent-log.log'
level = 'verbose'

[gateway]
id = '5d6cb82################'
key = 'ec15554e-1d57-4956-################'
secret = '4725780baceabab2a7a####################################'

[store]
path = '/data/losant-edge-agent-store.db'

docker run called as follows:

sudo docker run -d --restart always --name docs-agent   -v /home/nated0g/losant/data:/data   -v /home/nated0g/losant/config.toml:/etc/losant/losant-edge-agent-config.toml   losant/edge-agent

Same issue, docker container reboots continuously, logs show only:

Illegal instruction (core dumped)
Illegal instruction (core dumped)
Illegal instruction (core dumped)

I tried an identical installation process on my Ubuntu 19.04 development machine and everything works properly and I can see the Edge Agent from Losant.

Not sure if relevant but I notice when I install docker, I get a warning:

aufs aufs_fill_super:912:mount[17219]: no arg
overlayfs: missing 'lowerdir'

@Edward_Szajner and @Nate_Usher,

I have been able to reproduce the Illegal instruction (core dumped) issue on our Edge Agent version 1.11.0. I tested this on an Advantech UTX-3115 gateway that is running Ubuntu 16.04 (I am not using snap). The issue appears to be limited to gateways that have an Atom processor.

I am working with Engineering on this issue now and will reply back as soon as I know more. Thank you for reporting this issue and your patience while we investigate.

Thank you Edward and Nate,
Kevin

1 Like

@Edward_Szajner and @Nate_Usher,

The Illegal Instruction (core dumped) error is limited to running the non-alpine version 1.10.0 or 1.11.0 of the Losant Edge Agent on an Atom device.

You can run the alpine version of the Losant Edge Agent. The alpine version does not come with the TensorFlow: Predict Node. The command to run the latest alpine version (1.11.0 at the time of this post) is sudo docker pull losant/edge-agent:1.11.0-alpine. You will have to change losant/edge-agent to losant/edge-agent:1.11.0-alpine at the end of your docker run command.

I will update this post when we have a fix for the non-alpine version.

Thank you,
Kevin

Kevin,

That worked except it did not like passing in the “configuration” (.toml) file. I had to use the following docker command

sudo docker run -d --restart always --name losant-agent -e ‘DEVICE_ID=5d5ea5081###############’ -e ‘ACCESS_KEY=42530772-d9a7-44ca-8feb-##############’ -e ‘ACCESS_SECRET=6bf8896ce0502155959c3f92c040a12d#########################’ losant/edge-agent:1.11.0-alpine

Here is the output from the docker logs command

admin@B7Z8N42:/home/losant-edge-agent$ sudo docker logs 17e3357953c0
2019-09-03T19:45:52.949Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T19:45:54.626Z [info] Agent Starting…
2019-09-03T19:45:54.629Z [info] Agent Version: 1.11.0
2019-09-03T19:45:54.844Z [info] Webserver started on port: 8080
2019-09-03T19:45:54.853Z [info] Workflows initialized and running…
2019-09-03T19:45:54.856Z [info] Connecting to: mqtts://broker.losant.com …
2019-09-03T19:45:55.412Z [info] Connected to: mqtts://broker.losant.com
2019-09-03T19:45:55.637Z [info] Workflows received from Losant and applied.

Edward,

I am glad you got it working! Thank you for your patience as we worked through that issue.

In regards to the toml file issue: in your post from August 30th, the Docker run command that you were using was:

sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.tom;:/etc/losant/losant-edge-agent-config.toml losant/edge-agent

In the Docker run command above, the l in .toml is missing. You had a ; in place of the l which might have been a typo? When you executed that Docker run command last, was the ; still there instead of the l? Your updated Docker run command should be:

sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine

Thank you Edward,
Kevin

Kevin,

I believe it was a typo. Here is the command I used from the “history”

sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine

ed

Edward,

The next thing to check would be to ensure the config.toml file is really a toml file instead of something like config.toml.iotgw since you previously renamed your file extension from .iotgw to .toml. I have tried renaming a file extension before and it actually added what I thought was the new extension to the file name instead of updating the actual file extension.

If you decide to try and run with the toml file again, please post the error(s) that you receive.

Thank you,
Kevin

Kevin,

I did try again just to see what the error initially was. Here is the current log output

2019-09-03T20:34:13.034Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:14.139Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:14.143Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:14.168Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required
2019-09-03T20:34:35.488Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:36.617Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:36.619Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:36.642Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required
2019-09-03T20:34:57.580Z [warn] This version of the edge agent does not support the TensorFlow Predict Node.
2019-09-03T20:34:58.666Z [warn] Unable to read and parse agent config file at expected path: /etc/losant/losant-edge-agent-config.toml
2019-09-03T20:34:58.670Z [warn] Attempting startup using configuration from only environment variables and command line flags.
2019-09-03T20:34:58.694Z [error] Losant Edge Agent was unable to start due to the following configuration errors:
Device Id is required

Edward,

The error that you are seeing states Unable to read and parse agent config file at expected path... means that either the toml file does not exist in the expected directory, that it is not a toml file (it has a different extension), or formatting is not correct in the file. I have listed a couple suggestions below - please try them in the order listed and continue to move on to the next suggestion if the current one does not work:

  1. In your post from Aug 29, it looks like your toml file on your gateway is in this file location: C:\Users\v111405\Downloads. Is it still in this location? This file should be located in /home/losant-edge-agent/ based off your current Docker run command.

  2. Ensure that /home/losant-edge-agent/ is the absolute path to your config.toml file and that there aren’t other folders/directories before the home folder.

  3. Did you confirm that your toml file is definitely a toml file and not something like config.toml.iotgw?

  4. Can you create a new toml file within the /home/losant-edge-agent/ directory (you can delete the old one and create a new config.toml file OR create a new file with a new name like config_test.toml - if you create a new file with a new name, just a reminder to update your docker run file to look like below:

sudo docker run -d --restart always --name losant-agent -v /home/losant-edge-agent/data:/data -v /home/losant-edge-agent/config_test.toml:/etc/losant/losant-edge-agent-config.toml losant/edge-agent:1.11.0-alpine

Thank you Edward,
Kevin

Kevin,

Here are the answers below. Also I cut-n-pasted the content of the .toml file from your documentation page. This was the last issue you mentioned that the formatting was incorrect.

The error that you are seeing states Unable to read and parse agent config file at expected path... means that either the toml file does not exist in the expected directory, that it is not a toml file (it has a different extension), or formatting is not correct in the file. I have listed a couple suggestions below - please try them in the order listed and continue to move on to the next suggestion if the current one does not work:

  1. In your post from Aug 29, it looks like your toml file on your gateway is in this file location: C:\Users\v111405\Downloads . Is it still in this location? This file should be located in /home/losant-edge-agent/ based off your current Docker run command.
    [ECS] The path “C:\Users\v111405\Downloads” was just the path I copied the file to in order for me
    to attach it to this thread. I was on my home computer but the GW is on the other side of a
    VPN so I was not able to copy directly. Here is the actual location of the “config.toml” file

        admin@B7Z8N42:/home/losant-edge-agent$ pwd
            /home/losant-edge-agent
    
        admin@B7Z8N42:/home/losant-edge-agent$ ls
            config.toml  data
    
  2. Ensure that /home/losant-edge-agent/ is the absolute path to your config.toml file and that there aren’t other folders/directories before the home folder.
    [ECS] See “ls” and “pwd” command output above under #1. It show that the “config.toml” file is in
    the “/home/losant-edge-agent/”

  3. Did you confirm that your toml file is definitely a toml file and not something like config.toml.iotgw?
    [ECS] I believe the file is a .toml file and not another extension. I will delete it and recreate it. My
    experience with Linux that there are not hidden extensions display. What you see is what you
    get.

  4. Can you create a new toml file within the /home/losant-edge-agent/ directory (you can delete the old one and create a new config.toml file OR create a new file with a new name like config_test.toml - if you create a new file with a new name, just a reminder to update your docker run file to look like below:
    [ECS] I will delete and re-create the .toml file as “config.toml”. Again, I will cut-n-paste the contents
    from your documentation page

Kevin,

I deleted the config.toml file and re-created it from scratch and used the command that you provided in the previous post. It is now working as expected. Obviously there was something wrong with the previous version. My take is that the “content” was not correct. Thanks for the help. Please let us know when there is a “bug fix” for the “alpine” issue so we can upload and test the new image/container

ed

Edward,

Thank you for the follow up letting us know that you got it working, great work! I will be sure to update this post when we have fixed the issue that we are seeing in the non-alpine Losant Edge Agent v1.10.0 and v1.11.0 when running on an ATOM processor.

Thank you for initially reporting this bug!
Kevin

Kevin,

Do you have any status on when a new Losant Edge Agent might be available that addresses/resolves the issue of the agent crashing with the “Illegal Instruction” when running on the GW with the ATOM processor? Just curious as we want to upgrade our lab environment as soon as possible

thanks

ed