After the rotation yesterday, we have not had much luck connecting our devices.
Access keys and secret should still work, right?
Yes, Access Keys and Secrets should still work. Are you getting any error messages? Would you be able to share with me what error you are getting?
The more information you can provide would be very beneficial.
Thank you,
Heath
we are trying to get some debug data from the devices as we speak.
Is there anywhere in Losant we could see if any devices have trouble to authenticate against the MQTTbroker?
Failed authentication attempts will be available in the Application Log.
Thank you,
Heath
I’m not aware of these being rotated before. Does this happen annually ?
It looks like the error message we’re getting is that it’s failing to verify the “peer certificate” and that “certificate is not correctly signed by the CA.”
Nothing related to the failing devices show up in the application log.
Something not working as before on your side?
I’m not aware of these being rotated before. Does this happen annually ?
The last time we rotated SSL certificates was approximately 2 years ago. Now, SSL policy does not allow for multi-year certificates. So, our next SSL certificate rotation will be in approximately 1 year.
Nothing related to the failing devices show up in the application log.
Something not working as before on your side?
Everything on our side is working as intended and is operational.
No devices showing up in your Application Log indicate that your devices are not establishing a connection. Would you be able to tell me what type of devices you are experiencing this on (i.e. make and model)? How are these devices connecting to Losant?
Thank you,
Heath
Heath,
This rotation caused our entire fleet of devices to go offline and not come back. We aren’t able to push a firmware update as we use an MQTT workflow to deliver firmware URL for the update process. Is it possible for there to be a temporary rollback for us to update our SSL certificates?
Hey @Zach_Jacobs1
The underlying certificate authority (CA), which is what should be on your devices to validate the certificate, has not been changed. Outside of the CA, we’re not sure what would be on your device that would require an update. Can you please provide additional details about your environment, or any additional details into why your devices are unable to reconnect?
Hey @Lars_Andersson,
Can you please copy/paste (or screenshot) the exact error you’re seeing? I’d like to see the exact wording of the message.
An announcement 15 minutes before it happens at 10 pm, is a bit tight.
Or did we miss some other announcements?
@Brandon_Cannaday I’m digging into this a bit more right now and will get some more info shortly.
The Status Page is the best place to monitor for upcoming scheduled maintenance. We’ll make sure to post those on the forums earlier for future updates.
With this specific update, there aren’t really actions that can be taken beforehand. The certificate we updated is not something that is distributed or something that is placed on any device. The root CA, which is the DigiCert Global Root CA, is the important file and what may potentially be on devices (if your device validates the certificate). That file is unchanged.
If our root CA does change, that would have significant impacts, which will require a long heads up.
We’ve been watching our analytics closely around connected devices and we’ve not seen any identifiable drop in connections. So we’re very interested in finding out what, specifically, you’re seeing.
Brandon,
The error that I’m seeing is: The certificate is not correctly signed by the trusted CA. We do indeed have a certificate on our devices to validate the certificate against the server. What is your preferred method for obtaining that certificate? We are currently using:
openssl s_client -showcerts -connect broker.losant.com:8883 </dev/null 2>/dev/null|openssl x509 -outform PEM > losant-root-ca.pem
Ah, I see. That command downloads the actual certificate, not the root CA. You were validating the certificate against itself, which is not a common practice or secure. This would not prevent a man-in-the-middle attack serving you a compromised certificate from somewhere between you and Losant’s servers.
Certificates are validated by making sure they are signed by a trusted CA. The CA is obtained directly from the certificate authority - separately from the certificate itself. There’s not really a command to obtain a CA. They are publicly available since they are not particularly sensitive. We also provide the CA in some of our MQTT libraries.
What you’ll need to do is download the CA from one of the links above and place that on your devices. This will result in a more secure environment and protect you from future certificate updates (which will happen yearly). CA files are good for much longer - 10 or so years.
Great, thanks. I was able to use the certificate from the “MQTT libraries” link and authenticate that way. Unfortunately I’m unable to roll this change out to my fleet with an OTA update since our existing code uses the old cert that is now expired. Is there a workaround for us to temporarily use the old certificate so we can update our devices that are offline?
The only way to accomplish that would be for us to roll back the certificate. While not impossible, it would be a risk we would prefer to avoid. Some, particularly strict, systems may not accept an older certificate once they’ve seen a newer one. So while it may allow your devices to connect, it may also have unforeseen consequences for other customers. This is the same certificate used for the Losant API, which facilitates a large number of system-to-system integrations.
@Brandon_Cannaday – Hello, Did the intermediate certificate also change? Earlier this week, the command mentioned in my previous post: openssl s_client -showcerts -connect broker.losant.com:8883 </dev/null 2>/dev/null|openssl x509 -outform PEM > losant-root-ca.pem gave me the intermediate certificate as shown here:
-----BEGIN CERTIFICATE-----
MIIElDCCA3ygAwIBAgIQAf2j627KdciIQ4tyS8+8kTANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
QTAeFw0xMzAzMDgxMjAwMDBaFw0yMzAzMDgxMjAwMDBaME0xCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxJzAlBgNVBAMTHkRpZ2lDZXJ0IFNIQTIg
U2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ANyuWJBNwcQwFZA1W248ghX1LFy949v/cUP6ZCWA1O4Yok3wZtAKc24RmDYXZK83
nf36QYSvx6+M/hpzTc8zl5CilodTgyu5pnVILR1WN3vaMTIa16yrBvSqXUu3R0bd
KpPDkC55gIDvEwRqFDu1m5K+wgdlTvza/P96rtxcflUxDOg5B6TXvi/TC2rSsd9f
/ld0Uzs1gN2ujkSYs58O09rg1/RrKatEp0tYhG2SS4HD2nOLEpdIkARFdRrdNzGX
kujNVA075ME/OV4uuPNcfhCOhkEAjUVmR7ChZc6gqikJTvOX6+guqw9ypzAO+sf0
/RR3w6RbKFfCs/mC/bdFWJsCAwEAAaOCAVowggFWMBIGA1UdEwEB/wQIMAYBAf8C
AQAwDgYDVR0PAQH/BAQDAgGGMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYY
aHR0cDovL29jc3AuZGlnaWNlcnQuY29tMHsGA1UdHwR0MHIwN6A1oDOGMWh0dHA6
Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RDQS5jcmwwN6A1
oDOGMWh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RD
QS5jcmwwPQYDVR0gBDYwNDAyBgRVHSAAMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8v
d3d3LmRpZ2ljZXJ0LmNvbS9DUFMwHQYDVR0OBBYEFA+AYRyCMWHVLyjnjUY4tCzh
xtniMB8GA1UdIwQYMBaAFAPeUDVW0Uy7ZvCj4hsbw5eyPdFVMA0GCSqGSIb3DQEB
CwUAA4IBAQAjPt9L0jFCpbZ+QlwaRMxp0Wi0XUvgBCFsS+JtzLHgl4+mUwnNqipl
5TlPHoOlblyYoiQm5vuh7ZPHLgLGTUq/sELfeNqzqPlt/yGFUzZgTHbO7Djc1lGA
8MXW5dRNJ2Srm8c+cftIl7gzbckTB+6WohsYFfZcTEDts8Ls/3HB40f/1LkAtDdC
2iDJ6m6K7hQGrn2iWZiIqBtvLfTyyRRfJs8sjX7tN8Cp1Tm5gr8ZDOo0rwAhaPit
c+LJMto4JQtV05od8GiG7S5BNO98pVAdvzr508EIDObtHopYJeS4d60tbvVS3bR0
j6tJLp07kzQoH3jOlOrHvdPJbRzeXDLz
-----END CERTIFICATE-----
This certificate is still showing as valid until 2023: Certificate Decoder - Decode certificates to view their contents
Certificate Information:
Common Name: DigiCert SHA2 Secure Server CA
Organization: DigiCert Inc
Country: US
Valid From: March 8, 2013
Valid To: March 8, 2023
Issuer: DigiCert Global Root CA, DigiCert Inc Write review of DigiCert
Serial Number: 01fda3eb6eca75c888438b724bcfbc91
This is what we have previously used to authenticate with and would not have any reason to expect this would expire (or not be usable) before the 2023 date.
When I currently run the SSL command mentioned, I do in fact now receive a client certificate which I am not even able to use to authenticate with the server.
Yes, the intermediate certificates did change.
Typically, there is no reason the download these as part of any validation process. As part of the TLS negotiation that’s performed on every connection, the entire certificate chain is sent to the client from the server. The only file that’s required, and the one that’s also not sent by the server, is the root CA.