The ESP 32 from Espressif is a nifty device that supports both WiFi and Bluetooth LE communication. This short note explains how to transmit to and receive data from AWS IoT on WiFi using MQTT protocol on an ESP32 device.

AWS IoT is a service that is being supported by more and more hardware vendors. It offers a comprehensive set of services to be able to manage devices, and send and receive data at scale. It integrates well into the rest of the ever growing AWS ecosystem to process, analyze, and visualize data. It takes some time getting used to all the nomenclature, but it has good documentation and support.

ESP 32 from Espressif is programmable straight from the Arduino IDE and a growing number libraries are being added to it everyday. With 2 cores, and both WiFi and BLE, it is a nifty SoC to use for IoT. The custom board we use also has a 3G cellular connectivity.

Here we will use the ESP32 to send and receive messages over MQTT from AWS IoT. MQTT is a lightweight Publish/Subscribe protocol that has been growing in popularity in the IoT/M2M area. AWS Iot supports it well with the ability to securely transmit bidirectional messages using PKI. AWS has released an SDK for embedded devices in C, and there a couple of implementations available, but we ran into issues building and using them.

Here we use a much simpler method using the WiFiClientSecure and PubSubClient libraries that can be easily added to the Arduino IDE.

AWS Prerequisites

There are a number of steps to be carried out on AWS IoT front that are not detailed here. There is plenty of documentation on the AWS IoT site. We assume that the following steps are done/understood:

  • Registration on AWS IoT Console
  • Creation/registration of a "Thing" in the console
  • Creation of a X.509 Certificate
  • Creation of a Policy to access AWS IoT topics
  • Attaching the Policy to the Certificate
  • Comfortable using the AWS Iot Console to test MQTT

AWS IoT Certificates

During the creation of a Certificate to use with the Thing, it is important to note that the public/private keys and the certificate are only available for download at the time of creation. If not downloaded then, there is no option but to discard them and use new ones. So ensure that all the keys and certificates are downloaded and stored in a safe location. The AWS Root Certificate is also required and should be downloaded along with the device certificate and key files. AWS IoT Thing Certificates

Another key step on the AWS IoT side is the security policy that has to be attached to the certificate. If the right permissions are not granted, it will make it difficult to access certain topics or perform certain actions. We've set the policy at its most permissive level as below. Once this works correctly, the settings can be changed to restrict certain actions and topics as necessary.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Prerequisites to Build

The two main libraries used are:

An older version on WiFiClientSecure library for ESP32 does not support Root CA functions. If you received a WiFiClientSecure::setCACert() not found error, ensure that you have the updated library.

The MQTT protocol used to connect to AWS Iot requires SSL and hence we need the WiFiClientSecure library over the regular WiFi library. With the device ("Thing") certificate, private key, and the AWS root CA certificate, it is possible to connect to AWS IoT.

There are a number of MQTT libraries available for Ardunino and a couple for ESP32. After searching a bit and failing with others, we finally settled with the generic PubSubClient as it offers a simple interface and builds without issues.

Use the Arduino IDE Library Manager and ensure that the following libraries are installed SPI, WiFi, WiFiClientSecure, and PubSubClient. Of course the ESP32 Dev Module framework has to be installed (and selected) to build for the ESP32.

Once the libraries are installed, there is a small change to make to the PubSubClient.h file. Find the location of PubSubClient.h on your platform. Inside you will see the definition

#define MQTT_MAX_PACKET_SIZE 128

This caused some confusion at first (when we received transmission errors), because the maximum packet size to publish information on the MQTT "shadow" topics on AWS IoT such as shadow/update/delta, shadow/get/accepted, etc. is 128 KB.

Then referring to the README of the library revealed that:

The maximum message size, including header, is 128 bytes by default. This is configurable via MQTT_MAX_PACKET_SIZE in PubSubClient.h.

With the confusion regarding the units cleared up, modify the PubSubClient.h and increase the packet size to 1024 bytes.

#define MQTT_MAX_PACKET_SIZE 1024

Implementation

There are a few examples available with the libraries, the overall flow of the Sketch is as below:

  • WiFi setup and connection
  • Set the certificates and keys in WiFiClientSecure
    • WiFiClientSecure::setCACert()
    • WiFiClientSecure::setCertificate()
  • Initialize MQTT client to connect to AWS IoT MQTT broker (gateway)
    • port is 8883 (secure connection)
    • AWS IoT endpoint is of the form xxxxxxxxxx.iot.us-east-1.amazonaws.com (confirm your endpoint in the AWS IoT Console)
  • Pub & Sub

In the code below, you will need to set the following variables:

  • WiFi connection information (SSID, PWD)
  • AWS Root Certificate, Device certificate and private key
  • Device name
  • AWS Iot Endpoint

You can also set some other topic names to publish and subscribe to. Here goes:

#include <WiFiClient.h>
#include <WiFiClientSecure.h>
#include <PubSubClient.h>

void connectAWSIoT();
void mqttCallback (char* topic, byte* payload, unsigned int length);

char *ssid = "<YOUR_SSID>";
char *password = "<YOUR_WIFI_PASSWORD>";

const char *endpoint = "<AWS_IOT_ENDPOINT>";
// Example: xxxxxxxxxxxxxx.iot.ap-northeast-1.amazonaws.com
const int port = 8883;
char *pubTopic = "$aws/things/<DEVICE_NAME>/shadow/update";
char *subTopic = "$aws/things/<DEVICE_NAME>/shadow/update/delta";

const char* rootCA = "-----BEGIN CERTIFICATE-----\n" \
"......" \
"-----END CERTIFICATE-----\n";

const char* certificate = "-----BEGIN CERTIFICATE-----\n" \
"......" \
"-----END CERTIFICATE-----\n";

const char* privateKey = "-----BEGIN RSA PRIVATE KEY-----\n" \
"......" \
"-----END RSA PRIVATE KEY-----\n";

WiFiClientSecure httpsClient;
PubSubClient mqttClient(httpsClient);

void setup() {
    delay(1000);
    Serial.begin(115200);

    // Start WiFi
    Serial.println("Connecting to ");
    Serial.print(ssid);
    WiFi.begin(ssid, password);

    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("\nConnected.");

    // Configure MQTT Client
    httpsClient.setCACert(rootCA);
    httpsClient.setCertificate(certificate);
    httpsClient.setPrivateKey(privateKey);
    mqttClient.setServer(endpoint, port);
    mqttClient.setCallback(mqttCallback);

    connectAWSIoT();
}

void connectAWSIoT() {
    while (!mqttClient.connected()) {
        if (mqttClient.connect("ESP32_device")) {
            Serial.println("Connected.");
            int qos = 0;
            mqttClient.subscribe(subTopic, qos);
            Serial.println("Subscribed.");
        } else {
            Serial.print("Failed. Error state=");
            Serial.print(mqttClient.state());
            // Wait 5 seconds before retrying
            delay(5000);
        }
    }
}

long messageSentAt = 0;
int dummyValue = 0;
char pubMessage[128];

void mqttCallback (char* topic, byte* payload, unsigned int length) {
    Serial.print("Received. topic=");
    Serial.println(topic);
    for (int i = 0; i < length; i++) {
        Serial.print((char)payload[i]);
    }
    Serial.print("\n");
}

void mqttLoop() {
    if (!mqttClient.connected()) {
        connectAWSIoT();
    }
    mqttClient.loop();

    long now = millis();
    if (now - messageSentAt > 5000) {
        messageSentAt = now;
        sprintf(pubMessage, "{\"state\": {\"desired\":{\"foo\":\"%d\"}}}", dummyValue++);
        Serial.print("Publishing message to topic ");
        Serial.println(pubTopic);
        Serial.println(pubMessage);
        mqttClient.publish(pubTopic, pubMessage);
        Serial.println("Published.");
    }
}

void loop() {
  mqttLoop();
}

Caveat

The format of the certificates and private key must be as follows. Each line should be properly terminated with a \n and surrounded by " ". Just copy-pasting the data from downloaded files will result in errors. For example, the AWS root certificate is set as follows:

const char* rootCA = \
"-----BEGIN CERTIFICATE-----\n"
"MIIE0zCCA7ugAwIBAgIQGNrRniZ96LtKIVjNzGs7SjANBgkqhkiG9w0BAQUFADCB\n"
"yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL\n"
"ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp\n"
"U2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW\n"
"ZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0\n"
"aG9yaXR5IC0gRzUwHhcNMDYxMTA4MDAwMDAwWhcNMzYwNzE2MjM1OTU5WjCByjEL\n"
"MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW\n"
"ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2ln\n"
"biwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJp\n"
"U2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9y\n"
"aXR5IC0gRzUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1\n"
"nmAMqudLO07cfLw8RRy7K+D+KQL5VwijZIUVJ/XxrcgxiV0i6CqqpkKzj/i5Vbex\n"
"t0uz/o9+B1fs70PbZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6/WhkcIz\n"
"SdhDY2pSS9KP6HBRTdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQG\n"
"BO+QueQA5N06tRn/Arr0PO7gi+s3i+z016zy9vA9r911kTMZHRxAy3QkGSGT2RT+\n"
"rCpSx4/VBEnkjWNHiDxpg8v+R70rfk/Fla4OndTRQ8Bnc+MUCH7lP59zuDMKz10/\n"
"NIeWiu5T6CUVAgMBAAGjgbIwga8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E\n"
"BAMCAQYwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAH\n"
"BgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVy\n"
"aXNpZ24uY29tL3ZzbG9nby5naWYwHQYDVR0OBBYEFH/TZafC3ey78DAJ80M5+gKv\n"
"MzEzMA0GCSqGSIb3DQEBBQUAA4IBAQCTJEowX2LP2BqYLz3q3JktvXf2pXkiOOzE\n"
"p6B4Eq1iDkVwZMXnl2YtmAl+X6/WzChl8gGqCBpH3vn5fJJaCGkgDdk+bW48DW7Y\n"
"5gaRQBi5+MHt39tBquCWIMnNZBU4gcmU7qKEKQsTb47bDN0lAtukixlE0kF6BWlK\n"
"WE9gyn6CagsCqiUXObXbf+eEZSqVir2G3l6BFoMtEMze/aiCKm0oHw0LxOXnGiYZ\n"
"4fQRbxC1lfznQgUy286dUV4otp6F01vvpX1FQHKOtw5rDgb7MzVIcbidJ4vEZV8N\n"
"hnacRHr2lVz2XTIIM6RUthg/aFzyQkqFOFSDX9HoLPKsEdao7WNq\n"
"-----END CERTIFICATE-----\n";

Some Explanation on the Code

WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
}

This is the WiFi connection setup. The SSL parameters (certificates and private key) are set in the WiFiClientSecure object. Currently these are input directly into the code, but should usually be stored in an EEPROM for security. As mentioned above, ensure that each line is properly terminated by newline characters.

httpsClient.setCACert(rootCA);
httpsClient.setCertificate(certificate);
httpsClient.setPrivateKey(privateKey);

Next is the intialization of the MQTT client. The host is the AWS IoT Endpoint and the port is 8883 for secure connections.

mqttClient.setServer(endpoint, port);

Once the client is initialized, we set the callback function, which forms the heart of the MQTT processing. The callback function is triggered when data is received on the subcribed topic(s).

mqttClient.setServer(endpoint, port);

At the moment the callback function is really simple, it just print's out the received data to the serial console.

void mqttCallback (char* topic, byte* payload, unsigned int length) {
    Serial.print("Received. topic=");
    Serial.println(topic);
    for (int i = 0; i < length; i++) {
        Serial.print((char)payload[i]);
    }
    Serial.print("\n");
}

The actual connection to the AWS IoT service is implemented as below:

void connectAWSIoT() {
    while (!mqttClient.connected()) {
        if (mqttClient.connect("ESP32_device")) {
            Serial.println("Connected.");
            int qos = 0;
            mqttClient.subscribe(subTopic, qos);
            Serial.println("Subscribed.");
        } else {
            Serial.print("Failed. Error state=");
            Serial.print(mqttClient.state());
            // Wait 5 seconds before retrying
            delay(5000);
        }
    }
}

The PubSubClient::connect method is used to connect to the MQTT broker. A Client ID is passed as an argument to the method. We don't think this has to be unique, but need to refer to the documentation. Once connected, the subscribe method is used to listen to the particular topic. A message received on the topic will trigger the callback function set above using the PubSubClient::setCallback method.

Once the connection is established, it is a simply a question of calling the PubSubClient::loop method.

mqttClient.loop();

Publishing to a topic is straightforward:

mqttClient.publish(pubTopic, pubMessage);

Currently we increment a counter every 5 seconds and print the value to a JSON formatted string. The string is published to the $aws/things/<Your_device_name>/shadow/update topic, essentially updating the state of the Thing Shadow. Since the subscribed topic is $aws/things/<Your_device_name>/shadow/update/delta, essentially we can read back the updated state that our published message has changed.

This can be further tested by stopping the publishing ever 5 seconds. Comment out the mqttClient.publish(pubTopic, pubMessage); line, and from the AWS IoT Console, manually publish a message to the $aws/things/<Your_device_name>/shadow/update topic. The delta (difference) show up on the subscribed topic on the device.

Conclusion

As can be seen, it is fairly straightforward to implement MQTT communication between ESP32 and AWS IoT. As there are a number of steps preparation steps involved on the AWS side, make sure that those are correctly done and the corresponding values input into the code.

There could be several common errors encountered over this simple implementation. Troubleshooting those will be a topic for another post.

Blog Comments powered by Disqus.

Next Post Previous Post