HTTPS API Data Fetching (Node-RED)

HTTPS API Data Fetching (Node-RED)

Overview

This section covers fetching weather data from REST APIs using Node-RED. The e-paper display shows weather data, which is first fetched from an API, processed in Node-RED, and then sent via MQTT to the ESP32. After completing this section, you will be able to:

• Configure the HTTP Request node in Node-RED
• Fetch data from public REST APIs (OpenWeatherMap)
• Handle API responses and error cases
• Prepare data for MQTT transmission to ESP32

Prerequisites

Before starting this section, please ensure:

• Node-RED is installed and running (see Chapter 09)
• MQTT Broker (Mosquitto) is running
• Basic understanding of REST APIs
• OpenWeatherMap API key (free registration at openweathermap.org)

Key Concepts

Architecture Overview

[Weather API] ──HTTPS──→ [Node-RED] ──MQTT──→ [ESP32] ──SPI──→ [E-Ink Display]
                             │
                             ↓
                      [Debug/Log]

Node-RED acts as the middleware that:

1. Periodically fetches weather data from the API
2. Parses the JSON response
3. Extracts relevant values (temperature, humidity, forecast)
4. Publishes the processed data via MQTT to the ESP32

Why Node-RED Instead of ESP32 Direct?

Consideration	ESP32 Direct	Node-RED Middleware
API key security	Exposed in firmware	Stored on server
Data processing	Limited memory	Full JavaScript
Multiple displays	Each fetches separately	One fetch, many subscribers
Error handling	Complex with delays	Easy with catch nodes
API rate limits	Hard to manage	Centralized control
Firmware updates	Needed if API changes	No firmware changes

Implementation Steps

Step 1: Register for OpenWeatherMap API

1. Go to openweathermap.org
2. Click Sign In → Create an Account
3. After registration, go to API Keys tab
4. Copy your default API key (or create a new one)

Important: New API keys may take 1-2 hours to activate. Test your key in a browser before using it in Node-RED.

Step 2: Get Location Coordinates

Find latitude and longitude for your location:

# Option 1: Use latlong.net
https://www.latlong.net/

# Option 2: Use OpenWeatherMap's geocoding API
# https://api.openweathermap.org/geo/1.0/direct?q={city}&limit=1&appid={API_KEY}

Example: London, UK → lat: 51.5074, lon: -0.1278

Step 3: Build the Node-RED Flow

Create a new flow in Node-RED with the following nodes:

Flow Diagram:

[Inject (timestamp)] ─→ [HTTP Request] ─→ [JSON Parse] ─→ [Function] ─→ [MQTT Out]
                                             │
                                             ↓
                                        [Debug]

Node 1: Inject (Trigger)

Create an Inject node to trigger the flow:

• Name: Trigger every 30 min
• Payload: timestamp
• Repeat: interval, every 30 minutes

Node 2: HTTP Request

Add an HTTP Request node to fetch weather data:

• Method: GET
• URL:

https://api.openweathermap.org/data/2.5/weather?lat=51.5074&lon=-0.1278&units=metric&appid=YOUR_API_KEY

• Name: Fetch Weather Data
• Return: A parsed JSON object

Parameters Explanation:

• lat: Latitude
• lon: Longitude
• units=metric: Return temperature in Celsius
• appid: Your API key

Node 3: JSON Parse

Add a JSON node to parse the response:

• Name: Parse JSON
• Action: Always convert to JSON object

Node 4: Function (Data Transformation)

Add a Function node to extract and format the data:

// Extract relevant data from OpenWeatherMap response
const weatherData = {
    temperature: msg.payload.main.temp.toFixed(1),
    feels_like: msg.payload.main.feels_like.toFixed(1),
    humidity: msg.payload.main.humidity,
    pressure: msg.payload.main.pressure,
    description: msg.payload.weather[0].description,
    icon: msg.payload.weather[0].icon,
    city: msg.payload.name,
    timestamp: new Date().toISOString()
};

// Set output as JSON string for MQTT
msg.payload = JSON.stringify(weatherData);

return msg;

Name: Format Weather Data

Node 5: MQTT Out

Add an MQTT Out node to publish the formatted data:

• Server: Select your Mosquitto Broker
• Topic: factory/weather/data
• QoS: 1
• Name: Publish Weather

Step 4: Complete Flow JSON (Import Ready)

Copy this JSON and import it into Node-RED:

[
    {
        "id": "trigger",
        "type": "inject",
        "name": "Trigger every 30 min",
        "repeat": "1800",
        "crontab": "",
        "once": true,
        "payload": "",
        "payloadType": "date",
        "wires": [["http_req"]]
    },
    {
        "id": "http_req",
        "type": "http request",
        "name": "Fetch Weather Data",
        "method": "GET",
        "url": "https://api.openweathermap.org/data/2.5/weather?lat=51.5074&lon=-0.1278&units=metric&appid=YOUR_API_KEY",
        "ret": "obj",
        "tls": "",
        "wires": [["json_parse"]]
    },
    {
        "id": "json_parse",
        "type": "json",
        "name": "Parse JSON",
        "action": "",
        "property": "payload",
        "wires": [["format_data"]]
    },
    {
        "id": "format_data",
        "type": "function",
        "name": "Format Weather Data",
        "func": "const weatherData = {\n    temperature: msg.payload.main.temp.toFixed(1),\n    feels_like: msg.payload.main.feels_like.toFixed(1),\n    humidity: msg.payload.main.humidity,\n    pressure: msg.payload.main.pressure,\n    description: msg.payload.weather[0].description,\n    icon: msg.payload.weather[0].icon,\n    city: msg.payload.name,\n    timestamp: new Date().toISOString()\n};\n\nmsg.payload = JSON.stringify(weatherData);\nreturn msg;",
        "wires": [["mqtt_out"]]
    },
    {
        "id": "mqtt_out",
        "type": "mqtt out",
        "name": "Publish Weather",
        "topic": "factory/weather/data",
        "qos": "1",
        "retain": "false",
        "broker": "your_broker_id",
        "wires": []
    }
]

Step 5: Test the Flow

1. Click Deploy in Node-RED
2. Click the button on the Inject node to trigger manually
3. Check the Debug panel for the output:

Expected Debug Output:

{
    "temperature": "25.5",
    "feels_like": "24.0",
    "humidity": 60,
    "pressure": 1013,
    "description": "scattered clouds",
    "icon": "03d",
    "city": "London",
    "timestamp": "2026-05-17T10:30:00.000Z"
}

Step 6: Verify MQTT Transmission

Use MQTT Explorer or command line to verify:

# Subscribe to the weather topic
mosquitto_sub -h localhost -t "factory/weather/data"

# Expected output (after trigger):
# {"temperature":"25.5","humidity":60,"description":"scattered clouds",...}

Handling API Errors

Add error handling to prevent broken data from reaching the ESP32:

Function Node with Error Handling:

// Check if API returned valid data
if (!msg.payload || !msg.payload.main) {
    // Return cached/default data on error
    const fallbackData = {
        temperature: "--.-",
        humidity: "--",
        description: "No data",
        error: true,
        timestamp: new Date().toISOString()
    };
    msg.payload = JSON.stringify(fallbackData);
    return msg;
}

// Normal processing
const weatherData = {
    temperature: msg.payload.main.temp.toFixed(1),
    humidity: msg.payload.main.humidity,
    description: msg.payload.weather[0].description,
    error: false,
    timestamp: new Date().toISOString()
};

msg.payload = JSON.stringify(weatherData);
return msg;

Using Static Test Data

During development, use static JSON to avoid hitting API rate limits:

Inject with Static JSON:

Create a second Inject node with static weather data:

{
    "temperature": "25.5",
    "feels_like": "24.0",
    "humidity": 60,
    "pressure": 1013,
    "description": "scattered clouds",
    "icon": "03d",
    "city": "London",
    "timestamp": "2026-05-17T10:30:00.000Z"
}

Connect this directly to the MQTT Out node for offline testing.

Verification

• Node-RED flow deploys without errors
• API returns valid JSON data
• Data is correctly formatted and published to MQTT
• MQTT Explorer shows the published messages
• ESP32 receives the MQTT data (if connected)

Troubleshooting

Issue 1: HTTP Request Returns Error

Symptom:

Error: ETIMEDOUT
Error: "Not found"

Solutions:

1. Check API key is correct
2. Verify API key has activated (>1 hour old)
3. Check internet connection
4. Verify URL parameters (lat, lon, units, appid)

Issue 2: Invalid JSON Received

Symptom:

Error: Unexpected token

Solutions:

• Add a debug node directly after the HTTP Request to see raw response
• Check if the API returned an error message instead of data
• Add the JSON parse error handling with a Catch node

Issue 3: MQTT Not Receiving

Symptom:

• MQTT Out node shows no errors
• ESP32 doesn’t receive the data

Solutions:

• Verify the MQTT broker is running: docker ps | grep mosquitto
• Check the topic name matches between publisher and subscriber
• Verify QoS settings match

Best Practices

• ✅ Use static test data during development — Avoid hitting API rate limits
• ✅ Add error handling in the Function node — Prevent broken data from propagating
• ✅ Set inject to “once: true” — Get initial data immediately on deploy
• ❌ Don’t fetch more than once per 10 minutes — Respect API rate limits
• ❌ Don’t hardcode API keys in shared flows — Use environment variables

Summary

1. Node-RED fetches weather data via HTTP Request node from public APIs
2. JSON data is parsed and transformed in Function nodes
3. Processed data is published via MQTT to the ESP32
4. Node-RED handles error cases and provides fallback data
5. 30-minute update intervals balance freshness with API rate limits

References

• OpenWeatherMap API Documentation
• Node-RED HTTP Request Node
• Node-RED Function Node

---

Target Audience: Alibaba.com IoT Pre-sales Engineers
Status: ✅ Completed