| title | description | author | ms.service | services | ms.topic | ms.date | ms.author |
|---|---|---|---|---|---|---|---|
Troubleshoot Azure IoT message routing |
How to perform troubleshooting for Azure IoT Hub message routing |
ash2017 |
iot-hub |
iot-hub |
conceptual |
05/06/2020 |
asrastog |
This article provides monitoring and troubleshooting guidance for common issues and resolution for IoT Hub message routing.
We recommend you monitor IoT Hub metrics related to message routing and endpoints to give you an overview of the messages sent. You can also create a diagnostic setting to send operations for routes in IoT Hub resource logs to Azure Monitor Logs, Event Hubs or Azure Storage for custom processing. To learn more about using metrics, resource logs, and diagnostic settings, see Monitor IoT Hub. For a tutorial, see Set up and use metrics and resource logs with an IoT hub.
We also recommend enabling the fallback route if you want to maintain messages that don't match the query on any of the routes. These can be retained in the built-in endpoint for the amount of retention days configured.
The following are the most common issues observed with message routing. To start troubleshooting, click on the issue for detailed steps.
- Messages from my devices are not being routed as expected
- I suddenly stopped getting messages at the built-in Event Hubs endpoint
To troubleshoot this issue, analyze the following.
All the IoT Hub metrics related to routing are prefixed with Routing. You can combine information from multiple metrics to identify root cause for issues. For example, use metric Routing Deliveries to identify the number of messages that were delivered to an endpoint or dropped when they didn't match queries on any of the routes and fallback route was disabled. Check the Routing Latency metric to observe whether latency for message delivery is steady or increasing. A growing latency can indicate a problem with a specific endpoint and we recommend checking the health of the endpoint. These routing metrics also have dimensions that provide details on the metric like the endpoint type, specific endpoint name and a reason why the message was not delivered.
Observe the Routes resource logs to get more information on the routing and endpoint operations or identify errors and relevant error code to understand the issue further. For example, the operation name RouteEvaluationError in the log indicates the route could not be evaluated because of an issue with the message format. Use the tips provided for the specific operation names to mitigate the issue. When an event is logged as an error, the log will also provide more information on why the evaluation failed. For example, if the operation name is EndpointUnhealthy, an Error code of 403004 indicates the endpoint ran out of space.
Use the REST API Get Endpoint Health to get health status of the endpoints. The Get Endpoint Health API also provides information on the last time a message was successfully sent to the endpoint, the last known error, last known error time and the last time a send attempt was made for this endpoint. Use the possible mitigation provided for the specific last known error.
To troubleshoot this issue, analyze the following.
Once a route is created, data stops flowing to the built-in-endpoint, unless a route is created to that endpoint. To ensure messages continues to flow to the built-in-endpoint if a new route is added, configure a route to the events endpoint.
The fallback route sends all the messages that don't satisfy query conditions on any of the existing routes to the built-in-Event Hubs (messages/events), that is compatible with Event Hubs. If message routing is turned on, you can enable the fallback route capability. If there are no routes to the built-in-endpoint and a fallback route is enabled, only messages that don't match any query conditions on routes will be sent to the built-in-endpoint. Also, if all existing routes are deleted, fallback route must be enabled to receive all data at the built-in-endpoint.
The fallback route sends all the messages that don't satisfy any of the query conditions on any of the existing routes to the built-in-Event Hubs (messages/events), that is compatible with Event Hubs. If message routing is turned on, you can enable the fallback route capability. If there are no routes to the built-in endpoint and a fallback route is enabled, only messages that don't match any query conditions on routes will be sent to the built-in-endpoint. Also, if all existing routes are deleted, the fallback route must be enabled to receive all data at the built-in-endpoint.
You can enable or disable the fallback route in the Azure portal by using the Message Routing blade for the IoT hub. You can also use the Azure Resource Manager for FallbackRouteProperties to use a custom endpoint for a fallback route.
[!INCLUDE iot-hub-include-last-known-errors]
The following are the operation names and error codes logged in the routes resource logs.
[!INCLUDE iot-hub-diagnostics-operation-names]
[!INCLUDE iot-hub-diagnostics-error-codes]
If you need more help, you can contact the Azure experts on the Microsoft Q&A and Stack Overflow forums. Alternatively, you can file an Azure support incident. Go to the Azure support site and select Get Support.