How to troubleshoot unhealthy webhooks
September 25, 2024
Webhooks are a powerful tool for getting real-time notifications about changes to the data you care about. When something goes wrong, however, we will mark those webhooks as unhealthy. This guide explains how to diagnose and troubleshoot unhealthy webhooks at Unified.to.
Before you begin
This guide assumes you have a basic understanding of:
Diagnose the webhook
The first place to look for debugging information is in the API call logs.
1. Check the webhook status
- Log in to your Unified.to account.
- Navigate to Webhooks.
- Make note of the unhealthy webhook and its connection ID.
2. Review API call logs
- Go to API Call Logs.
- Filter by the connection ID of the webhook in question. You should see a list of API calls attempted by the webhook (note: we only store logs from the past 60 days).
- Click on the latest log entry that threw an error - the status column will display an error code.
- Details about the API call will be shown. The Description field will detail the reason the webhook failed and the Status field will show the specific error code itself. For example:
You can also use the Unified API to retrieve a list of your webhooks and API call logs.
API reference: List all webhooks, List all API calls
Troubleshooting webhook error codes
Bad request (400) errors
These errors indicate that something is wrong with the request - in this case, it is likely not a scope or authentication issue but a problem with the integration itself. Please reach out to us or let us know in our Discord server. Thank you!
Authentication (401) and permission (403) errors
A 401 error may indicate that the connection's auth credentials were entered incorrectly, whether it be the API token, a secret, or another value depending on the integration.
A 403 error is often indicative of scope mismatches with the API provider, with Unified.to, or both.
Both of these errors are indicative of an issue with the connection itself, rather than the webhook. Please see: How to troubleshoot unhealthy connections
Subscribe to the notifications webhook to be alerted when your webhooks fail
To learn about any webhook errors as soon as they occur, we recommend that you subscribe to our Notifications webhook to get real-time updates about your webhooks' statuses.
- Navigate to Settings > Workspace.
- Under Notifications webhook URL, enter the URL of the endpoint where you would like to receive notifications.
- The payload that will be sent to this endpoint contains:
id: string // the id of the object or the integration's type nonce: string // random string sig: string // base64( hmac-sha1( id + nonce + event, workspace.secret ) ) event: enum //the event that was triggered
- The payload that will be sent to this endpoint contains:
- Under Notifications webhook events, select the events you want to be alerted to e.g.
WEBHOOK_UNHEALTHY - On your server, handle incoming notifications as you see fit e.g. set up logging, send a message to the engineering team, post on Slack or Discord, etc.
Understanding why webhooks fail
This section will go over why webhooks fail to provide a deeper understanding of how our webhooks work.
Failed to subscribe
Failing to subscribe happens during webhook creation and can be due to multiple reasons: the integration isn't available, insufficient permissions (i.e. scopes mismatch), or bugs.
Note: This only applies to native webhooks.
Failed to refresh token
Failing to refresh the auth token happens when we attempt to communicate with the API provider unsuccessfully. This happens when the API provider only sends us partial data about new events (e.g. the IDs of the data) and we need to use a connection to retrieve the rest of the data. In this case, it is likely an error with the connection itself.
Double-check that the auth credentials for the connection are correct, that the scopes are set correctly (i.e. they haven't been revoked), and then recreate both the connection and its associated webhook.
Failed to read
Failing to read happens when we attempt to retrieve new data from the API provider and are prevented from doing so, whether due to configuration errors (i.e. 401 or 403 errors) or problems with the integration itself.
Double-check that the auth credentials for the connection are correct, that the scopes are set correctly (i.e. they haven't been revoked), and then recreate both the connection and its associated webhook.
Failed to dispatch data
Failing to dispatch data happens when we attempt to send webhook data to your server but the server does not respond or responds with an error.
Make sure your webhook URL is configured correctly when creating the webhook. Respond with a 200 status when your endpoint successfully receives and processes the data. Otherwise, we will keep trying to POST to your server before marking the webhook as unhealthy (after approximately 2 weeks).
After checking the above, recreate the webhook and try again.
Failed to process data
Failing to process data indicates a problem with either the reading or dispatching steps. Refer to the troubleshooting steps above and then recreate the webhook and try again.
A note on our webhook retry mechanism
At Unified.to, we've implemented a robust retry mechanism to ensure reliable webhook delivery. Here's how it works.
When your webhook endpoint is unavailable
- If your endpoint is unavailable or returns an error, we'll retry up to 3 times immediately, with a 1-second delay between attempts.
- If the error persists, we switch to a Fibonacci backoff strategy:
- Initial delay: 1 minute
- Subsequent delays: 1 minute, 2 minutes, 3 minutes, 5 minutes, 8 minutes, etc.
- This continues for several days and up to 2 weeks, maximizing the chance of successful delivery.
When the API provider is unavailable
If we can't reach the API provider or have hit their rate limits, we will back off and retry based off of a Fibonacci delay sequence (outlined above), starting with 1 minute.