How to troubleshoot unhealthy webhooks

September 10, 2024

Webhooks are a useful tool for getting real-time notifications about changes to the data you care about. When something goes wrong, however, it is usually indicative of an issue with the connection rather than the webhook itself. In these situations, we will mark the webhook or connection as unhealthy.

This guide will go over how to diagnose and fix webhook and connection errors.

Before you begin

This guide assumes you have a basic understanding of:

Check the API call logs

The first place to look for debugging information is in the API call logs.

On app.unified.to, navigate to Integrations > Webhooks.
Make note of the unhealthy webhook and its connection ID.
Navigate to Integrations > API Call Logs.
Filter by the connection ID of the webhook in question. You should see a list of API calls attempted by the webhook (note: we only store logs from the past 60 days).
Click on the latest log entry that threw an error - the status column will display an error code.
Details about the API call will be shown. The Description field will detail the reason the webhook didn't work and the Status field will show the specific error code itself. For example:

You can also retrieve your API call logs via the API.

API reference: Get API calls

Reasons why webhooks fail

Webhooks can break down when they:

Failed to subscribe
Failed to refresh token
Failed to read
Failed to dispatch data
Failed to process data

Failing to subscribe happens during webhook creation and can be due to multiple reasons: the integration isn't available, insufficient permissions (i.e. scopes mismatch), or bugs.

We recommend double-checking your scopes on both the API provider and Unified.to to ensure that everything has been set correctly. See: Insufficient permissions (403) errors

Note: This only applies to native webhooks.

Failed to refresh token

Failing to refresh the auth token happens when we attempt to communicate with the API provider unsuccessfully. This happens when the API provider only sends us partial data about new events (e.g. the IDs of the data) and we need to use a connection to retrieve the rest of the data. In this case, it is likely an error with the connection itself.

Double-check that the auth credentials for the connection are correct, that the scopes are set correctly (i.e. they haven't been revoked), and then recreate both the connection and its associated webhook.

Failed to read

Failing to read happens when we attempt to retrieve new data from the API provider and are prevented from doing so, whether due to configuration errors (i.e. 401 or 403 errors) or problems with the integration itself.

Failed to dispatch data

Failing to dispatch data happens when we attempt to send webhook data to your server but the server does not respond or responds with an error.

Make sure your webhook URL is configured correctly when creating the webhook. Respond with a 200 status when your endpoint successfully receives and processes the data. Otherwise, we will keep trying to POST to your server before marking the webhook as unhealthy (after approximately 2 weeks).

After checking the above, recreate the webhook and try again.

Failed to process data

Failing to process data indicates a problem with either the reading or dispatching steps. Refer to the troubleshooting steps above and then recreate the webhook and try again.

Insufficient permissions (403) errors

Insufficient permission errors may arise due to scope mismatches with the API provider, with Unified.to, or both. These errors are normally due to faulty connections, so the steps below outline how to debug the connection that is tied to your webhook.

Double check your scope settings in your developer account

Log into your developer account for the API you're trying to access.
Navigate to the settings page where your developer app's authorization options are found.
Ensure that all necessary scopes are enabled - some APIs require additional scopes to be always on. You can find this information in our How-to Guides specific to the integration you are interested in (if a guide is missing, please let us know and we'll write one up for it!)
Make a note of the scopes you have enabled.

Double check your scope settings on Unified.to

If you are using our Embedded components to create connections, you'll need to ensure that the scopes on Unified.to match the ones you enabled in the API provider.

Navigate to Settings > Embedded Authorization.
Click Options.
Select the permission scopes that match the scopes you enabled in the developer account (see note below on mappings).
Select the webhook scope.

If you are generating an auth URL to redirect your customers to our auth flow, include scopes as a query parameter and list all the Unified scopes you need. Behind the scenes, Unified.to will map the Unified scopes you pass to the scopes on the API provider's end (see note below on mappings).

For example:

https://api.unified.to/unified/integration/auth/
{WORKSPACE_ID}/{INTEGRATION e.g. hubspot}?redirect=true
&scopes=webhook,{YOUR_SCOPES e.g. crm_deal_read, crm_event_read}

API reference: Create an authorization URL

After addressing these issues, re-create the connection and/or webhook and then try again.

Note: A mapping of the Unified scopes to the API provider (i.e. integration's) scopes can be found on the integration details page under the OAuth 2 Credentials tab e.g. here are the mappings for HubSpot scopes.

Invalid authentication (401) errors

A 401 error may indicate that the connection's auth credentials were entered incorrectly, whether it be the API token, a secret, or another value depending on the integration.

Double check that your customer input the right values when going through the auth flow and then recreate the connection and webhook and try again.

Bad request (400) errors

These errors indicate that something is wrong with the request - in this case, it is likely not a scope or authentication issue but a problem with the integration itself. Please reach out to us or let us know in our Discord server. Thank you!

To know about webhook errors as soon as they occur, we recommend that you subscribe to our Notifications webhook to get real-time updates about your webhooks' statuses.

Navigate to Settings > Workspace.

Under Notifications webhook URL, enter the URL of the endpoint where you would like to receive notifications.

The payload that will be sent to this endpoint contains:

id: string // the id of the object or the integration's type
nonce: string // random string
sig: string // base64( hmac-sha1( id + nonce + event, workspace.secret ) )
event: enum //the event that was triggered

Under Notifications webhook events, select the events you want to be alerted to e.g. WEBHOOK_UNHEALTHY
On your server, handle incoming notifications as you see fit e.g. set up logging, send a message to the engineering team, post on Slack or Discord, etc.

A note on our webhook retry mechanism

At Unified.to, we've implemented a robust retry mechanism to ensure reliable webhook delivery. Here's how it works.

When your webhook endpoint is unavailable

When sending data to your webhook endpoint:

If your endpoint is unavailable or returns an error, we'll retry up to 3 times immediately, with a 1-second delay between attempts.
If the error persists, we switch to a Fibonacci backoff strategy:
- Initial delay: 1 minute
- Subsequent delays: 1 minute, 2 minutes, 3 minutes, 5 minutes, 8 minutes, etc.
- This continues for several days and up to 2 weeks, maximizing the chance of successful delivery.

When the API provider is unavailable

If we can't reach the API provider or have hit their rate limits, we will back off and retry based off of a Fibonacci delay sequence (outlined above), starting with 1 minute.