Understanding native and virtual webhooks January 10, 2024
A webhook is a mechanism that allows a server to send real-time notifications to another server when a specific event occurs. It is sometimes called a "web callback" or "push API". Webhooks enable instant communication between different systems, making them crucial for maintaining data synchronization and triggering automated workflows. For instance, Stripe's payment processing system uses webhooks to notify a merchant's server immediately when a customer completes a purchase, allowing for instant order fulfillment and inventory updates.
Think of webhooks as a delivery service that brings packages directly to your doorstep. In contrast, traditional REST APIs are like repeatedly checking your mailbox to see if you've received any mail. With webhooks, you don't need to keep asking, "Is there any new data?" Instead, the data comes to you as soon as it's available, saving time and resources.
Webhooks can make your synchronization strategy less complicated since your application will receive updated data directly, eliminating the need for frequent polling and reducing the risk of missing important updates.
The traditional way of polling for new data from an API is by calling the List endpoint with the updated_gte: Date filter - this returns data that has been updated since that date.How often you poll for new data should be determined by how important that data is to your application as well as your budget. If the data isn't time-sensitive i.e. you don't need to have the data the moment it becomes available, then this strategy may be sufficient. But you need to take care of retry strategies when the API returns a rate limit error, which can make this method more complicated than using a webhook. You'll also need to handle any other network errors that could occur.
Modern APIs and applications leverage webhooks for synchronization because of their simplicity and timeliness. You, the application developer, do not have to create a retry mechanism to actively poll the server for new data or deal with rate-limiting and other network errors. The hard work is done by the third-party API provider to send you updates as they come.
Webhooks operate on a subscription model, where your application registers to receive updates about specific events from a third-party provider or vendor. Here's a step-by-step breakdown of how webhooks typically work:
Create a webhook endpoint on your server that can receive POST requests. This endpoint will handle the incoming webhook data. Subscribe to the events you want to monitor by registering your webhook endpoint's URL with the third-party API provider. This is usually done through the provider's developer dashboard or via their API. Secure your webhook by implementing measures to verify that incoming webhook requests are legitimate and sent by the intended provider. This often involves checking cryptographic signatures or using shared secrets. Configure event types and specify which types of events you want to receive notifications for. This could be anything from new customer signups to payment confirmations, depending on the service. Receive and process data. When a relevant event occurs, the provider sends a POST request to your registered endpoint. Your server receives this data in real-time and can process it according to your application's needs. Handle the response. Your endpoint should respond promptly to the webhook request, typically with a 200 OK status, to acknowledge receipt. This helps prevent unnecessary retries from the provider. By following this process, your application can receive and react to important events as they happen, without the need for constant polling. This real-time communication enables more efficient and responsive systems, allowing you to build more dynamic and interconnected applications.
Webhooks sound great, right? However, there's an important catch: Not all software vendors support webhooks. 😢
Fortunately, Unified.to has standardized the webhook experience by creating virtual webhooks for most of our integrations, no matter if they natively support them or not. That means you, the application developer, don't have to worry about doing any sort of polling and can just focus on ingesting data with webhooks.
Our proprietary technology monitors your customers' connections for updates and then notifies your webhook endpoint of those changes. You subscribe to and manage virtual webhooks exactly the same way as you would with native (i.e. "real") webhooks. We'll manage all of the polling, rate-limiting, and state management for you. Lastly, because of our strict security policies, we don't store any of your customers' data on our servers.
Note that, since we are polling the vendor on your behalf (under the hood), you are able to control the frequency at which we do so and thus control your costs per customer. Virtual webhooks are especially well-suited for non-real time tasks. If you believe we are missing support for a native webhook, then let us know and our team will get right on it!
You can find the list of features supported by an integration by going to that integration's page and then clicking on "Feature Support." Under the "Webhooks" section, virtual webhooks will have the word "virtual" in their name:
Similarly, native webhooks will have the word "native" in their name.
Once a customer authorizes an integration for your application with the Unified.to authorization system, you will get a connection_id that represents that new connection.
You can then call the CreateWebhook API
POST /unified/webhook?include_all=true
{
hook_url: `${my_webhook_url}`,
connection_id: `${connection_id}`
object_type: 'candidate',
event: 'updated',
interval: 60,
fields: ''
}
The API will return a webhook object with an id; you can use this id to delete the webhook when it is no longer needed. If you delete the connection, any associated webhooks will also be deleted. To view your webhooks, call the ListWebhooks API
If you prefer to create webhooks through the Unified.to dashboard, you can also do that and view your registered webhooks at https://app.unified.to/webhooks .
When you first integrate with a customer's third-party account, you'll want to use the Unified.to API to retrieve relevant information. The first step is to initially retrieve all of the existing data and then the next step is to retrieve the updated data when it is changed.
Initial retrieval
When you create a webhook with the include_all parameter, Unified.to will retrieve all of the existing requested data. Our system will POST this data to your webhook in chunk sizes up to the limit supported by the integration as described under "Feature Support." Depending on the amount of data, the integration's API's rate-limiting rules, and your webhook server's performance, this could take a very long time. When this step is finished, your webhook will receive a final payload object with{type: 'INITIAL-COMPLETE'}.
If you do not want to get all of the existing data (for example, you already have it or don't need it), then do not send the include_all parameter.Updated data retrieval
When updated data is available, your webhook will be called with a list of up to 100 entries of that requested object. If there are more than 100 entries, then your webhook will be called multiple times.Triggering a retrieval
If you don't want to wait the whole hour while testing webhooks on a Tester plan, you can use a manual trigger to re-schedule a webhook call right away. Simply click the "Trigger" icon next to your webhook. This can also be accomplished in the API.If your webhook in unavailable or returns an error when Unified.to sends data to it, we will retry up to 3 times, waiting 1 second in between. If the error persists, we will back off and retry based off of a Fibonacci delay sequence, starting with 1 minute. If the integration's API is unavailable or returns an error (such as a Rate Limit warning), we will back off and retry based off of a Fibonacci delay sequence, starting with 1 minute. The Connection Logs show all webhook attempts and errors. health status indicator with possible values of Healthy and Unhealthy visible on the webhook page .Light audit trail that is limited to the last 100 activities Last-time that the webhook was run/checked
