Unified.to
Blog

How to traverse hierarchical data in APIs

November 7, 2024

This guide explains how to retrieve and traverse hierarchical data structures when working with APIs that support parent-child relationships.

Overview

Many APIs organize their data in tree-like structures, where entities can have parent-child relationships. For example:

  • File storage systems with folders and files
  • Messaging platforms with channels and messages
  • Knowledge management systems with spaces and pages

This guide will show you how to efficiently traverse these hierarchical structures to retrieve all the data you need.

Before you begin

This guide assumes you have:

Understanding parent-child relationships

When an API endpoint supports a parent_id parameter, it typically indicates that the data is organized in a hierarchical structure. To fully traverse this structure, you'll need to:

  1. Get the top-level entities (parent nodes)
  2. For each parent, retrieve its children by passing the parent's ID
  3. Recursively repeat this process for any children that can also be parents

Example: Messaging platforms

Let's look at how this works with Discord:

  1. Discord servers (also called "guilds") are top-level parents
  2. Channels exist within servers as children
  3. Messages are also related to channels through their parent_id
  4. To get all channels:
    • First retrieve all guilds (servers)
    • Then get channels for each guild using the guild's ID as the parent_channel_id
async function getAllChannels(connectionId) {
    // Get all top-level guilds first
    const guilds = await sdk.messaging.listMessagingChannels({
        connectionId,
    });

    let allChannels = [];

    // For each guild, get its channels
    for (const guild of guilds.data) {
        const channels = await sdk.messaging.listMessagingChannels({
            connectionId,
            parentChannelId: guild.id,
        });
        allChannels = allChannels.concat(channels.data);
    }

    return allChannels;
}

API reference: Messaging API

Example: File storage systems

File storage systems are another common example of hierarchical data. Here's how to recursively traverse a file system to get all files:

async function getAllFiles(connectionId, parentId) {
    const response = await sdk.storage.listStorageFiles({
        connectionId,
        parentId,
    });

    if (!response.data) {
        return [];
    }

    let allFiles = [];

    for (const item of response.data) {
        if (item.type === 'FOLDER') {
            // Recursively get files from subfolders
            const subfolderFiles = await getAllFiles(connectionId, item.id);
            allFiles = allFiles.concat(subfolderFiles);
        } else {
            allFiles.push(item);
        }
    }

    return allFiles;
}

API reference: File Storage API

Best practices

When working with hierarchical data:

  1. Implement pagination: Some nodes might have many children. Use the limit and offset parameters to handle large datasets.
async function getAllChannelsWithPagination(connectionId, parentId = undefined) {
    let allChannels = [];
    let offset = 0;
    const limit = 100;

    while (true) {
        const response = await sdk.messaging.listMessagingChannels({
            connectionId,
            parentChannelId: parentId,
            limit,
            offset,
        });

        if (!response.data || response.data.length === 0) {
            break;
        }

        allChannels = allChannels.concat(response.data);
        offset += limit;
    }

    return allChannels;
}
  1. Handle rate limits: When recursively fetching data, you might hit API rate limits. Implement backoff strategies and respect the provider's limits.
  2. Cache parent IDs: If you need to traverse the same structure multiple times, consider caching the parent-child relationships to reduce API calls.

See also

Blog