Federated search queries multiple external APIs in parallel. If you query 10 sources, your search is only as fast as the slowest API.

If 9 databases return results in 200ms, but a 10th database stalls for 5 seconds, the user (or the AI agent) experiences a 5-second delay. In production, this latency is unacceptable.

To build a reliable federated search system, you must design for partial failures. You cannot let a single slow API block the entire search. You need timeouts, parallel executions, and graceful quality degradation. By standardizing these source APIs into a unified interface (read more on eliminating the connector tax), we can easily wrap them in generic query handlers.

Implementing Timeout-Bounded Parallel Queries

In JavaScript or TypeScript, you can bound request times using an AbortController combined with a timeout promise.

Here is a resilient query wrapper that dispatches search requests in parallel, enforces a hard timeout for each, and collects whatever results complete in time:

export interface SearchQuery {
  term: string;
}

export interface SearchResult {
  source: string;
  title: string;
}

export interface Connector {
  name: string;
  query(search: SearchQuery, signal: AbortSignal): Promise<SearchResult[]>;
}

export async function federatedSearchWithTimeout(
  connectors: Connector[],
  query: SearchQuery,
  timeoutMs: number = 800
): Promise<{ results: SearchResult[]; failedSources: string[] }> {
  
  const failedSources: string[] = [];

  const promises = connectors.map(async (connector) => {
    // Create an AbortController for this specific request
    const controller = new AbortController();
    const id = setTimeout(() => controller.abort(), timeoutMs);

    try {
      const results = await connector.query(query, controller.signal);
      clearTimeout(id);
      return results;
    } catch (error: any) {
      clearTimeout(id);
      failedSources.push(connector.name);
      
      // Log the specific failure reason for observability
      if (error.name === 'AbortError') {
        console.warn(`Connector ${connector.name} timed out after ${timeoutMs}ms.`);
      } else {
        console.error(`Connector ${connector.name} failed:`, error.message);
      }
      return [];
    }
  });

  const allResults = await Promise.all(promises);
  return {
    results: allResults.flat(),
    failedSources
  };
}

Designing the Fallback Response

When a source times out, how should your agent behave?

  1. Provide a Degradation Status: Do not hide query failures. Return a metadata list of sources that failed or timed out. Your agent should know that its context might be incomplete.
  2. Execute a Background Sync: If a query times out, do not just discard the pending request. Let it run in the background (or trigger a retry) to populate your local database cache. The next time the user makes the query, the cache will be warm.
  3. Graceful UI Rendering: In human-facing portals (like Science.gov), render the results as they arrive. If 8 agencies return in 300ms, display them. Show a spinner for the remaining sources, but allow the user to read what is already loaded.

Enforcing Circuit Breakers

If a specific external database is experiencing an outage, continuing to query it wastes resources and pollutes logs. A circuit breaker pattern tracks failure rates and temporarily disables the connector if it fails repeatedly.

A simple circuit breaker has three states:

  • Closed: Requests flow normally. If a request fails, we increment a error counter.
  • Open: If errors exceed a threshold (e.g., 5 consecutive failures), the breaker trips. Requests to this source are immediately rejected with cached or empty data, bypassing the API call.
  • Half-Open: After a cooldown period (e.g., 60 seconds), we allow a single trial query. If it succeeds, we reset the error counter and close the breaker. If it fails, we trip it again.

By combining timeouts, circuit breakers, and explicit fallback notifications, you ensure that your federated retrieval engine remains online and fast—even when external directories (such as Querying EUR-Lex SOAP systems, which are notoriously prone to high latency spikes) go offline.