When teams build RAG pipelines, they usually start by writing custom integration scripts for each source. One script pulls from the SEC EDGAR API, another queries PubMed, and a third crawls internal Confluence pages.

This works when you have two or three sources. But at scale, you hit the connector tax. This overhead accumulates because AI agents cannot reliably reason over raw web scrapes, making normalized schemas a necessity (read more on why web snippets fail).

Every external database has its own quirks:

  • SEC EDGAR requires strict User-Agent headers declared in a specific format, and rejects queries that exceed rate limits.
  • PubMed E-utilities requires specific API parameters like db=pmc or retmode=json, and queries must be URL-encoded in a legacy structure.
  • SOAP/XML endpoints in legal directories require parsing deep XML trees.

If one of these external services changes its endpoint, rate limits, or response schema, your query pipeline fails. Instead of building agent reasoning, your engineers spend time fixing data plumbing.

Designing a Unified Connector Interface

The solution is to decouple source integration from your agent code. Every data source—whether it is a public compliance directory or a private PostgreSQL instance—should implement a single interface.

Here is the TypeScript interface we use to standardize search queries and normalize payloads:

export interface SearchQuery {
  term: string;
  limit?: number;
  filters?: Record<string, string>;
}

export interface NormalizedResult {
  source: string;
  id: string;
  title: string;
  abstract: string;
  author: string;
  url: string;
  rawPayload: Record<string, any>;
}

export interface SourceConnector {
  id: string;
  name: string;
  query(search: SearchQuery): Promise<NormalizedResult[]>;
}

Implementing a Concrete Connector (SEC EDGAR)

By implementing the SourceConnector interface, each source encapsulates its own authentication, rate-limiting, and schema parsing.

Here is a simplified example of how we wrap the SEC EDGAR company filing API:

import fetch from 'node-fetch';

export class SecEdgarConnector implements SourceConnector {
  id = 'sec_edgar';
  name = 'SEC EDGAR';
  private userAgent = 'Deep Web Technologies support@deepwebtech.com';

  async query(search: SearchQuery): Promise<NormalizedResult[]> {
    const url = `https://data.sec.gov/submissions/CIK${search.term}.json`;
    
    const response = await fetch(url, {
      headers: { 'User-Agent': this.userAgent }
    });

    if (!response.ok) {
      throw new Error(`SEC EDGAR failed with status ${response.status}`);
    }

    const data = await response.json();
    return this.normalize(data);
  }

  private normalize(data: any): NormalizedResult[] {
    // SEC EDGAR filings are returned as an array of recent submissions
    const recentFilings = data.filings.recent;
    return recentFilings.accessionNumber.map((accNum: string, index: number) => ({
      source: this.name,
      id: accNum,
      title: `${recentFilings.form[index]} - ${recentFilings.filingDate[index]}`,
      abstract: `Filing for ${data.name} (CIK: ${data.cik})`,
      author: data.name,
      url: `https://www.sec.gov/Archives/edgar/data/${data.cik}/${accNum.replace(/-/g, '')}/${recentFilings.primaryDocument[index]}`,
      rawPayload: data
    }));
  }
}

Decoupling the Agent

With this architecture, your core query pipeline does not need to know how SEC EDGAR or PubMed works. It simply executes queries against a registry of connectors:

class FederatedSearchEngine {
  private connectors: SourceConnector[] = [];

  register(connector: SourceConnector) {
    this.connectors.push(connector);
  }

  async search(query: SearchQuery): Promise<NormalizedResult[]> {
    const promises = this.connectors.map(async (connector) => {
      try {
        return await connector.query(query);
      } catch (error) {
        console.error(`Connector ${connector.name} failed:`, error);
        return []; // Fail gracefully, returning empty results for this source
      }
    });

    const results = await Promise.all(promises);
    return results.flat();
  }
}

If the SEC EDGAR API changes its schema tomorrow, you only update the SecEdgarConnector class. Your agent’s core code, pipelines, and evaluations remain untouched.

However, running parallel queries across multiple external connectors exposes your system to latency spikes if any single database stalls. To mitigate this risk, see our blueprints for resilient federated search using circuit breakers and timeouts.

Stop writing custom scraper scripts. Build once, maintain centrally, and keep your agents decoupled.