Data Ingestion API

The Data Ingest API is the recommended connector-facing API for Inspire. It is designed to feel like a modern infrastructure API: one bearer credential per connector, predictable JSON, stable request IDs, and clear separation between connector management and data submission.

Design goals

• Simple bearer-token authentication
• Connector-scoped credentials
• Predictable request and response shapes
• Fast writes for high-frequency machine traffic
• Explicit request IDs for support and tracing
• Clean separation from the frozen Legacy Optymyse Connector API

Base URL

Through Edge:

https://inspire.example.com/datapool

Direct to DataPool:

https://datapool.example.com

Authentication

Each connector receives one ingest credential:

Authorization: Bearer ik_<key_id>_<secret>

Example:

Authorization: Bearer ik_5f8d4c2a91be_pULjS4J8VbD2WXQYjM4f9m1i3p4lQ7mN1k2sT6uR

Notes:

• ik_ identifies a data ingest credential
• <key_id> is the public selector used for fast lookup
• <secret> is the private secret portion
• Credentials are connector-scoped, not tenant-global
• Rotating a connector key invalidates the previous ingest key immediately

Primary endpoint

POST /api/ingest

Headers

Request body

{
  "stream": "agents",
  "items": [
    {
      "id": "agent-1001",
      "name": "Alice Smith",
      "fields": {
        "state": "Ready",
        "queue": "Sales",
        "callsHandled": "14"
      },
      "tags": {
        "site": "London"
      }
    }
  ]
}

Response

{
  "status": "accepted",
  "requestId": "req_a1b2c3d4e5f6g7",
  "itemsReceived": 1
}

HTTP status:

• Success: 202 Accepted
• Auth failure: 401 Unauthorized
• Validation failure: 409 Conflict for field/item size violations
• Rate limiting may return 429 Too Many Requests when the ingest limiter is exceeded

The response also includes the request ID in the X-Request-Id header.

Log endpoint

POST /api/ingest/log

Use this to attach operational logs to the connector in Inspire.

{
  "entries": [
    {
      "level": "info",
      "message": "Poll completed",
      "timestamp": "2026-04-09T11:08:00Z",
      "context": {
        "rows": "52"
      }
    }
  ]
}

Heartbeat endpoint

POST /api/ingest/heartbeat

Use this when the connector needs to signal liveness without sending data.

{}

Webhook ingest

The webhook surface is still available for generic integrations:

POST /api/webhook/{stream}
Authorization: Bearer ik_<key_id>_<secret>
Content-Type: application/json

The JSON body is accepted as one webhook item. Top-level primitive fields are stored directly, nested objects and arrays are stored as raw JSON strings, and the item ID comes from the id field by default unless a different idField query parameter is supplied.

Connector bootstrap

Connectors can fetch their managed configuration from Edge:

GET /api/connector/config
Authorization: Bearer ik_<key_id>_<secret>

Example response:

{
  "connectorId": 42,
  "name": "Warehouse SQL",
  "configuration": "{\n  \"Version\": \"7.0\",\n  ...\n}",
  "defaultConfiguration": false,
  "dataPoolAddress": "https://datapool.example.com"
}

Connectors can also publish their default configuration back to Edge:

PUT /api/connector/config
Authorization: Bearer ik_<key_id>_<secret>
Content-Type: application/json

{
  "configuration": "{\n  \"Version\": \"7.0\",\n  ...\n}"
}

Connector usage pattern

An Inspire data-ingest connector should typically follow this sequence:

1. Read its connector ingest key from config or environment.
2. Call GET /api/connector/config on startup.
3. Poll or receive data from the source system.
4. Convert source records into Inspire items.
5. Send batched updates to POST /api/ingest.
6. Send POST /api/ingest/heartbeat between data pushes if needed.
7. Send POST /api/ingest/log for operational visibility.

Example: minimal connector loop

const baseUrl = process.env.INSPIRE_BASE_URL!
const ingestKey = process.env.INSPIRE_INGEST_KEY!
 
const headers = {
  Authorization: `Bearer ${ingestKey}`,
  'Content-Type': 'application/json',
}
 
await fetch(`${baseUrl}/api/connector/config`, { headers })
 
await fetch(`${baseUrl}/api/ingest`, {
  method: 'POST',
  headers,
  body: JSON.stringify({
    stream: 'agents',
    items: [
      {
        id: 'agent-1001',
        name: 'Alice Smith',
        fields: {
          state: 'Ready',
          queue: 'Sales',
          callsHandled: '14',
        },
      },
    ],
  }),
})

Limits

Current request guards:

• max 30 MB request body size
• max 200 fields per item
• max 5 MB per field value
• max 10 MB total field payload per item
• ingest rate limiting per connector credential

Best practices

• Keep one connector per upstream system or polling responsibility
• Batch multiple item changes into one ingest request where practical
• Use stable item IDs so updates replace state instead of creating churn
• Use name for operator-friendly display labels
• Put rendering/display metadata in tags, not in the item ID
• Log source failures to POST /api/ingest/log so operators can diagnose issues

Error handling

Treat 202 Accepted as the success state. Any non-202 response should be logged and retried according to your connector policy.

Recommended retry behavior:

• 401: stop and alert, the credential is invalid or rotated
• 409: fix payload shape or field size
• 429: back off and honor Retry-After
• 5xx: retry with exponential backoff