Skip to content

Ingesting Records

Records are rows of data that conform to a schema. Every field is encrypted individually using keys derived from your keyring — the Blind Insight API stores only ciphertext.

All encryption happens locally in the Blind Proxy. You must have a running Blind Proxy to ingest encrypted records.

Prerequisites

Quickstart

The fastest way to ingest records is via the CLI:

blind record create \
  --schema <schema_id> \
  --data '{"name": "Alice", "age": 30, "height": 1.68}'

The Proxy encrypts the record locally before sending it to the API. Blind Insight never sees the plaintext.

Single Record Ingest

Via CLI

blind record create --schema <schema_id> --data '{"name": "Bob", "age": 42}'

Pass data inline with --data or from a file:

blind record create --schema <schema_id> --data @record.json

Via Proxy API

The Proxy exposes a local REST API at http://localhost:8080 (or wherever you've configured it). The /api/records/ingest/ endpoint accepts plaintext and returns the ID of the created encrypted record.

POST /api/records/ingest/ HTTP/2
Host: localhost:8080
Authorization: Basic <base64(email:password)>
Content-Type: application/json

[
  {
    "schema": "/api/schemas/iKi3YbiG7Ax9cEZ7YPVNgw/",
    "data": {"name": "Alice", "age": 30, "height": 1.68}
  }
]

Note

/api/records/ingest/ is a proxy-only endpoint. Requests must go to your local Blind Proxy, not directly to api.app.blindinsight.io.

The record limit for this endpoint is 100 records per request. For larger ingests, use Batch Upload.

Batch Upload

For ingesting more than 100 records, use the batch upload job system. Jobs process records asynchronously and report progress.

Via CLI

# JSON file
blind jobs upload --data records.json

# CSV file
blind jobs upload --data records.csv

# Control batch size (default: 10)
blind jobs upload --data records.json --X-Batch-Size 50

JSON Format

Each object must include a schema field with the schema URL:

[
  {
    "schema": "/api/schemas/iKi3YbiG7Ax9cEZ7YPVNgw/",
    "data": { "name": "Alice", "age": 30 }
  },
  {
    "schema": "/api/schemas/iKi3YbiG7Ax9cEZ7YPVNgw/",
    "data": { "name": "Bob", "age": 42 }
  }
]

CSV Format

Include a header row with field names matching your schema. Add a magic schema_id:<id> header as the last column to specify which schema to use:

name,age,schema_id:iKi3YbiG7Ax9cEZ7YPVNgw
Alice,30
Bob,42

Warning

The CSV header row must include schema_id:<id> as the final column. Omitting it causes a validation error.

Via Proxy API

POST /api/jobs/upload/ HTTP/2
Host: localhost:8080
Authorization: Basic <base64(email:password)>
Content-Type: application/json

[
  {"schema": "/api/schemas/<id>/", "data": {"name": "Alice", "age": 30}},
  {"schema": "/api/schemas/<id>/", "data": {"name": "Bob",   "age": 42}}
]

On success, the response returns the job ID and status URLs:

HTTP/2 201 Created
X-Job-ID: 7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a
X-Job-Status-URL: /api/jobs/7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a/
X-Job-Websocket-URL: /api/ws/jobs/7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a/

{"job_id": "7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a"}

Monitor job progress

Poll the status endpoint:

GET /api/jobs/7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a/ HTTP/2
Host: localhost:8080

Or connect via WebSocket for real-time updates:

ws://localhost:8080/api/ws/jobs/7b8e9f2d-4a3c-4e5f-8d1a-6b9c0d3e2f1a/

Both return the same progress payload:

{
  "status": "processing",
  "processed": 42,
  "total": 100,
  "current_batch": 5,
  "total_batches": 10
}

Valid status values: uploading, processing, completed, failed.

Very Large Files (TUS Protocol)

For files larger than ~2 GB, use the TUS resumable upload protocol. Point any TUS-compatible client at /api/files/ on your running Blind Proxy.

Supported file formats are the same as batch upload: JSON and CSV.

Create an upload

POST /api/files/ HTTP/2
Host: localhost:8080
Tus-Resumable: 1.0.0
Upload-Length: <file_size_in_bytes>
Upload-Metadata: filename <base64_filename>, filetype <base64_filetype>

Response:

HTTP/2 201 Created
Location: http://localhost:8080/api/files/<file_id>
X-Job-Id: <job_id>
X-Job-Status-Url: http://localhost:8080/api/jobs/<job_id>/
X-Job-Websocket-Url: http://localhost:8080/api/ws/jobs/<job_id>/

Send file chunks

PATCH /api/files/<file_id>/ HTTP/2
Host: localhost:8080
Tus-Resumable: 1.0.0
Content-Type: application/offset+octet-stream
Upload-Offset: 0

<binary file contents>

Response:

HTTP/2 204 No Content
Upload-Offset: <bytes_received>

Repeat with Upload-Offset set to the value from the previous response until all bytes are sent. Track job progress the same way as batch uploads.

Using Sample Data

To test the ingest flow with pre-built datasets, clone the demo datasets repository:

git clone https://github.com/blind-insight/demo-datasets.git
cd demo-datasets/datasets

Each dataset includes schemas/ and data/ directories. Create the schema first, then ingest the data:

# Create the schema
blind schema create --name "Patient" --dataset <dataset_id> \
  --schema medical/schemas/patient.json

# Ingest the records
blind jobs upload --data medical/data/patient_data.json

Searching and Decrypting Records

Once records are ingested, you can run encrypted searches and decrypt results through the Blind Proxy. See the Using the Blind Proxy guide for full search and decrypt workflows.

Next Steps