Docs › Reference

Sources, connections and harvesting

The API calls behind sources, connections and harvesting.

Part of the HTTP API reference.

Routes:

Everything below is generic: "There is no mention of Snowflake, of a table, or of any asset type: a connector declares what it can do, the connection holds the mapping from its words to yours, and harvest_plan does the rest. Adding a connector is adding a file in connectors/ — this file does not change."

And the division of responsibility that holds it up:

connectors.available answers the first [question] — what is INSTALLED — by looking at the directory, and it remains the only thing that may. This answers the second, and it is the only thing that may do that. Nothing in connectors/ is told: a connector cannot read this table, cannot name this permission, and cannot make itself offered. Were the filtering pushed down into the package to save a few lines, a connector would be one import away from deciding its own availability, and the design would be over.

The order of calls

Sources: GET /api/connectors and GET /api/sources in parallel, then GET /api/gone-settings and PUT /api/gone-settings for the panel, then POST /api/sources from the new-source form.

Connectors: GET /api/connectors/availability, then PUT /api/connectors/availability.

A source: GET /api/sources/{code}, then on demand POST /api/sources/{code}/test, POST /api/sources/{code}/preview or /run, POST /api/sources/{code}/push/preview or /push , GET /api/runs/harvest/{uid} to poll, POST /api/runs/harvest/{uid}/stop, GET /api/sources/{code}/tagmap, POST /api/sources/{code}/intake-key, GET /api/sources/{code}/inbox?limit=1, POST /api/connectors/{kind}/fragment, and POST /api/sources for every save .

A run: GET /api/runs/harvest/{uid}?full=1&limit=1000, re-polled while it is running, and GET /api/runs/harvest/{uid}/lines.csv as an <a href>.

Connections: GET /api/connections, then POST, POST …/test, PUT …, DELETE … .

GET /api/connectors — one entry per file in connectors/

No parameters, no permission check.

Docstring — why the list is complete rather than filtered:

Everything installed is returned, not just what is offered, and that is deliberate. The Sources screen uses this list to put a NAME against an existing connection's kind — filter it here and every source on a connector somebody stopped offering would read as "no connector", which is the sentence this application uses for a file that has been deleted. Two different situations must not produce the same words.

So the list is complete and each entry says what it is allowed to do. The New source screen shows the cards it should; the endpoints below refuse what they should; neither takes the other's word for it.

Response

{
  "connectors": [          // connectors.describe(mod), plus:
    {"kind": "snowflake", "name": "Snowflake", "blurb": "…",
     "…": "the rest of describe — [not documented] here",
     "availability": {
       "state": "available",         // available | not_offered | withdrawn
       "says": "Offered. Anybody who manages sources may connect to it.",
       "methods": ["keypair", "password"],   // only the ones BOTH allowed and still offered
       "all_methods": true,          // the rule says null, meaning "all of them"
       "note": null,                 // the reason somebody gave
       "decided_by": null, "decided_at": null,
       "undecided": true,            // nobody has ruled on it
       "new": false                  // first seen on THIS request
     }}
  ],
  "newly_installed": ["databricks"]  // kinds this repository had never seen before
}

The three states (STATE_SAYS) — one wording, in one place, "because the screen explains it and the refusals quote it and two wordings that have to agree are one wording that has not been written down yet":

all_methods versus a list — "Null means 'all of them', and the screen says something different for that than for a list that happens to name all of them today — the first survives a connector adding a way in, the second does not."

Side effect (_notice_connectors): every connector this repository has not seen before gets a row with no decision, and that absence is the flag:

Called wherever the list is read, which is how a connector that ARRIVES in an upgrade announces itself without an upgrade step somebody has to remember to write. The row it writes carries no decision… So the default stays "offered", an unattended repository behaves exactly as it did, and the next person to open the screen is told what turned up.

The state is written out explicitly rather than left to the column default, "even though the two say the same thing today. They are two defaults that have to agree, and a check that reintroduced the bug proved it: changing the one above changed nothing anybody could observe, because the row written here quietly supplied the other. One place decides what an unruled connector gets, and this is not it."

GET /api/connectors/availability — the decision screen

Permission: CONNECTOR_ALLOW — "decide which connectors may be used".

Docstring: "Behind the permission, unlike almost every other read in this application. The facts a person needs in order to USE a connector are on /api/connectors and are open to everybody; what is here is the decision itself — who took it, when, and how many sources it would strand — and that is the sysadmin's screen rather than a view of the estate."

Response

{
  "connectors": [          // every kind that is installed, OR has a decision, OR has sources
    {"kind": "snowflake", "name": "Snowflake", "blurb": "…",
     "installed": true,    // false when the file is gone but a decision outlives it
     "auth": [{"code": "keypair", "name": "Key pair",
               "unattended": true,     // can run to a schedule
               "warn": null}],         // a warning to show beside it
     "state": "available",
     "says": "Offered. …",
     "methods": null,      // the RULE's own list — null means all
     "note": null,
     "first_seen": "…",
     "decided_by": null, "decided_at": null,
     "undecided": true,
     "new": false,
     "sources": 3,         // connections on this kind
     "active_sources": 2}
  ],
  "states": ["available", "not_offered", "withdrawn"],
  "newly_installed": [ … ]
}

installed: false is kept rather than dropped: "A connector can be uninstalled, and a decision taken about it outlives the file. Saying so beats showing a row that looks like a connector and is not one."

sources/active_sources are "What a decision would actually cost. 'Not offered' strands nothing; 'withdrawn' stops these running tonight, and the number belongs on the screen before the button is pressed."

PUT /api/connectors/availability — rule on connectors

Permission: CONNECTOR_ALLOW.

Body

{"decisions": [
  {"kind": "snowflake",
   "state": "not_offered",
   "methods": ["keypair"],      // or null/absent to mean ALL of them
   "note": "Pending a security review."}
]}

Docstring: "Rule on connectors. The whole screen at once, like every other workbook… the decision a person actually takes is 'this is the shape of it', and saving it a row at a time leaves a half-applied policy behind whenever something in the middle is refused."

And:

Submitting a connector unchanged is not a no-op. It stamps who looked and when, which is what clears the "nobody has ruled on this" flag — reviewing a connector that arrived in an upgrade and deciding to leave it offered is a real decision, and this is where it is recorded.

Refusals

The last is refused rather than quietly dropped: "A sign-in method the connector does not offer is somebody working from a stale screen, and saving it would record a policy about something that does not exist."

state defaults to "available" when absent. note is stripped and null when empty.

The previous state is read before the upsert "so the log can say what it changed FROM. A subquery in RETURNING would be reading the row this statement is in the middle of writing, which is a question with no good answer."

Response

{"saved": [{"kind": "snowflake", "state": "not_offered"}],
 "summary": "snowflake: not_offered, 1 of its ways in"}

GET /api/sources — the list

No parameters.

Reading is open; opening one is not:

Reading is open, as it is for the metamodel and for workflows. Seeing that a source exists is not the same as running it, and a list nobody without a global role can load is a screen most people find broken. The secret never leaves _public; the actions below are where the permissions are.

Opening one is not: the list is trimmed to a name, a kind and its runs for anybody without SOURCE_VIEW, and the endpoints behind a row refuse.

Response — with SOURCE_VIEW

{
  "sources": [               // ORDER BY code; webhooks excluded
    {
      // _public(row, mod=None) — the secret never leaves that function
      "uid": "…", "code": "SNOW_PROD", "name": "Snowflake production",
      "kind": "snowflake", "auth_method": "keypair",
      "settings": { … },     // the connector's own non-secret settings
      "mapping": {           // the connector's words → yours
        "types": {"table": "TABLE", "schema": "SCHEMA"},
        "relations": { … },
        "offered": ["table", "schema", …]   // what the connector OFFERED, ever
      },
      "scope": {"bring": {"shares": true}, …},
      "on_gone": "mark",     // report | mark | delete
      "schedule": "0 3 * * *",
      "is_active": true,
      "schedule_words": "every day at 03:00",
      "schedule_by": "r.iyer",     // whose authority a scheduled run uses
      "next_run_at": "…",
      "next_run_words": "Fri 03 Oct 2026, 03:00",   // the SERVER's own time
      "has_secret": true,
      "secret_keys": ["private_key", "passphrase"],  // WHICH credentials, never their values
      "secret_unreadable": false,   // this installation's key cannot open it
      "secret_changed_at": "…", "secret_changed_by": "r.iyer",
      "seen_as": {"user": "KRISID_SVC", "role": "KRISID_READER"},  // who the SOURCE said we were
      "seen_at": "…",
      "created_by": "…", "created_at": "…",
      "connector": null,     // describe(mod) only on the detail endpoint

      "last_run": {"kind": "sync", "status": "done",
                   "created": 412, "updated": 18, "linked": 900, "gone": 3,
                   "message": "…", "started_at": "…", "finished_at": "…"},
      "running": {"uid": "…", "kind": "sync", "actor": "r.iyer",
                  "progress": { … }, "started_at": "…",
                  "stop_asked_at": null},   // null when nothing is running
      "known": true          // the connector file is installed
    }
  ],
  "may_open": true,          // _may_bool(SOURCE_VIEW)
  "may_manage": false        // _may_bool(CONNECTION_MANAGE)
}

Response — without SOURCE_VIEW: each source is trimmed to uid, code, name, kind, is_active, plus running as a boolean, plus last_run and known.

next_run_words is in the server's own time, like the schedule itself: "Converted by the browser it read '03:30' on the schedule and '08:00' beside it."

seen_as is deliberately not a secret: "a username is the first thing anybody reviewing a connection asks for, and hiding it protects nothing while making the question unanswerable."

A running row is only reported when its heartbeat is inside RUN_SILENCE, "so somebody arriving at the screen for the first time today can see a run they did not start."

GET /api/sources/{code} — one source

Permission: SOURCE_VIEW (_see_source) — "Opening an established source. The list and the New source form are everybody's; this is the door behind a row."

Refusals: 404 "There is no source called '<code>'."; 400 from _connector when the connector file is missing.

Response — the _public shape above, with connector filled in, plus:

{
  "mapping_added": ["agent"],    // kinds the connector has GAINED since this mapping was saved
  "availability": {
    "state": "available", "says": "…", "note": null,
    "decided_by": null, "decided_at": null,
    "methods": ["keypair"], "all_methods": false,
    "method_still_allowed": true    // the method THIS source uses is still permitted
  },
  "gaps": {                      // ticks the metamodel cannot hold
    "shares": ["DATA_SHARE", "(nothing mapped to 'share')"]
  },
  "grants": "GRANT USAGE ON …",  // the privileges to ask an administrator for
  "runs": [                      // the FIRST PAGE, newest first — see below
    {"uid": "…", "kind": "sync", "status": "done", "actor": "r.iyer",
     "created": 412, "updated": 18, "linked": 900, "gone": 3,
     "message": "…", "started_at": "…", "finished_at": "…",
     "scope": { … }, "progress": { … },
     "heartbeat": "…", "stop_asked_at": null, "stop_asked_by": null,
     "stalled": false,
     "parts": [                  // the passes this run made, oldest first
       {"uid": "…", "kind": "catalogue", "status": "done",
        "pass_name": "SALES_DB", "created": 412, "updated": 0, "linked": 0,
        "gone": 0, "message": null, "started_at": "…", "finished_at": "…",
        "stalled": false}
     ]}
  ],
  "total": 148, "limit": 100, "offset": 0, "days": 90, "has_more": true,
  "running": { … } | null        // the first run that is running and not stalled
}

runs is the first page of GET /api/sources/{code}/runs, in the same shape, so the screen paints without a second request.

availability is on the source and not only on the list "because the person who needs to know a connector has been withdrawn is the person standing in front of a Sync button wondering why it will not work."

method_still_allowed: the method this source signs in with may itself have been disallowed since. It keeps working — "the decision governs what may be CHOSEN" — and saying so "beats letting somebody discover it the next time they touch the form."

gaps (_model_gaps) is answered "before anything runs, so a harvest never gets as far as finding something it has nowhere to put."

grants is "generated from the ticks that are actually on. Read-only and harmless to show — it is a request, not a credential — and it is the thing that turns 'what will this run' from a meeting into a paste." If it cannot be worked out, the field is the string "-- This could not be worked out: <error>".

mapping_added is reported before gaps on purpose: "a kind the mapping has never heard of is not a gap in the MODEL — it is a mapping that predates the connector gaining the kind, and reporting it as the former sends somebody to the metamodel screen to add a type that is already there."

GET /api/sources/{code}/runs — a page of runs

Permission: SOURCE_VIEW.

Query: limit (default 100, capped at 1,000), offset, days.

Response:

{"runs": [ { …a run…, "parts": [ …its passes… ] } ],
 "total": 148, "limit": 100, "offset": 0, "days": 90, "has_more": true}

One row per thing somebody started. A sync, a preview or a push is a row; the catalogue and lineage rows a run writes are parts of it. A walk writes one of each per database, so a twelve-database sync is one row with about fourteen parts — and pass_name says which database each part was, or every <kind> for the pass that judges which of them the source still has.

A part is matched to the newest run that had already started when the part began. That catches the phase row a failure writes before any plan exists — a row no run lists, and the one somebody most wants to see.

The window. days defaults to the repository's own log_keep_days setting, so it follows what an administrator has already decided the useful past is. days=0 is every run ever: harvest runs are never pruned, unlike the log the default is taken from.

The order is started_at DESC, id DESC — the id because two runs in the same millisecond are ordered by nothing without it, and page two would then repeat a row from page one.

POST /api/sources — create or change a source

Permission: CONNECTION_MANAGE — "manage sources".

Docstring: "Holding this is holding the credentials. Deliberately a different permission from running a harvest: widening what a source may pull in is a decision, and refreshing the estate on a Monday is an operation."

Body

Refusals

The state governs CREATING, not editing:

"Not offered" means no new connections; the sources already on it keep running, and a source that keeps running is one somebody has to be able to re-scope, rename and rotate the password on. Refusing the edit would leave a live connection nobody may touch, which is worse than either decision.

And an edit is only an edit of this connector: "Reusing a code to point an existing source at a different system is a creation as far as this decision is concerned, and letting it through on the strength of the code alone would be a way round the whole thing."

Credentials, one at a time (_merge_secret):

A secret typed replaces that one; one left empty is kept; one named in secret_clear is removed. So rotating a Snowflake key's passphrase does not mean pasting the key again, and a screen that cannot show a secret never deletes one because a box was left empty.

The way in decides which secrets belong: switching from a password to a key pair drops the password, since a credential nobody can see and nothing uses is one nobody rotates.

A stored secret this installation's key cannot open is replaced by whatever is typed and otherwise left alone — it may open with the right key, and deleting it because it could not be read would lose it for good.

The secret is stored sealed (vault.seal), and secret_changed_at / secret_changed_by are written only when something actually changed. A system logbook line names which keys were set, replaced or removed — never their values.

schedule_by is "whoever last SET the schedule, not whoever last renamed the source" — a scheduled run uses their authority.

On a pushing card: on_gone is forced to "report" and scope.bring is removed, "because a push reads nothing, so it never notices an absence and the setting can have no consequence… this makes sure the column does not end up holding an answer nobody gave, which would read later as a decision somebody made."

mapping.offered records what the connector offered at the moment of saving, "because somebody may hand in a mapping with a kind left out on purpose, and that decision has to survive" — _freshen_mapping later fills in only kinds that were never offered at all.

Response: {"uid": "…", "code": "SNOW_PROD"}.

POST /api/sources/{code}/test — connect, and say what it can see

Permission: SOURCE_VIEW and HARVEST_RUN — "test a source".

Docstring — the failure mode it exists for:

This exists because of one failure mode: a role that cannot see a schema is told nothing, not refused, so an under-granted connection produces a harvest that succeeds and is wrong. "Nothing visible" and "not allowed to look" are reported as different sentences on purpose.

Refusals: 404 no such source; 409 the withdrawn sentence (_refuse_withdrawn); 409 from _credentials when the secret cannot be unsealed.

The test uses "the same scope the harvest will be given, including which way in was chosen. Testing with a different one from running is how a connection passes its test and then fails at 2am."

Response

{
  "ok": true,                 // rep.ok AND no model gaps
  "lines": [                  // [level, sentence]; "blocked" is the level that matters
    ["found", "3 databases, 41 schemas, 412 tables"],
    ["blocked", "'shares' is ticked, and the model has nowhere to put it: DATA_SHARE."]
  ],
  "counts": {"tables": 412, …},
  "gaps": { … },
  "identity": {"user": "KRISID_SVC", "role": "KRISID_READER"}
}

A failure to connect is a 200 with {"ok": false, "lines": [["blocked", "<why>"]], "counts": {}} — not an HTTP error.

identity is stored on the connection as seen_as/seen_at, but only on a test that got through: "a failed connection has nothing to report, and overwriting a good answer with an empty one would lose the last thing anybody knew."

POST /api/sources/{code}/preview and POST /api/sources/{code}/run

Both are one line over _start(code, kind, what, x_actor).

Permission: SOURCE_VIEW and HARVEST_RUN.

Docstring: "Everything that must be answered before a run begins, then the thread. The refusals stay in the request: a person who may not run a harvest, a withdrawn connector, a model that cannot hold what is ticked and a run already in flight are all answers somebody is waiting for, and answering them from a background thread would mean a screen that says 'started' and a failure nobody sees for a minute. Everything after that — which is all of the time — happens in the thread."

Refusals

The pushing-card refusal is there because "The screens do not offer it — but a POST is a POST, and this one got through: a real repository had a sync and a catalogue run against a push card, both failed, both for a reason nobody could read."

The model-gap check is skipped for a preview and enforced for a sync.

Response: {"run": "<run uid>", "status": "running"}.

Scheduled runs (_start_scheduled) go through the same refusals, under the authority of whoever set the schedule, and every refusal is written to the catalogue log channel "because nobody is at the screen to read it." One extra refusal exists only there: 409 "<who>, who set the schedule, is not an active user any more" and 403 "<who>, who set the schedule, may no longer run a harvest (HARVEST_RUN)".

One bug worth knowing about, in _run_schedules: the loop used to be a list comprehension, and a 404 from one deleted source aborted it — "So a nightly sync could be missed for a night, or for ever, because a different source was deleted — and nothing anywhere said so." Each source is now caught in its own right.

POST /api/runs/harvest/{uid}/stop — ask a run to stop

Permission: SOURCE_VIEW and HARVEST_RUN — "Whoever may run one may stop one. Somebody who can start a sync and cannot end it has half a button."

Docstring:

Asked, not killed. The thread looks between items while it reads and between phases while it writes, so whatever a phase committed stays committed and nothing is half-written — the same contract Ctrl-C on rebuild-context has, which is what makes this safe to put on a screen.

Until this existed the only way out of a harvest doing the wrong thing was to kill the application, which ends every other run with it, leaves the row saying 'running' for ever, and is not available at all to the person whose repository it is.

Refusals: 404 "There is no such run."; 409 "That run is already <status>. There is nothing to stop." — "saying 'stopped' about a run that had already finished is how somebody comes back tomorrow believing they cancelled something they did not."

Response

{"asked": true,
 "said": "Asked it to stop. It finishes what it is in the middle of and ends within a few seconds — whatever it had already written stays written."}

GET /api/runs/harvest/{uid} — one run

Permission: SOURCE_VIEW — "read a run record".

Why there is a permission at all — the most pointed note in the file:

A run record is the whole plan: every asset key, every name, every field that changed and what it changed from. It is the most detailed thing this repository will hand anybody about a source's estate, and it went out to any signed-in account at all — these two endpoints declared x_actor and then never asked who it was, while every neighbouring source endpoint asked. Measured: two accounts holding neither SOURCE_VIEW nor ACCESS_VIEW read a full record and downloaded its CSV.

Query parameters

full=1 is the run's own page asking for lines, a page at a time… Everything else — the source screen, the progress poll — gets the counts and no lines at all.

It used to be one of two answers, and the second was every line. The poll asks this every two seconds to draw one sentence, and SELECT h.* decompressed the whole plan — a hundred megabytes on a large estate — parsed it into Python, counted it and threw the lines away, in the process that was also running the harvest. The lines are rows now, so a count is a count and a page is a page.

The detail column is not in the main SELECT: it is fetched separately and only for the parts that are small — the phase list, the blockers, the warnings, the counts — or for a run recorded before schema v67, whose lines genuinely are in there.

The ceiling exists "because full=1&limit=2000000 is a request somebody can make by hand."

Refusal: 404 "There is no such run."

Response

{
  "uid": "…", "connection_id": 4, "kind": "sync",
  "status": "done",                  // queued | running | done | failed
  "actor": "r.iyer",
  "scope": { … },
  "created": 412, "updated": 18, "linked": 900, "gone": 3,
  "message": "…",
  "progress": {"step": "writing 412 tables", …},
  "heartbeat": "…", "started_at": "…", "finished_at": "…",
  "stop_asked_at": null, "stop_asked_by": null,
  "source": "SNOW_PROD", "source_name": "Snowflake production",
  "stalled": false,                  // nothing said for RUN_SILENCE

  "phases": ["catalogue", "lineage"], // THE ORDER THEY HAPPENED IN
  "by_phase": {
    "catalogue": {
      "blockers": [ … ], "warnings": [ … ],   // never paged, never cut
      "counts": {"added": 412, "changed": 18, "removed": 3},
      "removed": [ … ],              // only when there are few enough; see below
      "added": [ … ], "changed": [ … ]   // only with full=1
    }
  },

  "full": true,
  "paged": true,                     // only with full=1 on a v67+ run
  "after": 0, "limit": 1000,
  "line_counts": {"catalogue": {"added": 412, "changed": 18, "removed": 3}},
  "lines_total": 433,

  "seconds": 41.2                    // only when it has finished
}

phases is read from the stored order, not from the object's keys: "jsonb sorts its keys by length and then alphabetically, so reading the order off the stored object put lineage before cataloguing."

Every phase always carries all three line buckets, even when the page has nothing in one of them: "A caller reading by_phase[ph]["added"] should not have to know whether the page it asked for happened to contain an add — that was the shape before the lines were paged, and a missing key is a KeyError somewhere rather than an empty list where one belongs."

What _capped cuts, and what it never cuts — this applies only to a run recorded before v67:

Not a shorter list — no list at all. The first version of this sent a sample of 25 per kind and the screen drew them, and on a real sync that is still a wall of text under the one sentence somebody wanted: 2,397 to add. Every line is in the run's own record, on its own page and in a file, which is where somebody who wants a line goes looking.

Blockers and warnings are NOT cut, ever. They are the part that stops a run or changes what you would do next, and there are never thousands of them.

Neither is a SHORT list of what the source no longer has. "no longer there: 3 assets" is the one count on this screen that changes what happens to something — on a real sync those three are marked Missing in Source, or deleted, depending on what the connection was told to do — and a number with no names is the wrong shape for it: the only response available is to open another page and look. Three names cost nothing. Three thousand are a wall, and at that point the count IS the answer and the file has the rest.

GET /api/runs/harvest/{uid}/lines.csv

Permission: SOURCE_VIEW — "download a run record". "The same door as the run's own page. A download is not a lesser read."

No query parameters.

Docstring: "The screen shows counts and a sample; the page shows all of it; this is for the third question, which is not 'what happened' but 'prove it' — a change somebody has to sign off, diff against last month, or send to whoever asked why a column disappeared."

Refusal: 404 "There is no such run."

Response: a streamed text/csv, filename krisid-run-<source-lower>-<kind>-<YYYYMMDD-HHMM>.csv, with these columns:

phase, action, kind, key, field, was, now, detail

action is add, change or gone; a stored blocked action is written as gone.

Why it streams:

It used to build the whole parsed document and the whole finished CSV in memory before sending a byte, so a download of a large run was two multi-hundred-megabyte allocations in the process that also runs harvests — and the first byte arrived only after both.

A generator and a SERVER-SIDE cursor: Postgres holds the result set and hands over a few thousand rows at a time, this holds one CSV row, and the browser starts receiving immediately.

And a bug worth quoting, because the failure mode is silent:

By NAME, not by position. This connection hands back dictionaries, so r[0] was a KeyError — raised inside a streaming generator, where it does not reach a handler and the download simply arrives empty. A file that is silently empty is worse than a download that fails.

GET /api/gone-settings

No permission check beyond a session.

Response: {"days": 30, "may_configure": false}.

What it used to carry, and why it does not:

This used to report how many assets are marked and how many are past the cut-off, because a number makes a setting mean something. It also scans the whole asset table on a JSONB key, on a screen people open to look at their sources — and on a real estate "how many" is both expensive to ask and not an answer anybody can read.

The docstring above it still says "the count is on the same answer as the setting on purpose", which the code no longer does — that is the one place in this file where the docstring is behind the body.

PUT /api/gone-settings

Permission: ACCESS_MANAGE — "change how long gone is gone".

Body: {"days": 30}.

Refusals: 400 "A number of days."; 400

"Between nothing and ten years. Nought means whatever is marked may be cleared
 however recently it went, which is a decision somebody can make — and ten years
 is long enough that the setting has stopped being one."

So 0 <= days <= 3650.

Why it is a setting at all (_clear_days): "the right answer is a property of the client's estate: a nightly sync against a stable warehouse is confident after a week, and a development account whose schemas CI drops and rebuilds wants a month. Nought is allowed and is not the default — the case the delay defends against is a harvest that read a thinner estate than it should have and marked a thousand present things as missing."

Response: the same shape as the GET.

POST /api/sources/{code}/intake-key — a lineage intake key

Permission: CONNECTION_MANAGE — "manage sources".

Refusals: 404 no such source; 400 "<Name> does not receive anything — it goes and reads its source. An intake key would be a key to nothing." (the connector has no WANTS_INBOX).

Docstring:

Once, and this is the only place it is ever readable. It is stored hashed, exactly as a password and an invitation link are, so nothing in this application can show it again — which is the property that makes losing it safe and makes finding it in a backup useless.

Making a second one replaces the first. There is deliberately no way to have two: "which of these four keys is the one Airflow uses" is a question nobody should have to answer during an incident.

Response

{"key": "<the raw key, this once only>",
 "made": true,
 "said": "This is the only time this key is shown. It is stored hashed, so nothing here can show it again — make another one if it is lost, which replaces this one."}

GET /api/sources/{code}/inbox — what has arrived

Permission: SOURCE_VIEW and HARVEST_RUN — "look at a source".

Query parameters

Docstring:

The failure a receiving source has that a pulling one does not: a producer pointed at the wrong URL does not fail HERE. It fails at the producer, and the first sign is an inbox that looks exactly like a quiet week. This is the screen that tells those apart.

Response

{
  "has_key": true, "key_made": "…",
  "events": 18402, "last": "…",
  "namespaces": 3, "jobs": 41,
  "recent": [            // ORDER BY received_at DESC, id DESC
    {"received_at": "…", "source": "endpoint",
     "event_type": "COMPLETE", "job_namespace": "airflow-prod",
     "job_name": "load_customers", "producer": "…"}
  ]
}

GET /api/sources/{code}/tagmap — what becomes what

Permission: SOURCE_VIEW and METAMODEL_VIEW — "see what a push would write".

Refusal: 400 when the card does not push (_push_row):

"'<code>' is a <Name>, which reads. Pushing tags is a different card: it holds
 different credentials and needs a role that may write, which is the whole
 reason the two are not one."

Plus 400 when the card's saved search no longer exists (see below).

Response

{
  "mappings": [
    {"uid": "…",
     "source": "a:CONTAINS_PII",           // which fact becomes the tag
     "label": "Contains PII",              // or the error sentence from tag_push.Bad
     "tag": "GOVERNANCE.TAGS.PII",
     "create_approved": true,              // KRISID may create this tag in Snowflake
     "approved_by": "r.iyer", "approved_at": "…"}
  ],
  "facets": [ … ],         // facets.catalogue(cur) — what `source` may be
  "search": "Finance only",// the saved search this card is pointed at, by name; "" for none
  "may_push": false,       // _may_bool(TAG_PUSH)
  "object_types": [ … ]    // mod.OBJECT_TYPES
}

Why a saved search (_search_conditions): "A push covers the whole account unless somebody names a saved search, which is the answer to 'push the Finance domain and not the rest' without inventing a second way of saying which assets somebody means. The search is the one they already wrote on Ask."

If the named search is gone: 400 "This card is set to push what the saved search '<name>' finds, and there is no search by that name. Either save one, or clear the box to push everything this account harvested." If it cannot be compiled: 400 "The saved search '<name>' cannot be turned into a condition on this estate: <facets.Bad>".

POST /api/sources/{code}/tagmap — say what a push writes

Permission: CONNECTION_MANAGE — "say what a push writes".

Body

Refusals

Response: {"uid": "<new uid>"}.

DELETE /api/sources/{code}/tagmap/{uid}

Permission: CONNECTION_MANAGE.

Docstring: "Stop sending it. What was already written stays written. Deliberately: this connector does not unset tags, and removing a mapping is not a back door to doing so. The record of what was pushed is kept too, so the tag can still be accounted for afterwards."

Refusal: 404 "There is no such mapping on this card."

Response: {"ok": true}.

POST /api/sources/{code}/tagmap/{uid}/approve

Permission: TAG_PUSH — "allow a tag to be created".

Body: {"approved": true} — defaults to true when absent.

Docstring: "A separate act from mapping to it, and from running the push. Creating an object in somebody's warehouse is the one thing here that leaves something behind which was not there before, so it is ticked by a named person at a known moment and the row remembers both."

Setting it to false clears approved_by and approved_at.

Refusal: 404 "There is no such mapping on this card."

Response: {"ok": true, "create_approved": true}.

POST /api/sources/{code}/push/preview and POST /api/sources/{code}/push

Both are one line over _start_push(code, kind, x_actor).

"Previewing needs only the permission to look; pushing needs the one whose consequences are in somebody else's warehouse."

Why it is not the same function as _start: "what must be true before a push differs from what must be true before a harvest, and a single function with a flag through it would be one place where a rule meant for reading quietly governed writing."

Refusals

Response: {"run": "<run uid>", "status": "running"}.

The push itself (_do_push) is worth one note: it used to hold a Postgres transaction open across every Snowflake round trip, "which on a large push is minutes… A connection sitting idle in transaction holds back the xmin horizon for the WHOLE database, so nothing that dies while it waits can be vacuumed — on a repository being harvested at the same time, which is precisely when a push runs." A tag nobody ticked is not created, and the rows that need it are not attempted either — "a SET against a tag that does not exist is a failure per row rather than one sentence."

DELETE /api/sources/{code}

Permission: CONNECTION_MANAGE — "manage sources".

Not called by any screen; see the final section.

404 "There is no source called '<code>'." Response {"deleted": true}.

GET /api/connections — what this repository can call out to

No permission check to read, and that is a decision:

Everybody signed in may see WHAT this repository can call out to. Not a permission, deliberately. A list of the places a repository is able to send things to is a governance fact — the same kind of fact as the list of sources it reads from, which has always been everybody's to see. What is behind a permission is the part that lets somebody USE one: the rest of the URL, the token, when it last ran and what it said, and every button.

The trimming happens here rather than in the screen, because a field the API returns is a field anybody can read whatever the screen draws.

Webhook connections only (kind = 'webhook'), ordered by code.

Response — with ACCESS_MANAGE

{
  "connections": [
    {"uid": "…", "code": "SLACK_GOV", "name": "Governance channel",
     "kind": "webhook",
     "url": "https://hooks.slack.com/services/T0/B0/XXXX",   // the WHOLE url
     "is_active": true,
     "created_by": "…", "created_at": "…",
     "has_secret": true,
     "secret_changed_at": "…", "secret_changed_by": "…",
     "used_by": 3,                 // workflow actions pointing at it
     "last_run": {"status": "done", "detail": "200 from hooks.slack.com",
                  "updated_at": "…"}}
  ],
  "may_manage": true
}

Response — without it: each row is {uid, code, name, kind, is_active, used_by, url, masked: true}, where url is only the scheme, host, port and a trailing /… if there was a path (_connection_host):

A Slack or Teams webhook carries its secret IN THE PATH — the URL is the credential, and handing the whole of it to everybody signed in would be handing them the token in a different box. The host is the part that answers the question the list exists to answer: what is this repository able to call out to? The path is the part nobody but a sysadmin needs.

Port kept: calling out to an internal service on 8443 is a different fact from calling out to the same host on 443, and it is not a secret.

last_run is withheld too, "because a failure detail comes back from the far system and quotes the request, so showing it would undo the masking by the back door."

POST /api/connections

Permission: ACCESS_MANAGE — "manage connections".

Body

Docstring: "Separate from the workflow because a definition is exported, copied and versioned — and a token that travels with it is a token that ends up in a spreadsheet. The secret is never read back out of this API."

Refusals: 400 "A connection needs a code and a URL."; 400 "A connection's URL has to be http or https."

Response: {"uid": "…", "code": "SLACK_GOV"}.

POST /api/connections/{code}/test

Permission: ACCESS_MANAGE — "test connections".

Docstring: "Call it once, with nothing but a test in it, and say what came back. The same request a workflow action makes — the same header, the same shape of body — because a test that goes a different way proves a different thing. It is marked as a test so whatever receives it can ignore it."

Refusals: 404 "There is no connection called <code>."; 409 "“<Name>” is switched off, so nothing would be sent to it."; 409 from vault.Unreadable.

What is sent: POST with content-type: application/json, an authorization: Bearer <token> header if a token is stored, a 10-second timeout, and this body:

{"test": true, "connection": "SLACK_GOV",
 "text": "A test from KRISID, sent by Rohan Iyer.",
 "sent_at": "2026-10-01T14:22:31+00:00"}

Response — always a 200:

{"ok": true, "detail": "200 from hooks.slack.com"}

or {"ok": false, "detail": "HTTP 404 from hooks.slack.com: <first 200 bytes>"} or {"ok": false, "detail": "URLError: <…>"}.

A system logbook line records the attempt either way, at level info or warn.

PUT /api/connections/{code} — switch it on or off

Permission: ACCESS_MANAGE — "manage connections".

Body: {"is_active": false} — required.

Docstring: "Switch one on or off, without deleting it. A workflow that calls a connection which is off records that it did not go, rather than failing silently — which is why switching off is worth having at all."

Refusals: 400 "Nothing to change."; 404 "There is no connection called <code>."

Response: {"code": "SLACK_GOV", "is_active": false}.

DELETE /api/connections/{code}

Permission: ACCESS_MANAGE — "manage connections".

No refusal for a code that does not exist — the DELETE simply matches nothing.

Response: {"deleted": true}.

Note the asymmetry with used_by on the GET: the list says how many workflow actions depend on a connection "so nobody deletes the one thing an approval depends on without being told", but this endpoint does not itself refuse.