Docs › Reference

Lineage and the explorer

The API calls behind lineage and the explorer.

Part of the HTTP API reference.

Three routes:

The order of calls on the Lineage screen

1. GET /api/lineage-views?uid=<uid> — every saved reading that applies here, narrowest first.
2. GET /api/lineage-views/{view}?on=<uid> — only when the address carried a view id.
3. Then one of two, decided by the mode: - connections mode → GET /api/graph/{uid}?depth=<hops> - lineage mode → GET /api/lineage/{uid}?direction=…&depth=…&grain=…
4. On demand: POST /api/lineage-views, PUT /api/lineage-views/{uid}, DELETE /api/lineage-views/{uid}.
5. On demand, download: GET /api/lineage/{uid}/export?….

The Explorer makes exactly one call: GET /api/explore/{uid}.

Two questions, two endpoints, on purpose:

/api/lineage — where did this come from, and what does it affect. Directed. Walks only the relations the metamodel declares as carrying data, and treats realisation as a free step sideways rather than a hop, so a Data Entity can reach the tables that realise it without appearing to be upstream of them.

/api/graph — what is this tangled up with. Undirected, every relation of every kind.

Both return the same shape — nodes and edges — so one drawing surface can render either. Which relations count as lineage is declared in the workbook, not decided here; there is no list of relation codes anywhere in this file.

GET /api/lineage/{uid} — where it came from and what it affects

Query parameters

Refusals: 400 "direction must be one of upstream, downstream, both"; 400 "grain must be one of column, table, business".

The three grains (GRAINS):

Three readings of one graph, coarsening in this order: the edges as recorded, the same edges rolled up to the objects that hold them, and the same edges again lifted onto whoever speaks for those objects in business words.

"Business lineage" is that third one and nothing else. It is not a second graph somebody draws by hand beside the technical one — that graph is right on the day it is drawn and wrong from the next harvest onwards, with nothing to say which of the two to believe. Which assets are business ones is declared, not decided here: asset_type.origin, in the workbook.

Response

{
  "focus": {"uid": "…", "name": "EMAIL", "type_name": "Column"},
  "mode": "lineage",
  "direction": "both", "depth": 4, "grain": "column",

  "nodes": [                     // ordered by depth, then type display order, then name
    {"depth": 0,
     "side": "here",             // here | upstream | downstream
     "arrived_by": "start",      // how the walk reached it
     "uid": "…", "name": "EMAIL", "external_key": "…",
     "type_code": "COLUMN", "type_name": "Column",
     "type_group": "Physical", "origin": "harvested",
     "inside": {                 // what this would become at a coarser grain; null if nothing
       "uid": "…", "name": "CUSTOMERS",
       "type_code": "TABLE", "type_name": "Table", "type_group": "Physical"
     },
     "chain": [                  // everything it sits inside, NEAREST FIRST
       {"uid": "…", "name": "CUSTOMERS", "type_code": "TABLE",
        "type_name": "Table", "type_group": "Physical"},
       {"uid": "…", "name": "SALES", "type_code": "SCHEMA", …}
     ]}
  ],

  "edges": [                     // ONLY between nodes that came back
    {"from": "…", "to": "…",
     "kind": "flow",             // "flow" | "same thing"
     "label": "feeds",           // at business grain: "feeds, through 4 technical steps"
     "method": "parsed_ddl",     // null at business grain
     "confidence": 0.9,          // null at table and business grain
     "weight": 1}                // column: 1; table: via_edges; business: via_steps
  ],

  "not_movement": [              // what this picture is deliberately NOT showing
    {"type_name": "Data Domain", "n": 3}
  ],

  "no_business_name": [          // business grain only; the busiest 12
    {"uid": "…", "name": "STG_ORDERS", "type_name": "Table", "edges": 37}
  ],
  "no_business_name_total": 44,
  "focus_has_no_business_name": false,

  "truncated": true              // something sits AT the depth limit
}

inside is not a lookup of anything called a column:

It is rolls_up_to and not a lookup of anything called a column, because this file must not know that a column is a column. That function already answers the same question for object grain: a leaf (something that contains nothing) folds into whatever contains it, and anything that contains something stays as itself. So a client whose workbook calls them FIELDs, or who puts partitions inside tables, gets the folding with no code change — and the folded picture agrees with the object-grain one because both are the same function.

chain is one query for every node, not one each: "A picture is a hundred boxes at its worst and a chain is four deep, so this is four hundred rows and one round trip" (_add_chains).

not_movement — why it exists:

A data domain owns a table; that is not data moving, so the domain is not on the lineage and never will be. But a diagram that silently omits two thirds of what an asset is attached to reads as an incomplete diagram rather than a focused one. Count what was left out, by type, and let the screen say so.

no_business_name — at business grain the omission is the point:

An asset that moves data and that nothing in the business layer speaks for cannot be drawn here — there is no business name to write in the box. Dropping it silently would make this diagram a lie by omission. Naming it turns the same omission into the most useful thing on the screen: the list of things carrying data through this business that nobody has yet described in the business's own words.

focus_has_no_business_name exists because "the focus itself may be one of them, and then the empty picture has a cause worth saying out loud rather than leaving the reader to conclude that nothing flows through it."

The twelve are the twelve that carry the most, ordered by edges — how many movements in this picture touch them — and no_business_name_total counts all of them. It used to be ORDER BY t.display_order, a.name, which on a 613,461-asset repository meant the twelve shown were the twelve whose names begin with A, and the view the reader was looking for was never among them however much data went through it. A sample chosen by spelling is a sample chosen at random, on the one list this screen calls the work outstanding. The Excel export re-asks without the limit and carries every one of them: a screen may summarise, a file may not.

Two short-circuits, both measured.

An asset with no lineage at all does not run the walk (_has_lineage):

0076's guard already empties every branch inside the function, but a SQL function's body is planned on each call, and planning six references to views over a ten-million-row table costs about six hundred milliseconds whether or not a single row is then read. So the cheap question is asked first and the expensive one is skipped: an asset with no lineage is its own one-row answer, in the same shape, from the same query. On a harvested estate that is 10,001,054 assets out of 10,001,150.

A one-node answer asks for no edges at all:

each of these reads a view over the whole of relation, and the one-element array does not make the scan smaller — v_lineage_edge_rolled filtered to a single id still took 0.88 seconds at ten million. An asset cannot be linked to itself… so a single-node answer has nowhere for an edge to go.

Same-thing edges come back separately rather than as flow edges somebody has to filter: they "are drawn differently and carry no direction." They are de-duplicated by the unordered pair.

GET /api/graph/{uid} — what it is tangled up with

Query parameters

Response

{
  "focus": {"uid","name","type_name"},
  "mode": "graph", "depth": 2,
  "nodes": [ … ],                 // same node shape as lineage, with `inside` and `chain`
  "edges": [
    {"from": "…", "to": "…",
     "kind": "flows to",          // coalesce(rt.lineage, 'structure')
     "label": "holds",
     "relation_code": "TABLE_HOLDS_COLUMN",
     "method": "declared", "confidence": null, "weight": 1}
  ],
  "cut": false,                   // the picture filled up BEFORE the hop limit
  "cap": 1200,                    // GRAPH_ASSETS
  "truncated": true               // something sits AT the hop limit
}

cut and truncated are not the same fact: "truncated means something sits at the hop limit, which the hop picker can undo. cut means the picture was full before the hop limit was reached, which it cannot."

Why there is a cap on the answer at all — the longest comment in the file, and the measurements are the argument:

/api/graph   depth=1          2 assets        2.0 kB     0.05s
             depth=2         50 assets       57.0 kB     0.07s
             depth=3        250 assets      230.9 kB     0.23s
             depth=4     10,101 assets   11,550.7 kB     7.63s
             and depth=6 was allowed

/api/explore on the same column, no depth
  the component                        10,001,010 assets
  the walk alone, counted in SQL                    58.9s
  the request itself                               222.2s, then
    HTTP 503 "Cannot reach Postgres … consuming input failed"
  kernel:  Memory cgroup out of memory: Killed process (postgres)
           anon-rss 5,749,560 kB

One GET. The backend holding the result set reached 5.7 GB, the OOM killer took it, and what the person saw was their repository reporting that the database could not be reached. The earlier lesson — an unbounded version of this "hung the server hard enough to need its backends killed" — had been answered with a limit on depth, which is the dimension that was never the problem.

So the cap is on the ANSWER. It is also the number above which a diagram stops being a diagram: a person reads a hundred boxes and skims a thousand, and nobody reads ten thousand. It is reported rather than applied quietly, because both screens carried a sentence a quiet cap would have turned into a lie — "N assets connected", and "All N assets are loaded, nothing has been left out".

GRAPH_ASSETS = 1200.

How the walk behaves at the edge (_walk_out): breadth first, a hop at a time, the two ends asked separately "because a condition on either of two columns is a condition on neither index — the shape that took speaks_for from 6.4 seconds to 87 milliseconds in 0074." Where a level does not fit, the ids that fit are taken in id order, "so the same request twice draws the same picture; an arbitrary half of a level would move under somebody's cursor." Taking part of a level rather than none of it is deliberate: "A table with three thousand columns would otherwise arrive with none of them, and a neighbourhood with none of its contents is not a safer answer, only an emptier one."

A note on cap=None in that function's signature: "a default argument is bound once when the function is defined: written the obvious way the cap became two numbers, and changing GRAPH_ASSETS moved one of them. Found by the check that moves it."

GET /api/explore/{uid} — the whole connected component

Query parameters

Docstring:

/api/graph answers "what is near this", and near is a number you pick. This answers "what is this part of", which has no number in it: the whole connected component, however far it runs. A repository is not one component — an unlinked business term is its own island — so this is bounded by the data rather than by a limit, and the client is told how big it is before it tries to draw it.

depth narrows that, and defaults to 0, meaning the whole thing — the question this endpoint exists to answer. Narrowing is for reading rather than for cost: it always says how far the component actually runs and how many assets were left outside, so a smaller number can never quietly look like the whole picture.

The walk always goes to EXPLORE_HOP_CAP = 24 hops and is narrowed here, "so that 'N further out, not shown' keeps the meaning it had: it counts what this walk actually found and left off the picture… Nothing readable is 24 hops wide." The answer is still capped at GRAPH_ASSETS = 1200.

Response

{
  "focus": {"uid","name","type_name"},
  "mode": "explore",
  "nodes": [                      // sorted by type display order then name, NOT by distance
    {"hops": 2,                   // this screen calls the distance `hops`
     "uid": "…", "name": "…", "external_key": "…",
     "type_code": "…", "type_name": "…", "type_group": "…", "origin": "…",
     "inside": { … } | null, "chain": [ … ]}
  ],
  "edges": [
    {"from": "…", "to": "…", "kind": "flows to" | "structure",
     "label": "holds", "relation_code": "…", "method": "declared",
     "contains": true}            // the estate tree's own rule: 1:N out, 1:1 back
  ],
  "depth": null,                  // the narrowing asked for, or null for "the whole thing"
  "reach": 7,                     // how far the component actually runs, in hops
  "component_assets": 1200,       // everything the walk found
  "beyond": 812,                  // found but outside the narrowing
  "cut": true,                    // the walk itself stopped early
  "cap": 1200,
  "repository_assets": 10001150,  // count(*) FROM asset
  "truncated": false              // always false here
}

Nodes are sorted by type and not by distance on purpose: "a diagram whose boxes moved would read as changed data."

contains comes back on the edge rather than being worked out in the browser from type names: "It is the same cardinality rule the estate tree uses (one parent, many children), so a new containing type declared in the workbook collapses on the diagram with no code change here."

When cut is true the screen must change its words, because "'N assets connected' and 'nothing has been left out' are both false here."

GET /api/lineage/{uid}/export — the picture, as a file

Reached as an <a href> from the canvas. This endpoint calls lineage itself rather than repeating its query — "so the file and the screen cannot answer differently; the commonest way an export becomes a lie is a second query that drifts from the first."

Query parameters

Refusals: 400 "The file is xlsx or csv."; 400 "part must be one of assets, links, omitted".

The three rules in the section header, which are what the query string is for:

What the picture leaves out travels with it. A diagram omits things on purpose; a spreadsheet of the same view that dropped them silently would be read as everything there is.

Collapsing a layer is a way of READING the picture, so it marks rows rather than removing them. A file that had quietly lost a third of its rows because somebody folded a band on screen would be a trap: the fold is one click to undo and impossible to see in a file a week later.

A bare name in collapsed is read as a layer "so a link made before collapsing reached asset types still says the right thing." collapse=hide is honoured "from a link made when the mode was one switch over the whole screen."

The three sheets

Assets — every node, with what the picture did with it:

Links — every edge:

From, From type, To, To type, Kind, How it is written (the label), Recorded by (the method), Confidence, Underlying links (the weight).

Left out — three columns, Why it is not on the picture, What, In other words, with a row per reason:

• "What is not data movement" / "3 × Data Domain" / an explanation
• "Nothing at the business layer speaks for it" / "STG_ORDERS (Table)" / …
• "Collapsed by the reader" / the band or type / hidden-vs-folded
• "Folded into what holds it" / "CUSTOMERS (+41), ORDERS (+12)" / …
• "Beyond the hop limit" / "4 hops" / …

Response

• fmt=xlsx → one workbook with all three sheets, filename krisid-lineage-<focus-slug>-<grain>-<YYYY-MM-DD>.xlsx, and a subtitle naming the counts, direction, hops, what was folded and the export time, plus the words "a report, not an upload".
• fmt=csv → the single sheet named by part, filename krisid-lineage-<focus-slug>-<grain>-<YYYY-MM-DD>-<part>.csv, with a UTF-8 BOM.

On folding being written twice: _nest_tiers, _first_level and _holder_at reimplement what the browser does, "on purpose and kept honest by a check rather than by hope: verify_phase0.py groups a picture through the API and through the export and insists the two agree. Sharing one implementation would mean the browser asking the server every time somebody moved a dropdown, which is a round trip for a question it can already answer from what it has."

One bug worth knowing about, recorded in the code: a uid is a UUID object here and a string in the query string, and said.get(UUID(...)) on a dict keyed by strings "is a lookup that misses every time and says nothing" — hence the str.

GET /api/lineage-views — the saved readings that apply here

Query parameters: uid — required, the asset being looked at.

Docstring: "Every saved reading that applies to this asset, narrowest first. The order IS the precedence, so the screen does not get to have its own opinion about which one opens: it takes the first."

The order is: views pinned to this asset first, then to this type, then everywhere; within each, yours before other people's, then most recently updated.

What makes these different from saved views:

Scope is which id is set. An asset, a type, or neither — and neither means everywhere. There is no scope word to fall out of step with the ids.

The narrowest applicable one opens the screen. "Every data product opens at business level" and "except this awkward one, which opens at two hops" are both sayable, and the second wins where it applies.

Private until shared. A saved view over a list is a team's agreement about how to read a type; a diagram setting is usually somebody's own way of working, so it starts private and its owner decides.

Visibility is can_see_view(view_id, username, asset_id) in the database.

Response

{
  "views": [
    {"uid": "…", "name": "Business level, both ways",
     "description": "…",
     "settings": { … },           // whatever the screen saved; opaque to the server
     "created_by": "r.iyer", "updated_at": "…",
     "asset_uid": "…",            // null unless scoped to an asset
     "asset_name": "CUSTOMERS",
     "type_code": "TABLE", "type_name": "Table",   // null unless scoped to a type
     "scope": "asset",            // "asset" | "type" | "everywhere"
     "mine": true,
     "shares": [                  // who else can see it
       {"kind": "everyone"|"person"|"team"|"role",
        "code": "a.khan"|"DATA_GOV"|"OWNER"|null,
        "name": "A. Khan"|"Data Governance"|"Owner"|"everyone"}
     ]}
  ],
  "type": {"code": "TABLE", "name": "Table"},
  "me": "r.iyer",
  "may": {"create": true, "share": false},
  "share_options": {              // EMPTY unless may.share
    "people": [{"code": "a.khan", "name": "A. Khan"}],
    "teams":  [{"code": "DATA_GOV", "name": "Data Governance"}],
    "roles":  [{"code": "OWNER", "name": "Owner", "scope": "resource"}]
  },
  "opens_with": "<uid>"           // the first view's uid, or null
}

may.create/may.share are may(username, 'VIEW_CREATE'|'VIEW_SHARE', asset_id) — judged against the asset, "which is what makes a steward of one domain able to save a view about it without holding anything repository-wide."

share_options is sent only to somebody who may share: "A screen that lists every colleague to a reader who cannot use the list is telling them something they did not ask for."

Shares are fetched in one query for all the views: "a person with thirty saved views should not cost thirty round trips to draw a picker" (_dress_views).

GET /api/lineage-views/{uid} — what a link opens

Query parameters: on — an asset uid, optional.

Docstring:

A link is not the same question as the picker. The picker asks "what applies to this asset"; a link says "this one, please", and it may be a view saved somewhere else entirely that somebody pasted into a ticket. So this fetches it by id and answers the only question that still matters: may this person see it.

on is the asset the reader is standing on, because a role share is judged there — the stewards of THIS table see it, whoever they are.

Refusals

The 403 names whose it is rather than hiding it: "knowing whose it is turns a dead end into a message you can send to one person."

Response: {"view": { … }} — one view in the shape above.

POST /api/lineage-views — save a reading

Permission: VIEW_CREATE, judged against the asset named by uid (or globally when there is none). VIEW_SHARE, also judged there, if shares is given.

Body

Refusals

SHARE_KINDS = ("everyone", "person", "team", "role"). A person share is matched on app_user.username, a team on user_group.code, a role on app_role.code.

Response: {"uid": "<new uid>"}.

PUT /api/lineage-views/{uid}

Permission — and the order of the two checks is load-bearing:

Who it is shared with is judged BEFORE the manager gate, not after. Asked the other way round, a manager asking to share somebody's private view is refused for not holding a permission — which reads as "get the permission and you may", and the answer is that nobody may. Whose view it is comes first; what you hold comes second.

So: shares in the body and not the owner → 403 "“<name>” is <creator>'s, and who else can see it is theirs to decide. You can rename it or tidy it away, with METAMODEL_MANAGE, but not hand it to anybody." Otherwise, not the owner → METAMODEL_MANAGE.

Body — every field optional; at least one of name, settings, description, shares.

Refusals: 404 "There is no saved view with that id."; 400 "Nothing to change."

Response: {"uid": "<uid>"}.

DELETE /api/lineage-views/{uid}

Owner always; anybody else needs METAMODEL_MANAGE.

404 "There is no saved view with that id." Response {"removed": "<uid>"}.