Docs › Administer

Connections and harvesting

Connect Snowflake, Databricks, Power BI, Tableau, Fabric or OpenLineage, and keep them in sync.

A connection, called a source on screen, is how KRISID reads your physical estate from the systems that hold it: databases, tables, columns, reports, jobs, and the lineage between them. This page covers setting one up, running it safely, and reading what it did.

Before you start

• Everything is on Administration → Sources.
• Everybody can see the list of sources: each source's name, the system it reads and when it last ran. Anybody can also press New source to see what a source asks for, but Create source stays greyed unless you hold Manage sources.
• Open source connections (SOURCE_VIEW) lets you open an established source: its settings, scope, mapping, runs, inbox and tag map. It is also needed to test, preview or sync one. In the starter model it comes with Sysadmin and Metadata Manager. Without it, a source's name in the list is plain text, not a link.
• What a source brought in is read with View assets, like everything else in the catalogue. Tables, columns, reports and lineage are ordinary assets, shown under Browse the physical estate and on each asset page. Every role in the starter model carries View assets, and holding it over anything at all — one data domain is enough — carries it everywhere; somebody with no role at all sees none of it.
• Manage sources (CONNECTION_MANAGE) lets you create a source, hold its credentials and decide what it brings. In the starter model only Sysadmin has it.
• Who can read each object is off by default and is the one tick whose cost depends on how you read the estate. From ACCOUNT_USAGE it is a single statement for the whole account, and it needs IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE — which the tags, lineage and usage ticks already need. Read per database there is no account-wide view of grants to ask, so it is one SHOW GRANTS per database, schema, table and view in scope: a round trip each, and it refuses above five thousand objects rather than spend hours on them.
• Run a harvest (HARVEST_RUN) lets you Test connection, Preview and Sync now on a source you can open. It comes with Sysadmin, Metadata Manager and Technical Custodian, but a Technical Custodian also needs Open source connections to use it from the screen. Scheduled syncs run regardless.
• Decide which connectors may be used (CONNECTOR_ALLOW) controls which systems can be connected at all, on Which connectors may be used.

These are separate on purpose. Somebody who refreshes the estate each week can't widen what a source is allowed to reach.

What the system is called

The Name you give a source is what its system asset is called in the estate — Tableau in Dev, not us-east-1.online.tableau.com (nbcudataanalyticsadminsdev). Rename the source and the next harvest renames the asset; its key never changes, so nothing is duplicated and every table under it stays where it is.

Power BI and Fabric ask for What to call this tenant in their own settings. If you fill that in, it wins — you have already answered the question in a narrower place.

What you can connect

A separate card, Snowflake — push tags, writes to Snowflake instead of reading from it. See Pushing tags back to Snowflake.

Create a source

1. On Sources, press New source.
2. Under Which system, pick a card. Only the connectors your repository offers are shown.
3. Name it: give it a Code, which is short and never changes, and a Name people will recognise.
4. How to sign in: choose a way in and fill in its fields. Methods marked needs a person, such as signing in with a browser, need somebody at the screen.
5. Where to look: the account, workspace or server, and anything to include or leave out.
6. What to bring: tick what you want read. The right-hand column shows what the source will need to grant for each tick. None of it grants access to your data, only to metadata.
7. If something is no longer there: choose what happens to things the source stops showing (see below).
8. Press Create source.

Passwords, keys and tokens you type are never shown back. The source page shows each one as dots, with when it last changed, and you can replace it there later: see See and change the sign-in details. They are stored encrypted, with a key kept outside the database, so a copy of the database or one of its backups does not contain them. Back that key up with the application — see Upgrading and licensing.

On the source's own page, open What to ask the administrator for. It holds the exact grants to request, worked out from your ticks, ready to Copy and send to whoever runs that system. For OpenLineage, it shows the settings to give Airflow, Spark or dbt instead.

See and change the sign-in details

Every source shows its credentials under How it signs in, so when a password, key or client secret is rotated you can put the new one in where the source is. You don't need to delete the source, so its schedule, history and mapping stay as they are. The names of the boxes come from the connector, so they differ between Snowflake, Databricks, Power BI, Fabric and Tableau.

What you see:

• Ordinary details, such as the user or the client ID, as they are set.
• Each secret, such as a password, private key or client secret, as a row of dots marked encrypted. The dots are always the same length: the value never comes back to the screen. A secret the way in needs that isn't set says so in red.
• A line saying when the secrets last changed and who changed them. Use it to see whether a credential is due for rotation.

To change them:

1. Press Change sign-in details.
2. Correct any of the ordinary boxes. They open showing what is set now.
3. Type a new value into the secret you are rotating. Leave the others empty to keep them. Show lets you check what you typed.
4. To remove an optional secret, such as a key passphrase that is no longer used, tick remove it.
5. Press Save sign-in details, then Test connection.

If the connector offers more than one Way in, you can switch to another, for example from a password to a key pair. Type the new way's secrets. The old way's secrets are removed when you save.

Each change is recorded on the System tab of the activity log, naming which secret changed but never its value.

A source that receives lineage

An OpenLineage source has no password. Producers send it events with an intake key, shown as dots with the date it was made. Press Make a key, or Make a new key to rotate it. The key is shown once, so copy it into the producer's settings straight away. Making a new one stops the old one working at once.

Test, preview, then sync

Each source page has three buttons. Use them in this order:

1. Test connection signs in and lists What these credentials can see. It reads nothing. Under How it signs in, the page then compares what you asked for with who the source says you are. If the source quietly gave you a different role or warehouse, a warning explains it — that usually means the one you asked for was never granted and the source fell back to a default, so everything afterwards succeeds while reading less than you think.

For Snowflake, the panel also shows a locator — UF26308 and obkzmft-hx91949 are two names for the same account, and only the second is the one you type. The locator is shown and never compared, so a correct connection is never accused of a mismatch. If the account ever is flagged, that means the Account box names a different account, not that anything fell back: a session cannot connect to one account and be served by another. 2. Preview reads the source and works out what would change, but writes nothing. It says Nothing has been written. This is exactly what Sync now will do. 3. Sync now does the same, then writes it.

A run carries on in the background. You can leave the page or close the browser. The Sources list shows a live marker on anything running, and the page picks the run up again when you come back.

A run says where it has got to as it goes, and keeps saying it is alive even during the parts that produce nothing to count — some sources spend a long time on a single question, and the panel says nothing new for 4 min rather than going quiet. If a run says nothing at all for twenty minutes, the server that was doing it has gone, and the run is shown as stalled rather than running. Nothing needs clearing up: starting the next sync on that source closes it and says so.

Stopping one

While a run is going, the panel has Stop this run. It asks the run to stop rather than killing it: the run finishes the item or the phase it is in the middle of and ends within a few seconds, and whatever it had already written stays written. A phase that finished is kept; the one it was in the middle of is left where it stands. Nothing is undone.

The run is then recorded as stopped, not failed, and says who asked and what it had got through — because a run somebody ended on purpose and a run that broke are different things, and a screen that cannot tell them apart teaches people to ignore red.

Stopping needs the same rights as starting: Run a harvest and Open source connections. Somebody who can start a sync and cannot end it has half a button. A run that has already finished refuses rather than pretending, so nobody comes back tomorrow believing they cancelled something they did not.

If a test or a sync says Reading … needs a driver, and it is not installed in the Python this application is running, Snowflake and Databricks each need a library that KRISID does not ship — about 90 MB, for a source most installations never have. Everything else about the connection works without it: the settings, the grants script, what it would bring. Only connecting refuses.

Whoever runs the repository installs it, and does not have to find out this way: bash run.sh doctor names it, and a release installs the drivers for the sources this repository actually has. The message itself gives the exact command, including which Python — pip install on its own usually installs into a different one, and then the message does not change. The application has to be restarted afterwards.

If a run says This source presented … things, and working out what to do with them needs about … GB, the source is bigger than this machine can work out in one go. Nothing was changed. The message says how many things would fit, and there are two ways on.

Reading a large source one database at a time

Under Narrowing there is a box for it, named after what the source calls the thing a pass covers:

Set it to read one … per pass and the sync reads the source one of those at a time — only the biggest one has to fit in memory, however large the source is. It is already on for sources you create from now on, and off for sources you set up before, so nothing changes under you — an older source shows read it all in one pass until you choose otherwise, which is what it is really doing.

A pass narrows through the same box you could have typed into yourself — Workspaces, Projects, Databases — so a pass asks the source for nothing a narrowed sync of your own would not, and needs no permission beyond what the connection already has.

What that gets you, and the things worth knowing:

• The run page says where it is: reading the source — database 4 of 12, PHARMA_GOLD_DB.
• Every database is catalogued before any link is drawn. The run reads and catalogues database 1, then 2, then 12; then it checks which databases the account still has; and only then does it draw the links, database by database in the same order. The catalogue is what makes an estate searchable and it is what people are waiting for — arrows into a database nobody has read yet cannot be drawn anyway.
• Each pass is kept on its own. If a run stops or is stopped, the databases it finished are catalogued and stay catalogued, and the links it was holding for them are drawn before it ends. The run says how far it got, and the next run carries on from there rather than starting the hour again. It only carries on if the list of databases has not changed since — otherwise it starts again and says so.
• Nothing is reported missing in a database a pass was not asked about. After the last pass, one quick extra pass checks which databases the account still has at all, so a database that really has been dropped is still noticed.
• Links between two databases are ordinary links now. A view in one database reading a table in another used to be unplaceable in the pass that read it, and was held back for a replay at the very end. Since the catalogue is finished before anything is drawn, both ends exist and it is simply one of that database's links.
• The run reads each database once. Keeping every reading until the end would undo the whole point of the mode; reading each database twice would double the half of a harvest that is spent waiting on the source. Neither happens: a pass keeps its links, which are a small fraction of what it read, and they are kept in the repository rather than in memory.
• A Preview of such a source previews the first database only, planned exactly as the run will plan it, and says so. Previewing all twelve would cost what the sync costs.

The other way on is to narrow what the source brings at all — fewer databases, fewer schemas, a pattern — which is the right answer when you never wanted the rest of it.

This applies to Preview as well as to Sync, and for the same reason: working out what a harvest would do is the expensive half, not writing it. A preview of a scope too large is refused rather than attempted.

When two relation types could hold the same link

A metamodel can name more than one link between the same two kinds of thing: a semantic dimension derives from a column and is realised by one, and both are relation types from Semantic Dimension to Column.

A source that reports the link says which it means — derives_from — and the harvest reads it as the relation type whose own words are derives from. It says so under the run's warnings, naming both it could have been, so you can see what was decided and name the other one in the mapping if the source means something different.

If the word the source uses names neither of them, nothing is guessed: the run says A '…' link from … to … could be any of: …, and those links wait until you choose on the source's Mapping screen. An arrow on a lineage diagram that nobody can tell is wrong is worse than a missing one.

If the page says This source is set to bring things the model has nowhere to put., the metamodel is missing a type the source needs. Under What this connector needs from the model, See what it would add shows the rows it proposes, and Add them adds them. This needs Change the metamodel (METAMODEL_MANAGE). Or untick that item under What it brings.

When something disappears from the source

A table dropped in the source may really be gone, or your scope may have narrowed, or a grant may have been lost. So you decide what happens:

• Report it only (the default): it is listed in the run and nothing is written. Somebody looks.
• Mark it as absent: its Source status becomes Missing in Source. This shows at the top of its page and beside its name on every list, and you can filter on it. It goes back to In Source as soon as the source shows it again.
• Delete it: the sync that finds it gone deletes it, and everything inside it, the same way a person deleting it would. Only what the person running the sync may delete is deleted. Where a deletion workflow governs that kind of asset, a request is raised instead, and the asset stays until somebody decides. Anything not deleted is marked Missing in Source. The activity log says how many were deleted, and why any were kept. While the source is narrowed, the deletion is held and the things are marked instead, and the run says so — see below.

Important: Deleting also removes what people added to those assets: descriptions, links to business terms, owners. Choose it only for a source whose contents nobody curates.

Narrowing, and the difference between two boxes that look alike

A sync can only say something has gone from the source if it looked at all of the place it would have been. When you narrow a source, it has not — so it says so, and nothing it did not look at is judged. You will see a line like this on the run:

This harvest did not look at everything, so nothing was judged missing where it did not look: the Databases setting.

That is the sync telling you what it is not claiming. Two boxes under Narrowing look alike and mean opposite things:

That is deliberate: excluding something by name is how you get it out of your repository, and it would stop working if the two were treated the same. The same difference applies to the two patterns: Include names matching is "only these", and narrows; Exclude names matching is "not these", and does not.

So: narrowing never deletes or marks anything by itself, and widening a source again does not need any repair — the things it could not see were never judged. Preview before you sync either way.

Clearing up what has gone

Mark it as absent is the right setting for most sources, and it leaves you with something to do: after a while, the things that really did go are still here, marked. That is on purpose — the mark is an observation, not a verdict — but a repository with ten thousand rows reading Missing in Source is one where nobody reads the column any more, and the few that matter are hidden among the ones nobody got to.

So you can clear them, in bulk, when you have looked and are sure.

Pick the thing they sit under. Open a source system, a database, a schema or a table, and press Clear what has gone… beside its name.

The button does not carry a number, and that is deliberate. Working out how many would go means walking every asset under what you are looking at, on every page view, for a figure nobody checks against their own records — and on a schema holding two hundred thousand columns it is slow enough to be felt. What you need before you press it is the RULE: what it catches and what it leaves alone. The panel says that, and the answer afterwards says what actually happened.

Only what is marked Missing in Source is deleted. Everything still in the source stays exactly where it is, under the same parent, even when the thing above it is going. If the asset you picked is itself missing, it goes too.

Or clear a whole kind at once. Open any list — Tables, Views, Columns — and if some of what it is showing has gone, a line above the table offers to clear them. That covers what a single branch cannot: a warehouse decommissioned across four databases.

Before anything happens the panel tells you the rule, not a tally:

Everything below this that is marked Missing in Source, and stopped being found more than 30 days ago, will be deleted — however much that is.

It deliberately does not say how many. Working that out means walking the whole tree under what you picked, and on a schema holding two hundred thousand columns that is slow enough to be felt on every page — for a number nobody checks against their own records. What you need before pressing it is what the rule catches and what it leaves, and the answer afterwards says what actually happened: how many were cleared, how many were held for approval, how many you were not allowed to delete.

What is not touched:

• anything still in the source, which keeps its place even where the thing above it is going;
• anything that went more recently than the cut-off;
• anything you may not delete;
• anything a deletion workflow governs — held as one request, not one per asset.

How long gone is gone

Something that vanished last night may be a deployment half-done, or a harvest that read less of the source than it should have. So nothing can be cleared in bulk until it has been missing for a while — 30 days unless somebody changes it.

The setting is on the Sources screen, under How long gone is gone, for whoever manages access. Nought is allowed and means whatever is marked may go however recently — a decision, not a default.

Like the button, this screen states the rule rather than counting against it. Counting how many assets across every source are marked, and how many are past the cut-off, is a scan of the whole estate every time somebody opens the screen.

You can also widen or narrow the cut-off for one clear, in the panel, without changing the setting.

Step by step, the first time

1. Look at what is marked. Browse to the source system, or open any list and filter Source status to Missing in Source. Nothing here is a verdict yet — this is the observation the syncs have been making.
2. Satisfy yourself they really went. A run that read less of the source than it should have marks things absent too. The Runs tab on the source says what each run read and whether it refused anything.
3. Decide the cut-off. Thirty days is the default. If a source has been quiet through a migration, widen it for this clear rather than changing the setting for everybody.
4. Pick the thing they sit under — a source system, a database, a schema, a table — and press Clear what has gone…. Or, for something no single branch covers, open the list for that kind and use the line above the table.
5. Read the rule on the panel. It names what will go and what it will leave, in the words above. Change the cut-off here if you decided to at step 3.
6. Press Clear what has gone. It is one request, however much is involved.
7. Read the answer. It says how many were cleared, how many were held for approval, how many you were not allowed to delete, and how many the database refused. Anything held or refused is still there and still marked.
8. If anything was held, somebody with the right to approve finds one request — not one per asset — in Requests. Nothing of that workflow's share goes until they decide.
9. Check History. Every deletion is recorded against whoever did it, with what it was.

What it will not do

Clearing goes down exactly the same road as deleting one asset by hand:

• You need the permissions. Delete assets (ASSET_DELETE) over each thing, and Delete harvested assets (HARVESTED_DELETE) as well, which only Sysadmin holds in the starter model. What you may not delete stays, and is named.
• Workflows still decide. Where a deletion workflow governs any of it, a request is raised and nothing of that workflow's share is deleted until somebody approves. It is one request for the whole clear, not one per asset — four hundred identical rows in an approval queue is not a control.
• The database still refuses. Anything something else still depends on stays, and is named in the answer.
• It is recorded. Every deletion goes into History against whoever did it.

Important: clearing removes what people added to those assets too — descriptions, links to business terms, owners. The asset going back to In Source by itself is no longer possible once it is gone; if the source ever presents it again, it arrives as a new asset with nothing written on it.

From the command line

For a large clean-up there is bash run.sh clear-gone. It says what would go and writes nothing until you add --apply:

bash run.sh clear-gone                            everything, everywhere, dry
bash run.sh clear-gone --source SF.ACME           one source
bash run.sh clear-gone --under SF.ACME.SALES_DB.PUBLIC    one branch, by its key
bash run.sh clear-gone --days 90                  a stricter cut-off
bash run.sh clear-gone --apply                    do it

It needs the application running, because it goes through the same endpoints the screens do — so every permission, every workflow and every refusal applies exactly as it would on screen. It runs as a sysadmin unless you name somebody with --as, and whatever it deletes is recorded against that name.

Run it on a schedule

A source can harvest by itself. On its page, under When it runs, choose how often and press Save schedule:

• Every hour, or Every few hours (2, 3, 4, 6, 8 or 12).
• Every day at a time you choose.
• Every week on a day and at a time you choose.

Times are the server's own. The page then says when it next runs, and whose name it runs under: whoever saved the schedule. A scheduled run is an ordinary Sync now, so everything that would stop that person pressing it stops the schedule too. If they are switched off or lose Run a harvest, the run does not start and the refusal is written to the Cataloguing tab of the activity log.

Choose By hand only to stop it. Saving where the source looks, or what it brings, leaves the schedule as it is.

A source that signs in a way that needs a person at the screen, such as a browser sign-in, cannot run on a schedule, and the page says so. A card that pushes tags to Snowflake is never scheduled either.

Read a run

The Runs table on a source page has one row per thing somebody started — a sync, a preview, a push — with Started, Ended, Who, New, Changed, Linked and Gone.

• Started and Ended are in your own time zone, labelled with it, and Ended also says how long the run took — 23 Sep, 21:45 GMT+5:30 · 4m.
• A run that is still going says running… rather than leaving the column blank.
• A run that stopped without ever finishing — a server restarted, a process was killed — says never finished. That is a fact worth seeing, not a gap.
• Click the number under Gone to see exactly which things were not found.

The passes are folded underneath. A sync is made of passes — the catalogue and lineage halves, and on a source read one database at a time, one of each per database. The number beside a run says how many it made; press it to open them. Each one says which database it was, has its own start, its own finish and its own page, so "how long did PHARMA_GOLD_DB take last night?" is a question the screen answers.

Every run has a page of its own. Click a run's kind — sync, catalogue, lineage, preview, pushed tags — to open it. That includes a run that changed nothing: "it looked and everything already matched" is an answer, and the page says so in as many words.

The list is paged, and it has a window. The table shows the last 90 days by default — or whatever your administrator has set logs to be kept for, since that is the same decision — 100 runs to a page, changed with Rows under the table. Show every run takes the window off: harvest runs are never pruned, so everything this source has ever done is there to look at.

The run's page lists every line:

• add: something new.
• change: a field that changed, from its old value to its new one.
• gone: something the source no longer shows.

Lines are grouped into Cataloguing (the things themselves) and Lineage (the links between them). Lineage runs only if cataloguing worked, because a link needs two things to exist. If one phase fails, the page says what was kept. Runs also appear in the Cataloguing and Lineage tabs of the activity log.

A first harvest of a warehouse can be hundreds of thousands of lines, so the page draws a thousand at a time and Show the next 1,000 brings the rest — the heading beside each phase always says how many there are in total, so you know what you are working through before you start. Your place on the page is kept.

Download the whole list saves every line as a file, however many there are. It starts arriving immediately rather than after the whole file has been put together, so a download of a very large run is something you can watch rather than something that looks stuck. Use this rather than the page when the question is prove it — a change somebody has to sign off, a diff against last month, or an answer for whoever asked why a column disappeared.

Good to know

• What a source says about each asset (its owner, who changed it and when, its description, size, data type and so on) is stored as read-only attributes, such as Owner at Source and Description from Source. The first time a source brings one, KRISID adds it to that asset type in the metamodel, marked harvested. A harvest never writes the steward's own Description. See Reading an asset.
• Which source catalogued an asset is recorded once, by the harvest that first brought it in, and does not change when another source later looks at the same thing. A BI tool that reports the warehouse tables its dashboards read does not take those tables over; Last seen (the asset's harvested date) is what moves. So "how many assets catalogued from Snowflake" means what it says, and stays still.
• The Runs column on the Sources list says how often each source runs by itself, or by hand. Last run beside it says when the last one started and when it ended, so you can tell a source that is stale from one that is merely quiet without opening it.
• A schedule saved before this release does not start until the source's schedule is saved again on its page.
• If a connector is later withdrawn, its sources stop testing, previewing and syncing. Nothing is deleted. A connector that is not offered keeps its existing sources running, but new ones can't be created.

Related

• Pushing tags back to Snowflake
• The metamodel
• Lineage and connections
• The activity log