Atlas Data
Norway's civil-society sector as a REST API — municipality-level indicators of humanitarian need, and the NGOs and local chapters that respond to them.
Installs the Atlas data platform as one application: a PostgreSQL database with its migrations, a Dagster code location running the ingest and dbt transforms, and a PostgREST API over the curated api_v1 views.
The data is an open semantic layer over Norwegian public sources — SSB statistics, FHI public-health and living-conditions indicators, Bufdir child-poverty figures and the Brønnøysund business register — joined to NGO supply data, so that humanitarian need and the organisations responding to it can be queried together at kommune level.
The API is self-describing: meta_sources, meta_endpoints and meta_dimensions let a consumer discover what is available without out-of-band documentation.
A fresh install serves an empty API until the first pipeline run — the schema and grants exist from install, the data arrives on the ingest schedule. Exports api-url for other applications to consume.
Installing does not start anything: the schedules and sensors ship stopped, so no data is fetched until an operator turns them on. That is a deliberate go-live decision, not a side effect of installing — and enabling them starts the NEXT scheduled run rather than backfilling, so a fresh install stays empty until the first-data jobs are launched by hand.
terchrisPrerequisites
Template README
Atlas serves Norway's civil-society sector as a REST API — municipality-level indicators of humanitarian need, and the NGOs and local chapters that respond to them.
The data is an open semantic layer over Norwegian public sources: SSB statistics, FHI public-health and living-conditions indicators, Bufdir child-poverty figures and the Brønnøysund business register, joined to NGO supply data, so that need and the organisations responding to it can be queried together at kommune level.
Installing it with UIS brings up the whole application under one name:
- a PostgreSQL database with its migrations applied,
- a Dagster code location for the ingestion pipelines,
- a PostgREST API over the resulting
api_v1views, - and an exported
api-urlother applications can consume.
Install
uis template install atlas
⚠️ A fresh install serves an empty API until the first pipeline run
This is correct behaviour, not a broken install. The schema and grants exist from install; the data arrives on the first pipeline run, which an operator has to start — see the next section. The API answers and returns zero rows until then.
Stated here because it is the state most likely to be read as failure by someone installing for the first time.
The API is also self-describing — meta_sources, meta_endpoints and meta_dimensions let a
consumer discover what is available without any out-of-band documentation.
What happens after you install
Installing starts nothing. The Dagster schedules and sensors ship stopped. No data is fetched, and no external service is contacted, until an operator turns them on. Turning them on is the go-live decision.
⚠️ Turning them on does not backfill. It starts the next scheduled run, so a fresh install stays empty until the first-data jobs below are launched by hand.
First ingest loads roughly 4.1 million rows across 52 raw BASE TABLEs and 61 marts BASE
TABLEs (plus 5 marts views), from about 41 sources. That is imac's measured 2,906,194 plus the
1,173,878-record Enhetsregisteret bulk load, so the row count is arithmetic on two measured figures —
the combined wall time is not measured and is deliberately not stated.
A "table" here means a BASE TABLE; views are excluded and counted separately. That rule is stated because two figures in the artifact used to disagree with each other and with the database while no counting method was written down anywhere.
Once schedules are on, Atlas polls on this cadence (Europe/Oslo) — mirroring
operational.cadence in the artifact at pin v20260913-e0ef430:
Every row names the job that owns it, and that is not decoration — see the warning below the table.
| when | job | what |
|---|---|---|
| Sunday 02:00 | annual_sources_refresh | ~37 annual public-sector sources — SSB, FHI, Bufdir |
| 1st of month, 01:00 | klass_refresh | SSB Klass classifications (kommune/fylke) |
| Every 30 min (:00, :30) | brreg_change_feed | Brønnøysundregistrene change feed — only what moved, ~114 records per cycle at the measured 3.8 changes a minute. Appends to raw only; rebuilds nothing and cannot affect the public API |
| Every 30 min (:10, :40) | brreg_transform | Brreg reconciliation into marts — ONE incremental model, ten minutes behind each poll. Rows are inserted and deleted, never the table replaced, so no api_v1 view is disturbed |
| Daily 04:00 | brreg_change_feed | Frivillighetsregisteret — a full re-walk of ~72,800 organisations (~727 requests) |
| Daily 05:00 | transform_and_publish | dbt transform and publish — no external calls. ⚠️ The only scheduled job that rebuilds marts, and therefore the only one during which api_v1 views are recreated — each restored immediately after its own mart, so the gap is milliseconds per view |
🔴 Two rows, one job — and reading them as one thing has already cost real time. The 04:00
Frivillighetsregisteret row and the half-hourly feed row are the same job, brreg_change_feed,
with different sources and different automation conditions. Two agents independently read the Brreg
rows as a single job that both polled every half hour and rebuilt marts, and one of them costed a
public-API outage at 48× its real rate and nearly weighed a cadence rollback against it (urb-agents
#780, #786).
The ten-minute offset is deliberate. The reconciliation runs at :10/:40 rather than alongside
the feed because firing on the same tick would reconcile data the feed had not yet written, leaving
the dimension permanently one cycle behind.
⚠️ Frivillighetsregisteret is daily and did not move to the feed's cadence. It has no change feed, so every refresh is the whole register — ~727 requests. At half-hourly that would be ~35,000 requests a day against a public-sector API, a 48× increase on someone else's service. The two Brreg ingests are on separate schedules for that reason and must stay that way.
The external services Atlas calls are SSB (Statistics Norway), FHI (Folkehelseinstituttet), Bufdir and Brønnøysundregistrene. All are public-sector APIs. The cadence is deliberately conservative — an annual statistical table is polled weekly, not nightly, because fetching annual tables every night would be roughly 15,000 pointless requests a year against services Atlas depends on staying welcome at.
Two sources are deliberately unscheduled (redcross-branches, frr) pending a credential. They
never self-trigger; invoking them by hand on a cluster without the private data repo fails, and that
is correct.
⚠️ The cadence lives in the image, not in this entry. It changes when the pinned tag changes,
with nothing here to review. If that matters to you, diff cadence.py between tags.
Getting data on day one
Enabling the schedules does not backfill. Every schedule is on_cron, which means next fire —
so a Thursday install waits until Sunday 02:00 for raw data, and until the 1st for the monthly
sources. The API stays empty in the meantime, with nothing to explain why.
To get data immediately, launch these six jobs from the Dagster UI, serially, in exactly this order:
| order | job | ~time |
|---|---|---|
| 1 | annual_sources_refresh | 6.3 min |
| 2 | klass_refresh | 1.0 min |
| 3 | seed_sources_refresh | 0.8 min |
| 4 | brreg_bootstrap | not yet measured |
| 5 | brreg_change_feed | not yet measured |
| 6 | transform_and_publish | 2.8 min |
The order is not arbitrary, for two separate reasons:
seed_sources_refreshcontainsraw/_migrationsand runs third, so the migrations apply after two source jobs have already written. That is safe because they are idempotent, but reordering has not been tested.brreg_change_feedruns immediately afterbrreg_bootstrapbecause the bootstrap seeds the feed's watermark from the snapshot's own date. The feed has nothing to start from until the bootstrap has run — and run first, it fails loudly rather than silently walking history: it refuses to start without a watermark.
Together these produce ~4.1M rows across 52 raw and 61 marts BASE TABLEs (plus 5 marts
views) from ~41 sources — the 1,173,878-record Enhetsregisteret bulk load on top of imac's
measured 2,906,194.
✅ The two figures in the artifact now agree, and this page no longer has to choose between them. Earlier pins had
install.first_loadsaying 60 marts whilefirst_data.takessaid 64, and this page used 64 with a note saying why. Both now say 61 with the counting rule beside them, derived from the repository with no database and matchingimac's live measurement exactly. The note that used to sit here is gone because the disagreement is gone — a caveat about a resolved problem is the same stale-label failure as the problem it described.
⚠️ Do not read "~11 minutes" as the total. That figure is imac's measurement of the four jobs
that existed when it was taken — annual_sources_refresh, klass_refresh, seed_sources_refresh
and transform_and_publish — on a clean cluster (10.8 min, 2,906,194 rows, zero failures; urb-agents
#507, #520). It predates both Brreg jobs. The bootstrap's download alone is 210 MB / 52 s and its
write time is not measured, and the change feed is not measured at all, so the combined wall time is
unknown rather than estimated.
(Stated by naming the four jobs rather than "the first four", because that phrase silently stopped being true when the list grew from four to six.)
🔴 brreg_bootstrap is the one most likely to be skipped
It has no schedule and no automation condition, on purpose. Re-running a 1.17M-record bulk load against a populated database is the only genuinely destructive-looking operation in this pipeline, so nothing self-triggers it. Run it once, here.
- Skipping it leaves the organisation register empty, with nothing saying why.
- Running it twice is safe — it upserts and never truncates — but pointless.
It is deliberately not in the unscheduled list further down: that list means cannot run.
brreg_bootstrap is the opposite — it must run once, on day one, and then be left alone.
Maintainer note — do not trim the paragraph above as duplication. It restates, in this page's own words, why nothing self-triggers
brreg_bootstrap. That looks redundant with the artifact'sfirst_data.how, and the redundancy is deliberate:first_data.howdoes not render at the end of an install — onlyuis template infoshows it — so an operator about to run the chain may see this page's reason and nowhere else's (urb-agents #802).This is the opposite call to the troubleshooting block, which this page deliberately does not copy, and the difference is rot. A copied remedy goes stale silently when its source changes; a sentence in this page's own voice about a property that does not change — a 1.17M-record bulk load must not self-trigger — stays true whether or not the artifact still says it. Copy nothing that has a current source; keep your own words for what would otherwise have none.
brreg_change_feed is different again — run it once here, then leave it to its schedule
Unlike the bootstrap, the change feed is scheduled: once automation is on it runs daily at 04:00. You launch it by hand exactly once, in the sequence above, because a fresh install has no data for the first nightly run to apply a delta to.
So the three kinds of job on this page are not interchangeable:
| runs by itself? | run by hand on day one? | |
|---|---|---|
| the four source/transform jobs | yes, on their crons | yes |
brreg_bootstrap | never — no schedule, no automation condition | yes, exactly once |
brreg_change_feed | yes, nightly at 04:00 | yes, once, after the bootstrap |
redcross-branches, frr | no — parked, cannot run | no |
⚠️ This list mirrors
operational.first_datain the artifact at pinv20260913-e0ef430. It is duplicated here, by hand, because as of that pinuis template inforenders none of the artifact'soperationalblock, so this page is the only place an operator can read it. It is therefore capable of going stale on the next bump — the artifact is the source of truth. Generating this section from the artifact at authoring time is the intended fix; seePLAN-application-catalogue.md.
If a run fails, the artifact tells you what to do
The install definition carries a troubleshooting block, and UIS renders it at install as of
1.6.67 — so the remedy reaches the operator rather than living only on this page. It covers the two
failure modes that have actually happened:
- a transform failing with a dbt schema error naming
dim_brreg_enhet, after an upgrade adds a column to an incrementally-materialised model, and - the public API returning 404 even after the database has been repaired and the view confirmed to exist with rows and grants.
The authoritative text is the artifact's, not this page's. It is deliberately not copied here: it
is long, it is precise about which database user must run the rebuild and about the ordering that
people miss, and a stale copy of a 02:00 remedy is worse than a pointer to a current one. Read it from
uis template info atlas, or from the definition at the pinned digest.
Installing alongside an atlas that is already running
If a tenant is already deployed, installing under the same name configures the same database and overwrites the running code location. To bring up a parallel tenant instead, pass a different application name:
uis template install atlas --param app_name=atlas-t
app_name isolates everything — database, secret prefix, url prefix and code location —
so an existing tenant is untouched. Verified by imac on urb-agents #481.
Upgrading an existing install
An upgrade happens when this catalogue entry's pin moves and you re-install. What that costs you depends on the release, and it is not always nothing.
Every atlas upgrade so far has required no operator action. The pins through 2026-09-13 were text-only or additive: you re-install and the data is untouched.
⚠️ Do not read that as a property of atlas upgrades. It is a property of the releases so far, and
it is the more dangerous half of this page precisely because three silent upgrades in a row teach an
expectation. A release that changes the shape of an incrementally-materialised model requires a
--full-refresh on an existing install, and dim_brreg_enhet runs with on_schema_change='fail'
— so an operator who upgrades without one gets a failed model doing exactly what it is configured to
do, not a bug.
Two surfaces answer two different questions, and this is the earlier one:
| you are asking | read |
|---|---|
| should I take this upgrade, and what will it cost? | this page, before you re-install |
| a run just failed — what now? | uis template info atlas, which renders the artifact's troubleshooting block, and the install output itself |
When a release needs operator action, this section will say so and name the cost, because by the time the install output tells you, you have already committed to it.
What this directory is, and is not
This directory is a catalogue pointer, not the application. There is no atlas source
code here. The install definition is the OCI artifact named in template-info.yaml:
ghcr.io/terchris/atlas-data/uis:v20260909-853c696
sha256:def7b9d2839180258f962555bd30ed5ca037fc42c52ebe0dd75f75c83d3a6c54
UIS pulls the digest. The tag is shown to people and never pulled by, because a tag can
be re-pointed at a registry and a digest cannot. Bumping atlas therefore means editing
source.digest in a reviewed change — that review is the only thing establishing that a
human approved this artifact.
The full reasoning lives in uis-applications/README.md in the dev-templates repository —
referenced as a path rather than a link, because this page is rendered on the documentation
site where a relative link to a repository file does not resolve.
params: and provides: are not listed here on purpose. They live in the artifact's
own template-info.yaml; there is one source of truth for what atlas installs and it is
not this repository.
Provenance of this entry
The artifact coordinates above are atlas's published values as verified by imac against
the real registry (urb-agents #481) — the 853c696 artifact, which superseded 4b11f3f
partway through that test round.
The three services listed above are not inferred: they are the plan
uis template install atlas --dry-run produces from this pin, as run by tor-agent against the
published registry (urb-agents #486) — deploy postgresql, configure postgresql,
configure postgrest --schemas api_v1 --url-prefix api-atlas, deploy postgrest, deploy dagster,
the atlas-data code location, then deploy dagster again.
The entry's description and abstract are atlas's own words, supplied on urb-agents #489.
Change them only on atlas's word.
⚠️ Two earlier versions of this description were wrong, and the sequence is worth keeping:
dev-templatesfirst wrote "Norwegian business-register data", inferred from thebrreg_enhetertable in the install records.- Corrected to "the Norwegian NGO sector, in one place" from the repository's own one-liner — better, but still not the framing atlas uses.
- atlas supplied the text above and explained why the first attempt was worst:
brreg_enheteris one raw table out of 47. FHI contributes 21 and SSB 17. Brønnøysund supplies organisation identity — how an NGO gets a stableorgnr— so leading with it described the smallest source as though it were the subject.
The lesson recorded rather than the words: a domain inferred from a table name reads plausibly and is not evidence. The prose in this repository is a stand-in until the owning application supplies its own.