--- url: /scarab/tutorials/getting-started.md --- # Getting started Load a typed config from a file with `scarab-config-file`. The whole flow is three steps: define your serde model, pick a [`Discovery`](/reference/config-port#discovery) spec, load. ## 1. Add the crates Scarab isn't published to crates.io — depend on the git repo, like every org consumer does (`myco`, `tarsi`, …): ```toml [dependencies] scarab-config = { git = "ssh://git@github.com/9aia/scarab.git" } scarab-config-file = { git = "ssh://git@github.com/9aia/scarab.git" } scarab-config-yaml = { git = "ssh://git@github.com/9aia/scarab.git" } scarab-config-toml = { git = "ssh://git@github.com/9aia/scarab.git" } serde = { version = "1", features = ["derive"] } ``` ## 2. Define your config model Scarab stays generic — your repo owns the `Config` type: ```rust use serde::Deserialize; #[derive(Debug, Deserialize)] struct MyConfig { name: String, port: u16, } ``` ## 3. Load it ```rust use scarab_config_file::FileSource; let cfg: MyConfig = FileSource::for_app("myapp").load()?; ``` `FileSource::for_app("myapp")` wires the org conventions for you: env var `MYAPP_CONFIG`, search dirs `[".", "~/.config/myapp"]`, and the standard format pair (YAML preferred, TOML fallback). For `myapp` it resolves, in order: 1. `$MYAPP_CONFIG` — an explicit file path 2. `./myapp.yaml`, `./myapp.yml`, `./myapp.toml` 3. `~/.config/myapp/myapp.yaml`, `.yml`, `.toml` The first existing file wins. See [reference: config port](/reference/config-port) for the full contract and [how-to: customize discovery](/how-to/customize-config-discovery) for custom search specs. ## Next steps * [Layer configs from several files](/how-to/layered-config) — merge `~/.config//` with a project file instead of first-match. * [Write a template pack](/how-to/write-a-template-pack) — ship rendered config files as a tamias `type: template` package. * [Deploy a template pack](/how-to/deploy-a-template-pack) — the driver verbs, variable layers, and uninstall semantics. --- --- url: /scarab/how-to.md --- # How-to guides * [Customize config discovery](customize-config-discovery.md) * [Layer configs from several files](layered-config.md) * [Write a template pack](write-a-template-pack.md) * [Deploy a template pack](deploy-a-template-pack.md) --- --- url: /scarab/how-to/customize-config-discovery.md --- # Customize config discovery `Discovery::for_app` covers the common case. When it doesn't — a different env var, extra search dirs, a different format set, or an explicit `--config` path — build the [`FileSource`](/reference/config-port#filesource-t) directly. ## Custom env var and dirs ```rust use scarab_config::{Discovery, expand_tilde}; use scarab_config_file::FileSource; let discovery = Discovery::for_app("myapp") .env_var("MYAPP_CFG") // override $MYAPP_CONFIG .dirs([".", "~/.config/myapp", "/etc/myapp"]); let src = FileSource::new(discovery); ``` `dirs` entries may contain `~` — expanded via `expand_tilde` at search time. `.env_var("…")` replaces the env step; `.without_env()` removes it entirely. `Discovery::bare("myapp")` starts with no env var and no dirs — fill both yourself. ## Explicit path wins over everything Set `explicit` on the source (a `--config`-style flag) and it takes precedence over the env var and all discovery candidates: ```rust let mut src = FileSource::new(discovery); src.explicit = Some("/etc/myapp/overrides.toml".into()); ``` An explicit path that doesn't exist is an error naming the path — a named path that misses is a config error, not a silent skip. ## Custom format set or order `FileSource::new` uses the standard pair — `[YamlFormat, TomlFormat]`. `with_formats` takes any `Vec>>`; earlier formats win candidate ties within a directory, and each format's `extensions()` are tried in declared order. To make TOML preferred: ```rust use scarab_config_toml::TomlFormat; use scarab_config_yaml::YamlFormat; let src = FileSource::with_formats( discovery, vec![Box::new(TomlFormat), Box::new(YamlFormat)], ); ``` With TOML first, `myapp.toml` beats `myapp.yaml` in the same directory. ## Extensionless explicit paths `load_path` dispatches on extension; a path with no extension parses with the first (preferred) format — the back-compat seam for `--config` pointing at a suffix-less file. An extension no format claims is an error: `no config format claims '.json'`. ## Parse a specific file Skip discovery entirely when the caller already resolved a path: ```rust let cfg: MyConfig = src.load_path(Path::new("/etc/myapp.toml"))?; ``` ## Inject env lookups in tests `locate_with` / `load_with` take `env: &dyn Fn(&str) -> Option` so tests never touch the process environment: ```rust let cfg = src.load_with(&|k| (k == "MYAPP_CONFIG").then(|| "/tmp/x.yaml".into()))?; ``` --- --- url: /scarab/how-to/layered-config.md --- # Layer configs from several files `FileSource` picks the *first* matching file — one config wins, the rest are ignored. `LayeredFileSource` is the merge counterpart: it collects **one file per search dir**, folds them into a single `serde_json::Value` tree, applies programmatic overrides, and deserializes into your model. Use it when a base config (say, `~/.config/myapp/myapp.yaml`) should be *overridden* by a project or environment file rather than replaced by it. ## Swap the source ```rust use scarab_config_file::LayeredFileSource; // same Discovery conventions as FileSource::for_app let cfg: MyConfig = LayeredFileSource::for_app("myapp").load()?; ``` Under `Discovery::for_app` the dirs are `[".", "~/.config/myapp"]` and the **last dir is the base**: `~/.config/myapp/myapp.yaml` loads first, `./myapp.yaml` merges over it. Within each dir only the preferred file counts — `myapp.yaml` beats `myapp.yml` beats `myapp.toml`, and the losers are not merged. ## Precedence, lowest to highest 1. `/.` per search dir, last dir first 2. the explicit path / `$_CONFIG` file — merged **on top** of every dir layer (unlike `FileSource`, where it replaces discovery) 3. `overrides` — flat dotted-path keys, the highest layer ```rust use scarab_config_file::LayeredFileSource; use serde_json::json; use std::collections::BTreeMap; let mut src = LayeredFileSource::::for_app("myapp"); src.explicit = Some("./deploy/prod.yaml".into()); // top file layer src.overrides = BTreeMap::from([ ("server.port".to_string(), json!(8443)), ("server.tls.enabled".to_string(), json!(true)), ]); let cfg = src.load()?; ``` ## Merge semantics Layers merge as `serde_json::Value`, never on your typed model — `MyConfig` only needs `DeserializeOwned` (no `Serialize`). `merge_value` does the folding: * **objects merge key-wise, recursively** — a key the overlay omits survives from the base * **scalars, arrays, and null replace** — arrays never concatenate; set `[c]` over `[a, b]` and you get `[c]` Each layer file must parse to a mapping — a top-level array or scalar is an error naming the path (`config layer is not a mapping`). Mixed formats across dirs are fine: a TOML user config under a YAML project config merges cleanly. A parse error names the file it came from. ## Inspect the layers `layer_paths()` returns the resolved layer paths in merge order (lowest precedence first) — handy for logging what will be merged: ```rust for p in src.layer_paths()? { println!("layer: {}", p.display()); } ``` `parse_layer(path)` parses one file into a `serde_json::Value` if you need to inspect or transform a layer yourself; `load_with(env)` takes the same injected env lookup as `FileSource` for tests. ## When not to layer * You want "one file wins, done" — that's `FileSource`; don't pay for a merge you don't need. * Your model can't round-trip through `serde_json::Value` — e.g. it deserializes from a non-self-describing format lossily. The merge happens on JSON, so exotic serde attributes (`#[serde(with)]`, borrowed data) may not survive the intermediate `Value`. --- --- url: /scarab/how-to/write-a-template-pack.md --- # Write a template pack A template pack is a tamias package with `type: template` whose `tamias.yaml` carries a `template:` table. The `tamias-driver-template` driver renders each declared `src` through the engine and writes it at `into`. ## Pack layout ``` repo-bootstrap/ ├── tamias.yaml ├── app.toml.tmpl └── notes.md.tmpl ``` ## Manifest ```yaml name: repo-bootstrap type: template version: "0.1.0" template: vars: env: dev # defaults — lowest-precedence layer files: - src: app.toml.tmpl into: ./config/app.toml - src: notes.md.tmpl into: docs/NOTES.md ``` * `template.vars` — flat `name → value` defaults (any JSON value: strings, numbers, bools, lists, tables). Overridable per deploy. * `template.files[]` — `src` is a path inside the pack payload, `into` the render target relative to the deploy target dir. Both `src` and `into` must be safe relative paths: absolute paths, empty paths, and `..` components are rejected at validation time — before anything is written. ## Template syntax The engine is [minijinja](https://github.com/mitsuhiko/minijinja) — `{{ var }}`, `{% if %}`, `{% for %}`. It is sandboxed: pack templates are third-party content and cannot reach the filesystem or network. Undefined variables render as empty. ```jinja # app.toml.tmpl env = "{{ env }}" repo = "{{ repo }}" user = "{{ user }}" ``` ## Inherit from a parent pack `template.extends` names another template pack by tamias ref — the child renders the parent's files too, with the child's `into`s and `vars` winning: ```yaml template: extends: git:9aia/scarab//libraries/rust-repo vars: channel: nightly # overrides the parent's "stable" default files: - src: extra.toml.tmpl into: ./extra.toml ``` `tamias install` resolves the ref and materializes the parent's payload inside yours at `.scarab-parent/` (recursively for grandparents). The driver renders ancestor-first: parent files write first, yours last — on an `into` conflict **you win**, and the state digest records your version. See [template packs — composition](/reference/template-packs#composition) for the full contract, including the vendored-parent case (`.scarab-parent/` present without `extends`). ## Variable layers Later layers win: 1. ancestor `template.vars` defaults (extends chain, ancestor-first), then this pack's `template.vars` 2. auto vars — `repo` (git remote basename or dir name of the target repo) and `user` (`$USER`/`$LOGNAME`) 3. `--config` vars file (JSON / TOML / YAML by extension) 4. `$TEMPLATE_VAR_` env vars (lowercased name) 5. trailing `key=value` args See [deploy a template pack](/how-to/deploy-a-template-pack) for the full invocation. ## Validate before shipping `tamias-driver-template install --dir --name ` parses the manifest, checks every `src` exists and compiles, resolves the var context, and previews the render plan — writing nothing. Run it against your pack dir before `tamias publish`. --- --- url: /scarab/how-to/deploy-a-template-pack.md --- # Deploy a template pack Template packs deploy through tamias: `tamias install` fetches the package and dispatches the `type: template` verbs to `tamias-driver-template`. You can also run the driver binary directly against a local payload dir. ## Via tamias ```bash # the shipped example pack — starter configs for a new Rust workspace tamias install git:9aia/scarab//libraries/rust-repo ``` Tamias calls the driver's `install` verb for validation, then `deploy` for the actual render+write. `uninstall`/`remove` reverses it. ## Directly ```bash tamias-driver-template --dir --name \ [--config ] [--target ] [--dry-run] [key=value …] ``` | Verb | What it does | | --------------- | --------------------------------------------------------------------------------------------------------------- | | `install` | Validate the pack and preview the render plan. Writes nothing. | | `deploy` | Render each `template.files[]` entry and write it at `into` under the target dir. | | `uninstall` | Remove files still matching the recorded sha256 digests. | | `config-schema` | Print the JSON Schema for the `--config` vars file. | | `capabilities` | Print `{"name": "template", "types": ["template"], "verbs": […], "engine": "minijinja"}` — the discovery probe. | On an `extends` chain, `install`/`deploy` render every ancestor pack inside `.scarab-parent/` first — the child's files win on `into` conflicts. See [template packs — composition](/reference/template-packs#composition). ## Choosing the target dir `deploy`/`uninstall` write relative to, in order: 1. `--target ` 2. `$TEMPLATE_TARGET` 3. the current working directory ## Overriding variables Vars resolve in layers (later wins): ancestor `template.vars` defaults (extends chain) → this pack's `template.vars` → auto vars (`repo`, `user`) → `--config` file → `$TEMPLATE_VAR_` env → `key=value` args. ```bash # vars file — parsed as JSON/TOML/YAML by extension (default YAML) tamias-driver-template deploy --dir ./repo-bootstrap --name repo-bootstrap \ --config prod.yaml --target ~/myrepo # env + inline overrides env=prod TEMPLATE_VAR_replicas=3 \ tamias-driver-template deploy --dir ./repo-bootstrap --name repo-bootstrap \ --target ~/myrepo replicas=5 ``` `--dry-run` prints what would be written without touching the target. ## Uninstall semantics `deploy` records `.scarab-template-state.json` at the target root — each `into` path mapped to the sha256 of what was written. `uninstall` deletes only files whose current digest still matches; files modified since deploy are left in place and reported as kept. State entries for already-removed files are dropped, and the state file itself is removed once nothing remains. --- --- url: /scarab/reference.md --- # Reference * [Crates](crates.md) — generated from workspace Cargo.tomls * [Config port](config-port.md) — `ConfigSource`/`ConfigFormat` traits, `Discovery`, `FileSource`, `LayeredFileSource` * [Template packs](template-packs.md) — `template:` manifest schema, var context, safe paths, `extends` chains * [Template driver](driver.md) — `tamias-driver-template` argv contract, state file * [Testkit](testkit.md) — `scarab-testkit` conformance suites + golden fixtures --- --- url: /scarab/reference/crates.md --- # Crates Generic config + template expertise — ports are traits, format and engine impls are separate crates; consumers plug in their own config model `T` and template pack. | crate | layer | purpose | |---|---|---| | `scarab-config` | port | Config port — ConfigSource\/ConfigFormat generic over the config model, plus the shared file-discovery spec | | `scarab-template` | port | Template port — TemplateEngine trait + template-pack model (files, var defaults), generic over the render context | | `scarab-config-file` | adapter | File impl of ConfigSource — discovery (explicit → env → project → user dir) + extension dispatch to ConfigFormat impls | | `scarab-config-toml` | adapter | TOML impl of ConfigFormat — parse/render any serde config model | | `scarab-config-yaml` | adapter | YAML impl of ConfigFormat — parse/render any serde config model | | `scarab-template-minijinja` | adapter | minijinja impl of scarab_template::TemplateEngine — Jinja-style {{ var }} / {% if %} syntax, sandboxed by default | | `tamias-driver-template` | driver | tamias content driver for type: template — renders template-pack files into declared targets at deploy | | `scarab-testkit` | testkit | Conformance suites + golden fixtures for the scarab config ports — assert\_\* functions any ConfigFormat/ConfigSource impl runs to prove itself | | `scarab-xtask` | tooling | Repo chores: check / test / coverage / install / package / smoke / site | --- --- url: /scarab/reference/config-port.md --- # Config port The config plane is two traits plus a search spec, generic over the config model `T` (`Deserialize + Serialize`). Scarab never names a consumer's `Config` type. ## `ConfigFormat` Syntax-level: which extensions a format claims, `parse`/`render` over serde. Object-safe via a blanket impl — one `TomlFormat` / `YamlFormat` value serves every model. ```rust pub trait ConfigFormat: Send + Sync { fn extensions(&self) -> &'static [&'static str]; fn parse(&self, text: &str) -> Result; fn render(&self, config: &T) -> Result; } ``` | Impl | Crate | Claims | | ------------ | -------------------- | ----------------------------- | | `YamlFormat` | `scarab-config-yaml` | `yaml`, `yml` (in that order) | | `TomlFormat` | `scarab-config-toml` | `toml` | `scarab_config::format_for(path, formats)` returns the format claiming `path`'s extension, or `None`. ## `ConfigSource` Source-level: `load() -> Result`. Impls compose formats — `scarab-config-file`'s `FileSource` (first-match) and `LayeredFileSource` (merge-all) are the file-backed impls; env/remote sources are new impls of the same trait. ## `Discovery` The shared search spec every file-backed source uses. Nothing is hardcoded per app — basename, env var, and dirs are caller-supplied. | Field | Meaning | | --------- | ------------------------------------------------------------------------------------- | | `name` | Config file basename — `myco` → `myco.yaml`, `myco.yml`, `myco.toml` | | `env_var` | Env var naming an explicit config file (`$MYCO_CONFIG`); `None` disables the env step | | `dirs` | Directories searched in order for `.`; `~` expands via `expand_tilde` | Constructors and builders: * `Discovery::for_app(name)` — env var `_CONFIG` (uppercased), dirs `[".", "~/.config/"]` * `Discovery::bare(name)` — no env var, no dirs * `.env_var(var)` / `.without_env()` — set or disable the env step * `.dirs(iter)` — replace search dirs, in order ## `FileSource` File-backed `ConfigSource`: discovery + extension dispatch over a set of `ConfigFormat`s. | Field | Meaning | | ----------- | ------------------------------------------------------------- | | `discovery` | Where to search | | `explicit` | Explicit `--config`-style path; wins over `discovery.env_var` | | `formats` | Syntax formats in preference order | Constructors: * `FileSource::new(discovery)` — standard formats `[Yaml, Toml]` * `FileSource::with_formats(discovery, formats)` — caller-chosen set/order * `FileSource::for_app(name)` — `Discovery::for_app` + standard formats * `FileSource::standard_formats()` — the `[YamlFormat, TomlFormat]` pair Requires `T: DeserializeOwned + Serialize` for `new`/`for_app` (the `ConfigSource` impl bound). ## `LayeredFileSource` The merge counterpart, in `scarab-config-file`. Where `FileSource` picks the *first* existing file, `LayeredFileSource` collects **one file per search dir** (same per-dir preference rules), folds them low→high precedence into a `serde_json::Value` tree, applies programmatic dotted-path overrides, then deserializes into `T`. Formats are held as `ConfigFormat` — layers merge as JSON, so `T` only needs `DeserializeOwned` (no `Serialize`). | Field | Meaning | | ----------- | --------------------------------------------------------------------------------------------------------------------- | | `discovery` | Where to search — same `Discovery` spec | | `explicit` | Explicit path — the **top file layer**, merged over all dir layers (unlike `FileSource`, where it replaces discovery) | | `formats` | Syntax formats over `serde_json::Value`, in preference order | | `overrides` | `BTreeMap` — flat dotted-path keys (`"server.port"`), the highest layer | Constructors mirror `FileSource`: `new(discovery)`, `for_app(name)`, `with_formats(discovery, formats)`, `standard_formats()`. Precedence, lowest to highest: 1. `/.` per dir, **last dir first** — under `Discovery::for_app` (`[".", "~/.config/"]`) the user config is the base and `./.yaml` overrides it 2. the explicit path / `$_CONFIG` file 3. `overrides` — dotted-path leaf values Methods: `layer_paths()` / `layer_paths_with(env)` (merge order, lowest first), `parse_layer(path)` (one file → `Value`; a non-mapping document is an error), `load()` / `load_with(env)`. Same `load_with` env-injection seam as `FileSource` for tests. See [how-to: layered config](/how-to/layered-config) for the merge semantics and a worked example. ## Merge helpers Pure functions in `scarab-config` — the half of layered loading that never touches the filesystem: * `merge_value(base, overlay)` — recursive JSON merge: objects merge key-wise (overlay wins per key, recursing into shared objects); scalars, arrays, and null **replace** — arrays never concatenate * `apply_overrides(base, overrides)` — flat dotted-path keys (`"server.port" → 8080`) set leaves, creating intermediate objects; a scalar sitting where a path needs to descend is replaced by an object. Call after the file layers are merged ## Resolution order `locate` picks the config path: 1. `explicit` path, else `$_CONFIG` — a named path that doesn't exist is an error (`config file not found: …`), not a silent skip 2. `/.` per search dir, in dir order; within a dir, formats in preference order and each format's extensions in declared order — YAML before TOML by convention `load` reads the file and dispatches parsing by extension; an extensionless explicit path parses with the first (preferred) format; an unclaimed extension errors with `no config format claims '.'`. When nothing is found, the error names the app, the claimed extensions, and the search steps actually configured. ## Path helpers * `home_dir()` — `$HOME`, else `$USERPROFILE`, else `/` * `expand_tilde(path)` — `~`/`~/…` expansion; `~user`, relative, and absolute paths pass through untouched * `user_config_dir(name)` — `~/.config/` (tilde expanded at search time) --- --- url: /scarab/reference/template-packs.md --- # Template packs A template pack is a tamias package with `type: template` whose `tamias.yaml` carries a `template:` table. The port lives in `scarab-template` (pure domain — no I/O, no engine specifics); impls own the syntax. ## Manifest schema ```yaml name: repo-bootstrap type: template version: "0.1.0" template: vars: # optional — flat name → JSON value map env: dev replicas: 1 extends: git:9aia/base//libraries/base-pack # optional — parent pack ref files: # required, non-empty - src: app.toml.tmpl # path inside the pack payload into: ./config/app.toml # render target, relative to target dir ``` ### `template.vars` `BTreeMap` — defaults layer, lowest precedence. Values may be strings, numbers, bools, arrays, or tables; templates see them as engine-native values. ### `template.files[]` | Field | Meaning | | ------ | ------------------------------------------------ | | `src` | Template path inside the payload root | | `into` | Render target, relative to the deploy target dir | `into` must normalize to a non-empty relative path — absolute paths, `..`, root/prefix components, and empty results are `TemplateError::UnsafeTarget`. `src` follows the same rule inside the payload (`TemplateError::UnsafeSource`). ### `template.extends` Optional tamias ref naming a **parent template pack** (`git:org/repo//path`, `cf:…`, a store name, …). The driver never resolves refs itself — resolution is tamias-side. See [Composition](#composition) below. ## Composition (`extends` chains) A pack may inherit from another template pack. `tamias install` resolves `template.extends` and materializes each ancestor's payload inside the child's at `.scarab-parent/` — recursively, so a grandparent lands at `.scarab-parent/.scarab-parent/`, and so on. The driver then renders the chain **ancestor-first**: * parent `template.vars` are the lowest-precedence defaults — folding vars ancestor→child means the child's defaults win * parent files write first, the child's last — on an `into` conflict the **child wins**; distinct `into`s all render * the state file records only the final (child) digest for a contested path, which is exactly what `uninstall` needs Edge cases: * `.scarab-parent/` present **without** `extends` is honored as a vendored parent — useful for local composition and tests * `extends` declared but `.scarab-parent/` absent is an error: the payload never went through `tamias install`, and the driver can't fetch refs * `extends: ""` (empty) fails manifest validation ## `TemplateEngine` The engine port — one impl per syntax family: ```rust pub trait TemplateEngine: Send + Sync { fn name(&self) -> &'static str; fn render(&self, source: &str, ctx: &serde_json::Value) -> Result>; } ``` `ctx` is a JSON object; engines map it to their native value. `MinijinjaEngine` (`scarab-template-minijinja`) is the v1 impl — sandboxed, keeps the template's trailing newline so rendered config files stay POSIX-clean. ## Variable context `merge_context(defaults, auto, file, args)` merges flat `name → JSON value` layers, later winning: ``` ancestor template.vars … this pack's template.vars < auto vars < --config file < env+args ``` On an `extends` chain the driver folds every ancestor's `template.vars` into `defaults` ancestor-first, so nearer packs win. The driver supplies the layers: 1. `template.vars` pack defaults — every ancestor's folded ancestor-first, then this pack's (nearest wins) 2. auto vars — `repo` (git `remote.origin.url` basename of the target dir, else canonical dir name) and `user` (`$USER`/`$LOGNAME`) 3. `--config` file — JSON by `.json`, TOML by `.toml`, YAML otherwise (incl. extensionless); must be a flat map 4. `$TEMPLATE_VAR_` env vars, name lowercased, value parsed as JSON falling back to string 5. trailing `key=value` args — value parsed as JSON, falling back to string ## Errors | Variant | Meaning | | -------------- | -------------------------------------------------- | | `Manifest` | Pack manifest or `template:` table failed to parse | | `UnsafeTarget` | `into` escaped the deploy dir | | `UnsafeSource` | `src` escaped the payload root | | `NotUtf8` | Rendered output isn't valid UTF-8 text | --- --- url: /scarab/reference/driver.md --- # Template driver `tamias-driver-template` is the tamias content driver for `type: template` packs — it renders a pack's files into declared targets. Discovered by tamias as `tamias-driver-template` on `PATH`/`drivers/` like every content driver. ## argv contract Tamias verbs always arrive as: ``` tamias-driver-template --dir --name \ [--config ] [--target ] [--dry-run] [key=value …] ``` | Flag | Meaning | | ----------- | ---------------------------------------------------------------------------------------- | | `--dir` | Pack payload dir (contains `tamias.yaml`/`.yml`) — required for install/deploy/uninstall | | `--name` | Package name — required for install/deploy/uninstall | | `--config` | Vars file — JSON/TOML/YAML by extension, YAML default | | `--target` | Deploy target dir — overrides `$TEMPLATE_TARGET` and cwd | | `--dry-run` | Print actions without writing (deploy/uninstall) | | `key=value` | Trailing var overrides, highest-precedence layer | ## Verbs | Verb | Behavior | Exit | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `install` | Validate the pack *chain* (each manifest parses, every `src` compiles, vars resolve); print the merged render plan. Writes nothing — the payload dir is the evictable fetch cache. | 0 / 1 / 2 | | `deploy` | Render each `template.files[]` entry across the whole `extends` chain (ancestors first — the child wins on `into` conflicts), write at `into` under target dir, record state. | 0 / 1 / 2 | | `uninstall` | Remove files still matching recorded sha256 digests; modified files kept + reported. | 0 / 1 / 2 | | `config-schema` | Print JSON Schema (draft 2020-12) for the `--config` vars file — a flat `name = value` map. | 0 | | `capabilities` | Print `{"name": "template", "types": ["template"], "verbs": ["install","deploy","uninstall","config-schema"], "engine": "minijinja"}`. | 0 | Exit codes: `0` success, `1` operational error (message on stderr — tamias wraps it as `driver … failed: …`), `2` usage/arg error. ## `extends` chains `install` and `deploy` walk the pack's chain: each level's payload may nest its parent at `.scarab-parent/` (materialized by `tamias install` from `template.extends`). `uninstall` needs no chain — it reverses whatever the state file recorded. * `install` reports the merged plan — every pack's `into`s in render order, with a count of files overridden by a later pack * `deploy` renders ancestor-first; on an `into` conflict the child (later pack) writes last and its digest is what state records * `extends` declared but `.scarab-parent/` missing → error ("install through `tamias install`"); `.scarab-parent/` present without `extends` → honored as a vendored parent * var layers fold ancestor `template.vars` first, so nearer packs win — see [template packs](/reference/template-packs#composition) ## State file `deploy` writes `.scarab-template-state.json` at the target root: ```json { "pack": "repo-bootstrap", "files": { "config/app.toml": "", "docs/NOTES.md": "" } } ``` The file is shared per target dir — `deploy` merges its entries into any existing state (multiple packs can coexist). `uninstall` removes entries as files are deleted or found missing; the state file itself is removed once `files` is empty. ## Driver errors All failures surface as `OpsError` — a readable message on stderr: * missing `--dir`/manifest, `type` isn't `"template"`, empty `template.files`, bad `template:` table shape * unsafe `src`/`into` paths (rejected before any write) * missing/unreadable payload files, render failures * malformed `--config` vars file (must parse to a map) * I/O errors on read/write/remove --- --- url: /scarab/reference/testkit.md --- # Testkit `scarab-testkit` is the behavioral backbone for the config ports: conformance suites any implementation runs to prove itself, plus committed golden fixtures. Same pattern as `sepia-testkit` / `tamias-testkit`. Dev-dependency only — it never ships in a consumer's binary. ```toml [dev-dependencies] scarab-testkit = { git = "ssh://git@github.com/9aia/scarab.git" } ``` ## Format conformance `assert_config_format_conformance(format, &FormatCase)` checks a \[`ConfigFormat`] against one scenario: * extensions are claimed, non-empty, and unique * `parse(text)` produces `expected` * `render(expected)` re-parses to `expected` (round-trip) * every `bad_inputs` entry fails with `Err` — never a panic ```rust use scarab_config_yaml::YamlFormat; use scarab_testkit::{FormatCase, assert_config_format_conformance}; assert_config_format_conformance( &YamlFormat, &FormatCase::new("app.yaml", "name: app\nport: 8080\n", cfg) .bad(["name: [unclosed"]), ); ``` ## Discovery conformance `assert_discovery_conformance(&case, make)` scripts a whole filesystem + env scenario into a temp dir — files, search dirs, env vars (paths resolve against the case root), an optional explicit path — then runs `load_with_env` on whatever source `make` builds from the \[`PreparedCase`]. The env lookup is injected; the process environment is never touched. ```rust use scarab_testkit::{DiscoveryCase, assert_discovery_conformance}; let case = DiscoveryCase::loaded("yaml beats toml", expected) .file("myapp.toml", "name = \"toml\"\n") .file("myapp.yaml", "name: yaml\n") .dir("."); assert_discovery_conformance(&case, |pc| FileSource::new( Discovery::bare("myapp").dirs(pc.dirs().to_vec()), )); ``` `DiscoveryCase::error(name, needle)` expects a failure whose message contains `needle`. Works for `FileSource`, `LayeredFileSource`, and any future source implementing \[`EnvAwareSource`] (`load_with_env`). ## Source contract For sources needing no injected env: `assert_config_source_conformance(source, &expected)` and `assert_config_source_error(source, "needle")` cover the minimal \[`ConfigSource`] contract. ## Merge conformance The pure half of layered loading — run once against `scarab_config::{merge_value, apply_overrides}` themselves, or against your own merge impl: * `assert_merge_value_conformance(&[MergeCase])` — objects merge key-wise recursively; scalars/arrays/null replace; arrays never concatenate * `assert_apply_overrides_conformance(&[OverrideCase])` — dotted-path leaves set in place, intermediate objects materialized ## Template engine conformance `assert_template_engine_conformance(engine, &[TemplateCase])` is engine-agnostic — cases carry source + JSON context + expected output *or* expected error; only success/failure is contracted, never error text. `TemplateCase::output` / `::error` build cases; `assert_template_renders(engine, src, ctx, expected)` is the one-off assertion. ## Fixtures and the canonical model * `fixture_path(rel)` / `fixture_text(rel)` — committed golden files under `scarab-testkit/fixtures/` (`app.yaml`, `app.toml`, `layered/…`) * `TestConfig` / `ServerConfig` — the canonical model the fixtures describe; reuse it, or define your own and reuse the `assert_*` suites unchanged The suites panic on the first violated check — they're test helpers, not libraries. `scarab-testkit`'s own `tests/conformance.rs` runs every suite against the workspace impls and is the worked example. --- --- url: /scarab/explanation.md --- # Explanation * [Architecture](architecture.md) — the two planes, the ports, the invariants * [Concepts](concepts.md) — the domain vocabulary, defined once * [Boundaries](boundaries.md) — owns / doesn't-own, neighbors named * [Decisions](decisions.md) — ADR-lite: decision → alternative → consequence --- --- url: /scarab/explanation/architecture.md --- # Architecture Scarab is ports-and-adapters in two planes: **config loading** and **templated content**. Both are generic — scarab never names a consumer's types. ## The config plane ``` ConfigSource ──load()──▶ T (the port) ▲ ▲ │ impl │ consumer's serde model scarab-config-file MyConfig, TamiasConfig, … │ dispatches by extension ConfigFormat ──parse/render──▶ text (syntax port) ▲ scarab-config-toml / scarab-config-yaml ``` * `scarab-config` is pure: traits, the `Discovery` spec, path helpers, and the JSON merge half (`merge_value` / `apply_overrides`) — no I/O, no formats, no models. * `scarab-config-file` is the discovery impl every org repo copy-pasted: `FileSource` picks the *first* matching file; `LayeredFileSource` merges *every* dir's file plus explicit/override layers into one `T`. Both dispatch by extension across a format list. * `scarab-config-toml` / `-yaml` own only syntax — serde parse/render over any `T`. Formats are separate crates so a consumer can carry just the syntax it wants. Layered loading parses each layer as `serde_json::Value` through the same blanket impls, so `T` itself never needs `Serialize`. Generic-over-`T` is the load-bearing decision: one `FileSource` serves `myco`, `tamias`, `tarsi`, and every future repo because the model stays in the consumer. The trait objects (`&dyn ConfigFormat`) keep `formats` heterogeneous without generics exploding at the call site. Discovery is a *spec*, not a search loop — `FileSource` interprets it, and env/remote `ConfigSource` impls can reuse the same vocabulary (`env_var`, `dirs`, preference order) without inheriting file semantics. ## The template plane ``` tamias pack (type: template) │ tamias.yaml `template:` table ▼ scarab-template ── PackTemplate/TemplateFile/TemplateEngine (pure) │ ▼ tamias-driver-template ── verbs, vars, state file (the ops impl) │ ▼ scarab-template-minijinja ── the one engine impl (v1) ``` * `scarab-template` holds the pack model and path safety — pure domain, engine-agnostic. `merge_context` defines layer precedence; `safe_into`/`safe_src` define what "inside the target" means. `template.extends` composes packs: tamias materializes each ancestor's payload at `.scarab-parent/`, and the driver renders the chain ancestor-first so children override parents on `into` conflicts — inheritance without a fetch capability in the driver. * `tamias-driver-template` is where I/O lives: manifest loading, chain walking, var resolution (ancestor defaults → pack defaults → auto → file → env → args), render+write, digest state. It implements the tamias driver contract, so packs distribute through tamias like every other content type — no bespoke install path. * `scarab-template-minijinja` is deliberately the only engine: minijinja is small, sandboxed by construction, and its `{{ }}`/`{% %}` syntax covers config-file templating without a plugin system. The port exists so a second syntax family can land without touching the model or the driver. ## Why the split Config and templates share a shape — *declared inputs in, files out* — but not a mechanism. Config is *read*: find a file, parse to a typed model. Templates are *write*: render declared sources to declared targets with a variable context. Keeping them as separate ports means a consumer can adopt one without the other, and each plane's invariants (discovery order vs. path safety + digest state) stay local. ## Invariants worth knowing * **Formats prefer YAML.** Within a directory, `yaml` → `yml` → `toml`; across dirs, earlier dirs win regardless of format. * **Named paths fail loudly.** An explicit/`$*_CONFIG` path that misses is an error; discovery candidates skip silently. * **Templates can't escape.** `..`/absolute/empty targets are rejected at validation, before any write. * **Uninstall is digest-exact.** Only files still matching their recorded sha256 are removed; user edits are preserved and reported. * **Layers merge as JSON, not as `T`.** `LayeredFileSource` folds `serde_json::Value` trees — objects key-wise, scalars/arrays replace — before one final deserialize, so merge semantics stay format- and model-agnostic. * **Secrets stay names.** Config surfaces carry secret *names*, never values — cuora owns the values. ## Where it sits Scarab is a **library** in the org's layering — below the engines, consumed through tamias. The stack is **tamias** (package & content manager) → **engines** (myco, kuda, mimus, sepia, sturn, hypha, tarsi, cicada, cypera, peregrin) → **orchestration** (peregrin campaigns) → **surfaces** (pavo, MCP fronts). Scarab doesn't run anywhere itself: engines call `scarab-config` to load their `.yaml`/`.toml`, and tamias distributes `type: template` packs that `tamias-driver-template` renders. It sits beside the engines as shared expertise — the config port every engine agrees on, the template mechanism every pack shares. --- --- url: /scarab/explanation/concepts.md --- # Concepts The domain vocabulary, each term defined once. Field-level contracts live in [Reference](../reference/). ## ConfigSource / ConfigFormat The two config-plane traits. `ConfigSource` finds and loads a `T` (one `load()`); `ConfigFormat` owns one syntax — serde parse/render over text, dispatched by file extension. Both are generic over the consumer's model: scarab never names a `Config` type. See [Architecture](architecture.md#the-config-plane). ## Discovery The spec, not the search loop: `env_var`, `dirs`, format preference, explicit-path handling. `FileSource` interprets it for the filesystem; other `ConfigSource` impls can reuse the same vocabulary. `Discovery::for_app(name)` wires the org conventions — `$_CONFIG`, `[".", "~/.config/"]`, YAML preferred. ## Layered loading `LayeredFileSource` merges one file *per* search dir plus explicit/override layers — JSON-value merge (objects key-wise, scalars/arrays replace) before one final deserialize — so user config is the base and project config lands on top. Contrast `FileSource`, which takes the *first* match. ## Template pack A tamias package of `type: template`: a `template:` table in `tamias.yaml` declaring sources, targets (`into`), variables, and optional `extends` chains. Packs are ordinary tamias content — installed, deployed, and uninstalled by `tamias-driver-template`. See [Reference: template packs](../reference/template-packs). ## TemplateEngine The render port: `{{ }}` vars + `{% %}` blocks over a context. `scarab-template-minijinja` is the only v1 impl — sandboxed by construction, no filesystem or network reach. ## merge_context Layer precedence for template variables: ancestor defaults → pack defaults → auto → file → env → args. Defined once in `scarab-template` so every driver agrees. ## Path safety `safe_into`/`safe_src`: targets and sources must stay inside the pack/target roots — `..`, absolute, and empty paths are rejected at validation, before any write. ## Digest state The driver's record of what it wrote (sha256 per file). Uninstall is exact: only files still matching their digest are removed; user edits survive and are reported. ## Secret names, never values Config surfaces and template vars carry secret *names* (or references); [cuora](https://9aia.com/cuora/) owns resolution. This is a convention the whole org enforces — see [Boundaries](boundaries.md). --- --- url: /scarab/explanation/boundaries.md --- # Boundaries Scarab owns two things: **config loading** (discovery + formats, generic over the model) and **templated content packs** (the `type: template` driver). Everything adjacent belongs to a neighbor. ## Neighbors | Repo | Owns | Scarab's relation | | --------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **tamias** | Content distribution — refs, fetch, store, install pipeline, driver dispatch | Scarab ships *inside* tamias's world: `tamias-driver-template` is a content driver; template packs are tamias packages. Scarab never reimplements fetch/install/store. | | **cuora** | Secret values — sealed, not spilled | Config carries secret *names*, never values. A template var named `api_key` holds a reference/name; cuora resolves it. | | **tarsi** | Release plane — changelog → bump → tag → publish | A consumer: `tarsi.toml`/profile loading rides `scarab-config`. | | **myco / kuda / mimus** | Work ledger, code-map, verification | Consumers of the config port for their own `.yaml`/`.toml` discovery — the exact problem P-017 generalized. | | **hypha** | Pure distributed substrate | The P-015 config crates were ported *here*, not kept there — hypha stays substrate-only. | | **peregrin / sepia / pavo** | Orchestration, sessions, agent UI | May consume either plane; neither defines scarab's contracts. | ## What scarab deliberately is not * **Not a config schema registry.** It loads and renders; it doesn't validate semantics. Your serde model *is* the schema. * **Not a secrets store.** See cuora above — the boundary is enforced by convention (`secret names, never values`) and by cuora owning resolution. * **Not a template marketplace.** Packs are ordinary tamias packages; distribution, provenance, and digests are tamias's job. * **Not consumer-aware.** No crate in scarab may name another repo's `Config` type — that coupling is what the port exists to prevent. ## The naming story *Scarabaeus sacer* — the sacred scarab rolls the ball into place; scarab amulets were seals and signets. Config as the seal that validates what runs: discovery finds the right file, the format parses it, the model validates it. Design history lives in the org ledger — proposal P-017 (supersedes P-015). --- --- url: /scarab/explanation/decisions.md --- # Decisions ADR-lite: the load-bearing decisions, each with the alternative it beat and the consequence it carries. Design history lives in the org ledger — proposal P-017 (supersedes P-015). ## Generic over `T`, never a consumer's model **Decision:** `ConfigSource`/`ConfigFormat` are generic over the consumer's serde model — scarab never names a `Config` type. * *Alternative:* a scarab-owned universal config schema. * *Consequence:* one `FileSource` serves myco, tamias, tarsi, and every future repo; each consumer keeps only its serde model. Cost: trait objects at the call site to keep generics from exploding. ## Discovery is a spec, not a search loop **Decision:** `Discovery` declares `env_var`, `dirs`, format preference, explicit-path handling; `FileSource` interprets it. * *Alternative:* hard-coded per-repo search logic (the `gen-*.py`/copy-paste status quo). * *Consequence:* env/remote `ConfigSource` impls reuse the same vocabulary without inheriting file semantics. ## Formats are separate crates **Decision:** `scarab-config-toml` and `-yaml` own only syntax; `scarab-config` stays pure (traits, `Discovery`, merge helpers). * *Alternative:* one crate with all formats baked in. * *Consequence:* a consumer carries just the syntax it wants; the port stays I/O-free and unit-testable. ## Config and templates are separate planes **Decision:** config is *read* (find, parse, model); templates are *write* (render sources to targets with a var context) — two ports, two invariants. * *Alternative:* one unified "content" port. * *Consequence:* a consumer adopts one plane without the other; discovery order vs. path-safety+digest invariants stay local. ## MiniJinja is the only engine (v1) **Decision:** `scarab-template-minijinja` is deliberately the sole `TemplateEngine` impl — sandboxed by construction. * *Alternative:* a plugin system for template syntax. * *Consequence:* `{{ }}`/`{% %}` covers config-file templating without attack surface; the port exists so a second syntax family can land without touching model or driver. ## Uninstall is digest-exact **Decision:** the driver records sha256 per written file; uninstall removes only files still matching. * *Alternative:* remove everything the pack declared. * *Consequence:* user edits survive uninstall and are reported — templated files are owned only until touched. ## Secrets stay names **Decision:** config surfaces and template vars carry secret *names*, never values; cuora owns resolution. * *Alternative:* templated secret injection. * *Consequence:* packs are safe to distribute and review; the boundary is convention plus cuora owning the values.