# apiscp-borg reference

Exhaustive reference for every command, verb, configuration key, installer, and
automation shipped in this repository. For the design rationale, and for how
Borg's own metadata handling changes the approach taken by the sibling
apiscp-kopia project, see `docs/DESIGN.md`.

## 1. Overview

apiscp-borg is a backup engine for ApisCP built on top of
[BorgBackup](https://www.borgbackup.org/). Unlike kopia, Borg preserves POSIX
ACLs and extended attributes natively on Linux, so no ACL/xattr sidecar
mechanism is needed to protect the permission data ApisCP encodes in those
mechanisms: `borg create` captures them and `borg extract` restores them on its
own. The plugin's value is instead: running ApisCP's own per-site database
export before archiving so the logical `.sql` dumps are current, archiving
each site as its own named set of Borg archives so single-site restore and
per-site retention are possible, and wiring retention, maintenance, integrity
checking, scheduling, notifications, one-time key backup, and both a
CLI and a native ApisCP panel integration around the `borg` binary. It ships
as two independent layers: a standalone POSIX-sh engine (Layer 1, three
binaries plus a systemd timer, with zero ApisCP coupling) and a native ApisCP
integration (Layer 2, a module plus two GUI apps installed into upgrade-safe,
git-ignored locations). Layer 2 provides two editions of the panel:

- **Appliance-admin edition ("Borg Backups")**: full repository configuration,
  backup selection, scheduling, retention (prune), maintenance (compact),
  integrity checking, notifications, one-time key backup, and restore (site,
  account, system, or database) for any site on the box. `PRIVILEGE_ADMIN`.
- **Site-owner edition ("My Borg Backups")**: a scoped self-service panel for
  a single signed-in site owner to restore their own files, databases, or
  whole account, and to trigger a backup of their own site on demand. It never
  accepts a site identifier from the caller; the site is always derived from
  the authenticated session. `PRIVILEGE_SITE`.

Borg's repository model is simpler than kopia's: there are no backend
"types", no repository server, and no SFTP transport. A repository is just a
location (`BORG_REPO`, a local path or an `ssh://` URL), a `BORG_PASSPHRASE`,
a `BORG_ENCRYPTION` mode (`repokey-blake2` by default, so the encryption key
lives inside the repository and only the passphrase is needed to recover it),
and an optional `BORG_RSH` for SSH transport options. There are no tagged
snapshots either: Borg has named archives, and this engine names them with a
structured prefix (`siteN-shadow`, `siteN-info`, `siteN-db`, `_system`,
`_custom-<slug>`, each suffixed with Borg's `{now}` timestamp placeholder) so
per-site operations are glob-addressable. Restore in this release always uses
the **newest** archive of a given prefix: there is no point-in-time archive
selection and no browse/subpath picker in the panel (a current limitation,
noted throughout this document and in the GUI itself).

## 2. Installation

### `install.sh` (Layer 1: the engine)

Installs the standalone engine. Must be run as root. Idempotent.

```sh
sudo sh install.sh
```

Unlike apiscp-kopia's installer, this script does **not** install `borg`
itself: it only checks whether `borg` is on `PATH` and prints a warning if it
is not. Install `borgbackup` yourself first (`dnf install borgbackup` /
`apt install borgbackup`).

Flags:

| Flag | Effect |
|---|---|
| (none) | Installs the library, binaries, and systemd units; seeds the config. |
| `-h`, `--help` | Prints the header usage comment and exits 0. |

What it installs, in order:

| Item | Source | Destination | Mode |
|---|---|---|---|
| shared library | `lib/apiscp-borg-common.sh` | `/usr/local/lib/apiscp-borg/apiscp-borg-common.sh` | 0644 |
| engine binary | `bin/borg-apiscp-backup` | `/usr/local/bin/borg-apiscp-backup` | 0755 |
| repo binary | `bin/borg-apiscp-repo` | `/usr/local/bin/borg-apiscp-repo` | 0755 |
| restore binary | `bin/borg-apiscp-restore` | `/usr/local/bin/borg-apiscp-restore` | 0755 |
| systemd service | `systemd/borg-apiscp-backup.service` | `/etc/systemd/system/borg-apiscp-backup.service` | 0644 |
| systemd timer | `systemd/borg-apiscp-backup.timer` | `/etc/systemd/system/borg-apiscp-backup.timer` | 0644 |
| config seed | `etc/apiscp-borg.config.example` | `/etc/apiscp-borg/config` | 0600, only if absent (never overwrites an existing config) |

`systemctl daemon-reload` is run after the units are installed. The script
does **not** enable the timer, install `borg`, or create/connect the Borg
repository; it prints the manual next steps (install `borgbackup`, set
`BORG_REPO`/`BORG_PASSPHRASE`, initialise the repository, back up the
passphrase/key off-machine, one test run, `systemctl enable --now
borg-apiscp-backup.timer`).

### `install-layer2.sh` (Layer 2: native ApisCP integration)

Installs the panel module, both GUI apps, both menu links, the account
lifecycle hooks, and a scoped sudoers file. Run as root, after `install.sh`.
Idempotent. Respects `CP_ROOT` (default `/usr/local/apnscp`) and `PANEL_USER`
(default `apnscp`).

| Step | Source | Destination | Notes |
|---|---|---|---|
| Module | `src/modules/borg.php` | `$CP_ROOT/lib/modules/surrogates/borg.php` | git-ignored by ApisCP (`surrogates/*`), survives `upcp` |
| Admin GUI app | `src/apps/borg/` (borg.php, application.yml, views/index.blade.php) | `$CP_ROOT/config/custom/apps/borg/` | whole tree ignored |
| Site-owner GUI app | `src/apps/myborgbackups/` | `$CP_ROOT/config/custom/apps/myborgbackups/` | whole tree ignored |
| Admin menu link | `src/templates/admin.php` | `$CP_ROOT/config/custom/templates/admin.php` | only written if the file does not already exist; if it exists and already references `/apps/borg`, left untouched; if it exists, does not reference `/apps/borg`, and contains a closing `?>`, a warning is printed and nothing is appended; otherwise the `create_link()` call is appended |
| Site menu link | `src/templates/site.php` | `$CP_ROOT/config/custom/templates/site.php` | same append-not-clobber logic, keyed on `/apps/myborgbackups` |
| Account hooks | `src/hooks/{addDomain,suspendDomain,deleteDomain}.sh` | `$CP_ROOT/config/custom/hooks/<event>.sh` | only installed if the destination does not exist, or exists but does not already contain the string `apiscp-borg` (an operator's own hook is left untouched) |
| Sudoers | generated inline | `/etc/sudoers.d/apiscp-borg` | 0440 root:root, validated with `visudo -cf` before install; installation is skipped with a warning if validation fails |

After installing the sudoers file, the script forces the engine config to be
root-owned (`chown root:root` + `chmod 0600` on `/etc/apiscp-borg/config` if
it exists), then restarts the panel (`systemctl restart apnscp`) so the new
GUI apps and menu entries are picked up; the module itself needs no restart
and is picked up on the next CLI/API call. Site owners see the "My Borg
Backups" menu entry after their next login (the menu is session-cached).

**Upgrade-safe locations.** Every Layer 2 destination is one ApisCP
git-ignores, so a `git pull` / `upcp` upgrade never touches it:
`lib/modules/surrogates/*`, and the entirety of `config/custom/` (via its own
`.gitignore` of `*` plus `!.gitignore`).

## 3. Uninstall

`uninstall.sh`, run as root, idempotent.

```sh
sudo sh uninstall.sh            # default: keep config, staging, key backups
sudo sh uninstall.sh --purge    # also delete config, staging dir, key backup dir
```

Always removed (both modes):

- Layer 2: the module surrogate, both GUI app directories (`borg` and
  `myborgbackups`), the account hooks (only the ones containing the string
  `apiscp-borg`), the scoped sudoers file, and the `apiscp-borg` lines
  stripped out of `admin.php`/`site.php` (the template file itself is deleted
  only if nothing but `<?php` and whitespace remains after stripping). The
  panel is restarted afterwards.
- Layer 1: the timer is disabled and stopped, the service is stopped, both
  systemd unit files and any `borg-apiscp-backup.timer.d` schedule drop-in are
  removed, `systemctl daemon-reload` runs, the three binaries and the shared
  library directory are removed, and the Prometheus textfile metric
  (`apiscp-borg.prom`) is deleted from any of the well-known collector
  directories.

Kept by default (i.e. **not** removed unless `--purge`):

- `/etc/apiscp-borg/config` (holds the repository passphrase)
- `/var/lib/apiscp-borg` (the staging area; also where the deleted-accounts
  audit log lives)
- the key backup directory (`KEY_BACKUP_DIR`, read out of the config before
  it would be deleted)

With `--purge`, the config directory and staging directory are deleted; the
key backup directory is deliberately **not** auto-deleted even under
`--purge` (its path is only printed, with an explicit `rm -rf` suggestion),
because it may be the only offline copy of the repository passphrase/key.

**Guarantee, always true, in both modes:** the Borg repository and every
archive in it are never touched by `uninstall.sh`, and the `borg` binary
itself is never removed. Uninstalling apiscp-borg does not affect your
ability to recover backups with plain `borg`, or to reinstall and pick up
where you left off.

## 4. Configuration reference

Config lives at `/etc/apiscp-borg/config`, POSIX-sh `KEY=value` syntax (no
spaces around `=`), sourced directly by the three binaries. Every key has a
built-in default so the engine runs unconfigured on a stock ApisCP box (other
than `BORG_REPO`, which is required before any real operation).
`etc/apiscp-borg.config.example` documents the same defaults and is seeded
(never overwritten) by `install.sh`.

`CONFIG_ALLOWED` in `bin/borg-apiscp-repo` (`config-set`) is the authoritative
whitelist of keys settable through the CLI/GUI (the module's
`Borg_Module_Surrogate::ALLOWED_KEYS` mirrors it exactly). A `config-set` /
`set_option` call for any other key is rejected. Several engine-internals keys
(`VIRTBASE`, `STAGE`, `SYSTEM_PATHS`, `SYSTEM_GLOBS`, `APNSCP_CMD`,
`APNSCP_ROOT`, `DB_EXPORT_CMD`, `LOCK_FILE`) are read from the config file if
present but are **not** in `CONFIG_ALLOWED`; they can only be changed by
editing the config file directly, not via `config-set`, `set_option`, or
either GUI.

### Repository

| Key | Meaning | Default | Example |
|---|---|---|---|
| `BORG_REPO` | Repository location: a local path or an `ssh://` URL | (none, required) | `ssh://borg@backup.example.net:22/./apiscp-borg` |
| `BORG_PASSPHRASE` | Repository passphrase, exported into the environment for every `borg` invocation | (none) | generated via `gen-passphrase` |
| `BORG_ENCRYPTION` | Encryption mode used only when **creating** a new repository | `repokey-blake2` | `keyfile-blake2` |
| `BORG_RSH` | Extra SSH options passed to `borg` via `BORG_RSH`, for `ssh://` repositories | `ssh -o BatchMode=yes` | `ssh -o BatchMode=yes -i /root/.ssh/borg_key` |
| `BORG_COMPRESSION` | Compression algorithm passed to `borg create --compression` | `zstd` | `lz4`, `zstd,15`, `none` |
| `KEY_BACKUP_DIR` | Where `backup-key` writes the one-time passphrase/key export | `/root/apiscp-borg-keys` | (none) |

`repokey-blake2` (the default) stores the encryption key inside the
repository itself, so only the passphrase is needed to recover it.
`keyfile-blake2` stores the key under `~/.config/borg` on the machine that
created the repository and must be backed up separately (`borg key export`,
handled automatically by `backup-key`/`key-bundle`).

### What to back up

| Key | Meaning | Default | Example |
|---|---|---|---|
| `BACKUP_SITES` | `all`, `none`, or a comma-separated list of site ids | `all` | `site1,site3` |
| `BACKUP_SYSTEM` | Back up the shared system paths as well as per-site data: `1` or `0` | `1` | `0` |
| `BACKUP_DATABASES` | Dump every site's databases via ApisCP's native per-site export and archive them: `1` or `0` | `1` | `0` |
| `BACKUP_PATHS` | Extra absolute paths to back up, comma or space separated | (empty) | `/srv/extra,/opt/thing` |
| `SYSTEM_PATHS` | Shared, non-per-site paths, space separated (not settable via CLI/GUI) | `/etc /opt` | (as default) |
| `SYSTEM_GLOBS` | Shell globs (space separated) expanded at run time; non-matching globs are skipped, not passed literally to `borg` (not settable via CLI/GUI) | `/var/log/mailer_table* /var/lib/mysql/mysql-grants* /var/lib/pgsql/*/backups /root/apnscp* /root/license*` | (as default) |
| `CAPTURE_META_SIDECAR` | Whitelisted config key reserved for an optional ACL/xattr text sidecar capture, off by default; Borg already preserves ACLs/xattrs natively in every archive it creates, and no sidecar-capture logic is implemented in the current engine regardless of this setting | `0` | `1` |

### Retention (`borg prune`)

Applied per archive-name prefix (each site's `shadow`, `info`, and `db`
archives separately, plus `_system-*` and `_custom-*`), both automatically
after every backup run and on demand via the `prune` verb/button. Any unset
or blank value means no limit for that bucket. There is no `KEEP_LATEST` and
no `KEEP_HOURLY` bucket (unlike kopia).

| Key | Meaning | Default | Example |
|---|---|---|---|
| `RETENTION_KEEP_WITHIN` | Keep every archive created within this duration (Borg's `--keep-within`, a number plus `H`/`d`/`w`/`m`/`y`) | (none / unlimited) | `7d` |
| `RETENTION_KEEP_DAILY` | Keep 1 per day for this many days (`--keep-daily`) | (none / unlimited) | `30` |
| `RETENTION_KEEP_WEEKLY` | Keep 1 per week for this many weeks (`--keep-weekly`) | (none / unlimited) | `52` (1 year) |
| `RETENTION_KEEP_MONTHLY` | Keep 1 per month for this many months (`--keep-monthly`) | (none / unlimited) | `24` (2 years) |
| `RETENTION_KEEP_ANNUAL` | Keep 1 per year for this many years (Borg's `--keep-yearly`) | (none / unlimited) | `5` |

### Excludes

| Key | Meaning | Default | Example |
|---|---|---|---|
| `BACKUP_EXCLUDES` | `borg create --exclude` glob patterns, comma-separated on disk (one per line in the GUI textarea) | (empty) | `*.log,cache/,node_modules/` |

`*/.borg-restore` and `/.borg-restore` are **always** excluded (hardcoded in
both `create_archive`/`_create_with_excludes`), so site-owner self-service
restores staged under that path are never recaptured into a subsequent
archive.

### Schedule

| Key | Meaning | Default | Example |
|---|---|---|---|
| `BACKUP_SCHEDULE` | Time to run backups: `HH:MM` for a daily time, or a full systemd `OnCalendar` expression | `03:30` | `Mon *-*-* 03:30:00` |

### Notifications

| Key | Meaning | Default | Example |
|---|---|---|---|
| `NOTIFY_EMAIL` | Recipient address; empty disables email notification entirely | (empty) | `ops@example.com` |
| `NOTIFY_ON` | `failure` (default, only email on a failed/incomplete run), `always`, or `never` | `failure` | `always` |
| `NOTIFY_FROM` | Override the `From:` address | `root@$(hostname)` | `backups@example.com` |

### Engine internals

| Key | Meaning | Default | Settable via CLI/GUI? |
|---|---|---|---|
| `VIRTBASE` | ApisCP virtual host base; sites are `$VIRTBASE/site<N>` | `/home/virtual` | yes |
| `STAGE` | Reserved working area for the engine (not exercised by any active code path in this release) | `/var/lib/apiscp-borg` | yes |
| `METRICS_DIR` | Prometheus node_exporter textfile collector directory; blank autodetects common locations, and metrics are skipped if none is found | (empty / autodetect) | yes |
| `RUN_DB_EXPORT` | Run ApisCP's per-site database export (`backup_dbs.php`) before archiving: `1` or `0` | `1` | yes |
| `DB_EXPORT_CMD` | Path to `backup_dbs.php` | `$APNSCP_ROOT/bin/scripts/backup_dbs.php` | no |
| `APNSCP_CMD` | ApisCP CLI binary, used for native per-site database verbs (`list_databases`/`export`/`import`) | `/usr/local/apnscp/bin/cmd` | no |
| `APNSCP_ROOT` | Path to the ApisCP installation | `/usr/local/apnscp` | no |
| `LOCK_FILE` | Single-instance lock file guarding against overlapping backup runs (via `flock`, if available) | `/run/apiscp-borg.lock` | no |

Two further variables are read only via the environment, not the config file
whitelist: `APISCP_BORG_LIB` (override the shared-library search path) and
`APISCP_BORG_CONFIG` (override the config file path itself, default
`/etc/apiscp-borg/config`), plus `APISCP_BORG_LOG_TAG` (the `logger` tag used
by `akp_log`, default `apiscp-borg`) and `APISCP_BORG_DELETED_LOG` (used only
by the `deleteDomain` hook, default `/var/lib/apiscp-borg/deleted-accounts.log`).
`BORG_RELOCATED_REPO_ACCESS_IS_OK` is exported by the engine (default `no`,
overridable via the real environment) so an unattended run does not block on
Borg's interactive "repository moved" prompt.

## 5. CLI reference (Layer 1)

All three binaries source `lib/apiscp-borg-common.sh` (search order:
`$APISCP_BORG_LIB`, `../lib/` relative to the binary,
`/usr/local/lib/apiscp-borg/`, `/usr/lib/apiscp-borg/`) then
`$APISCP_BORG_CONFIG` (default `/etc/apiscp-borg/config`). Every subcommand
below matches the code exactly.

### `borg-apiscp-backup`

Takes no subcommands; running it (with no arguments) performs one full backup
run and exits 0 if every archive was created, 1 if any failed. Driven by the
systemd timer/service, or launched detached by `borg-apiscp-repo run`.

Sequence on every run:

1. Acquire `$LOCK_FILE` via `flock` (non-blocking; a second concurrent run
   aborts immediately) if `flock` is available.
2. If `RUN_DB_EXPORT=1`, run `$DB_EXPORT_CMD` (ApisCP's `backup_dbs.php`) so
   per-site `.sql` dumps are current before they are archived.
3. For every `$VIRTBASE/site<N>` directory (skipping any not selected by
   `BACKUP_SITES`):
   - archive `shadow` (if present) as `site<N>-shadow-{now}`;
   - archive `info` (if present) as `site<N>-info-{now}`;
   - if `BACKUP_DATABASES=1`: enumerate the site's `mysql` and `pgsql`
     databases via `$APNSCP_CMD -o json -d <domain> <engine>:list_databases`,
     export each to a site-relative staging path
     (`/tmp/.apiscp-borg-db/<engine>/<db>.sql`, resolved inside the site's
     `fst` namespace by ApisCP), archive that staging tree from its real host
     location (`<site>/fst/tmp/.apiscp-borg-db`) as `site<N>-db-{now}`, then
     delete the staging tree from the site;
   - apply per-archive-prefix retention (`borg prune`) for
     `site<N>-shadow-*`, `site<N>-info-*`, and `site<N>-db-*` right away, so
     one site's daily churn is pruned every run rather than waiting for a
     separate step.
4. If `BACKUP_SYSTEM=1`: expand `SYSTEM_PATHS` and matching `SYSTEM_GLOBS`
   entries to those that actually exist, archive them together as
   `_system-{now}`, then prune `_system-*`.
5. If `BACKUP_PATHS` is non-empty: for each existing custom path, archive it
   as `_custom-<slug>-{now}` (slug via `akp_slug`), then prune
   `_custom-<slug>-*`.
6. Write Prometheus textfile metrics (always, even on failure or interrupt,
   via an `EXIT`/`INT`/`TERM` trap) and send an email notification per
   `NOTIFY_EMAIL`/`NOTIFY_ON`/`NOTIFY_FROM`.

Every archive is created with `borg create --compression
"${BORG_COMPRESSION:-zstd}"`, plus `--exclude '*/.borg-restore' --exclude
'/.borg-restore'` and one `--exclude` per `BACKUP_EXCLUDES` entry.

Metrics written to `$METRICS_DIR/apiscp-borg.prom` (or autodetected):
`apiscp_borg_run_success`, `apiscp_borg_archives_ok`,
`apiscp_borg_archives_failed`, `apiscp_borg_duration_seconds`,
`apiscp_borg_db_export_success` (1 ok / 0 failed / -1 skipped), and
`apiscp_borg_last_run_timestamp_seconds`.

### `borg-apiscp-repo`

Manages the repository and everything that is not a data restore.

| Subcommand | Syntax | Description |
|---|---|---|
| `status` | `borg-apiscp-repo status` | Print `repository: $BORG_REPO` followed by `borg info` (first 30 lines, indented). |
| `init` | `borg-apiscp-repo init` | Create (initialise) the repository at `BORG_REPO` with `BORG_ENCRYPTION`. Requires `BORG_PASSPHRASE` already set. Idempotent: if `borg info` already succeeds against the repository, logs and returns 0 without re-initialising. |
| `check-access` | `borg-apiscp-repo check-access` | Print `ok` if the repository is reachable and the passphrase is correct, else `error: cannot access ...`. Always exits 0 so a UI can read the message. |
| `gen-passphrase` | `borg-apiscp-repo gen-passphrase` | Generate a strong random passphrase (`openssl rand -base64 30`, or `/dev/urandom` fallback), store it as `BORG_PASSPHRASE` via `config-set`, and print **only** the passphrase to stdout. |
| `key-status` | `borg-apiscp-repo key-status` | Print `yes` if the one-time key/passphrase backup sentinel exists at `$KEY_BACKUP_DIR/.apiscp-borg-key-backed-up`, else `no`. |
| `backup-key` | `borg-apiscp-repo backup-key [DEST] [--force]` | Export `repository.passphrase` (if `BORG_PASSPHRASE` is set), `repository.key` (via `borg key export`, which is a no-op warning for `repokey` repositories since the key already lives in the repository), and a `README.txt` to `DEST` (default `$KEY_BACKUP_DIR`), all 0600. Refuses to redo (logs and returns 0) unless `--force`. |
| `key-bundle` | `borg-apiscp-repo key-bundle` | Print a self-contained recovery bundle (instructions, the plaintext passphrase, the repository location, and an exported key via `borg key export :: /dev/stdout`) to stdout. For the GUI's one-time post-creation display/download. |
| `config-get` | `borg-apiscp-repo config-get` | Print every `KEY=VALUE` line from the config file with surrounding quotes stripped (comments and malformed lines skipped). |
| `config-set` | `borg-apiscp-repo config-set KEY VALUE` | Set one config key, rejecting any key not in `CONFIG_ALLOWED` and any value containing a CR or LF. Value is stored single-quoted (shell-escaped) so the config, which is sourced by the engine, cannot be injected. Writes with mode 0600. |
| `sites` | `borg-apiscp-repo sites` | List `site<N>\t<domain>` for every ApisCP site under `$VIRTBASE` (domain falls back to the site id if `siteinfo` cannot be read). |
| `run` | `borg-apiscp-repo run [--site SITE \| --path PATH]` | No args (or `--all`): `systemctl start --no-block borg-apiscp-backup.service` (returns immediately). `--site SITE`: launch a detached one-shot run scoped to just that site (`BACKUP_SITES=SITE BACKUP_SYSTEM=0`); `SITE` must match `site[0-9]*` or be the literal `none`. `--path PATH`: launch a detached one-shot backing up only that one absolute, existing, readable path (`BACKUP_SITES=none BACKUP_SYSTEM=0 BACKUP_PATHS=PATH`). Detached runs use `systemd-run --collect` if available, else a backgrounded `env ... &`. |
| `running` | `borg-apiscp-repo running` | Print `systemctl is-active borg-apiscp-backup.service` (e.g. `active`, `inactive`). |
| `prune` | `borg-apiscp-repo prune` | Apply retention (`RETENTION_KEEP_*`, numeric/duration values only) across every archive prefix: each site's `shadow`/`info`/`db` archives separately, plus `_system-*` and `_custom-*`. If no retention key is set, logs and does nothing. |
| `maintenance` | `borg-apiscp-repo maintenance` | Run `borg compact`, detached (`systemd-run --collect` if available, else backgrounded). |
| `verify` | `borg-apiscp-repo verify [--data]` | Run `borg check` (repository and archive consistency), detached. With `--data`, adds `--verify-data` to also read file contents back (slow, thorough); without it, only metadata is checked. |
| `set-schedule` | `borg-apiscp-repo set-schedule [SPEC]` | Set the backup timer's `OnCalendar`. `SPEC` is `HH:MM` (validated 00-23:00-59, becomes `*-*-* HH:MM:00`) or a full `OnCalendar` expression (validated against an allowed character set); falls back to `BACKUP_SCHEDULE` if `SPEC` is omitted. Writes `/etc/systemd/system/borg-apiscp-backup.timer.d/override.conf`, reloads systemd, restarts the timer. Prints the resulting `OnCalendar` value. |
| `notify-test` | `borg-apiscp-repo notify-test` | Send a test email to `NOTIFY_EMAIL` to confirm mail delivery; fails if `NOTIFY_EMAIL` is unset. |
| `list-databases` | `borg-apiscp-repo list-databases SITE` | Print `<engine>\t<db>` (engine `mysql` or `pgsql`) for every database ApisCP currently knows for `SITE`, via native per-site enumeration (a host-level dump cannot see per-site databases). `SITE` must match `site[0-9]*`. |
| (no args / `-h`/`--help`/`help`) | | Print the full header usage comment (via `awk`) and exit 0. |
| (anything else) | | Print `unknown command: <cmd>` to stderr and exit 2. |

There is no `connect`, `create`, `types`, `repo-info`, `validate-path`,
`browse`, or `apply-policy` subcommand: those either do not apply to Borg's
simpler repository model, or (in the case of `repo-info` and `validate-path`)
are computed directly in the Layer 2 module without a root round trip (see
section 6).

### `borg-apiscp-restore`

Restores data from a Borg archive. Borg stores archived paths relative
(leading `/` stripped) and preserves POSIX ACLs and extended attributes
natively as part of every archive, so restore is a plain `borg extract` with
`--strip-components` to rebase the archived path under a staging target: no
separate metadata-replay step is needed (that is the kopia engine's job, not
Borg's). Archive naming (from the engine): `siteN-shadow-{now}`,
`siteN-info-{now}`, `siteN-db-{now}`, `_system-{now}`,
`_custom-<slug>-{now}`. "Newest" means the most recent archive matching a
given prefix (`borg list --glob-archives PREFIX-* --last 1`).

| Subcommand | Syntax | Description |
|---|---|---|
| `list` | `borg-apiscp-restore list [PREFIX]` | List archives, optionally filtered to `PREFIX-*` (e.g. `site1`), else every archive in the repository. |
| `list-json` | `borg-apiscp-restore list-json PREFIX` | Print raw `borg list --json --glob-archives PREFIX-*`. `PREFIX` is required. Intended for the module/GUI to parse. |
| `site` | `borg-apiscp-restore site SITE TARGET [--subpath P ...]` | Restore `SITE`'s newest `shadow` and `info` archives into `TARGET/{shadow,info}`. With one or more `--subpath`, restores only those paths (relative to the site root, under `shadow/`) instead of the whole tree; each subpath is rejected if it starts with `/` or contains `..`. Fails if nothing was restored. |
| `account` | `borg-apiscp-restore account SITE TARGET` | Restore a whole account: files (`shadow`+`info`, fails hard if none exist) plus the newest `SITE-db` archive as a tree into `TARGET/databases` (warns, does not fail, if no database archive exists). |
| `system` | `borg-apiscp-restore system TARGET` | Restore the newest `_system` archive into `TARGET` via `borg extract --numeric-ids`. |
| `restore-db` | `borg-apiscp-restore restore-db SITE ENGINE DB TARGET` | Non-destructive: extract `ENGINE/DB.sql` from the newest `SITE-db` archive into `TARGET/DB.sql`, print the restored path. `ENGINE` must be `mysql` or `pgsql`; `DB` must match `[A-Za-z0-9_-]+`. |
| `import-db` | `borg-apiscp-restore import-db SITE ENGINE DB` | **Destructive.** Restores the newest dump of `SITE`'s `DB` to a temp directory (there is no `DUMPFILE` argument: the newest archived dump is always the one imported), stages it inside the site's own `fst` tree, and imports it into the site's live database via `$APNSCP_CMD -d <domain> <engine>:import`, then removes the staged copy. |
| `owner-files` | `borg-apiscp-restore owner-files SITE [--subpath P ...]` | Site-owner variant of `site`: restores into the site's own filesystem under `/.borg-restore/files-<epoch>/`, chowned to the site owner, and prints the in-site relative path. |
| `owner-account` | `borg-apiscp-restore owner-account SITE` | Site-owner whole-account restore into `/.borg-restore/account-<epoch>/` (files, plus databases under `databases/`), chowned to the owner. Prints the in-site relative path. |
| `owner-restore-db` | `borg-apiscp-restore owner-restore-db SITE ENGINE DB` | Site-owner, non-destructive: restore the newest dump of the site's own database into `/.borg-restore/db-<epoch>/DB.sql`, chowned to the owner. Prints the in-site relative path. |
| `owner-import-db` | `borg-apiscp-restore owner-import-db SITE ENGINE DB` | Site-owner, **destructive**: equivalent to `import-db` for the site's own database. |
| (no args / `-h`/`--help`/`help`) | | Print the header usage comment and exit 0. |
| (anything else) | | Print `unknown command: <cmd>` to stderr and exit 2. |

Every "owner-*" subcommand takes `SITE` as an argument, but in every real
invocation path that argument is supplied by the trusted Layer 2 module
(`borg.php`), which derives it from `$this->getAuthContext()->site` (the
authenticated session), never from a value a site user could control. The
Layer 1 binaries additionally validate `SITE` matches `site[0-9]*`.

**Current limitation, by design in this release:** restore always uses the
newest archive of a matching prefix. There is no way, from the CLI or the
GUI, to restore an older archive by id/time, and `owner-files`/`site` support
subpath filtering only within the `shadow` tree, not a general browse picker.
`import-db`/`owner-import-db` take no dump-file argument at all: they always
restore-and-import the newest `SITE-db` archive's dump for the named
database.

## 6. Panel/API reference (Layer 2)

`Borg_Module_Surrogate` (`src/modules/borg.php`, installed as
`lib/modules/surrogates/borg.php`) exposes every verb below as `cpcmd
borg:<verb>` and to the two GUI apps (`Page_Container::__call` forwards
`$this->borg_<verb>(...)` calls to the module). Every verb runs one of the
three Layer 1 binaries as root via a scoped `sudo -n`, with a positional
printf-style format string so `Util_Process` escapes every argument (see
section 9).

### Admin verbs (`PRIVILEGE_ADMIN`)

| Verb | Parameters | Description |
|---|---|---|
| `get_config` | | Return the current config as an associative array, with any key containing `PASSPHRASE` masked to `********`. |
| `set_option` | `$key, $value` | Set one whitelisted config key (client-side newline check only; `config-set` is the authoritative validator). |
| `status` | | Return `borg-apiscp-repo status` output. |
| `check_access` | | Return `borg-apiscp-repo check-access` output (`ok` or an error message). |
| `key_backed_up` | | `true` if `key-status` reports `yes`. |
| `backup_key` | `$dest = null, $force = false` | Run `backup-key`, optionally to a custom destination, optionally forced. |
| `key_bundle` | | Return the one-time recovery bundle text (secrets included by design). |
| `run` | `$site = '', $path = ''` | Trigger a backup: `$path` takes precedence over `$site`; with neither, runs everything. `$site` is validated against `^site\d+$`. |
| `is_running` | | `true` if the backup service is currently active. |
| `sites` | | Return `siteN => domain` for every ApisCP site, sorted by key. |
| `site_snapshots` | `$site` | Archives available for a site's files, newest first, as `[{id,time}, ...]`; the archive name is the id. Informational only: restore always uses the newest. |
| `restore_site` | `$site, $target` | Restore a site's newest `shadow`+`info` archives into a staging directory. There is no point-in-time selection or subpath picker exposed here (unlike the Layer 1 `site` subcommand's `--subpath`); the whole site tree is always restored. |
| `init_repo` | | Create (initialise) the configured repository. Idempotent; requires a passphrase already set. |
| `gen_passphrase` | | Generate and store a strong repository passphrase; returns it once. |
| `set_backup_selection` | `$sites, $system, $paths = []` | Set `BACKUP_SITES` (collapsing to `all` if the selection covers every known site, `none` if empty, otherwise a CSV), `BACKUP_SYSTEM`, and `BACKUP_PATHS` (each path re-validated with `validate_path`). |
| `set_schedule` | `$when` | Validate `$when` as `HH:MM` or a safe `OnCalendar`-like expression, store it in `BACKUP_SCHEDULE`, then call `set-schedule`. |
| `maintenance` | | Run `borg compact`, detached. |
| `verify` | `$data = false` | Run `borg check`, detached; `$data` adds `--verify-data` (reads file contents back). |
| `notify_test` | | Send a test notification email. |
| `prune` | | Apply retention (`borg prune`) across every archive prefix right now, per the configured `RETENTION_KEEP_*` settings. Unlike kopia's global policy, there is no separate "apply policy" step: retention lives in config, and `prune` (also run automatically after each backup) is what enforces it. |
| `repo_info` | | Return `['connected'=>bool, 'location'=>string]`, computed from `get_config()['BORG_REPO']` and a live `check-access` call; there is no repository "type" in the Borg model. |
| `validate_path` | `$path` | `true` if the path is a plausible absolute path (non-empty, starts with `/`, no CR/LF). Unlike kopia's `validate-path`, there is no corresponding root-level CLI check: this is a best-effort structural check performed entirely in PHP, not authoritative (a path that does not actually exist when the engine runs is simply skipped with a warning in the log). |
| `sites_with_snapshots` | | Site ids that have at least one `siteN-shadow` archive (for greying out empty sites in the restore picker). |
| `site_databases` | `$site` | `[{engine, db}, ...]` currently known for a site (live, from ApisCP, not from an archive). |
| `database_backups` | `$site` | `[{id, time}, ...]` (newest first) of dated `siteN-db` archives for a site. Informational: restore/import always use the newest. |
| `restore_database` | `$site, $engine, $db, $target` | Non-destructive restore of one database dump to a staging path. |
| `restore_account` | `$site, $target` | Restore a whole account (files + all databases) to a staging directory. |
| `import_database` | `$site, $engine, $db` | **Destructive** import of the newest archived dump into the live database. No dump-file parameter: `import-db` always restores and imports the newest `siteN-db` archive itself. |
| `restore_system` | `$target` | Restore the shared system archive to a staging directory. No subpath parameter is exposed (unlike kopia's `restore_system`). |

### Site-owner verbs (`PRIVILEGE_SITE`)

Every verb below resolves the caller's own site from the ApisCP auth context
via a private `authSite()` helper (`$this->getAuthContext()->site`, validated
against `^site\d+$`) and never accepts a site identifier as a parameter. Where
a database is named, it is additionally checked to belong to that resolved
site (`ownsDatabase()`) before use.

| Verb | Parameters | Description |
|---|---|---|
| `my_site` | | `['site'=>siteN\|'', 'domain'=>string]` for the caller. |
| `my_snapshots` | | `[{id,time}, ...]` archives available for the caller's own site. Informational only; restore always uses the newest. |
| `restore_my_files` | | Restore the caller's own files (newest archive) into their own site filesystem (chowned to them), returning the in-site relative path. No point-in-time or subpath selection in this release. |
| `my_databases` | | `[{engine,db}, ...]` for the caller's own site. |
| `my_database_backups` | | `[{id,time}, ...]` dated database archives for the caller's own site. |
| `restore_my_database` | `$engine, $db` | Non-destructive restore of one of the caller's own databases (newest archive) into their own filesystem; rejected if the database is not one the site currently owns. |
| `import_my_database` | `$engine, $db, $confirm = false` | **Destructive**: import the newest archived dump of one of the caller's own databases back into their live database. Requires `$confirm === true`. |
| `restore_my_account` | | Restore the caller's whole account (files + all databases, newest archives) into their own filesystem, chowned to them. |
| `run_my_backup` | | Trigger a detached backup run scoped to only the caller's own site. |

## 7. Automations

### Scheduled backup (systemd timer)

`systemd/borg-apiscp-backup.timer` runs `borg-apiscp-backup.service`
(`Type=oneshot`, `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=0` so a
long backup is never killed as if hung).

| Setting | Value | Purpose |
|---|---|---|
| `OnCalendar` | `*-*-* 03:30:00` | Nightly at 03:30, before jitter |
| `RandomizedDelaySec` | `1800` | Up to 30 minutes of jitter so a fleet of servers does not hit shared storage simultaneously |
| `Persistent` | `true` | Catch up with a missed run if the machine was off at the scheduled time |

`BACKUP_SCHEDULE` overrides `OnCalendar` without editing the shipped unit:
`borg-apiscp-repo set-schedule [SPEC]` (or `cpcmd borg:set_schedule`, or the
GUI time picker) writes
`/etc/systemd/system/borg-apiscp-backup.timer.d/override.conf` containing an
empty `OnCalendar=` (clearing the base unit's value) followed by
`OnCalendar=<computed value>`, then runs `systemctl daemon-reload` and
`systemctl restart borg-apiscp-backup.timer`. `SPEC` may be `HH:MM` (becomes
`*-*-* HH:MM:00`) or a full `OnCalendar` expression.

### Retention (`borg prune`)

There is no separate "apply policy" step as with kopia's global policy.
Retention lives directly in the `RETENTION_KEEP_*` config keys, and `borg
prune` is what enforces it, in two places:

1. **Automatically, after every backup run**, per archive-name prefix (each
   site's `shadow`, `info`, and `db` archives pruned separately, plus
   `_system-*` and `_custom-<slug>-*`), right inside
   `borg-apiscp-backup`, so retention never falls behind the schedule.
2. **On demand**, via `borg-apiscp-repo prune` (`cpcmd borg:prune`, or the
   GUI's "Prune now" button), which sweeps every site's `shadow`/`info`/`db`
   prefixes plus `_system-*`/`_custom-*` in one pass, using whichever
   `RETENTION_KEEP_*` values are currently set. If none are set, it logs and
   does nothing.

### Maintenance and verify (detached)

`borg-apiscp-repo maintenance` runs `borg compact` (reclaims space freed by
pruned archives); `borg-apiscp-repo verify [--data]` runs `borg check`
(repository and archive consistency; `--data` also reads back every file's
content, slower but thorough). Both are launched via `systemd-run --collect`
when available (falling back to a backgrounded shell job) specifically so a
web/API caller (the GUI, `cpcmd`) returns immediately rather than blocking on
a potentially long-running repository operation.

### Email notifications

Configured via `NOTIFY_EMAIL` / `NOTIFY_ON` / `NOTIFY_FROM`. Sent via the
local `sendmail` (tried at `/usr/sbin/sendmail`, `/usr/lib/sendmail`, or
`sendmail` on `PATH`); if none is found, mail is skipped as best-effort. After
every `borg-apiscp-backup` run (success, failure, or an interrupted/crashed
run, all handled by an `EXIT`/`INT`/`TERM` trap), `notify()` sends a plain
text summary (host, result, finish time, archives ok/failed, database export
result, duration, repository location) if `NOTIFY_ON` calls for it: never
(`never`), always (`always`), or only when the run did not fully succeed
(`failure`, the default; an interrupted run that never reached its normal end
also counts as a failure). `borg-apiscp-repo notify-test` (`cpcmd
borg:notify_test`, GUI "Send test email") sends a one-off confirmation
message independent of an actual backup run.

### Account-lifecycle hooks

ApisCP runs `config/custom/hooks/<event>.sh` with the site id (`siteN`) as
`$1` after the named account event. All three hooks below always exit 0
(hooks cannot interrupt or fail the underlying ApisCP operation) and run
strictly **after** the event has already completed, not before or during it.

| Hook | Source | Timing | Action |
|---|---|---|---|
| `addDomain.sh` | `src/hooks/addDomain.sh` | After a domain/account is created | Kicks off a detached backup scoped to just the new site (`borg-apiscp-repo run --site <site>`), so a brand-new account is protected immediately rather than waiting for the next scheduled run. |
| `suspendDomain.sh` | `src/hooks/suspendDomain.sh` | After a domain/account is suspended | Takes one final detached backup of the site (same `run --site` call) to capture its state at the moment of suspension, since a suspended account's files still exist on disk. |
| `deleteDomain.sh` | `src/hooks/deleteDomain.sh` | After a domain/account is deleted | Does **not** purge the site's archives. Only appends a `timestamp\tsite` line to an audit log (`$APISCP_BORG_DELETED_LOG`, default `/var/lib/apiscp-borg/deleted-accounts.log`) and logs to syslog. Existing archives are retained under the repository's normal retention policy; any purge is a deliberate, manual (or retention-window) decision, never automatic. |

All three are installed only if the destination hook file does not already
exist, or exists but does not already contain the string `apiscp-borg` (an
operator's own custom hook is never overwritten).

## 8. Both GUIs

### Appliance-admin panel: "Borg Backups" (`src/apps/borg/`)

Registered at `/apps/borg` via `config/custom/templates/admin.php`, under the
admin menu's "System" category (internal id `services`). Every action maps
to one `PRIVILEGE_ADMIN` verb from section 6. Sections, top to bottom:

- **Repository**: a compact connected/not-connected status line with a "Show
  details" toggle (raw `borg info`). When a repository is already connected,
  the connection fields are prepopulated and disabled behind a "Reconfigure
  repository" checkbox that must be ticked to unlock them. There is only one
  repository shape (a path or an `ssh://` URL): fields are the repository
  location, passphrase, an optional "generate a strong passphrase" checkbox,
  an encryption-mode selector (`repokey-blake2` / `keyfile-blake2`), and SSH
  options (`BORG_RSH`); there is no backend "type" selector, no server mode,
  and no SFTP key upload, unlike the kopia panel. "Save settings" persists;
  "Check access" confirms the repository is reachable with the current
  passphrase. Creating a new repository requires ticking an explicit
  acknowledgement ("I understand that if the repository passphrase/key is
  lost, the backups are permanently irrecoverable") before "Initialise
  repository" is enabled; immediately after creation, a one-time
  red-bordered panel displays the full recovery bundle in a read-only
  textarea plus a "Download recovery key" button (client-side `Blob`
  download, no extra request), and is **not** shown again on reload.
- **What to back up**: a two-box Available/Backing-up multi-select over
  sites, "System information", and any configured custom paths, with arrow
  buttons to move entries between boxes and a "Save selection" button, plus
  a checkbox for whether site databases are included. A separate text field
  plus "Add custom path" validates and appends an arbitrary absolute path
  (via `validate_path`) to the backing-up box.
- **Retention and excludes**: a "Within" duration field (e.g. `7d`, `4w`,
  `6m`, `1y`) plus four archive-count fields (Daily/Weekly/Monthly/Annual;
  blank = no limit), and a textarea of exclude globs, one per line (stored
  comma-separated on disk); a note that `/.borg-restore` is always excluded.
  "Save retention and excludes" persists the config; "Prune now" runs `borg
  prune` immediately using whatever is currently saved.
- **Run**: a status badge (idle / a backup is currently running, which
  disables the button) and a mode selector (everything / one site / a custom
  path) with "Run now" (always detached).
- **Schedule and maintenance**: an HTML time input bound to `BACKUP_SCHEDULE`
  with "Save schedule" (a systemd `OnCalendar` expression can still be set
  from the CLI), plus "Run maintenance now" (`borg compact`) and a "verify
  file contents" checkbox with "Verify now" (`borg check`, both detached).
- **Notifications**: recipient email, an on-failure/always/never selector,
  a sender override, "Save", and "Send test email".
- **Repository key backup**: shows whether the one-time key/passphrase
  backup sentinel exists, a destination field (defaults to
  `KEY_BACKUP_DIR`), "Back up key now", and a "force (redo)" checkbox.
- **Restore**: a single workflow with a "What would you like to restore?"
  type selector (a site's files / an entire account / a single database /
  system information) that swaps in the matching sub-panel via JavaScript,
  rather than four separate pages. A note states plainly that Borg does not
  yet support point-in-time archive selection or a subpath picker in this
  release: every restore uses the newest matching archive.
  - *Site files*: pick a site (sites with no archives are disabled in the
    dropdown), "Show backups" lists archive dates for information (newest
    first, up to 6 shown); a target field and "Restore newest archive".
  - *Entire account*: pick a site (same archive-availability graying), a
    target directory, "Restore entire account" (files plus all databases
    under a `databases/` subdirectory).
  - *System information*: a target directory and "Restore system"
    (non-destructive).
  - *Databases*: pick a site, "List databases" shows each database with its
    engine, the dated backup history (up to 6 most recent shown, for
    information), and per-row "Restore dump to staging" (safe) versus
    "Import newest backup into live database" (destructive, requires a
    ticked confirmation checkbox **and** a JavaScript `confirm()` dialog).

A full-page "running" overlay (spinner + the clicked button's label) appears
on every form submit and blocks further clicks until the page reloads, since
Borg operations on a large repository can take a while; the page also
restores scroll position across the postback so a save action does not jump
the operator back to the top.

### Site-owner panel: "My Borg Backups" (`src/apps/myborgbackups/`)

Registered at `/apps/myborgbackups` via `config/custom/templates/site.php`,
under the site panel's "Account" category. Every action maps to one
`PRIVILEGE_SITE` verb from section 6, and the page never sends a site
identifier of its own. If the caller's site cannot be resolved from the auth
context, the whole page shows only a warning to contact the administrator.
Sections:

- **Restore files**: "Show my backups" lists the caller's own archive dates
  (informational, up to 6 shown); "Restore newest backup into my files (with
  permissions)" restores the newest archive into the owner's own site
  filesystem. A note states there is no point-in-time selection or file
  picker in this release.
- **Restore my entire account**: one button, restores all files plus all
  databases (newest archives) into `/.borg-restore/account-<date>/`
  (databases under a `databases/` subdirectory).
- **Databases**: "List my databases" shows engine/name plus dated backup
  history (informational); per-row "Restore dump to my files" (safe) and
  "Import newest backup into live database" (destructive, ticked
  confirmation plus a JS `confirm()`).
- **Back up now**: triggers an on-demand backup of just the caller's own
  site, in addition to the scheduled runs.

Every restore result banner states explicitly that the output landed under
`/.borg-restore/` in the owner's own file space and can be retrieved with the
File Manager or SFTP. The same "running" overlay and scroll-position
preservation as the admin app are used here too.

**Common restore landing convention**: every site-owner restore (files,
database dump, or whole account) is written inside that site's own `fst`
tree under a timestamped subdirectory of `/.borg-restore/`, and the entire
`/.borg-restore` directory is then recursively chowned to the site's admin
user (`akp_owner_chown`, preferring `chown --reference` against the owner's
home directory, falling back to the numeric uid:gid parsed from the site's
own `/etc/passwd`) so the owner can browse, download, and delete it through
their own File Manager or SFTP account.

## 9. Security model

- The Layer 2 module runs inside the ApisCP frontend as the unprivileged
  `apnscp` user, which cannot read the root-owned Borg repository config or
  list `/home/virtual`. Every verb therefore shells out to one of the three
  Layer 1 binaries via `Util_Process::exec('sudo -n <bin> %s %s ...', ...)`
  (`runTool()`/`runToolRaw()` in `borg.php`), where the binary path is a
  fixed class constant and every argument is passed through
  `Util_Process`'s own printf-style escaping, never string-concatenated into
  a shell command.
- `install-layer2.sh` installs a scoped `/etc/sudoers.d/apiscp-borg` granting
  exactly `apnscp ALL=(root) NOPASSWD: <the three binary paths>` (validated
  with `visudo -cf` before being installed) - nothing else is permitted to
  run as root. The three binaries themselves therefore are the privilege
  boundary: they validate every input (site id pattern, database engine
  must be `mysql`/`pgsql`, database name character set, target must be
  absolute, subpaths cannot contain `..` or a leading `/`) before acting.
- `/etc/apiscp-borg/config` (holds the repository passphrase) is kept
  root-owned (0600), never read or written directly by the panel user; all
  reads go through `config-get` and all writes through `config-set`, both of
  which run as root via the same sudoers grant.
- `config-set` (and the module's `set_option`) enforce a fixed whitelist of
  settable keys (`CONFIG_ALLOWED` in `bin/borg-apiscp-repo`, mirrored as
  `Borg_Module_Surrogate::ALLOWED_KEYS`) and reject any value containing a CR
  or LF, so a caller cannot smuggle extra config lines or inject shell
  metacharacters into a file that is later `.`-sourced by the engine; the
  stored value is additionally single-quote-escaped.
- Site-owner (`PRIVILEGE_SITE`) verbs never accept a site identifier as a
  parameter. Every one of them calls a private `authSite()` helper that reads
  `$this->getAuthContext()->site` (set by ApisCP's own auth framework to the
  authenticated session's site) and validates it against `^site\d+$`; if it
  cannot be resolved, the verb returns an error or an empty result rather
  than falling back to any caller-supplied value. Where a database name is
  also given (e.g. `restore_my_database`, `import_my_database`), it is
  cross-checked against that resolved site's current database list
  (`ownsDatabase()`) before use, so a site owner cannot reach another site's
  data even by guessing a database name. The corresponding Layer 1
  subcommands (`owner-files`, `owner-restore-db`, `owner-import-db`,
  `owner-account`) repeat the site-pattern and path-traversal checks
  independently, so the guarantee does not rest on the module alone.
- Per-site database isolation: ApisCP's own `<engine>:export`/`<engine>:import`
  verbs resolve their file argument **inside the target site's own filesystem
  namespace** (a chroot-like view via `file_make_path`), not the host
  filesystem, and a host-level dump/restore tool cannot see per-site
  databases at all. The engine and the CLI tools therefore always go through
  `$APNSCP_CMD -d <domain> <engine>:{list_databases,export,import}` in the
  site's own domain context: a dump is written to a site-relative path
  (e.g. `/tmp/.apiscp-borg-db/mysql/<db>.sql`) which the engine then reads
  back from its real host-side location under `<site>/fst/...` for
  archiving, and an import stages the dump under the site's own `fst` before
  calling the site-context import. This is what makes per-site database
  backup and restore correct in a multi-tenant ApisCP host where database
  names can collide or be invisible outside their own site.
- A Borg repository is locked to one operation at a time (Borg's own
  repository lock, in addition to the engine's local `$LOCK_FILE`), so
  concurrent site-owner actions against the same repository queue rather
  than run in parallel; there is no risk of two writers corrupting the
  repository.

## 10. Restore and recovery

A Borg repository is encrypted; if the passphrase (and, for keyfile-mode
repositories, the exported key) are both lost, the data is permanently
unrecoverable no matter how many copies of the encrypted archives exist.
`borg-apiscp-repo backup-key` (see section 5) exports everything needed to
recover onto a fresh machine, and refuses to silently redo that export more
than once (use `--force` to intentionally redo it after a credential
rotation).

To recover the repository itself on a brand-new machine, using either the
persistent key backup directory or the one-time `key-bundle` output:

```sh
# 1. install borgbackup (your package manager, e.g. dnf/apt install borgbackup)

# 2. obtain the repository passphrase (and, for keyfile-mode repositories,
#    the exported key), from either:
#      - the key backup directory (repository.passphrase + repository.key), or
#      - the one-time key-bundle text saved at repository-creation time

# 3. export the passphrase and the repository location
export BORG_PASSPHRASE="$(cat repository.passphrase)"
export BORG_REPO="<the repository location, e.g. /mnt/backups/apiscp-borg or ssh://...>"

# 4. only for keyfile-mode repositories (repokey-blake2 keeps the key IN the
#    repository, so this step is unnecessary for the default encryption mode):
borg key import :: repository.key

# 5. confirm the archives are visible
borg list

# 6. extract whatever is needed, e.g. the newest siteN-shadow archive
borg extract ::<archive-name>
```

From there, install apiscp-borg itself (`install.sh`, then point
`BORG_REPO`/`BORG_PASSPHRASE` at the reconnected repository) to regain the
per-site restore tooling (`borg-apiscp-restore`), or continue with plain
`borg list` / `borg extract` if only a bare `borg` binary is available: no
manual ACL/xattr replay step is needed, because Borg already restored that
metadata as part of `borg extract`. The repository and its archives are
never deleted by `uninstall.sh` (with or without `--purge`), so reinstalling
this project at any point picks up exactly where the repository left off.