diff --git a/README.md b/README.md index e8b58eb..63b0bca 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,13 @@ restore tools, Layer 2 panel integration (module + two GUIs + hooks), install / uninstall, and full reference documentation are being built out to track the apiscp-kopia feature set. Not yet ready for production use. +## Documentation + +- `docs/REFERENCE.md`: exhaustive reference for every command, verb, + configuration key, installer, and automation shipped in this repository. +- `docs/DESIGN.md`: design rationale, and how Borg's native ACL/xattr + handling changes the approach taken relative to apiscp-kopia. + ## License MIT. See `LICENSE`. diff --git a/bin/borg-apiscp-backup b/bin/borg-apiscp-backup index 7b26a6a..a9d2e49 100644 --- a/bin/borg-apiscp-backup +++ b/bin/borg-apiscp-backup @@ -32,8 +32,21 @@ done # --- configuration ------------------------------------------------------------- CONFIG_FILE="${APISCP_BORG_CONFIG:-/etc/apiscp-borg/config}" +# Capture run-scoped overrides passed via the environment (e.g. by +# `borg-apiscp-repo run --site X`, which sets these via systemd-run) BEFORE +# sourcing the config, then re-apply them AFTER, so a targeted run is not +# clobbered by the persistent config defaults. `${VAR-x}` (single dash) +# distinguishes unset (use config) from set-but-empty (an explicit override). +_env_sites="${BACKUP_SITES-__unset__}" +_env_system="${BACKUP_SYSTEM-__unset__}" +_env_paths="${BACKUP_PATHS-__unset__}" +_env_db="${BACKUP_DATABASES-__unset__}" # shellcheck source=/dev/null [ -r "$CONFIG_FILE" ] && . "$CONFIG_FILE" +[ "$_env_sites" != "__unset__" ] && BACKUP_SITES="$_env_sites" +[ "$_env_system" != "__unset__" ] && BACKUP_SYSTEM="$_env_system" +[ "$_env_paths" != "__unset__" ] && BACKUP_PATHS="$_env_paths" +[ "$_env_db" != "__unset__" ] && BACKUP_DATABASES="$_env_db" : "${APNSCP_ROOT:=/usr/local/apnscp}" : "${APNSCP_CMD:=/usr/local/apnscp/bin/cmd}" diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md new file mode 100644 index 0000000..d54ae95 --- /dev/null +++ b/docs/REFERENCE.md @@ -0,0 +1,765 @@ +# apiscp-borg reference + +Exhaustive reference for every command, verb, configuration key, installer, and +automation shipped in this repository. For the design rationale, and for how +Borg's own metadata handling changes the approach taken by the sibling +apiscp-kopia project, see `docs/DESIGN.md`. + +## 1. Overview + +apiscp-borg is a backup engine for ApisCP built on top of +[BorgBackup](https://www.borgbackup.org/). Unlike kopia, Borg preserves POSIX +ACLs and extended attributes natively on Linux, so no ACL/xattr sidecar +mechanism is needed to protect the permission data ApisCP encodes in those +mechanisms: `borg create` captures them and `borg extract` restores them on its +own. The plugin's value is instead: running ApisCP's own per-site database +export before archiving so the logical `.sql` dumps are current, archiving +each site as its own named set of Borg archives so single-site restore and +per-site retention are possible, and wiring retention, maintenance, integrity +checking, scheduling, notifications, one-time key backup, and both a +CLI and a native ApisCP panel integration around the `borg` binary. It ships +as two independent layers: a standalone POSIX-sh engine (Layer 1, three +binaries plus a systemd timer, with zero ApisCP coupling) and a native ApisCP +integration (Layer 2, a module plus two GUI apps installed into upgrade-safe, +git-ignored locations). Layer 2 provides two editions of the panel: + +- **Appliance-admin edition ("Borg Backups")**: full repository configuration, + backup selection, scheduling, retention (prune), maintenance (compact), + integrity checking, notifications, one-time key backup, and restore (site, + account, system, or database) for any site on the box. `PRIVILEGE_ADMIN`. +- **Site-owner edition ("My Borg Backups")**: a scoped self-service panel for + a single signed-in site owner to restore their own files, databases, or + whole account, and to trigger a backup of their own site on demand. It never + accepts a site identifier from the caller; the site is always derived from + the authenticated session. `PRIVILEGE_SITE`. + +Borg's repository model is simpler than kopia's: there are no backend +"types", no repository server, and no SFTP transport. A repository is just a +location (`BORG_REPO`, a local path or an `ssh://` URL), a `BORG_PASSPHRASE`, +a `BORG_ENCRYPTION` mode (`repokey-blake2` by default, so the encryption key +lives inside the repository and only the passphrase is needed to recover it), +and an optional `BORG_RSH` for SSH transport options. There are no tagged +snapshots either: Borg has named archives, and this engine names them with a +structured prefix (`siteN-shadow`, `siteN-info`, `siteN-db`, `_system`, +`_custom-`, each suffixed with Borg's `{now}` timestamp placeholder) so +per-site operations are glob-addressable. Restore in this release always uses +the **newest** archive of a given prefix: there is no point-in-time archive +selection and no browse/subpath picker in the panel (a current limitation, +noted throughout this document and in the GUI itself). + +## 2. Installation + +### `install.sh` (Layer 1: the engine) + +Installs the standalone engine. Must be run as root. Idempotent. + +```sh +sudo sh install.sh +``` + +Unlike apiscp-kopia's installer, this script does **not** install `borg` +itself: it only checks whether `borg` is on `PATH` and prints a warning if it +is not. Install `borgbackup` yourself first (`dnf install borgbackup` / +`apt install borgbackup`). + +Flags: + +| Flag | Effect | +|---|---| +| (none) | Installs the library, binaries, and systemd units; seeds the config. | +| `-h`, `--help` | Prints the header usage comment and exits 0. | + +What it installs, in order: + +| Item | Source | Destination | Mode | +|---|---|---|---| +| shared library | `lib/apiscp-borg-common.sh` | `/usr/local/lib/apiscp-borg/apiscp-borg-common.sh` | 0644 | +| engine binary | `bin/borg-apiscp-backup` | `/usr/local/bin/borg-apiscp-backup` | 0755 | +| repo binary | `bin/borg-apiscp-repo` | `/usr/local/bin/borg-apiscp-repo` | 0755 | +| restore binary | `bin/borg-apiscp-restore` | `/usr/local/bin/borg-apiscp-restore` | 0755 | +| systemd service | `systemd/borg-apiscp-backup.service` | `/etc/systemd/system/borg-apiscp-backup.service` | 0644 | +| systemd timer | `systemd/borg-apiscp-backup.timer` | `/etc/systemd/system/borg-apiscp-backup.timer` | 0644 | +| config seed | `etc/apiscp-borg.config.example` | `/etc/apiscp-borg/config` | 0600, only if absent (never overwrites an existing config) | + +`systemctl daemon-reload` is run after the units are installed. The script +does **not** enable the timer, install `borg`, or create/connect the Borg +repository; it prints the manual next steps (install `borgbackup`, set +`BORG_REPO`/`BORG_PASSPHRASE`, initialise the repository, back up the +passphrase/key off-machine, one test run, `systemctl enable --now +borg-apiscp-backup.timer`). + +### `install-layer2.sh` (Layer 2: native ApisCP integration) + +Installs the panel module, both GUI apps, both menu links, the account +lifecycle hooks, and a scoped sudoers file. Run as root, after `install.sh`. +Idempotent. Respects `CP_ROOT` (default `/usr/local/apnscp`) and `PANEL_USER` +(default `apnscp`). + +| Step | Source | Destination | Notes | +|---|---|---|---| +| Module | `src/modules/borg.php` | `$CP_ROOT/lib/modules/surrogates/borg.php` | git-ignored by ApisCP (`surrogates/*`), survives `upcp` | +| Admin GUI app | `src/apps/borg/` (borg.php, application.yml, views/index.blade.php) | `$CP_ROOT/config/custom/apps/borg/` | whole tree ignored | +| Site-owner GUI app | `src/apps/myborgbackups/` | `$CP_ROOT/config/custom/apps/myborgbackups/` | whole tree ignored | +| Admin menu link | `src/templates/admin.php` | `$CP_ROOT/config/custom/templates/admin.php` | only written if the file does not already exist; if it exists and already references `/apps/borg`, left untouched; if it exists, does not reference `/apps/borg`, and contains a closing `?>`, a warning is printed and nothing is appended; otherwise the `create_link()` call is appended | +| Site menu link | `src/templates/site.php` | `$CP_ROOT/config/custom/templates/site.php` | same append-not-clobber logic, keyed on `/apps/myborgbackups` | +| Account hooks | `src/hooks/{addDomain,suspendDomain,deleteDomain}.sh` | `$CP_ROOT/config/custom/hooks/.sh` | only installed if the destination does not exist, or exists but does not already contain the string `apiscp-borg` (an operator's own hook is left untouched) | +| Sudoers | generated inline | `/etc/sudoers.d/apiscp-borg` | 0440 root:root, validated with `visudo -cf` before install; installation is skipped with a warning if validation fails | + +After installing the sudoers file, the script forces the engine config to be +root-owned (`chown root:root` + `chmod 0600` on `/etc/apiscp-borg/config` if +it exists), then restarts the panel (`systemctl restart apnscp`) so the new +GUI apps and menu entries are picked up; the module itself needs no restart +and is picked up on the next CLI/API call. Site owners see the "My Borg +Backups" menu entry after their next login (the menu is session-cached). + +**Upgrade-safe locations.** Every Layer 2 destination is one ApisCP +git-ignores, so a `git pull` / `upcp` upgrade never touches it: +`lib/modules/surrogates/*`, and the entirety of `config/custom/` (via its own +`.gitignore` of `*` plus `!.gitignore`). + +## 3. Uninstall + +`uninstall.sh`, run as root, idempotent. + +```sh +sudo sh uninstall.sh # default: keep config, staging, key backups +sudo sh uninstall.sh --purge # also delete config, staging dir, key backup dir +``` + +Always removed (both modes): + +- Layer 2: the module surrogate, both GUI app directories (`borg` and + `myborgbackups`), the account hooks (only the ones containing the string + `apiscp-borg`), the scoped sudoers file, and the `apiscp-borg` lines + stripped out of `admin.php`/`site.php` (the template file itself is deleted + only if nothing but `` | `/home/virtual` | yes | +| `STAGE` | Reserved working area for the engine (not exercised by any active code path in this release) | `/var/lib/apiscp-borg` | yes | +| `METRICS_DIR` | Prometheus node_exporter textfile collector directory; blank autodetects common locations, and metrics are skipped if none is found | (empty / autodetect) | yes | +| `RUN_DB_EXPORT` | Run ApisCP's per-site database export (`backup_dbs.php`) before archiving: `1` or `0` | `1` | yes | +| `DB_EXPORT_CMD` | Path to `backup_dbs.php` | `$APNSCP_ROOT/bin/scripts/backup_dbs.php` | no | +| `APNSCP_CMD` | ApisCP CLI binary, used for native per-site database verbs (`list_databases`/`export`/`import`) | `/usr/local/apnscp/bin/cmd` | no | +| `APNSCP_ROOT` | Path to the ApisCP installation | `/usr/local/apnscp` | no | +| `LOCK_FILE` | Single-instance lock file guarding against overlapping backup runs (via `flock`, if available) | `/run/apiscp-borg.lock` | no | + +Two further variables are read only via the environment, not the config file +whitelist: `APISCP_BORG_LIB` (override the shared-library search path) and +`APISCP_BORG_CONFIG` (override the config file path itself, default +`/etc/apiscp-borg/config`), plus `APISCP_BORG_LOG_TAG` (the `logger` tag used +by `akp_log`, default `apiscp-borg`) and `APISCP_BORG_DELETED_LOG` (used only +by the `deleteDomain` hook, default `/var/lib/apiscp-borg/deleted-accounts.log`). +`BORG_RELOCATED_REPO_ACCESS_IS_OK` is exported by the engine (default `no`, +overridable via the real environment) so an unattended run does not block on +Borg's interactive "repository moved" prompt. + +## 5. CLI reference (Layer 1) + +All three binaries source `lib/apiscp-borg-common.sh` (search order: +`$APISCP_BORG_LIB`, `../lib/` relative to the binary, +`/usr/local/lib/apiscp-borg/`, `/usr/lib/apiscp-borg/`) then +`$APISCP_BORG_CONFIG` (default `/etc/apiscp-borg/config`). Every subcommand +below matches the code exactly. + +### `borg-apiscp-backup` + +Takes no subcommands; running it (with no arguments) performs one full backup +run and exits 0 if every archive was created, 1 if any failed. Driven by the +systemd timer/service, or launched detached by `borg-apiscp-repo run`. + +Sequence on every run: + +1. Acquire `$LOCK_FILE` via `flock` (non-blocking; a second concurrent run + aborts immediately) if `flock` is available. +2. If `RUN_DB_EXPORT=1`, run `$DB_EXPORT_CMD` (ApisCP's `backup_dbs.php`) so + per-site `.sql` dumps are current before they are archived. +3. For every `$VIRTBASE/site` directory (skipping any not selected by + `BACKUP_SITES`): + - archive `shadow` (if present) as `site-shadow-{now}`; + - archive `info` (if present) as `site-info-{now}`; + - if `BACKUP_DATABASES=1`: enumerate the site's `mysql` and `pgsql` + databases via `$APNSCP_CMD -o json -d :list_databases`, + export each to a site-relative staging path + (`/tmp/.apiscp-borg-db//.sql`, resolved inside the site's + `fst` namespace by ApisCP), archive that staging tree from its real host + location (`/fst/tmp/.apiscp-borg-db`) as `site-db-{now}`, then + delete the staging tree from the site; + - apply per-archive-prefix retention (`borg prune`) for + `site-shadow-*`, `site-info-*`, and `site-db-*` right away, so + one site's daily churn is pruned every run rather than waiting for a + separate step. +4. If `BACKUP_SYSTEM=1`: expand `SYSTEM_PATHS` and matching `SYSTEM_GLOBS` + entries to those that actually exist, archive them together as + `_system-{now}`, then prune `_system-*`. +5. If `BACKUP_PATHS` is non-empty: for each existing custom path, archive it + as `_custom--{now}` (slug via `akp_slug`), then prune + `_custom--*`. +6. Write Prometheus textfile metrics (always, even on failure or interrupt, + via an `EXIT`/`INT`/`TERM` trap) and send an email notification per + `NOTIFY_EMAIL`/`NOTIFY_ON`/`NOTIFY_FROM`. + +Every archive is created with `borg create --compression +"${BORG_COMPRESSION:-zstd}"`, plus `--exclude '*/.borg-restore' --exclude +'/.borg-restore'` and one `--exclude` per `BACKUP_EXCLUDES` entry. + +Metrics written to `$METRICS_DIR/apiscp-borg.prom` (or autodetected): +`apiscp_borg_run_success`, `apiscp_borg_archives_ok`, +`apiscp_borg_archives_failed`, `apiscp_borg_duration_seconds`, +`apiscp_borg_db_export_success` (1 ok / 0 failed / -1 skipped), and +`apiscp_borg_last_run_timestamp_seconds`. + +### `borg-apiscp-repo` + +Manages the repository and everything that is not a data restore. + +| Subcommand | Syntax | Description | +|---|---|---| +| `status` | `borg-apiscp-repo status` | Print `repository: $BORG_REPO` followed by `borg info` (first 30 lines, indented). | +| `init` | `borg-apiscp-repo init` | Create (initialise) the repository at `BORG_REPO` with `BORG_ENCRYPTION`. Requires `BORG_PASSPHRASE` already set. Idempotent: if `borg info` already succeeds against the repository, logs and returns 0 without re-initialising. | +| `check-access` | `borg-apiscp-repo check-access` | Print `ok` if the repository is reachable and the passphrase is correct, else `error: cannot access ...`. Always exits 0 so a UI can read the message. | +| `gen-passphrase` | `borg-apiscp-repo gen-passphrase` | Generate a strong random passphrase (`openssl rand -base64 30`, or `/dev/urandom` fallback), store it as `BORG_PASSPHRASE` via `config-set`, and print **only** the passphrase to stdout. | +| `key-status` | `borg-apiscp-repo key-status` | Print `yes` if the one-time key/passphrase backup sentinel exists at `$KEY_BACKUP_DIR/.apiscp-borg-key-backed-up`, else `no`. | +| `backup-key` | `borg-apiscp-repo backup-key [DEST] [--force]` | Export `repository.passphrase` (if `BORG_PASSPHRASE` is set), `repository.key` (via `borg key export`, which is a no-op warning for `repokey` repositories since the key already lives in the repository), and a `README.txt` to `DEST` (default `$KEY_BACKUP_DIR`), all 0600. Refuses to redo (logs and returns 0) unless `--force`. | +| `key-bundle` | `borg-apiscp-repo key-bundle` | Print a self-contained recovery bundle (instructions, the plaintext passphrase, the repository location, and an exported key via `borg key export :: /dev/stdout`) to stdout. For the GUI's one-time post-creation display/download. | +| `config-get` | `borg-apiscp-repo config-get` | Print every `KEY=VALUE` line from the config file with surrounding quotes stripped (comments and malformed lines skipped). | +| `config-set` | `borg-apiscp-repo config-set KEY VALUE` | Set one config key, rejecting any key not in `CONFIG_ALLOWED` and any value containing a CR or LF. Value is stored single-quoted (shell-escaped) so the config, which is sourced by the engine, cannot be injected. Writes with mode 0600. | +| `sites` | `borg-apiscp-repo sites` | List `site\t` for every ApisCP site under `$VIRTBASE` (domain falls back to the site id if `siteinfo` cannot be read). | +| `run` | `borg-apiscp-repo run [--site SITE \| --path PATH]` | No args (or `--all`): `systemctl start --no-block borg-apiscp-backup.service` (returns immediately). `--site SITE`: launch a detached one-shot run scoped to just that site (`BACKUP_SITES=SITE BACKUP_SYSTEM=0`); `SITE` must match `site[0-9]*` or be the literal `none`. `--path PATH`: launch a detached one-shot backing up only that one absolute, existing, readable path (`BACKUP_SITES=none BACKUP_SYSTEM=0 BACKUP_PATHS=PATH`). Detached runs use `systemd-run --collect` if available, else a backgrounded `env ... &`. | +| `running` | `borg-apiscp-repo running` | Print `systemctl is-active borg-apiscp-backup.service` (e.g. `active`, `inactive`). | +| `prune` | `borg-apiscp-repo prune` | Apply retention (`RETENTION_KEEP_*`, numeric/duration values only) across every archive prefix: each site's `shadow`/`info`/`db` archives separately, plus `_system-*` and `_custom-*`. If no retention key is set, logs and does nothing. | +| `maintenance` | `borg-apiscp-repo maintenance` | Run `borg compact`, detached (`systemd-run --collect` if available, else backgrounded). | +| `verify` | `borg-apiscp-repo verify [--data]` | Run `borg check` (repository and archive consistency), detached. With `--data`, adds `--verify-data` to also read file contents back (slow, thorough); without it, only metadata is checked. | +| `set-schedule` | `borg-apiscp-repo set-schedule [SPEC]` | Set the backup timer's `OnCalendar`. `SPEC` is `HH:MM` (validated 00-23:00-59, becomes `*-*-* HH:MM:00`) or a full `OnCalendar` expression (validated against an allowed character set); falls back to `BACKUP_SCHEDULE` if `SPEC` is omitted. Writes `/etc/systemd/system/borg-apiscp-backup.timer.d/override.conf`, reloads systemd, restarts the timer. Prints the resulting `OnCalendar` value. | +| `notify-test` | `borg-apiscp-repo notify-test` | Send a test email to `NOTIFY_EMAIL` to confirm mail delivery; fails if `NOTIFY_EMAIL` is unset. | +| `list-databases` | `borg-apiscp-repo list-databases SITE` | Print `\t` (engine `mysql` or `pgsql`) for every database ApisCP currently knows for `SITE`, via native per-site enumeration (a host-level dump cannot see per-site databases). `SITE` must match `site[0-9]*`. | +| (no args / `-h`/`--help`/`help`) | | Print the full header usage comment (via `awk`) and exit 0. | +| (anything else) | | Print `unknown command: ` to stderr and exit 2. | + +There is no `connect`, `create`, `types`, `repo-info`, `validate-path`, +`browse`, or `apply-policy` subcommand: those either do not apply to Borg's +simpler repository model, or (in the case of `repo-info` and `validate-path`) +are computed directly in the Layer 2 module without a root round trip (see +section 6). + +### `borg-apiscp-restore` + +Restores data from a Borg archive. Borg stores archived paths relative +(leading `/` stripped) and preserves POSIX ACLs and extended attributes +natively as part of every archive, so restore is a plain `borg extract` with +`--strip-components` to rebase the archived path under a staging target: no +separate metadata-replay step is needed (that is the kopia engine's job, not +Borg's). Archive naming (from the engine): `siteN-shadow-{now}`, +`siteN-info-{now}`, `siteN-db-{now}`, `_system-{now}`, +`_custom--{now}`. "Newest" means the most recent archive matching a +given prefix (`borg list --glob-archives PREFIX-* --last 1`). + +| Subcommand | Syntax | Description | +|---|---|---| +| `list` | `borg-apiscp-restore list [PREFIX]` | List archives, optionally filtered to `PREFIX-*` (e.g. `site1`), else every archive in the repository. | +| `list-json` | `borg-apiscp-restore list-json PREFIX` | Print raw `borg list --json --glob-archives PREFIX-*`. `PREFIX` is required. Intended for the module/GUI to parse. | +| `site` | `borg-apiscp-restore site SITE TARGET [--subpath P ...]` | Restore `SITE`'s newest `shadow` and `info` archives into `TARGET/{shadow,info}`. With one or more `--subpath`, restores only those paths (relative to the site root, under `shadow/`) instead of the whole tree; each subpath is rejected if it starts with `/` or contains `..`. Fails if nothing was restored. | +| `account` | `borg-apiscp-restore account SITE TARGET` | Restore a whole account: files (`shadow`+`info`, fails hard if none exist) plus the newest `SITE-db` archive as a tree into `TARGET/databases` (warns, does not fail, if no database archive exists). | +| `system` | `borg-apiscp-restore system TARGET` | Restore the newest `_system` archive into `TARGET` via `borg extract --numeric-ids`. | +| `restore-db` | `borg-apiscp-restore restore-db SITE ENGINE DB TARGET` | Non-destructive: extract `ENGINE/DB.sql` from the newest `SITE-db` archive into `TARGET/DB.sql`, print the restored path. `ENGINE` must be `mysql` or `pgsql`; `DB` must match `[A-Za-z0-9_-]+`. | +| `import-db` | `borg-apiscp-restore import-db SITE ENGINE DB` | **Destructive.** Restores the newest dump of `SITE`'s `DB` to a temp directory (there is no `DUMPFILE` argument: the newest archived dump is always the one imported), stages it inside the site's own `fst` tree, and imports it into the site's live database via `$APNSCP_CMD -d :import`, then removes the staged copy. | +| `owner-files` | `borg-apiscp-restore owner-files SITE [--subpath P ...]` | Site-owner variant of `site`: restores into the site's own filesystem under `/.borg-restore/files-/`, chowned to the site owner, and prints the in-site relative path. | +| `owner-account` | `borg-apiscp-restore owner-account SITE` | Site-owner whole-account restore into `/.borg-restore/account-/` (files, plus databases under `databases/`), chowned to the owner. Prints the in-site relative path. | +| `owner-restore-db` | `borg-apiscp-restore owner-restore-db SITE ENGINE DB` | Site-owner, non-destructive: restore the newest dump of the site's own database into `/.borg-restore/db-/DB.sql`, chowned to the owner. Prints the in-site relative path. | +| `owner-import-db` | `borg-apiscp-restore owner-import-db SITE ENGINE DB` | Site-owner, **destructive**: equivalent to `import-db` for the site's own database. | +| (no args / `-h`/`--help`/`help`) | | Print the header usage comment and exit 0. | +| (anything else) | | Print `unknown command: ` to stderr and exit 2. | + +Every "owner-*" subcommand takes `SITE` as an argument, but in every real +invocation path that argument is supplied by the trusted Layer 2 module +(`borg.php`), which derives it from `$this->getAuthContext()->site` (the +authenticated session), never from a value a site user could control. The +Layer 1 binaries additionally validate `SITE` matches `site[0-9]*`. + +**Current limitation, by design in this release:** restore always uses the +newest archive of a matching prefix. There is no way, from the CLI or the +GUI, to restore an older archive by id/time, and `owner-files`/`site` support +subpath filtering only within the `shadow` tree, not a general browse picker. +`import-db`/`owner-import-db` take no dump-file argument at all: they always +restore-and-import the newest `SITE-db` archive's dump for the named +database. + +## 6. Panel/API reference (Layer 2) + +`Borg_Module_Surrogate` (`src/modules/borg.php`, installed as +`lib/modules/surrogates/borg.php`) exposes every verb below as `cpcmd +borg:` and to the two GUI apps (`Page_Container::__call` forwards +`$this->borg_(...)` calls to the module). Every verb runs one of the +three Layer 1 binaries as root via a scoped `sudo -n`, with a positional +printf-style format string so `Util_Process` escapes every argument (see +section 9). + +### Admin verbs (`PRIVILEGE_ADMIN`) + +| Verb | Parameters | Description | +|---|---|---| +| `get_config` | | Return the current config as an associative array, with any key containing `PASSPHRASE` masked to `********`. | +| `set_option` | `$key, $value` | Set one whitelisted config key (client-side newline check only; `config-set` is the authoritative validator). | +| `status` | | Return `borg-apiscp-repo status` output. | +| `check_access` | | Return `borg-apiscp-repo check-access` output (`ok` or an error message). | +| `key_backed_up` | | `true` if `key-status` reports `yes`. | +| `backup_key` | `$dest = null, $force = false` | Run `backup-key`, optionally to a custom destination, optionally forced. | +| `key_bundle` | | Return the one-time recovery bundle text (secrets included by design). | +| `run` | `$site = '', $path = ''` | Trigger a backup: `$path` takes precedence over `$site`; with neither, runs everything. `$site` is validated against `^site\d+$`. | +| `is_running` | | `true` if the backup service is currently active. | +| `sites` | | Return `siteN => domain` for every ApisCP site, sorted by key. | +| `site_snapshots` | `$site` | Archives available for a site's files, newest first, as `[{id,time}, ...]`; the archive name is the id. Informational only: restore always uses the newest. | +| `restore_site` | `$site, $target` | Restore a site's newest `shadow`+`info` archives into a staging directory. There is no point-in-time selection or subpath picker exposed here (unlike the Layer 1 `site` subcommand's `--subpath`); the whole site tree is always restored. | +| `init_repo` | | Create (initialise) the configured repository. Idempotent; requires a passphrase already set. | +| `gen_passphrase` | | Generate and store a strong repository passphrase; returns it once. | +| `set_backup_selection` | `$sites, $system, $paths = []` | Set `BACKUP_SITES` (collapsing to `all` if the selection covers every known site, `none` if empty, otherwise a CSV), `BACKUP_SYSTEM`, and `BACKUP_PATHS` (each path re-validated with `validate_path`). | +| `set_schedule` | `$when` | Validate `$when` as `HH:MM` or a safe `OnCalendar`-like expression, store it in `BACKUP_SCHEDULE`, then call `set-schedule`. | +| `maintenance` | | Run `borg compact`, detached. | +| `verify` | `$data = false` | Run `borg check`, detached; `$data` adds `--verify-data` (reads file contents back). | +| `notify_test` | | Send a test notification email. | +| `prune` | | Apply retention (`borg prune`) across every archive prefix right now, per the configured `RETENTION_KEEP_*` settings. Unlike kopia's global policy, there is no separate "apply policy" step: retention lives in config, and `prune` (also run automatically after each backup) is what enforces it. | +| `repo_info` | | Return `['connected'=>bool, 'location'=>string]`, computed from `get_config()['BORG_REPO']` and a live `check-access` call; there is no repository "type" in the Borg model. | +| `validate_path` | `$path` | `true` if the path is a plausible absolute path (non-empty, starts with `/`, no CR/LF). Unlike kopia's `validate-path`, there is no corresponding root-level CLI check: this is a best-effort structural check performed entirely in PHP, not authoritative (a path that does not actually exist when the engine runs is simply skipped with a warning in the log). | +| `sites_with_snapshots` | | Site ids that have at least one `siteN-shadow` archive (for greying out empty sites in the restore picker). | +| `site_databases` | `$site` | `[{engine, db}, ...]` currently known for a site (live, from ApisCP, not from an archive). | +| `database_backups` | `$site` | `[{id, time}, ...]` (newest first) of dated `siteN-db` archives for a site. Informational: restore/import always use the newest. | +| `restore_database` | `$site, $engine, $db, $target` | Non-destructive restore of one database dump to a staging path. | +| `restore_account` | `$site, $target` | Restore a whole account (files + all databases) to a staging directory. | +| `import_database` | `$site, $engine, $db` | **Destructive** import of the newest archived dump into the live database. No dump-file parameter: `import-db` always restores and imports the newest `siteN-db` archive itself. | +| `restore_system` | `$target` | Restore the shared system archive to a staging directory. No subpath parameter is exposed (unlike kopia's `restore_system`). | + +### Site-owner verbs (`PRIVILEGE_SITE`) + +Every verb below resolves the caller's own site from the ApisCP auth context +via a private `authSite()` helper (`$this->getAuthContext()->site`, validated +against `^site\d+$`) and never accepts a site identifier as a parameter. Where +a database is named, it is additionally checked to belong to that resolved +site (`ownsDatabase()`) before use. + +| Verb | Parameters | Description | +|---|---|---| +| `my_site` | | `['site'=>siteN\|'', 'domain'=>string]` for the caller. | +| `my_snapshots` | | `[{id,time}, ...]` archives available for the caller's own site. Informational only; restore always uses the newest. | +| `restore_my_files` | | Restore the caller's own files (newest archive) into their own site filesystem (chowned to them), returning the in-site relative path. No point-in-time or subpath selection in this release. | +| `my_databases` | | `[{engine,db}, ...]` for the caller's own site. | +| `my_database_backups` | | `[{id,time}, ...]` dated database archives for the caller's own site. | +| `restore_my_database` | `$engine, $db` | Non-destructive restore of one of the caller's own databases (newest archive) into their own filesystem; rejected if the database is not one the site currently owns. | +| `import_my_database` | `$engine, $db, $confirm = false` | **Destructive**: import the newest archived dump of one of the caller's own databases back into their live database. Requires `$confirm === true`. | +| `restore_my_account` | | Restore the caller's whole account (files + all databases, newest archives) into their own filesystem, chowned to them. | +| `run_my_backup` | | Trigger a detached backup run scoped to only the caller's own site. | + +## 7. Automations + +### Scheduled backup (systemd timer) + +`systemd/borg-apiscp-backup.timer` runs `borg-apiscp-backup.service` +(`Type=oneshot`, `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=0` so a +long backup is never killed as if hung). + +| Setting | Value | Purpose | +|---|---|---| +| `OnCalendar` | `*-*-* 03:30:00` | Nightly at 03:30, before jitter | +| `RandomizedDelaySec` | `1800` | Up to 30 minutes of jitter so a fleet of servers does not hit shared storage simultaneously | +| `Persistent` | `true` | Catch up with a missed run if the machine was off at the scheduled time | + +`BACKUP_SCHEDULE` overrides `OnCalendar` without editing the shipped unit: +`borg-apiscp-repo set-schedule [SPEC]` (or `cpcmd borg:set_schedule`, or the +GUI time picker) writes +`/etc/systemd/system/borg-apiscp-backup.timer.d/override.conf` containing an +empty `OnCalendar=` (clearing the base unit's value) followed by +`OnCalendar=`, then runs `systemctl daemon-reload` and +`systemctl restart borg-apiscp-backup.timer`. `SPEC` may be `HH:MM` (becomes +`*-*-* HH:MM:00`) or a full `OnCalendar` expression. + +### Retention (`borg prune`) + +There is no separate "apply policy" step as with kopia's global policy. +Retention lives directly in the `RETENTION_KEEP_*` config keys, and `borg +prune` is what enforces it, in two places: + +1. **Automatically, after every backup run**, per archive-name prefix (each + site's `shadow`, `info`, and `db` archives pruned separately, plus + `_system-*` and `_custom--*`), right inside + `borg-apiscp-backup`, so retention never falls behind the schedule. +2. **On demand**, via `borg-apiscp-repo prune` (`cpcmd borg:prune`, or the + GUI's "Prune now" button), which sweeps every site's `shadow`/`info`/`db` + prefixes plus `_system-*`/`_custom-*` in one pass, using whichever + `RETENTION_KEEP_*` values are currently set. If none are set, it logs and + does nothing. + +### Maintenance and verify (detached) + +`borg-apiscp-repo maintenance` runs `borg compact` (reclaims space freed by +pruned archives); `borg-apiscp-repo verify [--data]` runs `borg check` +(repository and archive consistency; `--data` also reads back every file's +content, slower but thorough). Both are launched via `systemd-run --collect` +when available (falling back to a backgrounded shell job) specifically so a +web/API caller (the GUI, `cpcmd`) returns immediately rather than blocking on +a potentially long-running repository operation. + +### Email notifications + +Configured via `NOTIFY_EMAIL` / `NOTIFY_ON` / `NOTIFY_FROM`. Sent via the +local `sendmail` (tried at `/usr/sbin/sendmail`, `/usr/lib/sendmail`, or +`sendmail` on `PATH`); if none is found, mail is skipped as best-effort. After +every `borg-apiscp-backup` run (success, failure, or an interrupted/crashed +run, all handled by an `EXIT`/`INT`/`TERM` trap), `notify()` sends a plain +text summary (host, result, finish time, archives ok/failed, database export +result, duration, repository location) if `NOTIFY_ON` calls for it: never +(`never`), always (`always`), or only when the run did not fully succeed +(`failure`, the default; an interrupted run that never reached its normal end +also counts as a failure). `borg-apiscp-repo notify-test` (`cpcmd +borg:notify_test`, GUI "Send test email") sends a one-off confirmation +message independent of an actual backup run. + +### Account-lifecycle hooks + +ApisCP runs `config/custom/hooks/.sh` with the site id (`siteN`) as +`$1` after the named account event. All three hooks below always exit 0 +(hooks cannot interrupt or fail the underlying ApisCP operation) and run +strictly **after** the event has already completed, not before or during it. + +| Hook | Source | Timing | Action | +|---|---|---|---| +| `addDomain.sh` | `src/hooks/addDomain.sh` | After a domain/account is created | Kicks off a detached backup scoped to just the new site (`borg-apiscp-repo run --site `), so a brand-new account is protected immediately rather than waiting for the next scheduled run. | +| `suspendDomain.sh` | `src/hooks/suspendDomain.sh` | After a domain/account is suspended | Takes one final detached backup of the site (same `run --site` call) to capture its state at the moment of suspension, since a suspended account's files still exist on disk. | +| `deleteDomain.sh` | `src/hooks/deleteDomain.sh` | After a domain/account is deleted | Does **not** purge the site's archives. Only appends a `timestamp\tsite` line to an audit log (`$APISCP_BORG_DELETED_LOG`, default `/var/lib/apiscp-borg/deleted-accounts.log`) and logs to syslog. Existing archives are retained under the repository's normal retention policy; any purge is a deliberate, manual (or retention-window) decision, never automatic. | + +All three are installed only if the destination hook file does not already +exist, or exists but does not already contain the string `apiscp-borg` (an +operator's own custom hook is never overwritten). + +## 8. Both GUIs + +### Appliance-admin panel: "Borg Backups" (`src/apps/borg/`) + +Registered at `/apps/borg` via `config/custom/templates/admin.php`, under the +admin menu's "System" category (internal id `services`). Every action maps +to one `PRIVILEGE_ADMIN` verb from section 6. Sections, top to bottom: + +- **Repository**: a compact connected/not-connected status line with a "Show + details" toggle (raw `borg info`). When a repository is already connected, + the connection fields are prepopulated and disabled behind a "Reconfigure + repository" checkbox that must be ticked to unlock them. There is only one + repository shape (a path or an `ssh://` URL): fields are the repository + location, passphrase, an optional "generate a strong passphrase" checkbox, + an encryption-mode selector (`repokey-blake2` / `keyfile-blake2`), and SSH + options (`BORG_RSH`); there is no backend "type" selector, no server mode, + and no SFTP key upload, unlike the kopia panel. "Save settings" persists; + "Check access" confirms the repository is reachable with the current + passphrase. Creating a new repository requires ticking an explicit + acknowledgement ("I understand that if the repository passphrase/key is + lost, the backups are permanently irrecoverable") before "Initialise + repository" is enabled; immediately after creation, a one-time + red-bordered panel displays the full recovery bundle in a read-only + textarea plus a "Download recovery key" button (client-side `Blob` + download, no extra request), and is **not** shown again on reload. +- **What to back up**: a two-box Available/Backing-up multi-select over + sites, "System information", and any configured custom paths, with arrow + buttons to move entries between boxes and a "Save selection" button, plus + a checkbox for whether site databases are included. A separate text field + plus "Add custom path" validates and appends an arbitrary absolute path + (via `validate_path`) to the backing-up box. +- **Retention and excludes**: a "Within" duration field (e.g. `7d`, `4w`, + `6m`, `1y`) plus four archive-count fields (Daily/Weekly/Monthly/Annual; + blank = no limit), and a textarea of exclude globs, one per line (stored + comma-separated on disk); a note that `/.borg-restore` is always excluded. + "Save retention and excludes" persists the config; "Prune now" runs `borg + prune` immediately using whatever is currently saved. +- **Run**: a status badge (idle / a backup is currently running, which + disables the button) and a mode selector (everything / one site / a custom + path) with "Run now" (always detached). +- **Schedule and maintenance**: an HTML time input bound to `BACKUP_SCHEDULE` + with "Save schedule" (a systemd `OnCalendar` expression can still be set + from the CLI), plus "Run maintenance now" (`borg compact`) and a "verify + file contents" checkbox with "Verify now" (`borg check`, both detached). +- **Notifications**: recipient email, an on-failure/always/never selector, + a sender override, "Save", and "Send test email". +- **Repository key backup**: shows whether the one-time key/passphrase + backup sentinel exists, a destination field (defaults to + `KEY_BACKUP_DIR`), "Back up key now", and a "force (redo)" checkbox. +- **Restore**: a single workflow with a "What would you like to restore?" + type selector (a site's files / an entire account / a single database / + system information) that swaps in the matching sub-panel via JavaScript, + rather than four separate pages. A note states plainly that Borg does not + yet support point-in-time archive selection or a subpath picker in this + release: every restore uses the newest matching archive. + - *Site files*: pick a site (sites with no archives are disabled in the + dropdown), "Show backups" lists archive dates for information (newest + first, up to 6 shown); a target field and "Restore newest archive". + - *Entire account*: pick a site (same archive-availability graying), a + target directory, "Restore entire account" (files plus all databases + under a `databases/` subdirectory). + - *System information*: a target directory and "Restore system" + (non-destructive). + - *Databases*: pick a site, "List databases" shows each database with its + engine, the dated backup history (up to 6 most recent shown, for + information), and per-row "Restore dump to staging" (safe) versus + "Import newest backup into live database" (destructive, requires a + ticked confirmation checkbox **and** a JavaScript `confirm()` dialog). + +A full-page "running" overlay (spinner + the clicked button's label) appears +on every form submit and blocks further clicks until the page reloads, since +Borg operations on a large repository can take a while; the page also +restores scroll position across the postback so a save action does not jump +the operator back to the top. + +### Site-owner panel: "My Borg Backups" (`src/apps/myborgbackups/`) + +Registered at `/apps/myborgbackups` via `config/custom/templates/site.php`, +under the site panel's "Account" category. Every action maps to one +`PRIVILEGE_SITE` verb from section 6, and the page never sends a site +identifier of its own. If the caller's site cannot be resolved from the auth +context, the whole page shows only a warning to contact the administrator. +Sections: + +- **Restore files**: "Show my backups" lists the caller's own archive dates + (informational, up to 6 shown); "Restore newest backup into my files (with + permissions)" restores the newest archive into the owner's own site + filesystem. A note states there is no point-in-time selection or file + picker in this release. +- **Restore my entire account**: one button, restores all files plus all + databases (newest archives) into `/.borg-restore/account-/` + (databases under a `databases/` subdirectory). +- **Databases**: "List my databases" shows engine/name plus dated backup + history (informational); per-row "Restore dump to my files" (safe) and + "Import newest backup into live database" (destructive, ticked + confirmation plus a JS `confirm()`). +- **Back up now**: triggers an on-demand backup of just the caller's own + site, in addition to the scheduled runs. + +Every restore result banner states explicitly that the output landed under +`/.borg-restore/` in the owner's own file space and can be retrieved with the +File Manager or SFTP. The same "running" overlay and scroll-position +preservation as the admin app are used here too. + +**Common restore landing convention**: every site-owner restore (files, +database dump, or whole account) is written inside that site's own `fst` +tree under a timestamped subdirectory of `/.borg-restore/`, and the entire +`/.borg-restore` directory is then recursively chowned to the site's admin +user (`akp_owner_chown`, preferring `chown --reference` against the owner's +home directory, falling back to the numeric uid:gid parsed from the site's +own `/etc/passwd`) so the owner can browse, download, and delete it through +their own File Manager or SFTP account. + +## 9. Security model + +- The Layer 2 module runs inside the ApisCP frontend as the unprivileged + `apnscp` user, which cannot read the root-owned Borg repository config or + list `/home/virtual`. Every verb therefore shells out to one of the three + Layer 1 binaries via `Util_Process::exec('sudo -n %s %s ...', ...)` + (`runTool()`/`runToolRaw()` in `borg.php`), where the binary path is a + fixed class constant and every argument is passed through + `Util_Process`'s own printf-style escaping, never string-concatenated into + a shell command. +- `install-layer2.sh` installs a scoped `/etc/sudoers.d/apiscp-borg` granting + exactly `apnscp ALL=(root) NOPASSWD: ` (validated + with `visudo -cf` before being installed) - nothing else is permitted to + run as root. The three binaries themselves therefore are the privilege + boundary: they validate every input (site id pattern, database engine + must be `mysql`/`pgsql`, database name character set, target must be + absolute, subpaths cannot contain `..` or a leading `/`) before acting. +- `/etc/apiscp-borg/config` (holds the repository passphrase) is kept + root-owned (0600), never read or written directly by the panel user; all + reads go through `config-get` and all writes through `config-set`, both of + which run as root via the same sudoers grant. +- `config-set` (and the module's `set_option`) enforce a fixed whitelist of + settable keys (`CONFIG_ALLOWED` in `bin/borg-apiscp-repo`, mirrored as + `Borg_Module_Surrogate::ALLOWED_KEYS`) and reject any value containing a CR + or LF, so a caller cannot smuggle extra config lines or inject shell + metacharacters into a file that is later `.`-sourced by the engine; the + stored value is additionally single-quote-escaped. +- Site-owner (`PRIVILEGE_SITE`) verbs never accept a site identifier as a + parameter. Every one of them calls a private `authSite()` helper that reads + `$this->getAuthContext()->site` (set by ApisCP's own auth framework to the + authenticated session's site) and validates it against `^site\d+$`; if it + cannot be resolved, the verb returns an error or an empty result rather + than falling back to any caller-supplied value. Where a database name is + also given (e.g. `restore_my_database`, `import_my_database`), it is + cross-checked against that resolved site's current database list + (`ownsDatabase()`) before use, so a site owner cannot reach another site's + data even by guessing a database name. The corresponding Layer 1 + subcommands (`owner-files`, `owner-restore-db`, `owner-import-db`, + `owner-account`) repeat the site-pattern and path-traversal checks + independently, so the guarantee does not rest on the module alone. +- Per-site database isolation: ApisCP's own `:export`/`:import` + verbs resolve their file argument **inside the target site's own filesystem + namespace** (a chroot-like view via `file_make_path`), not the host + filesystem, and a host-level dump/restore tool cannot see per-site + databases at all. The engine and the CLI tools therefore always go through + `$APNSCP_CMD -d :{list_databases,export,import}` in the + site's own domain context: a dump is written to a site-relative path + (e.g. `/tmp/.apiscp-borg-db/mysql/.sql`) which the engine then reads + back from its real host-side location under `/fst/...` for + archiving, and an import stages the dump under the site's own `fst` before + calling the site-context import. This is what makes per-site database + backup and restore correct in a multi-tenant ApisCP host where database + names can collide or be invisible outside their own site. +- A Borg repository is locked to one operation at a time (Borg's own + repository lock, in addition to the engine's local `$LOCK_FILE`), so + concurrent site-owner actions against the same repository queue rather + than run in parallel; there is no risk of two writers corrupting the + repository. + +## 10. Restore and recovery + +A Borg repository is encrypted; if the passphrase (and, for keyfile-mode +repositories, the exported key) are both lost, the data is permanently +unrecoverable no matter how many copies of the encrypted archives exist. +`borg-apiscp-repo backup-key` (see section 5) exports everything needed to +recover onto a fresh machine, and refuses to silently redo that export more +than once (use `--force` to intentionally redo it after a credential +rotation). + +To recover the repository itself on a brand-new machine, using either the +persistent key backup directory or the one-time `key-bundle` output: + +```sh +# 1. install borgbackup (your package manager, e.g. dnf/apt install borgbackup) + +# 2. obtain the repository passphrase (and, for keyfile-mode repositories, +# the exported key), from either: +# - the key backup directory (repository.passphrase + repository.key), or +# - the one-time key-bundle text saved at repository-creation time + +# 3. export the passphrase and the repository location +export BORG_PASSPHRASE="$(cat repository.passphrase)" +export BORG_REPO="" + +# 4. only for keyfile-mode repositories (repokey-blake2 keeps the key IN the +# repository, so this step is unnecessary for the default encryption mode): +borg key import :: repository.key + +# 5. confirm the archives are visible +borg list + +# 6. extract whatever is needed, e.g. the newest siteN-shadow archive +borg extract :: +``` + +From there, install apiscp-borg itself (`install.sh`, then point +`BORG_REPO`/`BORG_PASSPHRASE` at the reconnected repository) to regain the +per-site restore tooling (`borg-apiscp-restore`), or continue with plain +`borg list` / `borg extract` if only a bare `borg` binary is available: no +manual ACL/xattr replay step is needed, because Borg already restored that +metadata as part of `borg extract`. The repository and its archives are +never deleted by `uninstall.sh` (with or without `--purge`), so reinstalling +this project at any point picks up exactly where the repository left off.