Encrypted Offsite Backup (restic)

The offsite service pushes an encrypted, deduplicated copy of a project's database and filestore off the machine. Encryption happens here, before anything leaves the container — the storage side only ever sees ciphertext.

Since 2026-08 this runs on restic. It replaced BorgBackup; there is deliberately only one mechanism, so nobody has to work out which of two is active on a given machine.

The shape of it

Odoo machine                                     backup server
┌───────────────────────────────┐                ┌──────────────────────────┐
│ offsite container             │                │ rest-server              │
│  restic backup                │──── HTTPS ────▶│  --append-only           │
│  (encrypts + deduplicates)    │   port 8000    │  --private-repos         │
└───────────────────────────────┘                └───────────┬──────────────┘
   sources (read-only):                                      │
     /source/pgbackrest pgBackRest repo (WAL + backups)       ▼
     /source/filestore  this database's filestore        <area>/db
     /source/dumps      optional / the fresh dump        <area>/files

Two repositories per area

A run writes into two repositories under the same customer area:

Repository	Contents	Snapshot tags
<area>/db/	pgBackRest repository, or the database dump	zodoo,db
<area>/files/	the filestore of this database	zodoo,files

The reason is monitoring, not tidiness. With everything in one repository, an arriving filestore hides a database dump that stopped coming: the area looks freshly written, and the part that matters is missing. Split, each stream has its own age, and the backup server alarms per stream — the mail then says which half stopped.

Both live in the same area and therefore share access credentials and passphrase. It stays one secret per project; two passphrases would be two things to lose, neither protecting anything the other does not.

OFFSITE_LAYOUT=flat restores the old single-repository behaviour. It exists for legacy installations that should not be moved, not as a preference.

Three properties matter, and they are the reason for this setup:

• The storage side cannot read the backup. restic encrypts on the source machine. Whoever holds the disk — a provider, a foreign NFS export, a stolen drive — sees ciphertext.
• The source machine cannot delete the backup. rest-server --append-only accepts writes and refuses removals. A compromised Odoo host can therefore not destroy the history. This is the part a plain NFS mount can never provide, and it is the main reason the target is our own server rather than a share.
• One customer cannot see another. With --private-repos every account is confined to the path matching its username; anything else answers 401.

The trade-off of append-only is that the client can no longer clean up. Retention has to run on the server side, and it has to actually run there — see Retention.

Targets

OFFSITE_REPO	What it is
rest:https://10.222.0.106:8000/<area>/	Our backup server. The normal case, append-only, one area per customer.
sftp:u12345@u12345.your-storagebox.de:23/…	Hetzner Storage Box or any SSH target. Key at $HOST_RUN_DIR/offsite/id_ed25519 (mode 0600).
/mnt/somewhere/repo	A mounted filesystem. Also set OFFSITE_LOCAL_DIR so the path exists inside the container.

For a path target the run refuses to start when the parent directory is missing. That is on purpose: an unmounted disk looks exactly like an empty repository, and restic would happily create a fresh one on the local disk — which is only discovered when the backup is needed.

What enrollment hands out

The backup server issues no repository key. odoo offsite register receives:

OFFSITE_REST_USER / OFFSITE_REST_PASSWORD	an upload account for this area
OFFSITE_WO_URL	the write-only receiver
OFFSITE_WO_RECIPIENT / OFFSITE_WO_DB_RECIPIENT	the two age public keys

Public means these may sit in a settings file and travel over the wire: they encrypt, they decrypt nothing. So a machine set up this way holds no secret that can read its own backup.

That also makes the handover far less delicate than it used to be. The access password is replaceable (restic-area passwd) — lose it and you set a new one. The only irreplaceable secrets are the private age keys, and those are never created here: they live in 1Password and nowhere else.

An older backup server that still issues repo_url and repo_key keeps working — register takes them if they arrive, and the run then uses restic as before.

Setting it up against our backup server

Do not wire this by hand. The machine asks for an area, a human approves it:

odoo offsite register

1. The first call files a request (area name derived from the project name) and remembers request id + pickup token in $HOST_RUN_DIR/offsite/enroll.json. No credentials exist yet.
2. An admin opens the backup server's admin page, checks the name and approves. Access password and repo key are created at that moment.
3. Both are shown exactly once. The admin files them in 1Password and confirms that with a checkbox. Without that confirmation the machine does not get its credentials — a repo key that exists only on the source machine is worthless precisely when it is needed.
4. Run odoo offsite register again: it collects credentials and the server certificate, writes them into the settings and sets RUN_OFFSITE=1. The server then forgets the repo key.

odoo reload && odoo build offsite
odoo offsite backup          # first run, creates the repository

On first contact there is no server certificate on the machine yet. It is fetched, pinned to $HOST_RUN_DIR/offsite/rest-server.crt and its fingerprint printed — the same trust-on-first-use as ssh accept-new. Any later change aborts the connection instead of being silently accepted.

If the machine already has a passphrase — zCICD and the shop generate one per project and keep it in the backend — that passphrase stays the truth and the server creates no second key. Two keys for one repository is a trap, not redundancy.

What gets backed up — and the check that must not be lost

A run collects, from read-only mounts:

• /source/pgbackrest — the pgBackRest repository (WAL archive + backups), only present with RUN_PGBACKREST=1
• the filestore of this database (filestore/$DBNAME), not the host-wide pool; on a machine with several instances the pool holds other customers' attachments
• /source/dumps only with OFFSITE_INCLUDE_DUMPS=1, or the single fresh dump that odoo offsite backup pulls when pgBackRest is off

The run aborts when no database state would end up in the snapshot. The filestore is always there, the database is not — and a snapshot of nothing but attachments looks like a backup until someone tries to restore. This is the failure this system had before; it must survive every future refactor.

OFFSITE_ALLOW_WITHOUT_DB=1 switches the check off. Only use it when the database is provably backed up elsewhere.

And the run aborts when no filestore would end up in the snapshot — for the same reason, in the other direction. A database restores fine without attachments; it is just incomplete, and in Odoo that shows up the first time someone clicks an invoice PDF. An empty or missing filestore directory is what an unmounted volume looks like, not what an empty instance looks like, so an empty directory does not count as a filestore. OFFSITE_ALLOW_WITHOUT_FILES=1 is the deliberate way out for an instance that genuinely has no attachments yet.

Both streams are attempted even when one fails, and the run then reports which. Otherwise a broken database upload would mask that the filestore did not go either, and one alarm would arrive where two belong.

The recommendation is RUN_PGBACKREST=1: it costs nothing extra (it runs on a disk that is already paid for) and adds point-in-time recovery on top.

Write-only filestore backup

The restic path has one property that cannot be configured away: a machine that can back up can also read its own backup history, because deduplication needs the repository index and the index is encrypted (see Keys). A compromised Odoo host therefore reaches not just the live database but every older state as well.

For the filestore that is avoidable, and cheaply, because Odoo already does the hard part: every attachment is named after the SHA-1 of its content (filestore/<db>/05/055ffc5c…). A file is written once and never changes, and "same content" is already "same name" — the deduplication is in the naming. So "what is missing at the far end?" is a pure name comparison, answerable from a local ledger without reading the target at all.

Set both of these and the filestore leaves the restic path:

Setting	What it is
OFFSITE_WO_URL	The write-only receiver, e.g. https://10.222.0.106:8444/<area>/
OFFSITE_WO_RECIPIENT	An age public key (age1…). Generate with age-keygen; the private key belongs in 1Password and nowhere else

odoo offsite filestore     # or automatically as part of `odoo offsite backup`

What the machine can then do, and what it cannot:

	restic path	write-only path
upload	yes	yes
read what it uploaded	yes, all of it	no — encrypted to a public key it has no private half of
delete	no (append-only)	no

When a write-only target is configured it replaces the restic files stream rather than running beside it. Two copies of the same attachments in two places is cost without redundancy, and it makes "which one do I restore from?" a question during an incident.

What travels, and what does not

New files go up as one bundle per run (tar → gzip → age), not one object per file: an instance with a million attachments would otherwise mean a million HTTP requests. Alongside it goes a manifest:

{ "run": "20260821T173222Z", "kind": "filestore",
  "bundle": "filestore-20260821T173222Z-85914f45a55c.tar.gz.age",
  "sha256": "85914f45…", "size": 665623,
  "files_added": 56, "files_total": 56, "ledger_sha256": "8f73c96f…" }

The manifest lists bundles, never file names. That is deliberate: a file name is the hash of its content, so a name list at the target would let someone confirm whether a particular known document is in the backup. Bundle names and checksums are enough for the receiving side to notice a missing bundle, which is what completeness means here.

The ledger

$HOST_RUN_DIR/offsite.state/filestore.ledger — one file name per line, appended only after a successful upload. A crash between upload and ledger write costs a repeated upload, never a file that is believed safe but never arrived.

Losing the ledger (volume wiped, machine rebuilt) means the next run uploads the whole filestore again. That is the price of never asking the target what it has; it costs traffic, not data.

Restoring

Needs the age private key from 1Password — this machine cannot do it:

age -d -i filestore.age-key -o bundle.tar.gz filestore-<run>-<sum>.tar.gz.age
tar xzf bundle.tar.gz -C $ODOO_FILES/filestore/<db>/

Unpack every bundle, oldest first. And the filestore checks itself: because each name is the SHA-1 of its content, sha1sum over the restored tree against the file names is a complete integrity check — no manifest, no checksum list, no key required.

The window against the database

WAL/PITR can recover the database to any moment; the filestore is pushed once a night. So an attachment created at 14:00 is in the backup once the night has passed, not before. The rule that follows: the filestore must be at least as new as the database recovery target. An older database with a newer filestore is always safe (a superset); the other way round, attachments are missing. Run the sync more often if the window matters — it is cheap, because it works on names.

Write-only database backup (WAL)

The same move as the filestore, for the half that matters more. With RUN_PGBACKREST=1 the database is captured as backups plus WAL segments, and both are immutable: a WAL object is written once and never changed, a backup directory never changes once pgbackrest has recorded its label in backup.info. So there is nothing to deduplicate — and therefore no need to read the target, which is what forces a readable key onto the machine in the restic path.

Note this is not pgBackRest's own repo-host topology. That one also keeps the key and the delete rights off this machine, but it needs pgbackrest running on the far side. This path uploads to a receiver that only ever stores what it is given and knows nothing about postgres — see 12-pgbackrest.md.

Setting	What it is
OFFSITE_WO_DB_RECIPIENT	age public key for the database stream. Deliberately a different key from the filestore one
OFFSITE_WAL_CRON	how often WAL is pushed. * * * * * — every minute

odoo offsite db     # backups + WAL (after the nightly pgbackrest backup)
odoo offsite wal    # WAL only, every minute via CRONJOB_OFFSITE_WAL

Two modes because they have different rhythms. WAL goes up every minute, so losing the machine costs a minute of transactions rather than a night. The run is cheap: nothing new means no upload, and it is silent, because it runs 1440 times a day.

Why a separate key from the filestore: the two have different value and are needed separately. "Somebody may restore the filestore but not the database" is a real request, and one shared key cannot express it.

What this buys beyond confidentiality

• Completeness is checkable without a key. WAL names are a sequence and the manifest declares, in the clear, which segments belong to which backup (wal_start/wal_stop, plus the backup type). In a restic repository the file names are encrypted, so only a key holder could ever notice a broken chain. On the backup server wo-check verifies: every declared object present and the right size, every base backup's begin_wal present, no gap inside a timeline, and manifests that only ever grow — a compromised machine cannot quietly declare less than it did yesterday.
• Retention becomes possible without a key. Whole generations can be dropped by name. An append-only restic repository cannot be pruned at all, because forget --prune needs the repo key.

Sizes, measured

WAL compresses extraordinarily well because a segment is a fixed 16 MiB regardless of how much is in it:

	raw	transferred
quiet WAL segment	16 MiB	~16 KB (factor ~1000)
busy WAL segment	16 MiB	~630 KB
base backup (small test DB)	9.6 MB datadir	5.8 MB

A base backup is a full copy every time, so its frequency is the only real lever on growth. WAL volume is the irreducible part — it is the actual write volume of the database.

The *.partial trap

pg_receivewal writes the segment currently being filled as streaming/<name>.partial. It is incomplete by definition, and uploading it would store a half-written segment under a name that is supposed to mean "complete". Only wals/ is ever read, and *.partial is excluded on top of that.

At a hundred instances

Measured on the real backup server, because guessing is worthless here: it writes 410 MB/s to its local disk and 8.5 MB/s to the NFS store (reads: 70 MB/s), consistent across 1M/8M/32M block sizes. That write figure is the ceiling for all customers together — 30 GB/hour, 184 GB in a six-hour night.

WAL is irrelevant at that scale (16 KB per segment). Base backups are the whole problem: 100 instances × 5 GB compressed = 16 hours, which collides with the next night. Incremental base backups are therefore the condition for a hundred instances, not an optimisation.

It is bandwidth, not locking: 100 concurrent uploads all answered 201, and the areas have separate ledgers and locks. They queue on throughput.

Two things follow for the client:

• Every instance uploaded in the same second, because they all run * * * * *. The WAL job now waits a stable offset derived from the project name — stable rather than random, so each machine keeps its own second and runs cannot overtake each other.
• One manifest per run does not scale: hundreds of files per day per area on a busy database, and the completeness check reads the manifests of all areas. Manifests are now one file per day (<date>-db.jsonl), one JSON line per run, appended. Appending fits the write-only model — adding yes, changing and deleting no — so earlier lines stay untouchable and "a source may never declare less than before" still holds.

Concurrency

The minutely WAL job gets its own container name, because docker rejects a duplicate name with a hard error and a job running every minute must not fail every minute while a nightly base backup upload is still going. Serialisation happens inside, on a lock in the state directory, where a busy lock is a quiet success rather than an error.

Nothing is spooled to disk

A base backup and the first filestore bundle are streamed: tar | gzip | age goes straight onto the wire. Spooling them to a file first would need their own compressed size in scratch space - for a 600 GB database that is ~170 GB, and it would land on the container's writable layer, i.e. the system disk.

Streaming means the checksum cannot be known in advance, because age is deliberately not reproducible (a fresh ephemeral key per encryption). So the source hashes what it streams, the receiver hashes what it received, and the two are compared - an end-to-end check that costs no disk. Where a file genuinely is needed (WAL segments, manifests), the scratch directory sits inside the state directory, next to the data rather than on the system disk.

Repeats and starting over

Before uploading, the source asks the receiver whether the object name is already taken (HEAD). That makes a repeat cheap: after a crash between upload and ledger write, or after a deliberate reset, what is already there is recognised and not re-sent.

odoo offsite reset            # forget the ledgers; next run offers everything again
odoo offsite reset db         # only base backups + WAL
odoo offsite reset filestore  # only the filestore

The reset deletes nothing on the receiver - this machine cannot, by design. The old ledgers are renamed rather than removed; they are the only record of what this machine ever reported.

One subtlety worth knowing: an object that was already there is not declared again in the new manifest. Its stored ciphertext was produced by an earlier encryption and therefore has a different checksum than one computed now - declaring our value for those bytes would make the completeness check report a mismatch that is not one. It stays declared by the run that uploaded it, and manifests are read cumulatively.

HEAD answers only whether a name is taken - no content, no size. There is still no GET.

Local space on the machine

pgBackRest writes into its local repository before anything is uploaded, so local space is needed for the local retention window. Three settings decide how much:

Setting	Default	Effect
PGBR_COMPRESS_TYPE	zst	roughly a third of the raw size at low CPU cost
PGBR_RETENTION_FULL	14 (days, ..._TYPE=time)	how far back a continuous point-in-time window reaches
PGBR_FULL_CRON / ..._DIFF_CRON	weekly / daily	weekly full plus daily differentials, rather than a full every night

The schedule is the real lever. A daily full of a 45 GiB database is about 16 GiB compressed, i.e. 5.8 TiB a year for one instance. Weekly full plus daily differentials costs a fraction of that and restores just as fast, because restore --delta only fetches the files that differ.

Retention needs no job of its own: expire runs as the last step of every backup. That is deliberate — a cleanup that has to be scheduled separately is a cleanup that eventually is not.

Ledger and restoring

$HOST_RUN_DIR/offsite.state/wal.ledger and base.ledger, appended only after a successful upload. Both are written by the container as root, so reading them from the host needs sudo.

Restoring needs the age private key from 1Password:

age -d -i db.age-key -o backup.tar backup-<label>.tar.age && tar xf backup.tar
for f in wal-*.age; do age -d -i db.age-key -o "${f%.age}" "$f"; done

The result is a pgBackRest repository again: put the backup directories under backup/<stanza>/ and the segments under archive/<stanza>/<version>-<n>/, point a pgbackrest.conf at it and run a normal restore.

Unlike the barman path the segments are already compressed by pgbackrest, so there is no gunzip step and the old "a segment must be exactly 16 777 216 bytes" sanity check no longer applies — the compressed size varies.

Switching the restic path off entirely

Once both streams are write-only, restic is not used at all — and then a run must no longer demand OFFSITE_REPO or OFFSITE_PASSPHRASE. That is the point of arriving here: the passphrase is the most expensive secret in the setup, and whoever does not need it should not have to hold it.

odoo setting OFFSITE_REPO=
odoo setting OFFSITE_PASSPHRASE=
odoo reload
odoo offsite backup      # runs both write-only streams, nothing else needed

On the backup server the old repositories have to be taken out of monitoring — otherwise nobody writes to them any more and the per-stream "no backup for 48 h" alarm fires from the next day onwards, for ever:

restic-area retire <area>

That moves db/ and files/ aside (.retired-db-<ts>), it does not delete them. Deleting would be irreversible, and the old stock is still worth having until the write-only path has proven itself over time. The area's access stays — the write-only receiver needs it.

Commands

Command	What it does
odoo offsite register	Request a customer area, then pick up credentials after approval.
odoo offsite filestore	Push the filestore to the write-only target. Needs no repository key.
odoo offsite db	Push base backups + WAL to the write-only target.
odoo offsite wal	Push newly archived WAL only. Runs every minute.
odoo offsite backup	Run a backup now. Same run as the nightly cron; a quiet no-op without RUN_OFFSITE=1.
odoo offsite init	Create the repository (the first backup does this anyway).
odoo offsite list	List snapshots.
odoo offsite info	Repository stats (size, deduplication).
odoo offsite check	Verify integrity by re-reading the data. Takes time and costs traffic.
odoo offsite prune	Apply retention. Refused against append-only targets, with an explanation.
odoo offsite restic <args>	Escape hatch: any restic command. OFFSITE_STREAM=db|files picks the repository (default db).

list, info, check and init run over both repositories and label which one they are reporting on.

The nightly run is OFFSITE_BACKUP_CRON (default 04:00), deliberately after the pgBackRest backup at 02:00 so it picks up the fresh state instead of yesterday's.

Retention

OFFSITE_KEEP_DAILY / _WEEKLY / _MONTHLY describe what should be kept.

• Against sftp: or a path target, the run applies them itself after each backup.
• Against our backup server (append-only) the client cannot — odoo offsite prune says so instead of failing silently. Retention runs on the server, in a maintenance window, with a separate access that is not append-only.

This is the point that gets forgotten in append-only setups: without server-side retention the repository grows without bound — at our volumes in months, not years.

Restoring

Two things are needed: the repository address and the passphrase. There is no separate key file to lose.

export RESTIC_PASSWORD="<passphrase from 1Password>"
BASE="rest:https://<user>:<password>@10.222.0.106:8000/<area>"

# the database
export RESTIC_REPOSITORY="$BASE/db/"
restic --cacert rest-server.crt snapshots
restic --cacert rest-server.crt restore <snapshot> --target /restore

# the attachments
export RESTIC_REPOSITORY="$BASE/files/"
restic --cacert rest-server.crt restore latest --target /restore

Same passphrase for both. Then restore the database from the dump or the pgBackRest catalog, and put the filestore back into $ODOO_FILES/filestore/<db>.

Check the dates of the two snapshots against each other. A filestore much newer than the database is harmless; a database much newer than the filestore means attachments are missing for everything created in between.

A backup that has never been restored is not a backup. Plan the rehearsal; do not wait for the emergency to be the first attempt.

Keys

Secret	Operational copy	Authoritative copy
OFFSITE_PASSPHRASE	project settings on the machine (0600)	1Password — without it the backup cannot be opened
OFFSITE_REST_PASSWORD	project settings on the machine	1Password (same item)
SSH key for sftp: targets	$HOST_RUN_DIR/offsite/id_ed25519	access credential only, not a data key
Server certificate	$HOST_RUN_DIR/offsite/rest-server.crt	public, gets distributed

One passphrase per project — a single shared one means one leak opens every customer. The passphrase has to sit on the source machine because the cron runs unattended; that is a deliberate, defensible trade-off, since whoever owns the machine already has the live database. The protection is aimed at the storage location and at the integrity of the history.

Troubleshooting

Symptom	Cause / fix
OFFSITE_REPO is empty	No target configured. Run odoo offsite register.
OFFSITE_REST_USER is empty	Area credentials missing — the registration never completed.
restic refuses the connection / certificate error	rest-server.crt missing or the server certificate changed. Re-run register; if the fingerprint really changed, find out why before trusting it.
Enrollment service … is unreachable	The enrollment service is only reachable over the zebroo VPN. Is this machine in a VPN group with the backup server?
Backup aborts with "no database state in the backup"	Working as intended. Set RUN_PGBACKREST=1, or use odoo offsite backup (pulls a dump itself).
Stale lock after a crash / reboot	The run breaks a hanging lock itself before starting. rest-server permits lock removal even in append-only mode.
Offsite target is a path and the run refuses to start	Parent directory missing — the disk is probably not mounted. Do not "fix" this by creating the directory.

Related

Internal documentation on the backup server itself (isolation, the enrollment service, monitoring, server-side retention, the 1Password vault) lives in Odoo Knowledge under Backup Plan → restic-backup — zentraler Backup-Server (append-only).