Data, retention and backups

The data directory

Everything lives in DATA_DIR:

• rum.db, with its -wal and -shm files: the one SQLite database, in WAL mode.
• secret: signs sessions and salts the IP hash.
• backups/: the automatic backups.
• Working folders for the load-test engine and node coordination.

Keep it outside the code directory. Then a fresh unpack or an rsync --delete of the code cannot take the database, the accounts and the secret with it.

How long data is kept

The rollup runs every minute and always recomputes the last few hours, so late beacons are included. The purge runs on its own thread, so nobody waits for it.

A period is deleted from a tier only once the tier above it holds that period in full and matches it: a raw hour once its hourly sums add up to it, an hourly day once its daily sums do, a day of sessions once its daily counts and its counts per country are written and checked against the sessions, a day of synthetic runs once each check's row for that day says what its runs say. A chart whose window reaches past the raw days reads the sums, a day a point where it reaches the daily tier. Over several days read from the daily counts, "visitors" adds each day's visitors, so a person who came on two days counts twice there.

So the long windows keep answering. The visits chart reads the hourly sums past the raw days, and the daily counts past the hourly sums' 400 days. The world map reads the counts per country for the days whose sessions are gone; a browser, device or operating-system filter, or "only sessions with errors", applies only to the sessions still kept, and the page says from when. A synthetic check's availability over a month or a year, its error budget and its timeline read the check's days past its raw runs. Session detail, cities and regions on the map, single runs and their screenshots are not kept behind the raw days.

One limit is worth knowing. Inside the raw retention, percentiles are exact. Beyond it, a chart point that spans several hours shows the highest of the hourly p75 values, which is the worst hour rather than a true combined percentile. The map's load p75 over days read from the counts per country is the worst day's, in the same way.

Limits

Settings → System health shows what is using the disk, by kind, and how fast each kind is growing. Its Kept by default card sets the three tiers above and how long synthetic runs stay raw. Its Retention and size card holds a disk quota and a number of days per kind. The quota is 10 GB unless you move it, and it can be turned off; the days per kind are off until you set them.

Disk quota

The quota is a slider, from off to a number of whole gigabytes (1 to the size of the disk, 1000 at most). An install that has never set it holds at most 10 GB. It measures what the database holds: its pages less the free ones. SQLite reuses the space a deletion frees rather than giving it back to the disk, so with a quota the file stops growing at about the quota rather than shrinking below it. Beside the slider the page shows what the install holds now, what is free on the disk, and roughly what a setting would trim.

At each purge, every six hours, and with Apply retention now, an install over its quota trims its oldest raw data, in this order, until it is under:

1. session recordings;
2. synthetic screenshots;
3. synthetic runs and the values they read;
4. backend traces and their spans;
5. log lines;
6. machine, process and container samples;
7. raw visitor data: sessions, page views, resources, requests, errors and events.

Each kind goes the way the normal retention takes it, behind its rollup: the hourly and daily sums, the log counts, each synthetic check's days, the daily visitor counts and the counts per country are written and checked first, and a period whose rollup does not match yet is kept for the next run. The rollups themselves are never trimmed by the quota. The newest 24 hours of every kind are never trimmed either. When the quota cannot be met above them, the card says so and by how much. Deletion runs in slices of about 500 rows with a pause between them, so the collector keeps writing. A run under the quota deletes nothing. IoT readings are not trimmed by the quota.

Days per kind

A maximum number of days can be set for visitor sessions, session replays, backend traces, synthetic checks, machines and processes, and IoT readings. These trim through the same guarded paths as the quota. Run rollup now and applying retention are on the same page. On the command line, use node server/cli.js rollup and purge.

How much space

As a rough guide, the demo data is about 8 kB per session: three page loads, seven resources, nine API calls and ten interactions. At 10,000 sessions a day that is about 80 MB a day of raw data, or about 2.4 GB at 30 days' retention. Hourly aggregates are small beside it. If it is too much:

• lower the application's retention;
• turn off resource timing with data-resources="0", the largest table;
• sample visitors;
• or lower the disk quota.

The collector accepts up to 120 batches a minute per address, 2 MB and 500 events per batch.

Automatic backups

Every install backs itself up once a day, unless BACKUP_OFF=1 is set. The backup:

• runs on the jobs' thread;
• uses VACUUM INTO, SQLite's online copy, because copying a live WAL database with cp is not safe;
• checks the copy with quick_check and row counts against the source;
• compresses it with gzip and stores the secret beside it;
• keeps the newest three in DATA_DIR/backups/.

It refuses to start when the disk could not hold the copy, and says so. BACKUP_HOURS changes the interval and BACKUP_KEEP the number kept. On an install with several nodes, each backup is also handed to every peer. See Nodes and mirrors. Copy backups off the machine as well. A backup on the same disk does not survive the disk.

Restoring

systemctl stop <watchdog> <collector service>   # however you run it
cd "$DATA_DIR"
mv rum.db rum.db.broken; rm -f rum.db-wal rum.db-shm
gunzip -c backups/rum-YYYYMMDD-HHMMSS.db.gz > rum.db
cp backups/rum-YYYYMMDD-HHMMSS.secret secret && chmod 600 secret
systemctl start <collector service> <watchdog>

Restore the secret that belongs to the backup. With a different secret, everybody is signed out, and visitor counting across the restore point breaks.

ClickHouse (optional)

For heavy log and process-metric volumes, a ClickHouse server can hold those tables. Set CLICKHOUSE_URL (with CLICKHOUSE_USER, CLICKHOUSE_PASSWORD and CLICKHOUSE_DB), and CH_STORE=logs,process_metrics to name what goes there. Nothing changes unless CLICKHOUSE_URL is set. The logs page says which store answered a search.