Logs and log patterns

The host agent reads two kinds of source: systemd journal units, and log files. Nothing is collected until you choose it. The agent finds and offers sources, it does not gather them by itself.

Where the agent looks

• systemd units, read from the journal with a cursor.
• System logs under /var/log.
• PM2 application logs in PM2's log directories, which are never under /var/log.
• Deploy directories: a logs/ or log/ folder under /opt, /srv, /var/www and home directories.

The search is not a walk of the filesystem. The roots are listed and checked for existence, and symbolic links are not followed. Rotated and compressed copies are skipped. To see what a machine would offer, run this on it:

node /opt/obscenter-agent/discover.cjs --sources

On Windows the agent reads the event log channels you name for that machine.

Worth switching on

A stock Ubuntu machine offers dozens of files and hundreds of units, and most of it is housekeeping. Each source is therefore classed as app, web, system or unit, with its size and when it was last written. Sources written in the last day that look like your own software go in a Worth switching on box at the top, with one button to collect them all. Security logs such as auth.log are always suggested, because a quiet auth log is the good case. System logs sit in a folded group below. You switch sources on and off on the host's settings page.

Reading logs

Backend → Logs has two tabs, Lines and Patterns & storage. On Lines, filter by level: all, warnings and worse, errors and worse, or critical. Lines with no level are counted separately. Above the lines, the Volume chart shows how many lines matched in each part of the window, split by level, by machine or by unit (the busiest three, and the rest as one bar). Click a bar to zoom into it, or drag across several. One host's page shows its own lines too. When the setting to keep what a request logged is on, the log lines a traced request wrote also appear in its trace.

Searching

The search box takes a short query language. Press Enter to run it; the ? beside the box shows the same summary as below.

• timeout refused: lines that contain both words. Upper and lower case count as the same for the letters A to Z.
• "timed out": the phrase, as typed.
• -health or -"GET /health": leave those lines out. NOT means the same.
• refused OR timeout: either. Terms next to each other must all match, and that binds tighter than OR, so a b OR c is "a and b, or c". Use parentheses to group: (refused OR timeout) host:web-1.
• /time ?out/: a regular expression, matched anywhere in the line; /…/i ignores case.
• host:web-1, unit:nginx (also finds nginx.service), source:…: where the line was written. A * in the value stands for any run of characters: host:web-*.
• service:checkout: the units that service writes under, on its machine. A service you cannot see matches nothing.
• level:error is exactly that severity, level:<=warning is warnings and worse, level:3 works too, and level:none finds the lines that carry no severity.
• trace:<32 hex digits>: the lines one request wrote. trace:* finds every line that names its request.
• Any other name:value reads a field of a line that is one JSON object: status:>=500, user.id:42, msg:"order failed", ok:true, code:* (the field is there). >, >=, < and <= compare numbers. To reach a JSON field named like one of the fields above, start it with a dot: .level:info.

To search for text that contains a colon, or that starts with a minus sign, put it in quotes: "error:", "-1". A mistake in a query is not run: the page says what was expected and marks the place in the query.

In a line, click a field of a structured line, or its level, to add it to the search. Saved keeps a search under a name, with its time range. Saved searches are your own, in the organisation you are looking at; what one finds is always limited to what you may see.

Regular expressions

The supported syntax: characters, ., classes such as [a-z] and [^0-9], d w s and their capitals, , ^ and $ (the start and end of the line), groups ( ) and (?: ), |, and the repetitions * + ? and {n,m} with counts up to 100. Back-references and look-arounds are not supported. The engine reads each line once from start to end, so the time an expression takes grows with the length of the line, never faster. A query may hold up to three expressions of up to 300 characters.

How much a search reads

A search that reads the text of the lines is bounded. With logs stored in SQLite it reads at most the newest 1,000,000 lines of the window (after the machine, source and level filters) and stops reading after about four seconds, reading the newest part of the window first. With logs stored in ClickHouse the line limit is 50,000,000. When either limit is reached, the chart and the list say how far back the search reached; everything from that moment on was searched and counted in full. Narrow the range, or add a machine or a level, to search further back. Filters by machine, unit, source and level alone are not limited this way.

Log patterns

Most log volume is the same few sentences with different numbers in them. The Patterns & storage tab reduces each line to its pattern: the sentence with numbers, ids, addresses and times replaced by <*>. For each pattern it shows its kind, how many lines it had, how many were not stored, a sample, a proposal and your verdict. The kinds are error, security, access, health, debug, periodic, lifecycle and other.

• Choose a mode: Store everything, except what I drop (the default), or Store only what I choose to store.
• Noise is proposed for dropping, and errors, security and access lines for storing. Mark each pattern keep or drop.
• keep every line at error level or worse is on by default, whatever a pattern says.
• SSH and auth lines, and access-log GET lines, are always stored, because the security pages read them.

Dropping happens at the collector. The agent still sends the line, and the collector counts it and keeps one sample per pattern per day. A pattern you drop by mistake can therefore be judged later, and you can see what dropping it saved. Changing verdicts is for administrators.

How long logs are kept

Two limits apply to the whole install, and the tighter one wins. Keep logs for (days) is 7 by default, from 1 to 365. Never more than (MB) is 1024 by default. Both are set on a host's settings page. A disk quota for the whole install can also be set; over it the oldest log lines go after their daily counts are written. See Limits.

Logs and security

Collected logs feed three security views. auth.log or the SSH journal feeds SSH logins. nginx access logs feed the map of who is scanning your servers. The Windows Security log feeds logons. See SSH logins, Windows logons and scanning.