As part of the effort to define and track performance metrics, this note defines the two metrics I'm using to measure the syncing pipeline: write throughput and data delay. The design follows the strategy we discussed in Performance Measurement: few logs, aggregation happens inside the software — everything below is a handful of counters and histograms the daemon maintains itself.

1. Sync write throughput (bytes per second)

This measures how fast we ingest content we don't have yet. The canonical scenario is the fresh start: you subscribe to a site you've never seen and the daemon has to catch up on its entire history.

What we count (the easy part)

The numerator is payload bytes: the sum of the sizes of blobs persisted by the sync path. We already store every blob's size, so we attribute bytes at persist time, and only for sync-originated writes — local edits and reindexing don't count.

Two related numbers already exist and are worth knowing about, but neither is the right numerator:

/debug/network shows the SQLite logical size at daemon start vs. now (session growth) and per-peer bitswap byte counters. DB growth includes write amplification — indexes, links, full-text search — so it moves when the schema changes, not only when sync gets faster. The bitswap counters measure the wire, including traffic we later discard.

/debug/sqlite does not track bytes at all — it tracks writer-slot time (who held the write lock, for how long). Useful for finding contention, not for throughput.

The ratio of DB growth to payload bytes is our write amplification factor. Worth watching separately, but as its own metric — mixing it into throughput would make sync look slower every time we add an index.

Measuring time (the hard part)

Collecting bytes is trivial; giving them meaning is not, because the obvious denominator — a per-session timer — lies twice:

Bursts and gaps. Data arrives in bursts. If we only run the clock while a discovery session is active, a daemon that syncs 100 MB in 10 seconds and then sits idle for 10 minutes reports a great number while the user experiences a slow system. The session clock measures the pipe; the user experiences the calendar.

Concurrency. Many sessions run in parallel. Summing session durations double-counts every overlapping second: two concurrent 60-second sessions moving 60 MB total would report 0.5 MB/s when the machine actually ingested 1 MB/s.

The busy clock

Instead of per-session timers we keep one global busy clock: the union of all intervals during which at least one sync session was active.

Implementation is a single atomic counter of active sessions. When it goes 0 → 1, stamp the start time. When it goes 1 → 0, add the elapsed time to a cumulative busy_seconds counter. Overlapping sessions never double-count (the clock is already running), and idle gaps never count (the clock is stopped). When reading the value, if the counter is above zero we add the currently-open interval, so a long in-flight catch-up is visible immediately.

This gives us three numbers instead of one:

Active throughput = payload bytes / busy seconds. How fast the pipe is when it's actually pumping. This is the engineering metric — it's what should improve when we optimize reconciliation, bitswap fetching, or the persist path.

Duty cycle = busy seconds / wall-clock seconds. What fraction of the time we were syncing at all. This captures scheduling cadence and burstiness.

Perceived throughput = payload bytes / wall-clock seconds. What the user actually experiences.

The three are linked by an identity: perceived = active × duty cycle. That factorization is the whole point, because it tells you where to optimize:

Perceived is low but active is high → the data path is fine; we simply aren't syncing often enough. Fix scheduling and discovery cadence, not the pipe.

Active is low → the pipe itself is slow. Profile the sync phases (reconcile, bitswap fetch, persist) — we already have per-phase histograms for these.

One more derived number closes the concurrency picture: accumulate plain session seconds (the sum of all session durations, counting overlaps multiple times) and divide by busy seconds to get average concurrency. If active throughput scales with concurrency, individual streams are the bottleneck; if it doesn't, sessions are contending with each other — for the SQLite writer slot or for bandwidth.

For the fresh-start scenario specifically, one headline number sits on top: time to catch up — from first discovery of a site to the point where the outstanding want-count drains to zero, reported together with the total bytes ingested.

2. Data delay (freshness)

The second metric answers a different question: how long after data is created somewhere in the network do we receive it?

Definition

Every blob carries a claimed creation timestamp, and we record the moment we persist it. Per blob:

delay = received time − claimed timestamp

We only observe blobs whose claimed timestamp falls after the current daemon session started. Blobs authored while we were offline would dominate the distribution with huge values that measure our downtime, not the network's propagation speed — catching up on old content is backlog, not lateness. The throughput metric above is the one that judges backlog; delay judges the live path.

Shape and caveats

Percentiles, not averages. The distribution is bimodal: content arriving via live sync lands in seconds, content picked up by the next periodic sync round lands minutes later. An average would report a number in between that never actually happens. We keep a histogram and read p50/p90/p99.

Claimed timestamps are self-reported. They come from the author's clock. Skewed clocks produce negative delays — we clamp those to zero and count them separately as a clock-skew signal. A buggy or malicious peer can claim anything, so this is a network-health metric, not a security boundary.

Label by arrival path where cheap (live subscription vs. periodic sync vs. explicit discovery), so we can tell whether a p99 regression means gossip broke or the periodic interval is just too long.

3. Every metric, broken down by site and by kind

A global number tells us the daemon is slow; a per-site number tells us which site. So every metric above gets a site dimension: per-site throughput, per-site busy clock, per-site delay percentiles. And within each site, we split media files from everything else — the two populations behave so differently that mixing them makes the numbers lie.

Attributing blobs to sites

Grouping by site requires knowing, for every blob we write, which site it belongs to. I verified this against a production database (~75k blobs, ~5.6 GB) using three attribution rules:

Blobs tied to a resource (Refs, Comments, Profiles, Contacts, most Changes): the resource IRI starts with the space, and the space is the site. Direct lookup.

Changes without a resource association: attributed through their genesis blob, which the resources table already knows.

Media (file roots and raw chunks — 99% of the bytes): no resource association at all, but they are reachable from structural blobs through the links table, so they inherit the site of whatever document embeds them.

Coverage on the production database: 99.7% of blobs and of bytes attribute to at least one site (74,369 of 74,602 blobs; only ~17 MB orphaned, consistent with the index-drift number /debug/network already shows).

For display, a space maps to its domain (seedteamtalks.hyper.media, gabo.es, …) via the siteUrl in its root document — 49 of the 108 spaces in my database have one; the rest are personal spaces and show as the account ID.

Media files vs. everything else

Within each site, every counter is split into two kinds: media (file roots and their raw chunks) and document data (Changes, Refs, Comments, Profiles, Contacts, Capabilities). Classification is free — it's the blob's codec, known the moment the blob arrives.

The two populations are opposites, and that's exactly why the split matters:

In the production database, media is 99% of the bytes but only a third of the blobs (~27k blobs, 5.55 GB). Document data is two thirds of the blobs but 1% of the bytes (~48k blobs, 55 MB).

Their bottlenecks differ. Media throughput is network-bound and genuinely a bytes-per-second story. Document blobs average ~1 KB, so their cost is per-blob — indexing and the SQLite writer slot — and the honest unit for that bucket is blobs per second, reported alongside bytes per second. A combined bytes/s number is dominated by media while the daemon's actual struggle is usually the flood of tiny structural blobs.

The split matches what the user perceives during a catch-up: text renders when document blobs arrive, media backfills afterwards (root-first discovery already fetches structure first for exactly this reason). Per-kind throughput turns "the page appeared but images took forever" into two separate curves instead of one muddled average.

One consequence for the delay metric: only document blobs carry a claimed timestamp — verified in production, every Ref/Change/Comment/Capability/Contact/Profile has one and media has none (0 of 2,440 file roots). So delay is defined on the document-data bucket only; media freshness is inherited from the document that references it.

Shared media

The one ambiguity is deduplication: the same image embedded by two sites is stored once and is reachable from both. In production data this is real but bounded — ~5,500 blobs (~11% of bytes) are reachable from more than one space.

For the live metrics the ambiguity disappears: at write time we attribute bytes to the site whose sync session fetched the blob. If a second site references it later, we never download or write it again — that site genuinely cost us zero bytes, so the accounting stays honest. Only retroactive storage accounting ("how much disk does this site occupy?") needs a policy for shared blobs, and that's a separate metric with its own rules.

Per-site clocks

Sync sessions are already scoped — a discovery session targets one document tree, hence one site — so each site gets its own busy clock next to the global one: the union of that site's session intervals. Per-site active throughput and duty cycle then work exactly like their global counterparts, and the delay histogram carries the same site label, since a blob's site is known the moment it's indexed.

The kind split does not get its own clock: a session fetches media and document blobs interleaved, so "media time" vs. "document time" would be fiction. Both kinds share the site's busy clock and only the numerators are split — media bytes/s plus document bytes/s add up to the site's total throughput over the same denominator.

Summary

Active throughput (bytes / busy seconds) — is the sync pipe fast? The number that judges sync-path optimizations.

Duty cycle (busy seconds / wall seconds) — are we syncing often enough? The number that judges scheduling.

Perceived throughput (their product) — what the user feels.

Average concurrency (session seconds / busy seconds) — are parallel sessions helping or contending?

Time to catch up (per fresh-start, with total bytes) — the headline fresh-start number.

Data delay (p50/p90/p99 of received − claimed, live blobs only) — how fresh is the data we serve?

All of it per site — every blob attributes to a site (99.7% coverage verified on production data), so we can tell "sync is slow" apart from "this site is slow".

And per kind within each site — media (99% of bytes, network-bound, bytes/s) split from document data (two thirds of blobs, indexer-bound, blobs/s + bytes/s); delay applies to document data only, since media carries no claimed timestamp.