All labs

One Pass, Bounded Memory

Streaming means, variances, quantile sketches, and count-min counting.

Ch 10 10.4
Builder
Systems & Performance
20 mindifficulty 3/5

Stream controls

Distribution
Concept drift

Mean shifts upward as the stream advances.

20000
500

How many rows arrive per Step or per animation tick.

Processed 500 of 20000 rows, one pass, no lookback.

Sketch sizes

300
64
64

Narrower width means more hash collisions and more overestimation.

3

Each extra row is an independent hash whose minimum we take.

Seed

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Running mean: Welford vs. exact batch

Recomputed batch statistics on the left axis are the ground truth; the streaming estimator should track them exactly.

40608010012005000100001500020000rows processedrunning mean
exact batch mean (recomputed each point)Welford running mean

Median: reservoir & bucket sketch vs. exact

0204060801001200100200300400valuecount
bucket sketch density

Heavy hitters: count-min vs. exact

00.200.400.600.8010246categorycount
exact countcount-min estimate (≥ exact)

Memory vs. accuracy

Bytes retained by each estimator against its relative error on the median, at the current cursor.

00.010.020.030.040.0501000200030004000Exact (all data): 4000 B, 0.0% errorReservoir (k=300): 2400 B, 1.3% errorBucket sketch (b=64): 256 B, 0.9% errorbytes retainedrelative error (median)
Exact (all data)Reservoir (k=300)Bucket sketch (b=64)
Exact mean
25.272
Welford mean
25.272
one running update per row
Exact variance
453.702
Naive sum-of-squares variance
453.700
offset +1.00e+6 exposes catastrophic cancellation
Exact median
19.441
Reservoir median
19.189
n=300 kept of 500
Exact p95
57.581
Bucket-sketch p95
57.708
64 equal-width buckets
Reservoir p95
59.074
Bucket-sketch median
19.623
Count-min max overestimate
sketch is 768 B, always ≥ truth
Rows processed
500
of 20000
One pass, bounded memory

Every streaming estimator here reads each row exactly once and never revisits it. Welford's update keeps the running mean and variance in four numbers total, immune to the cancellation that wrecks the naive sum-of-squares formula once values sit far from zero. The reservoir, bucket sketch, and count-min sketch each trade a fixed, small memory budget for an approximate — but bounded and provable — answer, which is the only option once n exceeds RAM.