Observable Web Latency

Recreating a custom graphic using Mosaic vgplot

The Observable Framework documentation includes a wonderful example about analyzing web logs, visualizing the latency (response time) of various routes on the Observable.com site. The marquee graphic is a pixel-level heatmap of over 7 million requests to Observable servers over the course of a week. The chart plots time vs. latency, where each pixel is colored according to the most common route (URL pattern) in that time and latency bin.

That said, a lot is going on in the original custom heatmap component:

• The data is pre-binned and aggregated for fast loading
• Observable Plot and HTML Canvas code are intermixed in non-trivial ways
• Frame-based animation is used to progressively render the graphic, presumably to combat sluggish rendering

Here we re-create this graphic with Mosaic vgplot, resulting in a simpler, standalone specification. We further leverage Mosaic's support for cross-chart linking and scalable filtering for real-time updates.

Select bars in the chart of most-requested routes above to filter the heatmap and isolate patterns. Or, select a range in the heatmap to show only corresponding routes.

Implementation Notes

While the original uses a pre-binned dataset, we might want to create graphics like this in a more exploratory context. So first we "reverse-engineered" the data into original units, with columns for time and latency values, in addition to route and request count. We can leverage DuckDB to re-bin and filter data on the fly!

We then implement the latency heatmap using a vgplot raster mark. Here is what that looks like when using a declarative Mosaic specification in YAML:

plot:
- mark: frame
  fill: black
- mark: raster
  data: { from: latency, filterBy: $filter }
  x: time
  y: latency
  fill: { argmax: [route, count] }
  fillOpacity: { sum: count }
  width: 2016
  height: 500
  imageRendering: pixelated
- select: intervalXY
  as: $filter
colorDomain: Fixed
colorScheme: observable10
opacityDomain: [0, 25]
opacityClamp: true
yScale: log
yLabel: ↑ Duration (ms)
yDomain: [0.5, 10000]
yTickFormat: s
xScale: utc
xLabel: null
xDomain: [1706227200000, 1706832000000]
width: 1063
height: 550
margins: { left: 35, top: 20, bottom: 30, right: 20 }

Key bits of the specification include:

• Binning to a pixel grid based on time (x) and latency (y).
• Mapping the pixel fill color to the route with largest request count per bin.
• Mapping the pixel fill opacity to the sum of counts within a bin.
• Interactive filtering using a selection ($filter). Setting colorDomain: Fixed ensures consistent colors; it prevents re-coloring when data is filtered.

However, this re-creation does diverge from the original in a few ways:

• The coloring is not identical. Ideally, vgplot should provide greater control over sorting scale domains (here, the list of unique route values).
• The re-creation above does not include nice tooltips like the original.