Writing
Open source

Catching a ticket pile-up with statistics: how Surge detects

Thirty "no internet" tickets arriving within the same minutes are one incident. How does Surge know?

November 20264 min readOpen source · Surge

When a switch fails, tickets reach the help desk not one by one but as a wave. The human eye notices after a while; I wanted Surge to notice in the first minute.

The window and the baseline

For every category and location, a baseline is calculated from the arrivals of recent weeks. If arrivals within a short window (10 minutes by default) climb to a clear multiple of that baseline, a candidate burst forms. Thresholds can be tuned per rule.

Similarity

The titles of candidate tickets are normalised and their word sets compared; "no internet", "wi-fi not working" and "3rd floor network" end up in the same group. Each ticket's match percentage is visible on screen, and a wrong match can be removed.

// simplified
score = 0.5*category + 0.2*location + 0.3*text_similarity
score >= threshold ? add to group : skip

One incident

The group becomes a major incident under a parent ticket: a problem record is opened, a shared follow-up goes to all of them, and a notice appears on the self-service portal so users do not open new tickets. When the parent ticket is solved, they all close together. The code is on GitHub; one package for 11 and 12.

open to new projectsLet's work together