A practical guide to security alarm triage

Every security team has the same failure mode, and it is not the one people expect. It is not that the system missed something. It is that the system reported everything, nobody could keep up, and the thing that mattered went past in a queue that had stopped being read.

Triage is the discipline of making that queue workable. It is mostly an operational problem, not a technical one.

Start by counting

Before changing anything, measure what you currently produce. For one normal week, per camera and per area:

• How many events were raised?
• How many were opened by a person?
• How many needed any response at all?

The ratio between the first and the third number is the one that matters. If you are raising four hundred events a night and three need a response, you do not have a detection problem. You have a configuration problem, and tightening the thresholds is usually the wrong fix for it.

Most sites have never measured this, and the numbers are routinely worse than anyone expects.

The rule: an event is a request for a decision

If an event does not require someone to decide something, it should not be an event. It can be a log entry, a daily digest, a report line — but not an interruption.

Applying that one rule honestly usually removes more noise than any amount of model tuning. Ask of each event type: if nobody looked at this, what would go wrong? If the answer is “nothing”, it is not an alert.

Cut the volume before you tune the detector

In rough order of effectiveness:

Narrow the areas. A camera watching its whole field of view reports everything in it. A camera watching a marked polygon — the fence line, the stock aisle, the apron in front of the goods door — reports what happens there. This is usually the single biggest reduction available, and it costs nothing.

Narrow the hours. Most areas only matter outside working hours. A stockroom that is busy all day and should be empty from 19:00 generates a sane number of events if you say so.

Narrow the subject. Person-only detection in an area where vehicles pass constantly removes an entire class of noise.

Fix the camera. If one camera is producing most of your false events — and on most sites, one camera is producing most of your false events — the cause is usually physical. Glare at a particular time of day, foliage moving in frame, insects at the lens at night, a light that swings. These are fixed with placement, lighting and cleaning, not with thresholds.

Lowering a confidence threshold to suppress a specific nuisance is the fix that feels fastest and costs the most later, because it suppresses real detections everywhere else too.

Give the operator enough to decide in one glance

The structural change that makes triage sustainable is attaching the evidence to the event. An event that arrives with the frame that caused it can be triaged in seconds. An event that requires opening a second system to understand will, in practice, be cleared without being understood.

What an event should carry:

• The frame at the moment of detection.
• The clip around it.
• The camera and the area, named in a way a human recognises — “goods door”, not “cam_14”.
• The time.
• What triggered it, in plain words.

Define the states, and record them

Three states cover nearly everything: acknowledged (seen, no action), resolved (dealt with), escalated (passed on, with a named destination).

The important part is that whoever changed the state is recorded against the event. The first time an incident has to be reconstructed, “who cleared this and when” is the question that gets asked, and “we don’t know” is an expensive answer.

Write the escalation path down

Most teams have an escalation path that lives in someone’s head. Put it in writing, per event type: who gets it, in what order, after how long, and what happens if nobody responds.

Then test it at the time it would actually be used. An escalation path that works at 3pm on a Tuesday and not at 3am on a Sunday is not an escalation path.

Review weekly at first

For the first month, look at the queue once a week and ask three questions:

1. Which event types produced nothing actionable? Turn them into report lines.
2. Which camera or area produced the most noise? Go and look at it physically.
3. What was missed? Work backwards to why, and be honest about whether it was coverage, configuration or the queue being too full to read.

After a month, this moves to monthly. The first month is where the real tuning happens, and it needs to be done against your own site’s volume rather than a default.

The measures worth keeping

• Events per night, per area. Trending down as tuning takes effect.
• Proportion opened by a person. If this is falling, the queue is becoming unworkable, whatever the other numbers say.
• Time from raised to opened. The practical measure of whether the workflow is alive.
• Events that needed a response. The number that justifies the system.

If the proportion opened starts falling, stop adding capabilities and go back to cutting volume. A queue that is read is worth more than a queue that is complete.

Where to start

With a week of counting, and a walk around the two or three cameras producing the most events. That costs nothing and tends to reshape the whole project — most sites discover that a small number of areas matter and a small number of cameras misbehave, and that both are addressable before any new capability is switched on.