Cuedis

Anomaly detection for small businesses, explained without the maths

"Anomaly detection" sounds like something only banks and big tech companies can afford. The idea behind it is simple, though, and every experienced manager already does a version of it in their head. They know when a day feels wrong.

This post explains what anomaly detection means for an ordinary business, how a system decides what is unusual, and why so many alert systems end up being ignored.

The idea in one sentence

An anomaly is something that is unusual for that particular thing.

Every word of that sentence matters:

  • Unusual, not "bad". A product suddenly selling three times as much is an anomaly too, and often good news.
  • For that particular thing. Selling 20 units a day is normal for one product and alarming for another. A supplier delivering in 10 days is normal for an overseas supplier and worrying for a local one.

So anomaly detection always starts with the same question: what is normal for this product, this supplier, this customer, this outlet, this cashier?

How a system learns "normal"

A simple, robust approach looks at each item's own history and asks two questions:

  1. What is its typical value? For example, the median daily sales over recent weeks. The median (the middle value) is better than the average because one huge day does not distort it.
  2. How much does it usually vary? Some items are steady, others swing wildly. A product that sells between 18 and 22 a day varies little. One that sells anywhere between 5 and 60 varies a lot.

A new value is then judged against both. Selling 30 is a big surprise for the steady product and nothing special for the one that swings. That is the key idea: the same change can be normal for one item and alarming for another.

Comparing like with like

Raw history is not enough, because businesses have rhythms:

  • Day of the week. A Saturday should be compared with other Saturdays, not with a Tuesday.
  • Month end and payday. Many businesses are busier around payday and quieter just before it.
  • Festivals and holidays. Chinese New Year, Hari Raya, Deepavali, Christmas and school holidays all change what normal looks like, in different ways for different products.
  • Seasons. Umbrellas, fans, mooncakes and school supplies all have their months.

A good system compares Saturdays with Saturdays and learns the yearly patterns, so that the busy month end it has seen many times stops raising alarms. (We wrote more about this in Busy or broken?.)

Three kinds of unusual

Not every problem looks the same in the data. It helps to watch for three different shapes:

1. Sudden surprises. One day, one delivery or one transaction far from normal: a refund ten times larger than usual, a product selling out in an hour. These are the easiest to spot.

2. Slow drifts. A margin losing a little every week, a customer ordering a bit less each month, delivery times creeping up. No single day is unusual, so day-by-day checks never catch them. They need a separate check that looks at the direction over weeks.

3. Many things at once. Thirty products all dropping on the same afternoon is not thirty problems. It is usually one cause: a till offline, a card terminal down, the shop closing early. Good systems group these into one incident instead of thirty alerts.

Why alert systems get ignored

Most people who have tried alerts have switched them off. The reasons are almost always the same:

  • Too many alerts. If every small wobble triggers a message, the important ones drown. People learn to ignore all of them.
  • Fixed thresholds. "Alert me when daily sales drop below 1,000" is wrong for every outlet and every day of the week except one.
  • The same alarm every year. If Chinese New Year triggers fifty alerts every year, the system has not learned anything.
  • No context. "Anomaly detected in SKU 40213" means nothing to an owner. "Sales of Brand X umbrellas tripled today" does.
  • No memory. An alert that fires every day for the same known problem is noise. Once a problem is known, it should be followed, not repeated.

What a good system does differently

From those failures, you can write a short checklist for any anomaly detection you use or build:

  1. Normal is learned per item, including weekday, month-end and festival patterns.
  2. Surprises, slow drifts and clusters are checked separately, because they look different.
  3. Known problems become beliefs, not repeated alarms. "Supplier X has been unreliable since 7 April" is stated once and then followed until it changes.
  4. It changes its mind when things improve, and keeps a record of when and why.
  5. It speaks in plain words, and the evidence is one step away.
  6. You can tell it when it is wrong. "Not a problem" and "watch this next time" should teach it something.

You do not need a data scientist

None of this needs a data team. The maths behind good anomaly detection for a business is mostly medians, comparisons and patience. The hard part is doing it for every item, every day, without getting tired or distracted.

That is the job Cuedis does. It reads new records from your MySQL or SQL Server database every few minutes, learns what is normal for each supplier, product, customer and person, and goes over the day every night: repeated signs become beliefs with dates and evidence, slow drifts are flagged as trends, and many alarms with one cause become one incident. You get a short morning briefing, and you can mark anything as right or not a problem so it learns.

Whether you use a tool or not, the main idea is worth keeping: unusual only means something compared with what is normal for that one thing. Everything else follows from that.