Service overview
Machine learning, predictive maintenance, computer vision, sensor analytics and forecasting. This service is part of that division and is scoped individually around what you need to learn, decide or publish.
Anomaly detection looks for observations that differ from normal behaviour. In engineering it appears as process monitoring, fault detection, quality screening and data validation. Because anomalies are rare, varied and often unlabelled, the problem differs from standard classification: there may be no examples of the faults you most fear, and the definition of normal itself changes with operating conditions.
The design of an anomaly detector is dominated by the cost trade-off. A sensitive detector catches more faults and produces more false alarms; operators who face many false alarms stop trusting the system. The detection threshold should therefore be linked to operational consequences, not chosen to maximise an abstract score. This service combines statistical and learning-based methods with that practical perspective.
If you plan to publish the outcome, see manuscript & publication support, for example computational & simulation manuscripts or original research article writing.
What this service covers
Problems this service can address
- Rare events with almost no labelled examples
- Normal operating modes that shift over time
- Alarm fatigue from false positives
Methods and analytical approaches
- Statistical process-control and residual-based detection
- Unsupervised and semi-supervised models
- Change-point and drift detection
- Threshold selection linked to operational cost
Applications
- Process monitoring
- Fault detection in machinery
- Quality-data screening
Relevant engineering disciplines
Scope, inputs and deliverables
Scope of work
- Data audit and definition of the prediction or decision task
- Model development with structure-aware validation and baselines
- Evaluation, error analysis and uncertainty reporting
- Documentation sufficient for others to reproduce the work
- Feasibility check on: normal-operation data and any known fault records
Typical inputs required from you
- Normal-operation data and any known fault records
- Definition of an anomaly in your context
- Tolerable false-alarm rate
Deliverables
- Detection approach and evaluation
- Threshold recommendation
- Examples of detected and missed events
Highly specialised requests are first checked for feasibility from this information. Please do not send confidential data through the public form; describe it first and agree secure transfer afterwards.
Project workflow and method
- 01Enquiry and requirement assessment
- 02Technical scoping and proposal
- 03Execution
- 04Quality review and delivery, with an agreed round of revisions
Read the full project workflow.
In depth: how the work is done
Defining normal and what counts as anomalous
We start by defining the system's normal operating envelope, including different modes such as start-up, steady operation and shutdown, and by deciding what deviations matter. Anomalies can be point anomalies, contextual anomalies that are abnormal only in a given context, or collective anomalies visible only as patterns. The definition determines the method, and getting it wrong leads to detectors that flag legitimate operating changes.
Statistical and residual-based detection
Statistical process-control charts monitor measured quantities against control limits. Multivariate methods such as Hotelling's T-squared and the squared prediction error from principal component models monitor many correlated variables at once. Residual-based approaches predict expected behaviour from a model and monitor the difference, which removes the effect of known operating changes. These methods are interpretable, and they are good baselines for more complex detectors.
Unsupervised and semi-supervised learning methods
Methods such as isolation forests, one-class support vector machines, clustering-based approaches and autoencoders learn the structure of normal data and score departures. They need data that are representative of normal operation and not contaminated by hidden faults. Performance depends on feature design and on the choice of thresholds. Where a few labelled anomalies exist, semi-supervised approaches can use them to tune thresholds and to evaluate.
Change-point and drift detection
Sudden changes in the statistical behaviour of a signal can be detected with change-point methods, and gradual shifts with drift detection, such as cumulative sum charts. These are relevant for sensor degradation, process shifts and changes in the environment. Distinguishing a real process change from sensor drift needs additional information, such as redundant sensors or physical models. We discuss how to respond when drift is detected, including retraining or recalibration.
Threshold selection and evaluation
Thresholds are chosen using the cost of false alarms and missed detections, estimated from operational information, and evaluated with data that include realistic anomalies, real or injected. Metrics such as precision, recall, false alarms per unit operating time and detection delay are reported. When few real anomalies exist, synthetic anomalies can test sensitivity but do not replace real events, and the limitation is stated. Alarm handling, such as persistence rules, can reduce nuisance alarms.
Quality control and good practice
Quality-control approach
- Data are audited for quality, leakage and drift before modelling.
- Validation respects data structure (by specimen, machine or time) and includes simple baselines.
- Performance is reported with uncertainty and failure cases, not only headline scores.
- Model limits and the conditions under which it should not be trusted are documented.
Common pitfalls this service helps avoid
- Treating all deviations as faults. Legitimate changes in operating mode will produce alarms unless modelled.
- Contaminated training data. If unrecognised faults are in the 'normal' data, the detector learns to accept them.
- Choosing thresholds by default. Thresholds not linked to operational costs can yield unusable alarm rates.
- Evaluating only with synthetic anomalies. Injected anomalies may not resemble real faults and can overstate performance.
What good reporting looks like
An anomaly detection report states the monitoring objective and the definition of normal, the data and operating modes, the methods compared, the training and validation design, the threshold rationale, performance measures including false alarm rate and detection delay, examples of detected and missed events, the limitations and recommendations for maintenance of the detector. It does not promise detection of faults not represented in the evaluation.
An illustrative scenario
Hypothetical example written for explanation. It is not a client case, a result or a testimonial.
Suppose an engineer monitors a furnace with twenty correlated temperature and flow signals. A principal-component model of normal operation produces a prediction-error statistic that rises sharply during a sensor fault and gradually during fouling. Setting the threshold to give a manageable number of alarms, the engineer finds that most alarms correspond to start-up transients, so a mode indicator is added and the model is trained per mode. The final detector produces few false alarms and detects the injected sensor fault within minutes.
Questions to settle before you begin
- What are the normal operating modes?
- Which anomalies would be costly to miss?
- How many false alarms per week can operators tolerate?
- Are there any labelled faults or past incidents for evaluation?
- How will the detector be updated as the process changes?
Key terms
- Anomaly
- An observation that departs from expected behaviour.
- Control chart
- A plot of a monitored quantity with limits for detecting abnormal variation.
- Autoencoder
- A neural network trained to reconstruct its input, used to score departures from normal data.
- False alarm
- A detection triggered when no fault exists.
- Change-point detection
- Identifying times at which statistical properties of a signal change.
Working together on AI, Machine Learning & Data Science projects
AI and data science engagements start with the decision the model is meant to support and with an audit of the data: sensors, sampling, labels, missing values, the structure of the data by specimen, machine or time, and how future data will differ from past data. We ask for a representative sample and a description of the process. Validation is designed around that structure to avoid leakage, and results are compared with simple baselines. Reports give performance with uncertainty, failure cases and the conditions under which the model should not be trusted. No performance figures are promised in advance.
Frequently asked questions
Can you promise to catch every fault?
No detector can. We quantify detection rates and false alarms on your data and state the limitations.
Can you do anomaly detection with no labelled faults?
Yes, by modelling normal behaviour. Evaluation is harder without examples, and we state the limits of what can be verified.
How do I handle changing operating conditions?
By modelling the influence of operating variables or monitoring residuals relative to expected behaviour.
Can the detector explain the anomaly?
Some methods provide contributions by variable, which indicate where the departure arises. Diagnosis usually needs domain knowledge.
Is this suitable for data quality screening?
Yes. Anomaly detection can flag suspect measurements before analysis, with the flagged data reviewed by a person.
How is this service scoped and priced?
After your enquiry we review what you need, ask for missing information and send a written scope and quotation. Price depends on complexity, deliverables and timeline. See the project workflow.