user6634447 user6634447 - 1 month ago 22
C# Question

Find anomaly in a series of numbers

I have list of numbers

var nums = new List<double> {1,2,2.4,2.6,1.5,3,1.9};


In the about series I want to filter out 3 which is an anomaly in range, It should be dynamic series values may change so can't hard code any value in a condition check.

Answer

If the distribution of nums items is assumed to be normal one, we can treat a value being anomaly if it's beyond [mean - k * sigma..mean + k * sigma] range (sigma stands for the standard deviation), where k is typically 2 (95%), 3 (99.76%), sometimes even 5. If it's your case, you can implement

public static IEnumerable<T> Anomaly<T>(IEnumerable<T> source, 
                                        Func<T, double> map, 
                                        double maxSigma = 3.0) {
  if (null == source)
    throw new ArgumentNullException("source");
  else if (null == map)
    throw new ArgumentNullException("map");

  T[] data = source.ToArray();

  if (data.Length <= 1)
    yield break;

  double s = 0.0;
  double s2 = 0.0;

  foreach (var item in data) {
    double x = map(item);

    s += x;
    s2 += x * x;
  }

  double mean = s / data.Length;
  double sigma = Math.Sqrt(s2 / data.Length - (s / data.Length) * (s / data.Length));
  double leftMargin = mean - maxSigma * sigma;
  double rightMargin = mean + maxSigma * sigma;

  foreach (var item in data) {
    double x = map(item);

    if (x < leftMargin || x > rightMargin)
      yield return item;
  }
}

And so you can find anomalies for different k:

var nums = new List<double> { 1, 2, 2.4, 2.6, 1.5, 3, 1.9 };

// k = 3 (typical criterium of 3 sigma deviation - 99.76%) -  empty output
Console.Write(string.Join(", ", Anomaly(nums, x => x, 3)));

// k = 1 (unusual criterium of 1 sigma deviation - 67%) the output is "1, 3"
Console.Write(string.Join(", ", Anomaly(nums, x => x, 1)));
Comments