Pethead Pethead - 9 months ago 83
C Question

Floating point rounding error moves reported result inside range

I am working on a function to report test results together with lower and upper limit for that specific test result. These three values will be converted with a specified formula (aX + b)/c where X is testResult/lowerLimit/upperLimit and a,b and c are floating point numbers.

If the reported test result is inside/outside the specified limits before the conversion it shall also be inside/outside the limit after the conversion in order to ensure the validity of the reported results.

I have identified two cases where an invalid test result will move inside the range after the conversion but I have yet to find a case where the test result is inside the range before the conversion and will be outside the specified limits after conversion. Can this case even occur? I don't believe so? Can it?

Below is some code that produces the two cases that I mentioned together with corrections to ensure the validity of the reported test result.

TLDR: Can the ((TRUE == insideLimitBefore) && (FALSE == insideLimitAfter)) case in the code below happen?

#include <stdio.h>
#include <stdint.h>

#define TRUE (uint8_t)0x01
#define FALSE (uint8_t)0x00

int32_t LinearMapping(const int32_t input);
void Convert(int32_t testResult, int32_t lowerLimit, int32_t upperLimit);

int main(void)
{

int32_t lowerLimit = 504;
int32_t testResult = 503;
int32_t upperLimit = 1000;

printf("INPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit);
Convert(testResult, lowerLimit, upperLimit);

lowerLimit = 500;
testResult = 504;
upperLimit = 503;

printf("INPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit);
Convert(testResult, lowerLimit, upperLimit);

return 0;
}

int32_t LinearMapping(const int32_t input)
{
float retVal;

const float a = 1.0;
const float b = 1.0;
const float c = 2.3;

retVal = a * input;
retVal += b;
retVal /= c;

return (int32_t)retVal;
}

void Convert(int32_t testResult, int32_t lowerLimit, int32_t upperLimit)
{
uint8_t insideLimitAfter;
uint8_t belowLowerLimit;
uint8_t insideLimitBefore = ((lowerLimit <= testResult) && (testResult <= upperLimit)) ? TRUE : FALSE;

if (FALSE == insideLimitBefore)
{
/* testResult is either below or above lowerLimit/upperLimit respectively */
if (testResult < lowerLimit)
{
belowLowerLimit = TRUE;
}
else /* testResult > upperLimit */
{
belowLowerLimit = FALSE;
}
}

testResult = LinearMapping(testResult);
lowerLimit = LinearMapping(lowerLimit);
upperLimit = LinearMapping(upperLimit);

insideLimitAfter = ((lowerLimit <= testResult) && (testResult <= upperLimit)) ? TRUE : FALSE;

if ((FALSE == insideLimitBefore) && (TRUE == insideLimitAfter))
{
if (TRUE == belowLowerLimit)
{
printf("OUTPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit+1, testResult, upperLimit);
}
else /* belowLowerLimit == FALSE => testResult > upperLimit */
{
printf("OUTPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit-1);
}
}
else if ((TRUE == insideLimitBefore) && (FALSE == insideLimitAfter))
{
/* Is this case even possible? */
}
else
{
/* Do nothing */
}
}

Answer Source

to find a case where the test result is inside the range before the conversion and will be outside the specified limits after conversion. Can this case even occur?

No, given sane a,b,c, lowerLimit, testResult, upperLimit.

Given 3 values lo,x,hi that are lo <= x <= hi before the linear conversion in LinearMapping() the lo_new <= x_new <= hi_new will maintain the same relationship as long as the conversion is (positively) linear (no division by 0, a,b, c are not Not-A-Numbers). No conversion of a float that is out of range of int32_t.

The primary reason is on edge cases of x just inside or at the limit, [lo...hi] , the LinearMapping() may reduce the effective precision of all 3. The new x may now equal lo or hi and == favors "in range". So no change in lo <= x <= hi.

OP originally found examples of "invalid test result will move inside the range after the conversion" because the x was just outside [lo...hi] and the effective precision reduction now made x equal to either lo or hi. Since == favors "in range", the move from outside to inside is seen.

Note: if the LinearMapping() has a negative slope like -1, then lo <= x <= hi can easily be broken. as 1 <= 2 <= 3 --> -1 > -2 > -3. This makes lowerLimit > upperLimit which "in range" cannot be satisfied for any x.


For reference, OP's code simplified:

#include <stdio.h>
#include <stdint.h>

int LinearMapping(const int input) {
  const float a = 1.0;
  const float b = 1.0;
  const float c = 2.3;
  float retVal = a * input;
  retVal += b;
  retVal /= c;
  return (int) retVal;
}

void Convert(int testResult, int lowerLimit, int upperLimit) {
  printf("Before %d %s %d %s %d\n", lowerLimit,
      lowerLimit <= testResult ? "<=" : "> ", testResult,
      testResult <= upperLimit ? "<=" : "> ", upperLimit);
  testResult = LinearMapping(testResult);
  lowerLimit = LinearMapping(lowerLimit);
  upperLimit = LinearMapping(upperLimit);
  printf("After  %d %s %d %s %d\n\n", lowerLimit,
      lowerLimit <= testResult ? "<=" : "> ", testResult,
      testResult <= upperLimit ? "<=" : "> ", upperLimit);
}

int main(void) {
  Convert(503, 504, 1000);
  Convert(504, 500, 503);
  return 0;
}

Output

Before 504 >  503 <= 1000
After  219 <= 219 <= 435

Before 500 <= 504 >  503
After  217 <= 219 <= 219
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download