 Pethead - 1 year ago 186
C Question

# Floating point rounding error moves reported result inside range

I am working on a function to report test results together with lower and upper limit for that specific test result. These three values will be converted with a specified formula (aX + b)/c where X is testResult/lowerLimit/upperLimit and a,b and c are floating point numbers.

If the reported test result is inside/outside the specified limits before the conversion it shall also be inside/outside the limit after the conversion in order to ensure the validity of the reported results.

I have identified two cases where an invalid test result will move inside the range after the conversion but I have yet to find a case where the test result is inside the range before the conversion and will be outside the specified limits after conversion. Can this case even occur? I don't believe so? Can it?

Below is some code that produces the two cases that I mentioned together with corrections to ensure the validity of the reported test result.

TLDR: Can the ((TRUE == insideLimitBefore) && (FALSE == insideLimitAfter)) case in the code below happen?

``````#include <stdio.h>
#include <stdint.h>

#define TRUE    (uint8_t)0x01
#define FALSE   (uint8_t)0x00

int32_t LinearMapping(const int32_t input);
void Convert(int32_t testResult, int32_t lowerLimit, int32_t upperLimit);

int main(void)
{

int32_t lowerLimit = 504;
int32_t testResult = 503;
int32_t upperLimit = 1000;

printf("INPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit);
Convert(testResult, lowerLimit, upperLimit);

lowerLimit = 500;
testResult = 504;
upperLimit = 503;

printf("INPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit);
Convert(testResult, lowerLimit, upperLimit);

return 0;
}

int32_t LinearMapping(const int32_t input)
{
float retVal;

const float a = 1.0;
const float b = 1.0;
const float c = 2.3;

retVal = a * input;
retVal += b;
retVal /= c;

return (int32_t)retVal;
}

void Convert(int32_t testResult, int32_t lowerLimit, int32_t upperLimit)
{
uint8_t insideLimitAfter;
uint8_t belowLowerLimit;
uint8_t insideLimitBefore = ((lowerLimit <= testResult) && (testResult <= upperLimit)) ? TRUE : FALSE;

if (FALSE == insideLimitBefore)
{
/* testResult is either below or above lowerLimit/upperLimit respectively */
if (testResult < lowerLimit)
{
belowLowerLimit = TRUE;
}
else /* testResult > upperLimit */
{
belowLowerLimit = FALSE;
}
}

testResult = LinearMapping(testResult);
lowerLimit = LinearMapping(lowerLimit);
upperLimit = LinearMapping(upperLimit);

insideLimitAfter = ((lowerLimit <= testResult) && (testResult <= upperLimit)) ? TRUE : FALSE;

if ((FALSE == insideLimitBefore) && (TRUE == insideLimitAfter))
{
if (TRUE == belowLowerLimit)
{
printf("OUTPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit+1, testResult, upperLimit);
}
else /* belowLowerLimit == FALSE => testResult > upperLimit */
{
printf("OUTPUT:\n\tLower limit:\t%d\t\n\tTest result:\t%d\t\n\tUpper limit:\t%d\t\n", lowerLimit, testResult, upperLimit-1);
}
}
else if ((TRUE == insideLimitBefore) && (FALSE == insideLimitAfter))
{
/* Is this case even possible? */
}
else
{
/* Do nothing */
}
}
`````` chux

to find a case where the test result is inside the range before the conversion and will be outside the specified limits after conversion. Can this case even occur?

No, given sane `a,b,c, lowerLimit, testResult, upperLimit`.

Given 3 values `lo,x,hi` that are `lo <= x <= hi` before the linear conversion in `LinearMapping()` the `lo_new <= x_new <= hi_new` will maintain the same relationship as long as the conversion is (positively) linear (no division by 0, `a`,`b`, `c` are not Not-A-Numbers). No conversion of a `float` that is out of range of `int32_t`.

The primary reason is on edge cases of `x` just inside or at the limit, `[lo...hi]` , the `LinearMapping()` may reduce the effective precision of all 3. The new `x` may now equal `lo` or `hi` and `==` favors "in range". So no change in `lo <= x <= hi`.

OP originally found examples of "invalid test result will move inside the range after the conversion" because the `x` was just outside `[lo...hi]` and the effective precision reduction now made `x` equal to either `lo` or `hi`. Since `==` favors "in range", the move from outside to inside is seen.

Note: if the `LinearMapping()` has a negative slope like -1, then `lo <= x <= hi` can easily be broken. as `1 <= 2 <= 3` --> `-1 > -2 > -3`. This makes `lowerLimit > upperLimit` which "in range" cannot be satisfied for any `x`.

For reference, OP's code simplified:

``````#include <stdio.h>
#include <stdint.h>

int LinearMapping(const int input) {
const float a = 1.0;
const float b = 1.0;
const float c = 2.3;
float retVal = a * input;
retVal += b;
retVal /= c;
return (int) retVal;
}

void Convert(int testResult, int lowerLimit, int upperLimit) {
printf("Before %d %s %d %s %d\n", lowerLimit,
lowerLimit <= testResult ? "<=" : "> ", testResult,
testResult <= upperLimit ? "<=" : "> ", upperLimit);
testResult = LinearMapping(testResult);
lowerLimit = LinearMapping(lowerLimit);
upperLimit = LinearMapping(upperLimit);
printf("After  %d %s %d %s %d\n\n", lowerLimit,
lowerLimit <= testResult ? "<=" : "> ", testResult,
testResult <= upperLimit ? "<=" : "> ", upperLimit);
}

int main(void) {
Convert(503, 504, 1000);
Convert(504, 500, 503);
return 0;
}
``````

Output

``````Before 504 >  503 <= 1000
After  219 <= 219 <= 435

Before 500 <= 504 >  503
After  217 <= 219 <= 219
``````
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download