C5H8NNaO4 - 1 year ago 59
Javascript Question

# Compute the double value nearest preferred decimal result

Let N(x) be the value of the decimal numeral with the fewest significant digits
such that x is the

`double`
value nearest the value of the numeral.

Given
`double`
values a and b, how can we compute the
`double`
value nearest N(b)-N(a)?

E.g.:

• If a and b are the
`double`
values nearest .2 and .3,

• the desired result is the
`double`
value nearest .1,

• 0.1000000000000000055511151231257827021181583404541015625,

• rather than than the result of directly subtracting a and b,

• 0.09999999999999997779553950749686919152736663818359375.

As a baseline: In Java, the `Double.toString()` provides the N(x) function described in the question, returning its value as a numeral. One could take the strings for a and b, subtract them with the elementary-school method, and convert the resulting string to `double`.
• Is there a function D(x) that returns the number of significant digits after the decimal place for the numeral described in N(x)? If so, can we multiply a and b by a power of ten determined by D(a) and D(b), round as necessary to produce the correct integer results (for situations where they are representable as `double` values), subtract them, and divide by the power of ten?
• Can we establish criteria for which `b-a` or some simple expression can be quickly rounded to something near a decimal numeral, bypassing the code that would be necessary for harder cases? E.g., could we prove that for numbers within a certain range, `(round(10000*b)-round(10000*a))/10000` always produces the desired result?