Flip Flip - 2 months ago 8
Ruby Question

Why is Ruby adding precision/digits when calculating a mod?

I need to make some calculations with coordinates and came across
this (at least for me) weird behaviour.
Can someone explain why this happens?

$ long
=> 49.0126760222489
$ long % long.floor
=> 0.012676022248896857


I was expecting the last line to evaluate to just the digits after the period, from
long
, but instead there is an additional
6857
tacked onto the end.

Why is it adding digits?

Where is the information coming from?

Answer

I suppose we simply see float's imprecions. See i.e. 0.0126760222489 % 1.0 and 1.0126760222489 % 1.0. You'd think the result should be the same, but no - IEEE754 floats/doubles don't guarantee perfect results, and by default these are used in Ruby for storing floating point values.

It's even somewhat shown in the docs

6543.21.modulo(137)      #=> 104.21
6543.21.modulo(137.24)   #=> 92.9299999999996

you can see that the second result has a small error. Actually, on Ruby 2.3.1 I ran the first line and got:

pry(main)> 6543.21.modulo(137)
=> 104.21000000000004

It's not necessarily related to modulo, and not always visible:

[30] pry(main)> 10.0126760222489 - 0.0
=> 10.0126760222489
[31] pry(main)> 10.0126760222489 - 1.0
=> 9.0126760222489
[32] pry(main)> 10.0126760222489 - 2.0
=> 8.0126760222489
[33] pry(main)> 10.0126760222489 - 3.0
=> 7.0126760222489
[34] pry(main)> 10.0126760222489 - 4.0
=> 6.0126760222489
[35] pry(main)> 10.0126760222489 - 5.0
=> 5.0126760222489
[36] pry(main)> 10.0126760222489 - 6.0
=> 4.0126760222489
[37] pry(main)> 10.0126760222489 - 7.0
=> 3.0126760222489004

Every piece of software that uses standard floats needs to account for these small errors. If you cannot handle that for some reason, then you could use bigdecimal (should be already included in your Ruby), fixed-point, or some similar numeric library

require 'bigdecimal'

BigDecimal('6543.21').modulo(137).to_s
=> "0.10421E3"

BigDecimal('6543.21').modulo(137).to_f
=> 104.21

Keep in mind that 'bigdecimal' may be slower and may use more memory.