Jump to content

[TOPIC: topicViewTemplate]
[GLOBAL: userSmallPhoto]
Photo

Weird Maths (0.02 - 0.01) = 0.0099999999999997
Started by Xorcist Oct 27 2019 03:32 PM

1 reply to this topic
[TOPIC CONTROLS]
[/TOPIC CONTROLS]
[modOptionsDropdown]
[/modOptionsDropdown]
[reputationFilter]
[TOPIC: post.html]
#1

Xorcist

[GLOBAL: userInfoPane.html]
Xorcist
  • Observer

  • 18 posts
  • Corona SDK

So, when I run this code:

 

  local j = {};
  j.x = 0.02;
  j.x = j.x - 0.01;
  print(j.x);

 

I get 0.01 output, but if I run this code:

 

local function MusicDown()
  if (_cfg.Music > 0.00then
    _cfg.Music = _cfg.Music - 0.01;
  print(_cfg.Music);
end
end

 

where _cfg.Music starts as 0.50, when I get down to 0.02, the next subtraction results in 0.0099999999999997, not 0.01. Yeah I get it... floating point numbers, but how is that any different that the first example?

 

I mean I literally had to do this:

_cfg.Music = math.round((_cfg.Music * 100) - 1) * 0.01;

 

in order to have the proper value save into _cfg.Music.



[TOPIC: post.html]
#2

XeduR @Spyric

[GLOBAL: userInfoPane.html]
XeduR @Spyric
  • Contributor

  • 981 posts
  • Corona SDK

You already answered your question.

 

You may find this video by Computerphile on floating point numbers helpful https://www.youtube.com/watch?v=PZRI1IfStY0.


  • roaminggamer, sporkfin, Alan PlantPot and 1 other like this


[topic_controls]
[/topic_controls]