Re: A86: calculation error

To: assembly-86@lists.ticalc.org
Subject: Re: A86: calculation error
From: Jimmy =?iso-8859-1?Q?M=E5rdell?= <yarin@acc.umu.se>
Date: Sat, 31 Oct 1998 04:43:33 +0100
Delivered-To: assembly-86-outgoing@towerguard.unix.edu.sollentuna.se
In-Reply-To: <4c20777d.363a7087@aol.com>
Reply-To: assembly-86@lists.ticalc.org

At 21:05 1998-10-30 -0500, you wrote:
>
>has anyone else noticed that when doing math calculations, sometimes the
>answer will be 4.000000000001 or 3.999999999999 instead of 4 (4 is just an
>example)? in other words, it's off by the smallest value possible.  any ideas
>why?

I don't see what this has to do with assembly, but the answer is
pretty easy. First you can try calculating (1/3+1/3+1/3)-1 on any
non-symbolic calculator, and you will get the anwer -1E-14 or some
such instead of 0. The reason is that a computer (and calculator)
uses a limited storage space for numbers (unlimited memory is
very hard to find :) ), and thus rounding errors occur. In the
case above you add .333+.333+.333 = .999, and then -1 = -0.001.

--
Real name:     Jimmy Mårdell
Email:         mailto:yarin@acc.umu.se
Homepage:      http://www.acc.umu.se/~yarin/

Icarus Productions homepage: http://icarus.ganymed.org/

References:

A86: calculation error

From: ComAsYuAre@aol.com