#### Chapter 11 Language of Descriptive Statistics

Section 11.1 Terminology and Language

# 11.1.3 Remarks on the Rounding Processes

As the following considerations and examples will show, we have to be very careful when calculating with rounded results. Let us consider the set $M=ℝ{}_{\ge 0}$ of all non-negative real numbers. On this set, let us define the multiplication

$M×M\mathrm{ }\to M\mathrm{ }\mathrm{ },\mathrm{ }\mathrm{ }\left(a,b\right)\mathrm{ }⟼\mathrm{ }a\odot b$

by the calculation rule

$a\odot b\mathrm{ }=\mathrm{ }\frac{1}{{10}^{2}}·\text{round}\left({10}^{2}·a·b\right)\mathrm{ }=\mathrm{ }\frac{1}{{10}^{2}}·⌊{10}^{2}·a·b+\frac{1}{2}⌋ ,$

i.e. the product $a\odot b$ is calculated by calculating the usual product $a·b$ first and subsequently rounding the result mathematically to two fractional digits.
The law of associativity no longer applies to the rounded multiplication. For the numbers $a=2.11$, $b=3.35$, and $c=2,61$, we have, for example,

$a\odot b\mathrm{ }=\mathrm{ }2.11\odot 3.35\mathrm{ }=\mathrm{ }7.07\mathrm{ }\text{and}\mathrm{ }\left(a\odot b\right)\odot c\mathrm{ }=\mathrm{ }7.07\odot 2.61\mathrm{ }=\mathrm{ }18.45 .$

Changing the brackets, however, results in

$b\odot c\mathrm{ }=\mathrm{ }3.35\odot 2.61\mathrm{ }=\mathrm{ }8.74\mathrm{ }\mathrm{ }\text{and}\mathrm{ }\mathrm{ }a\odot \left(b\odot c\right)\mathrm{ }=\mathrm{ }2.11\odot 8.74\mathrm{ }=\mathrm{ }18.44 .$

##### Info 11.1.11

Since calculators (and computers) always calculate using rounded results, this means that the law of associativity does not apply unrestrictedly to multiplication on calculators.

Likewise, false results can be caused by careless rounding. Let us consider the numbers $a=4.98$ and $b=1.001$. Then, we have

$a·b\mathrm{ }=\mathrm{ }4.98·1.001\mathrm{ }=\mathrm{ }4.98498\mathrm{ }\mathrm{ }\text{, i.e.}\mathrm{ }a\odot b\mathrm{ }=\mathrm{ }4.98\odot 1.001\mathrm{ }=\mathrm{ }4.98\mathrm{ }=\mathrm{ }a .$

Furthermore, we have

$a·{b}^{1000}\mathrm{ }=\mathrm{ }4.98·1.{001}^{1000}\mathrm{ }=\mathrm{ }a·\underset{1000\mathrm{ }\text{factors}}{\underset{⏟}{b·\dots ·b}}\approx 13.53028118 .$

Rounding after each multiplication to $2$ fractional digits results (due to $a\odot b=a$) in the wrong result:

$\left(\dots \left(\left(a\odot \underset{1000\mathrm{ }\text{factors}}{\underset{⏟}{b\right)\odot b\right)\dots \odot b}}\right)\mathrm{ }=\mathrm{ }a\mathrm{ }=\mathrm{ }4.98 .$

##### Info 11.1.12

In practice, this means that you must calculate at least with double precision (twice the required digits) and round the result only finally to the required digits.