On the Accuracy Calculation of Floating Point Numbers in Computer

1. Problems

• In the computer, we may occasionally encounter some decimal calculations, and the results obtained are not the results we want, such as:

  console.log(0.1 + 0.2);     // 0.30000000000000004
  console.log(0.1 + 0.7);		// 0.7999999999999999
  console.log(0.1 * 3);		// 0.30000000000000004
  

• In the above code, 0.1 is added to 0.2, and the result should be 0.3, but here there is an extra 0.000000000000000004.
• This is a problem caused by the way floating-point numbers are calculated in computers.

2. Principle

• We know that the underlying principle of the computer is the switch of the circuit, and the conversion into data is binary 0 and 1
• All numerical operations in the computer must be converted into binary before operation can be performed, and there will be some minor problems when converting floating-point numbers into binary decimals

1. When converting a decimal integer to a binary integer, divide the integer to be converted by 2 continuously until the number is 1, as the first digit of binary, and then connect the remainder from bottom to top to get a binary number. like:

   36/2 = 18 ---- 0 (余数)
   18/2 = 9  ---- 0 (余数)
   9/2  = 4  ---- 1 (余数)
   4/2  = 2  ---- 0 (余数)
   2/2  = 1  ---- 0 (余数)
   1
   36转成二进制为100100
   
   53/2 = 26 ---- 1
   26/2 = 13 ---- 0
   13/2 = 6  ---- 1
   6/2  = 3  ---- 0
   3/2  = 1  ---- 1
   1
   53转成二进制为110101
   

2. When converting a decimal number into binary, the decimal part is multiplied by 2 until the decimal part is all 0, and then the integer bits of each multiplication result are connected from top to bottom. You can get binary decimals. like:

   0.125 * 2 = 0.25 ---- 0 (取0.25的整数部分)
   0.25 * 2  = 0.5  ---- 0 (取0.5的整数部分)
   0.5 * 2   = 1.0  ---- 1 (取1.0的整数部分)
   0.125转化为二进制为0.001
   
   然而并不是所有的十进制小数都能转化为二进制小数
   
   如十进制的0.1：
   0.1 * 2 = 0.2 ---- 0
   0.2 * 2 = 0.4 ---- 0
   0.4 * 2 = 0.8 ---- 0
   0.8 * 2 = 1.6 ---- 1
   0.6 * 2 = 1.2 ---- 1
   0.2 * 2 = 0.4 ---- 0
   0.4 * 2 = 0.8 ---- 0
   0.8 * 2 = 1.6 ---- 1
   将十进制的0.1转化为二进制是0.00011001100110011...是个无限循环的二进制小数
   

• However, the computer has a limited number of digits for data storage. When an infinite decimal appears, the computer will process the part exceeding the maximum number of digits to the nearest value (binary state)
• Therefore, when we perform floating-point calculations, the data actually calculated by the computer is no longer our original data. After the calculation is completed, if the binary is converted back to decimal, precision problems will inevitably occur, and ideal data will not be obtained.

Three, solve

1. You can selectively ignore some operations that do not require high precision (not desirable)

2. You can convert the decimal number into an integer before performing the operation, because the computer's operation on the integer is accurate

3. When the decimal place of the ideal result is known, use x.toFixed(n)active rounding to retain n decimal places to achieve the ideal value

   var num = (0.1 * 0.3).toFixed(2) - 0;
   console.log(num);		// 0.3
   

4. Use the function directly x.toPrecision(1), specifying bits of precision

   var num = (0.1 * 0.3).toPrecision(1) - 0;
   console.log(num);		// 0.03
   

---

The above, inadequacies, please leave a message to point out

Further reading: The past and present of js floating-point precision problems?