5,442 questions
5
votes
2
answers
279
views
Why are C++ and Python giving me different answers, when given (I think) the same precision?
I have this code in C++:
#include <iostream>
#include <iomanip>
#include <limits>
#include <boost/multiprecision/mpfr.hpp>
using namespace boost::multiprecision;
template<...
- 53
1
vote
1
answer
132
views
Avoiding duckdb OutOfRangeException when multiplying Decimals
I'm working with DuckDB and have several client-provided SQL expressions that use DECIMAL(38,10) columns (fixed precision with 10 digits after the decimal point).
For example:
SELECT S1__AMOUNT * ...
- 139
0
votes
0
answers
37
views
TypeOrm softDelete only updates deletedAt column to seconds precision. Can this be made more accurate?
Currently using typeOrm 0.2.37 with a mariaDB 11 database.
I have entities set up using @DeleteDateColumn() which are formed with datetime(6) precision.
On calling softDelete() or softRemove() on an ...
0
votes
1
answer
295
views
Inaccuracy replicating Fortran mixed-precision expression in Rust
I have the following code in my Fortran program, where both a and b are declared as REAL (KIND=8):
a = 0.12497443596150659d0
b = 1.0 + 0.00737 * a
This yields b as 1.0009210615647672
For comparison, ...
- 356
0
votes
1
answer
33
views
How to change `MPFR_MAX_PREC` either via `R` or a re-compile?
The default value (in bits, I believe), as shown via Rmpfr:
.mpfr_maxPrec()
[1] 9.223372e+18
I'd like to find a way to change that value to something larger. Here's a real-world example where it ...
- 21.6k
3
votes
2
answers
196
views
How does Oracle convert decimal values to float?
If I have a float(5) column, why does 7.89 get rounded to 7.9 but 12.79 gets rounded to 13, not 12.8?
Binary forms are as follows for 3 examples:
7.89 0111.01011001 ------ round to------\> 7.9 ...
- 31
-1
votes
1
answer
155
views
Unexpected behaviour of Trunc() function [duplicate]
I am experiencing an unexpected result from the Trunc() function, due to the lack of precision of floating-point numbers, when the float value stored is just below the positive integer value:
var
u1, ...
- 61
1
vote
0
answers
104
views
std::norm gives two different results (One for rhel8 and another for ubuntu22) [duplicate]
I have been working on a bug that involves the library from C++ and std::norm. I have been observing a test failure (inaccuracy) due to precision differences which I observe because std::norm gives ...
- 107
0
votes
0
answers
50
views
Flutter decimal / fractions madness [duplicate]
I wonder how people handle Flutters "decimal madness" in num/double?
Recently I implemented an app with currency / small-currency.
Small currency is 100 times larger than it original value.
...
- 757
5
votes
2
answers
168
views
Is the Java float to double cast lossy?
I am creating a float value using raw int bits where the mantissa has all bits set to 1. However, casting to double flips the final bit to 0. Why does this happen? From what I've been seeing online, ...
- 126
2
votes
1
answer
92
views
Unity `Transform.position` changes value unexpectedly after assignment — from -0.05 to -0.05004883
I'm encountering a weird issue in Unity where setting a Transform.position to -0.05f causes it to change immediately to -0.05004883.
Here’s a minimal repro:
Vector3 EndPoint = new Vector3(-0.05f, 0f, ...
2
votes
3
answers
117
views
Raising a float near 1 to an infinite power in Python
1.000000000000001 ** float('inf') == float('inf')
while
1.0000000000000001 ** float('inf') == 1.0
What determines the exact threshold here? It seems to be an issue of float precision, where numbers ...
- 139
0
votes
2
answers
151
views
how to approximate 2^x and log2(x) where x is an FP number [closed]
I have 2 approximations for both functions, however the log2(x) takes in an FP number, and outputs a fixed point number; and the 2^x takes in an integer, and outputs a floating point number.
the ...
- 11
0
votes
1
answer
145
views
How to find the index of the minimum cumulative sum in the face of floating-point rounding errors?
Call arr an array of float64 between −1 and 1. I want to find the index of the minimum of the cumulative sums of arr. My problem is that I cannot do this easily as the numbers in arr are of very ...
- 21.5k
0
votes
1
answer
63
views
Why can't calculated values match exactly the precalculated ones? [duplicate]
I can't understand the cause of this error. I'm trying to recreate the column by calculating it using the specific formula, but I see troubles.
I have a dataset with columns 'A', 'B', and 'C'. Column '...
1
vote
1
answer
77
views
The error of manually computed inverse-FFT increase along x_axis
I am manually compute the inverse FFT using the NumPy FFT components by the below code, to compare the rebuild signal with the original signal (sin function). As shown in the results, the error (...
- 59
5
votes
1
answer
162
views
Haskell's double behaviour
It's a known fact that numbers like Double which correspond to IEEE 754 floating-point cannot represent numbers like 0.1 accurately.
But when I played around with it in REPL, I'm confused by the ...
- 49k
0
votes
1
answer
82
views
polynomial evaluation producing non-differentiable behavior in Python
I'm trying to evaluate a polynomial of degree 61. However, around the point sqrt(2) I get some weird behavior. The graph of the polynomial becomes erratic. What could be the reason for this. The ...
- 123
8
votes
1
answer
157
views
If x*y ≠ 2ⁿ, does it follow that x * y = ((x * y) / y) * y under IEEE 754 semantics?
This is a follow-up to my previous question here. Additional restrictions highlighted in bold.
Given two nonzero, finite, double-precision (a.k.a. binary64) floating point numbers x and y, is it ...
- 8,887
1
vote
1
answer
66
views
String formatting removes floating point errors?
I am trying to understand floating point arithmetic a little better. Can somebody please help me understand what is going on in the below example?
using System;
public class FloatingPointMadness
{
...
- 11
0
votes
2
answers
193
views
Why 5.0 / 2 returns 2.5000000000000000 (scale of 16) instead of 2.5 (scale of 1)?
Fiddle
select 5.0 / 2
, pg_typeof(5.0 / 2);
pg_typeof
2.5000000000000000
numeric
Why the result is 2.5000000000000000?
When we insert 2.5 into a numeric column or select 2.5 from a numeric column ...
- 27
1
vote
2
answers
133
views
How many digits can I rely on?
Context
I am trying to improve my ability to recognize the number of correct decimal digits after performing short calculations involving floating-point numbers. In particular, I find it challenging ...
- 23
4
votes
0
answers
65
views
Python - MariaDB result of query is rounded down
When getting data out of my MariaDB database, I lose some precision --> every number is rounded down.
stored data as seen by DBeaver:
(see atached picture)
or
1737210600 0.0911
1737209700 0.103
...
- 41
1
vote
3
answers
153
views
With IEEE 754 floating point numbers, is underflow still better than overflow? [closed]
In his classic works from the 1970s, Richard Hamming advises setting up floating point calculations to be more likely to underflow than overflow.
For example, if you want to calculate 1 - 1/(exp(x)+1),...
- 9,367
0
votes
2
answers
164
views
How to cast a float into a string while keeping all its digits?
I know that type conversion may lead small errors due to the way floating numbers are stored in binary. Yet I do have the use case of casting a float into a string, yet now I have the issue that this ...
- 67.6k