1,380 questions
2
votes
1
answer
179
views
Accurate computation of the inverse gamma function with the standard C math library
The inverse of the gamma function over the reals is multivalued with an infinite number of branches. This self-answered question is about the principal
inverse of the gamma function, Γ0-1(x), whose ...
- 27.1k
1
vote
0
answers
101
views
How does R round(1.535, 2) → 1.53? [duplicate]
My understanding is that R uses IEC 60559 rounding with "Round to nearest, ties to even", but it does not appear to be generating valid results. For decimal part .535 rounded to 2 places, my ...
- 1,711
1
vote
0
answers
79
views
Floating point accuracy changes based on how values are initialised [duplicate]
If I run x=0:0.1:1; x==0.2, I get equality at the third element (as expected). However, if I change it to x=-1:0.1:1; x==0.2 I no longer get any hits. Why the change? (I am familiar with some details ...
- 639
4
votes
1
answer
114
views
How to compute sign(a² - b * c) * sqrt(abs(a² - b * c)) accurately with floating point arithmetics?
Question
I was wondering if there is a numerically accurate way to compute
sign(a² - b * c) * sqrt(abs(a² - b * c))
with floating point arithmetics because it suffers from (ordered from most to least ...
- 131
0
votes
3
answers
167
views
Why does Double.CompareTo() use "==" internally?
One should not use == to compare real numbers, every C# developer knows that.
Then why the Double.CompareTo method, from the framework, is implemented this way for checking equality?
public int ...
- 540
3
votes
2
answers
246
views
Function which returns coefficients of Legendre Polynomials not accurate enough
I'm trying to write a function which returns the coefficients of the nth legendre polynomial.
// store previous polynomials to make the recursive process faster
unordered_map<int, vector<double&...
- 293
0
votes
2
answers
86
views
Matplotlib Makes Trippy Incorrect plots
I fill a numpy array with 0 or 1 depending if a computed array indexed value is more or less than a given value. This results in an array of zeros and ones. Matplotlib doesnt think so
The comparison ...
- 61
0
votes
0
answers
74
views
Reading Float data with float variable in stored procedure rounds numbers
In SQL Server Management Studio:
DECLARE @Amount FLOAT
SELECT SUM(Amount)
FROM Docs
WHERE DocID IN (104482, 104483, 104484, 104486, 104489, 104491, 104493)
-- displays in results window 178457,05
...
- 11
2
votes
1
answer
237
views
Accurate computation of the negative branch of the real-valued Lambert W function in standard C
This is a self-answered question, as encouraged on Stackoverflow for sharing knowledge. As such, it is a follow-on to my previous self-answered question regarding accurate computation of the principal ...
- 27.1k
2
votes
1
answer
474
views
Can Neumaier summation be sped up?
Neumaier summation is an improvement of Kahan summation for accurately summing arrays of floats.
import numba as nb
@nb.njit
def neumaier_sum(arr):
s = arr[0]
c = 0.0
for i in range(1, ...
- 21.5k
-1
votes
1
answer
78
views
Float calculation problem. 12.7 - 20 + 7.3 not equal zero [duplicate]
12.7 - 20 + 7.3 = -8.8817841970013e-016
I got this problem when I read Programming in Lua book, and I cannot figure it out how to get zero as the result of the calculation.
When I formatted the result ...
0
votes
1
answer
165
views
ceil(double(int)*double) or ceil(float(int*double))?
I had this C++ code:
int x;
double y;
// do sth
int z = ceil(x*y);
If x = 410, y = 1.1, then z = 452 when it should've been 451. I changed to z = ceil(float(x*y)) and it works. My friend suggests z ...
- 13
0
votes
0
answers
39
views
R: floating point errors in simple math operations [duplicate]
So of course:
1023.1 - 0.1 < 1023
[1] FALSE
but...
1024.1 - 0.1 < 1024
[1] TRUE
Is this a bug? Is it expected behaviour?
Is there some way I can avoid this, such as rounding all my numbers to ...
- 292
0
votes
1
answer
97
views
Calculating power of (1-1/x)^y for very large x,y
I am thinking about calculating $(1-1/x)^y$ for very large x,y numbers in python.
One way todo so, it to use exp(y*log1p(-1/x)), but I am not sure about its accuracy.
Are they any results on using ...
- 367
0
votes
1
answer
147
views
Rounding errors when computing intersection between two lines in python
I'm attempting to create a Python script that identifies all the unique intersection points, eliminating duplicates, among all the lines passing through a subset of points of a regular grid in 2D or ...
- 135
1
vote
2
answers
242
views
imshow plotting very large integers, but "dtype object cannot be converted to float"
I have the following code, plotting a function on a grid, where the function happens to have a very large integer value:
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter, ...
- 127
1
vote
3
answers
103
views
Without overflow and underflow, for a float number x, is x/4 always equals to x/2/2 (also for x*4 == x*2*2)?
For example, as I know, a float point x, which x/n^2 may not be equal to x/n/n, because in x/n/n, which would create an intermediate x/n first. For example:
document.write(1.23456789/49 == 1....
1
vote
0
answers
174
views
Different floating point precision in Matrix multiplication in Python and C++
I am trying to do a very simple thing
take a square shape 2d array
take its transpose
multiply the array with its transpose
I am trying to perform the above steps in C++ and Python as shown in the ...
- 5,777
3
votes
1
answer
113
views
python rounding error when sampling variable Y as a function of X with histogram?
I'm trying to sample a variable (SST) as a function of another variable (TCWV) using the function histogram, with weights set to the sample variable like this:
# average sst over bins
num, _ = np....
- 8,176
1
vote
1
answer
92
views
Exponential limits for C and Matlab
In computers, the math operation of floating numbers is actually processed using the base number and the exponential number separately, and then combine them together. We learn this in our computation ...
- 37
0
votes
2
answers
561
views
Rendering Accuracy Problem with Large Zoom in/Out Range in OpenGL Graphics
I have a simple camera based 'modern' OpenGL 3D graphics display, for rendering relatively simple objects that are constructed from collections of specified points, lines and curves (e.g. a cube, ...
- 107
-4
votes
1
answer
317
views
Multiplication of large numbers python
I decided to multiply two numbers:
1.0
17299991019999108999910899991119999
and as a result , I received in response:
1.729999101999911e+34
Code:
print(1.0*17299991019999108999910899991119999)
1....
user19345539
1
vote
1
answer
65
views
Program giving inaccurate value
The below program calculates 262 + 261 + 260 + ... + 21 + 20.
Using double type to store sum:
double sum = 0;
for (int i = 0; i < 63; i++) {
sum += pow(2.0, i);
// print("i : $i sum :...
user8234870
1
vote
0
answers
192
views
Alternatives to == in dplyr::filter, to accomodate floating point numbers
First off, let me say that I am aware that we are constrained by the limitations of computer arithmetic and floating point numbers and that 0.8 doesn't equal 0.8, sometimes. I'm curious about ways to ...
- 11
3
votes
1
answer
95
views
For any float number x,y,integer n, where const y=n-x, and 0<=x,y,n<=MAX_VALUE, is x+y==n always true?
For example:
const n=5;
const x=1.23;
const y=n-x;
document.write(x+y==n);
the code about prints true for n=5,x=1.23,y=5-1.23 (value after rounded). I think it is just a "lucky" ...
