BUG: Pearson correlation outside expected range -1 to 1

[madrjor02-bh]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

values = [{'col1': 30.0, 'col2': 116.80000305175781},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': 30.100000381469727, 'col2': 116.8000030517578},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None}]

data = pd.DataFrame(values)
data.corr(method='pearson')

Issue Description

In the code snipped I'm trying to calculate the correlation between a pair of columns. However, when using pearson correlation method for this particular example, the outputted correlation is outside the -1 to 1 expected range.

Expected Behavior

The output of the pearson correlation method should be inside the -1 to 1 range.

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.9.19.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.223-211.872.amzn2.x86_64
Version : #1 SMP Mon Jul 29 19:52:29 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.5.1
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.18.1
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.9.2
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
pyarrow : 14.0.1
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : 2.0.31
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[RaghavKhemka]

The bug does seem to exist in df.corr

Reason for the Unexpected Behavior

The reason is the Welford's method which is used for calculation which doesn't perform well when the datasets has extremely small differences or precision issues.

Finding Correlation using the standard Pearson Product-Moment Correlation

import pandas as pd
import numpy as np
import math

values = [{'col1': 30.0, 'col2': 116.80000305175781},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None},
 {'col1': 30.100000381469727, 'col2': 116.8000030517578},
 {'col1': None, 'col2': None},
 {'col1': None, 'col2': None}]

data = pd.DataFrame(values)

def corr_coef(X,Y):
    x1 = np.array(X)
    y1 = np.array(Y)
    x_m=x1.mean()
    y_m=y1.mean()
    numer=0
    v1sq=0
    v2sq=0
    for i in range(len(x1)):
        xx = (x1[i]-x_m)
        yy = (y1[i]-y_m)
        numer+=xx*yy
        v1sq+=xx*xx
        v2sq+=yy*yy
    return(numer/(math.sqrt(v1sq*v2sq)))

data = data.dropna()
corr_coef(data.iloc[:,0],data.iloc[:,1])

Result:

-0.7071067811865475

We get the same -0.707.. results if np.corrcoef() is used.

Though this is a rare case to happen in reality, I believe it should be fixed. I can work of fixing the issue by changing the Welford's method to the standard method for df.corr. It might slightly impact the time for running. Not sure if it will be acceptable by the team.

[rhshadrach]

Thanks for the report!

It might slightly impact the time for running.

If it is slight, I think the numerical stability would be valued.

[KevsterAmp]

take

[ashlab11]

take

[eicchen]

I'm not sure if a fix has been implemented yet. df.corr returns 1 and -1. But I can take a look as this thread is almost a year old and has an obvious fix

[eicchen]

Looks like a bandaid fix was put in for keeping values with in -1~1 but the underlying forumla used is still Welford's method

[Alvaro-Kothe]

The sample only contains 2 values, as it's ignoring NAs. If the sample only contains 2 values, the correlation is only $-1$ or $1$, assuming all values are different.

Proof

$$ \begin{align} \bar X &= \frac{1}{2} [ X_1 + X_2] \\ X_1 - \bar X &= \frac{X_1 - X_2}{2} \\ \text{Cov}(X, Y) &= \frac{1}{2} [ (X_1 - \bar X)(Y_1 - \bar Y) + (X_2 - \bar X)(Y_2 - \bar Y)] \\ &= \frac{1}{4}(X_1 - X_2)(Y_1 - Y_2) \\ \sqrt{Var(X) Var(Y)} &= \frac{1}{4} | X_1 - X_2 | | Y_1 - Y_2| \\ \text{Cor}(X, Y) &= \frac{(X_1 - X_2)(Y_1 - Y_2)}{|X_1 - X_2||Y_1 - Y_2|} \\ &= \text{sign}(X_1 - X_2) \quad \text{sign}(Y_1 - Y_2) \end{align} $$

---

For me, the result in Pandas is correct.