pandas 滚窗斯皮尔曼关联

本教程将介绍 pandas 滚窗斯皮尔曼关联的处理方法,这篇教程是从别的地方看到的,然后加了一些国外程序员的疑问与解答,希望能对你有所帮助,好了,下面开始学习吧。

pandas 滚窗斯皮尔曼关联 教程 第1张

问题描述

我要使用滚动窗口计算DataFrame的两列之间的Spearman和/或Pearson相关性。

我已尝试df['corr'] = df['col1'].rolling(P).corr(df['col2'])
(P为窗口大小)

但我似乎无法定义该方法。(添加method='spearman'作为参数会产生错误:

File "main.py", line 29, in __init__
_df['corr'] = g['col1'].rolling(P).corr(g['col2'], method = corr_function)
File "~Python36libsite-packagespandascorewindow.py", line 1287, in corr
**kwargs)
File "~Python36libsite-packagespandascorewindow.py", line 1054, in corr
_get_corr, pairwise=bool(pairwise))
File "~Python36libsite-packagespandascorewindow.py", line 1866, in _flex_binary_moment
return f(X, Y)
File "~Python36libsite-packagespandascorewindow.py", line 1051, in _get_corr
return a.cov(b, **kwargs) / (a.std(**kwargs) * b.std(**kwargs))
File "~Python36libsite-packagespandascorewindow.py", line 1280, in cov
ddof=ddof, **kwargs)
File "~Python36libsite-packagespandascorewindow.py", line 1020, in cov
_get_cov, pairwise=bool(pairwise))
File "~Python36libsite-packagespandascorewindow.py", line 1866, in _flex_binary_moment
return f(X, Y)
File "~Python36libsite-packagespandascorewindow.py", line 1015, in _get_cov
center=self.center).count(**kwargs)
TypeError: count() got an unexpected keyword argument 'method'

公平地说,我并没有期望这会起作用,因为在阅读文档时,没有提到rolling.corr支持方法...

考虑到数据帧相当大(>10M行),有什么建议吗?

推荐答案

rolling.corr,所以您可以使用它。对于Spearman,使用类似以下内容:

import pandas as pd
from numpy.lib.stride_tricks import as_strided
from numpy.lib import pad
import numpy as np
def rolling_spearman(seqa, seqb, window):
 stridea = seqa.strides[0]
 ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
 strideb = seqa.strides[0]
 ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
 ar = pd.DataFrame(ssa)
 br = pd.DataFrame(ssb)
 ar = ar.rank(1)
 br = br.rank(1)
 corrs = ar.corrwith(br, 1)
 return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)

例如:

In [144]: df = pd.DataFrame(np.random.randint(0,1000,size=(10,2)), columns = list('ab'))
In [145]: df['corr'] = rolling_spearman(df.a, df.b, 4)
In [146]: df
Out[146]: 
  a b  corr
0  429  922NaN
1  618  382NaN
2  476  517NaN
3  267  536  -0.8
4  582  844  -0.4
5  254  895  -0.4
6  583  9740.4
7  687  298  -0.4
8  697  447  -0.6
9  383350.4

解释:numpy.lib.stride_tricks.as_strided是一种老套的方法,在这种情况下,它为我们提供了一个序列的视图,它看起来像一个2D数组,其中包含我们正在查看的序列的滚动窗口部分。

从那时起,一切都很简单。Spearman相关相当于将序列变换为等级,并取皮尔逊相关系数。 pandas 在DataFrame上得到了快速的行式实现来完成这项工作,这很有帮助。然后在最后,我们用NaN值填充得到的Series的开头(这样您就可以将它作为列添加到您的数据帧中或其他任何地方)。

(个人笔记:我花了很长时间试图弄清楚怎么有效地处理麻木和僵硬的问题,直到我意识到你需要的一切都已经在 pandas 身上了……!)。

为了展示此方法相对于仅在滑动窗口上循环的速度优势,我制作了一个名为srsmall.py的小文件,其中包含:

import pandas as pd
from numpy.lib.stride_tricks import as_strided
import scipy.stats
from numpy.lib import pad
import numpy as np

def rolling_spearman_slow(seqa, seqb, window):
 stridea = seqa.strides[0]
 ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
 strideb = seqa.strides[0]
 ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
 corrs = [scipy.stats.spearmanr(a, b)[0] for (a,b) in zip(ssa, ssb)]
 return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)

def rolling_spearman_quick(seqa, seqb, window):
 stridea = seqa.strides[0]
 ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
 strideb = seqa.strides[0]
 ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
 ar = pd.DataFrame(ssa)
 br = pd.DataFrame(ssb)
 ar = ar.rank(1)
 br = br.rank(1)
 corrs = ar.corrwith(br, 1)
 return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)

和比较性能:

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: from srsmall import rolling_spearman_slow as slow
In [4]: from srsmall import rolling_spearman_quick as quick
In [5]: for i in range(6):
...:  df = pd.DataFrame(np.random.randint(0,1000,size=(10*(10**i),2)), columns=list('ab'))
...:  print len(df), " rows"
...:  print "quick: ",
...:  %timeit quick(df.a, df.b, 4)
...:  print "slow: ",
...:  %timeit slow(df.a, df.b, 4)
...:10  rows
quick: 100 loops, best of 3: 3.52 ms per loop
slow: 100 loops, best of 3: 3.2 ms per loop
100  rows
quick: 100 loops, best of 3: 3.53 ms per loop
slow: 10 loops, best of 3: 42 ms per loop
1000  rows
quick: 100 loops, best of 3: 3.82 ms per loop
slow: 1 loop, best of 3: 430 ms per loop
10000  rows
quick: 100 loops, best of 3: 7.47 ms per loop
slow: 1 loop, best of 3: 4.33 s per loop
100000  rows
quick: 10 loops, best of 3: 50.2 ms per loop
slow: 1 loop, best of 3: 43.4 s per loop
1000000  rows
quick: 1 loop, best of 3: 404 ms per loop
slow:

在一百万行(在我的机器上)上,快速( pandas )版本运行不到半秒。上面没有显示,但1000万只花了8.43秒。速度较慢的仍在运行,但假设线性增长持续,则1M需要大约7分钟,10M需要一个多小时。

好了关于 pandas 滚窗斯皮尔曼关联的教程就到这里就结束了,希望趣模板源码网找到的这篇技术文章能帮助到大家,更多技术教程可以在站内搜索。