Sparse array cumsum leads to infinite recursion 62669 #62703
Conversation
pandas/core/arrays/sparse/array.py
Outdated
| if not self._null_fill_value: | ||
| if isinstance(self, SparseArray) and self.fill_value == 0: | ||
| # special case where we can avoid max recursion depth | ||
| return SparseArray(self.to_dense().cumsum()) |
There was a problem hiding this comment.
| if not self._null_fill_value: | |
| if isinstance(self, SparseArray) and self.fill_value == 0: | |
| # special case where we can avoid max recursion depth | |
| return SparseArray(self.to_dense().cumsum()) | |
| if not self._null_fill_value and self.fill_value != 0: |
There was a problem hiding this comment.
I see now this was wrong, but the previous implementation also did not handle the case
arr = pd.arrays.SparseArray([1.0, 0.0, np.nan, 3.0], fill_value=0.0)correctly as it would have given the result [1.0, 1.0, nan, nan] whereas I believe docstring indicates the result should be [1.0, 1.0, nan, 4.0]. We can accomplish this by:
if not self._null_fill_value:
return SparseArray(self.to_dense(), fill_value=np.nan).cumsum()Note this changes the fill_value to NaN, but that adheres to the docstring. Anytime this method has been successful in previous versions, it was always returning an NA fill value regardless of the input fill value. So I think we should stick to that logic at least for now.
There was a problem hiding this comment.
Returning to this, an explicit test for changing the fill value was added to 33e11ad where this docstring was set.
Co-authored-by: Richard Shadrach <[email protected]>
pandas/core/arrays/sparse/array.py
Outdated
| if not self._null_fill_value: | ||
| if isinstance(self, SparseArray) and self.fill_value == 0: | ||
| # special case where we can avoid max recursion depth | ||
| return SparseArray(self.to_dense().cumsum()) |
There was a problem hiding this comment.
I see now this was wrong, but the previous implementation also did not handle the case
arr = pd.arrays.SparseArray([1.0, 0.0, np.nan, 3.0], fill_value=0.0)correctly as it would have given the result [1.0, 1.0, nan, nan] whereas I believe docstring indicates the result should be [1.0, 1.0, nan, 4.0]. We can accomplish this by:
if not self._null_fill_value:
return SparseArray(self.to_dense(), fill_value=np.nan).cumsum()Note this changes the fill_value to NaN, but that adheres to the docstring. Anytime this method has been successful in previous versions, it was always returning an NA fill value regardless of the input fill value. So I think we should stick to that logic at least for now.
|
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
|
@mroeschke - this should be ready. |
|
Thanks @santhoshbethi |
4 participants
doc/source/whatsnew/v3.0.0.rstfile if fixing a bug or adding a new feature.