Skip to content

Conversation

Copy link

Copilot AI commented Dec 21, 2025

Polars float_precision config was ignored for nested float types (Array[Float64], List[Float64]), causing inconsistent display between regular and nested columns.

Changes

Modified _format_polars_series in datatables_format.py:

  • Added detection for nested types with float inner dtype using dtype.is_nested() and dtype.inner
  • Uses Polars' native _s.get_fmt() method which is the same method Polars uses internally for string representation
  • Ensures perfect alignment with Polars' display formatting by reusing Polars' own formatting logic
  • Added escape_special_floats_in_string() helper to handle NaN/Inf encoding in formatted strings
  • Sets fmt_table_cell_list_len=1000 to avoid truncation in array/list output
  • Preserved None value handling (converts "null" string to None for proper JSON encoding)

Added test coverage:

  • Array[Float64] and List[Float64] with various precision levels
  • Non-finite value handling in nested types
  • None value preservation in lists

Example

import polars as pl
import math

df = pl.DataFrame({
    "array_col": [[1.234567890, 2.345678901, math.pi]]
}, schema={"array_col": pl.Array(pl.Float64, 3)})

# Before: precision config ignored
# [1.234567890123456, 2.345678901234567, 3.141592653589793]

with pl.Config(float_precision=2):
    show(df)  # Now: [1.23, 2.35, 3.14]

Benefits

  • Perfect alignment: ITables now displays arrays/lists exactly as Polars does
  • Simpler implementation: Uses Polars' internal method (~10 lines vs ~50 lines of custom code)
  • Future-proof: Automatically respects all Polars config options (float_precision, fmt_table_cell_list_len, etc.)
  • No divergence risk: Using Polars' own formatting logic eliminates any possibility of formatting inconsistencies
Original prompt

This section details on the original issue you should resolve

<issue_title>Apply the Polars formatting (e.g. floating point precision) to column of types array[f64] or list[f64]</issue_title>
<issue_description>> I have another request indeed. After more testing, the polars formatting is not applied to column of types array[f64] or list[f64]. Do you think it is feasible (and easy for you) to also forward the formatting to list column types ? I can give it a try also.

Originally posted by @paulduf in #447</issue_description>

<agent_instructions>@copilot, could you analyze https://github.com/pola-rs/polars and figure out where the conversion of floats and float arrays to string objects for display is done?

The objective here is to apply the exact same formatting in ITables as in Polars, in a way that is aware of the Polars config object, including but not only https://docs.pola.rs/api/python/stable/reference/api/polars.Config.set_float_precision.html.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Apply Polars formatting to array and list columns Apply Polars float_precision config to Array[f64] and List[f64] columns Dec 21, 2025
Copilot AI requested a review from mwouts December 21, 2025 17:19
Copy link
Owner

@mwouts mwouts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot I would prefer to use a Polars method to apply the formatting, can you do that?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you change that file? The change does not relate to the current PR?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in c1b63d6. These files were accidentally modified during the initial commit and are now restored to their original state.

values = x.round(precision).to_list()
return [escape_non_finite_float(v) for v in values]

# Array and List types with float inner dtype - apply formatting to nested values
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to know where in the Polars codebase this logic is applied. I would prefer to re-use the Polars implementation rather than to re-code it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored in c1b63d6 to use Polars' native _s.get_fmt() method. This method is called internally by Series.__str__() and respects all Polars config settings including float_precision and fmt_table_cell_list_len. The implementation is now much simpler (~10 lines vs ~50) and guarantees perfect alignment with Polars' display formatting.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't expect changes here either.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in c1b63d6. These changes were unintentional and have been removed.

@codecov-commenter
Copy link

codecov-commenter commented Dec 21, 2025

Codecov Report

❌ Patch coverage is 98.63014% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.99%. Comparing base (31483aa) to head (d7c9c53).

Files with missing lines Patch % Lines
src/itables/datatables_format.py 95.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #472      +/-   ##
==========================================
+ Coverage   93.84%   93.99%   +0.14%     
==========================================
  Files          46       46              
  Lines        2373     2446      +73     
==========================================
+ Hits         2227     2299      +72     
- Misses        146      147       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Author

Copilot AI commented Dec 21, 2025

@copilot I would prefer to use a Polars method to apply the formatting, can you do that?

Done in c1b63d6. Now using Polars' _s.get_fmt() method which is the same method Polars uses internally for string representation. This ensures ITables displays arrays/lists exactly as Polars does.

Copilot AI changed the title Apply Polars float_precision config to Array[f64] and List[f64] columns Apply Polars float_precision config to Array[f64] and List[f64] columns using native Polars formatting Dec 21, 2025
@github-actions
Copy link

Thank you for making this pull request.

Did you know? You can try it on Binder: Binder:lab.

Also, the version of ITables developed in this PR is available as a wheel artifact 📦 for easy installation.
Download it here, unzip it and then run pip install itables-xxx.whl in the unzipped directory.

A conda package is also available for easy installation:
Download it here, and then run conda install itables-xxx.tar.bz2 in the directory where you downloaded the file.

Copilot AI requested a review from mwouts December 21, 2025 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Apply the Polars formatting (e.g. floating point precision) to column of types array[f64] or list[f64]

3 participants