CSVDocumentCleaner deletes a row that says N/A, and both CSV components empty those cells

[L4XB]

Describe the bug

CSVDocumentCleaner deletes a row whose cells say N/A, and both it and CSVDocumentSplitter empty any cell holding one of pandas' default missing-value strings.

pd.read_csv treats the literal strings N/A, NA, n/a, NULL, None, NaN and nan as missing values. Neither component passes keep_default_na=False, and dtype=object does not change it — the string is gone before the dtype is applied. CSVDocumentCleaner then runs dropna(how="all") over those rows, so a row that says "not applicable" in every column is treated as an empty row and removed.

Error message

None. No error, no warning — the document simply comes back smaller.

Expected behavior

A cell that says N/A is a cell with content. N/A is how a person writes "not applicable" and NULL is what a database export writes, so those cells usually carry the answer someone would ask the document for. They should survive cleaning and splitting; only genuinely empty cells should count as empty.

Additional context

Measured on main (7476cef):

from haystack import Document
from haystack.components.preprocessors.csv_document_cleaner import CSVDocumentCleaner

# 1. a row that says N/A everywhere is deleted
csv = "Code,Status\nA1,ok\nN/A,N/A\nA3,ok\n"
CSVDocumentCleaner().run(documents=[Document(content=csv)])["documents"][0].content
# -> 'Code,Status\nA1,ok\nA3,ok\n'     the middle row is gone

# 2. a column of statuses is emptied
csv = "Code,Status,Note\nA1,N/A,keep\nA2,NULL,keep\nA3,NA,keep\n"
CSVDocumentCleaner().run(documents=[Document(content=csv)])["documents"][0].content
# -> 'Code,Status,Note\nA1,,keep\nA2,,keep\nA3,,keep\n'

CSVDocumentSplitter produces the same emptied content for case 2.

The fix is one merged default in each component, keeping the caller's override intact:

# cleaner
df = pd.read_csv(StringIO(document.content), header=None, dtype=object, keep_default_na=False)

# splitter
resolved_read_csv_kwargs = {
    "header": None, "skip_blank_lines": False, "dtype": object,
    "keep_default_na": False, **self.read_csv_kwargs,
}

A genuinely empty cell is unaffected — keep_default_na governs only the string list, not real blanks — so remove_empty_rows / remove_empty_columns keep doing their job.

I have this reproduced with tests and would normally open the PR, but the flood guard on my last PR asks me to hold off until the first two of my open ones are reviewed, so this is the report instead. Say the word and I will send it.

To Reproduce

The two snippets above, against main.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: macOS
• Haystack version: main at 7476cef
• Reproduced with the installed pandas from the dev extras

[PushpakAg]

It seems like CSVDocumentCleaner is removing rows with N/A and similar strings, treating them as empty. Would checking how keep_default_na is set in the CSV reading process help here?

[feiiiiii5]

Reproduced both symptoms on main at b717d006 (pandas 3.0.5), plus one caution about the fix as written in the issue body.

Confirmed. Real CSVDocumentSplitter(split_mode="threshold", row_split_threshold=1) on "Code,Status\nA1,ok\nN/A,N/A\nA3,ok\n\n":

• as shipped: 2 documents, the N/A row emptied and used as a split boundary — 'Code,Status\nA1,ok\n' + 'A3,ok\n'
• keep_default_na=False: 1 document, N/A,N/A preserved
• keep_default_na=False, na_values=[""]: same, with the trailing blank row still recognized

Caution: keep_default_na=False on its own drops a behaviour both components depend on. With it, an empty field parses as "" rather than NaN, so df.isnull().all(axis=1) reports zero all-null rows and dropna(how="all") removes nothing. Executed through the real CSVDocumentCleaner on input containing a blank ",\n" row:

• as shipped: 'Code,Status\nA1,ok\nA3,ok\n' — blank row removed
• keep_default_na=False: 'Code,Status\nA1,ok\n,\nA3,ok\n' — blank row now survives, remove_empty_rows silently stops working
• keep_default_na=False, na_values=[""]: 'Code,Status\nA1,ok\nA3,ok\n' — blank removed and N/A kept

So na_values=[""] has to travel with keep_default_na=False. #12772 already does exactly that for the cleaner.

For the splitter the same pair, placed before the caller's kwargs so an explicit override still wins:

resolved_read_csv_kwargs = {
    "header": None,
    "skip_blank_lines": False,
    "dtype": object,
    "keep_default_na": False,
    "na_values": [""],
    **self.read_csv_kwargs,
}

#12772 changes only csv_document_cleaner.py, so haystack/components/preprocessors/csv_document_splitter.py:105 is still open — flagging it here rather than opening a PR, since @L4XB says the reproduction and tests are already in hand.

Disclosure: AI-assisted investigation; each result above was executed against b717d006 in this session.