XLSX/XLS: keep repeated header names instead of Total.1, Item.1

[Cindy (cindykrafft)]

pandas renames duplicate column labels, and pandas 2 removed the option to turn that off, so a header row Item | Total | Total | Item is rendered as Item | Total | Total.1 | Item.1. The Markdown table no longer matches the spreadsheet.

When a label looks renamed, this re-reads that sheet's raw first row and uses the original labels. Eleven other sheet shapes compared give identical output.

This builds on the XLSX text-cells PR; the branch contains both commits, so please merge that one first or review just the last commit.

Tests

• test_xlsx_repeated_header_names_are_kept fails on main (| Item | Total | Total.2 | Item.1 | Total.1 | NA | Unnamed: 6 |) and passes with the change.
• With both changes: packages/markitdown 1098 passed, 12 skipped; packages/markitdown-ocr 108 passed. black clean.

Disclosure: this was found, reproduced and patched by an AI agent (Claude Code), which ran the reproduction and tests quoted above. It has not been separately reviewed by a human.

[Bicheng (Kenneth) (BichengWang)]

Reviewed 20b92625a3bcd4252a4b4a4430dc74cd9e5dd1ca (both commits), with one nonblocking coverage gap:

Repeated NA-like text headers still receive suffixes. The raw reread at _xlsx_converter.py:60–63 uses pandas' default NA parsing, so _restore_header sees missing values for literal NA, N/A, and NULL headers and retains their mangled names. For example, this workbook still produces | NA | NA.1 |:

import io
import openpyxl
from markitdown import MarkItDown, StreamInfo

workbook = openpyxl.Workbook()
workbook.active.append(["NA", "NA"])
workbook.active.append([1, 2])
stream = io.BytesIO()
workbook.save(stream)
stream.seek(0)
print(MarkItDown().convert_stream(
    stream, stream_info=StreamInfo(extension=".xlsx")
).markdown)

The same reproduction yields | N/A | N/A.1 | and | NULL | NULL.1 |. The helper docstring acknowledges this fallback, but the PR's stated behavior has this exception. Consider disabling NA parsing for the header reread while retaining a separate fallback for genuinely blank header cells, with parametrized coverage for these labels. Reading only the first row would also avoid parsing the entire sheet again.

Validation on Python 3.12.15 / pandas 3.0.6: core suite 1053 passed, 57 skipped with GITHUB_ACTIONS=1; OCR suite 108 passed; pre-commit run --all-files passed. These used direct pytest runs in an offline environment. The three additional header assertions above failed as described; header-only, numeric, date, and mixed text/numeric header probes converted successfully. I did not generate new .xls fixtures.

[Cindy (cindykrafft)]

Thanks for checking this so thoroughly. You're right; that was a gap. 2cee2ab addresses it:

• The header re-read now reads only the first row (nrows=1) with na_filter=False, so NA, N/A, NULL (and nan) header text stays as written.
• Blank header cells still keep pandas' Unnamed: n names. They come back as "" with NA parsing off, and a first row narrower than the sheet (empty trailing header cells) is padded the same way.
• New parametrized test test_xlsx_repeated_na_like_header_names_are_kept (NA, N/A, NULL, nan, blank) plus test_xlsx_repeated_headers_with_a_narrow_first_row. The four text labels fail on 20b9262 and pass now.

Validation (Python 3.12, pandas 3.0.6): core suite 1059 passed, 57 skipped with GITHUB_ACTIONS=1; OCR suite 108 passed; black 23.7.0 clean on the changed files. As before, I did not add new .xls fixtures; the XLS path uses the same helper with xlrd.

Disclosure: this fix and reply were produced by an AI agent (Claude Code), which ran the reproduction and tests quoted above.