Skip to content

Conversation

@010Soham
Copy link
Contributor

What changed

  • Add a client-side validation error when a PyArrow null type is used with format version < 3, including the field path and a hint to use a concrete type or format-version 3.
  • Add a unit test to verify the clear error message when converting a PyArrow schema with a null field.

Why

  • Prevents misleading REST errors like “Cannot parse type string… unknown” and points directly to the offending field.

Testing

  • make lint
  • uv run pytest tests/catalog/test_base.py -k rejects_null_type -v
  • make test (timed out at ~42%)

Closes #2539

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks pa.null() has been a long standing issue. This should provide some clarity

Comment on lines +215 to +217
assert "Null type" in message
assert "n1" in message
assert "format-version=3" in message
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the full error message

'Null type (pa.null()) is not supported in Iceberg format version 2. Field: n1. Use a concrete type (string, int, boolean, etc.) or set format-version=3.'

@kevinjqliu kevinjqliu requested a review from Fokko December 24, 2025 18:48
raise ValueError(
"Null type (pa.null()) is not supported in Iceberg format version "
f"{self._format_version}. Field: {field_path}. "
"Use a concrete type (string, int, boolean, etc.) or set format-version=3."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: wdyt about future proofing with a message like: Requires v3+ or use a concrete type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error message for failed table creation due to null columns is misleading

3 participants