Skip to content

[SPARK-57850][SQL] Return the correct fractional-second scale from EXTRACT(SECOND) on TIME#56931

Open
yadavay-amzn wants to merge 2 commits into
apache:masterfrom
yadavay-amzn:SPARK-57850
Open

[SPARK-57850][SQL] Return the correct fractional-second scale from EXTRACT(SECOND) on TIME#56931
yadavay-amzn wants to merge 2 commits into
apache:masterfrom
yadavay-amzn:SPARK-57850

Conversation

@yadavay-amzn

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

EXTRACT(SECOND FROM t) / date_part('SECOND', t) on a TIME(p) value returned a hardcoded DecimalType(8, 6) regardless of the TIME precision. This changes the result type to DecimalType(2 + p, p), so the fractional-second scale matches the value's precision (e.g. TIME(0) -> decimal(2,0), TIME(6) -> decimal(8,6)).

Why are the changes needed?

The scale was always 6, so EXTRACT(SECOND) on e.g. TIME(0) reported an incorrect decimal(8,6) type with spurious fractional digits. The scale should reflect the value's declared precision.

Does this PR introduce any user-facing change?

Yes. EXTRACT(SECOND FROM <TIME(p)>) / date_part('SECOND', <TIME(p)>) now returns DecimalType(2 + p, p) instead of always DecimalType(8, 6). TIMESTAMP behavior is unchanged.

How was this patch tested?

Unit tests in TimeExpressionsSuite asserting both the result DecimalType and the value across TIME precisions 0-6; the time.sql SQLQueryTestSuite golden was updated accordingly.

Was this patch authored or co-authored using generative AI tooling?

Authored with assistance by Claude Opus 4.8.

@MaxGekk MaxGekk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 0 non-blocking, 1 nits.
Correct, minimal, well-tested fix. The declared StaticInvoke return type and the runtime Decimal construction are changed in lockstep to DecimalType(2 + p, p), so the analyzer-declared type matches execution output; the TIMESTAMP paths are correctly left unchanged (single-precision, so a fixed scale is already exact). Tests assert both the declared DecimalType and the value across TIME(0-9), including the sub-microsecond 7-9 cases, and the time.sql golden is updated consistently.

Nits: 1 minor item (general comment, in an unchanged file):

  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:480-482: the SecondWithFractionNanos Scaladoc cites SecondsOfTimeWithFraction as a fixed-scale exemplar ("The scale is fixed ... like SecondsOfTimeWithFraction does for TIME values"). This PR changes that expression to a per-precision scale (DecimalType(2 + p, p)), so the cross-reference is now inaccurate for the very symbol it names. Since this PR changes the referenced behavior, consider fixing the comment here: drop the like [[SecondsOfTimeWithFraction]] does for TIME values clause, or restate it as a contrast (TIME now uses a per-precision scale; the nanos timestamp type keeps a fixed scale because it is single-precision).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants