[SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps#56951
Open
yadavay-amzn wants to merge 2 commits into
Open
[SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps#56951yadavay-amzn wants to merge 2 commits into
yadavay-amzn wants to merge 2 commits into
Conversation
…r nanosecond-precision timestamps
MaxGekk
reviewed
Jul 2, 2026
| import org.apache.spark.sql.catalyst.util.DateTimeConstants.{NANOS_PER_DAY, NANOS_PER_MICROS} | ||
|
|
||
| override def nullIntolerant: Boolean = true | ||
| override def inputTypes: Seq[AbstractDataType] = Seq(CalendarIntervalType) |
Member
There was a problem hiding this comment.
We shouldn't support the legacy type CalendarIntervalType.
Contributor
Author
There was a problem hiding this comment.
Done - replaced the custom CalendarIntervalToNanos with CalendarIntervalToMicros typed as DayTimeIntervalType, so the nanosecond session-end now goes through the supported, timezone-aware TimestampAddInterval path. No legacy CalendarIntervalType in the nanos computation; window/session tests pass.
…IntervalType for nanosecond windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Extends
window,session_window, andwindow_timeto accept nanosecond-precision timestamp columns (TimestampNTZNanosType,TimestampLTZNanosType). Window boundaries are computed on the nanosecond scale: durations are scaled x1000 withMath.multiplyExact, timestamp <-> epoch-nanos round-trips go throughPreciseTimestampNanosConversion, andwindow_timesubtracts 1 nanosecond (vs 1 microsecond for micros input). The window start/end preserve the input's nanosecond precision.This also updates the batch
session_windowaggregation iterators (MergingSessionsIterator/UpdatingSessionsIterator) to read and compare session boundaries via the nanosecond-awareTimestampNanosValaccessors through a sharedSessionNanosHelper. These sort-merge iterators are on the shared (non-streaming) batch aggregation path, andgetLongwould misread the variable-lengthTimestampNanosValrepresentation - so the feature cannot function correctly without them.Why are the changes needed?
Part of nanosecond-precision timestamp support (SPARK-56822), building on the event-time watermark support in SPARK-57830. Previously
window/session_window/window_timerejected nanosecond-precision timestamp columns.Does this PR introduce any user-facing change?
Yes -
window,session_window, andwindow_timenow accept nanosecond-precision timestamp columns, and the resulting window bounds preserve nanosecond precision.How was this patch tested?
New
DataFrameTimeWindowingSuiteandDataFrameSessionWindowingSuitetests for NTZ and LTZ nanosecond inputs, including value-levelcheckAnswerassertions for both (window bounds, session merging, andwindow_timeat nanosecond granularity). All existing window/session tests pass.Was this patch authored or co-authored using generative AI tooling?
Authored with assistance by Claude Opus 4.8.