[SPARK-56505][SQL][TESTS][FOLLOWUP] Make the classic session in SparkSessionBinderBase an overridable seam#56963
Open
cloud-fan wants to merge 1 commit into
Conversation
Contributor
Author
f186973 to
ae7ee1e
Compare
…derBase an overridable seam Follow-up to the SessionQueryTest refactor. SparkSessionBinderBase exposes the bound session via the abstract `spark`, but its per-test afterEach hook needs the classic session (sharedState.cacheManager) and reads the trait's private _spark field directly. That hard-codes the assumption that the shared beforeAll session is the one the hooks should act on, so a suite cannot bind a per-test session (e.g. under OneInstancePerTest, where beforeAll never runs on the per-test instance and _spark stays null). Promote the classic session to an overridable `protected def classicSpark` (default = _spark) and route afterEach through it. connect.SparkSessionBinder's existing private classicSpark becomes an override of it (no behavior change). Add PerTestSessionBinderSuite demonstrating a per-test cloned session via the override. No user-facing change; test-only.
ae7ee1e to
5a21790
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a follow-up to #56190 (which added
SessionQueryTestand theSparkSessionBinder/SparkSessionBinderBasetraits).This PR makes the classic
SparkSessionused by the test-session lifecycle hooks inSparkSessionBinderBasean overridable seam, so a suite can bind its session per test instead ofonly through the single shared
beforeAllsession.SparkSessionBinderBaseexposes the bound session through the abstractspark(typed as thesession-agnostic
SparkSession, so the connect binder can narrow it to a connect session). Itsper-test
afterEachhook, however, needs the classic session (sharedState.cacheManager), andtoday reads the trait's
private var _sparkdirectly. That is fine for the common case where thesession is created in
beforeAll, but it hard-codes the assumption that the shared field is thesession the hooks should act on.
Changes:
protected def classicSpark: classic.SparkSession = _sparktoSparkSessionBinderBaseandroute the per-test hook through it (
afterEach'scacheManager.clearCache()).connect.SparkSessionBinderalready computed the same value as aprivate def classicSpark;change it to
override protected def classicSparkof the promoted accessor (no behavior change).PerTestSessionBinderSuite, a small example suite that runs each test in a fresh instance(
OneInstancePerTest) bound to a per-test cloned session, by overridingclassicSpark. This bothdemonstrates the new flexibility and guards the seam.
Why are the changes needed?
The current binder can only bind the session created in
beforeAll. A suite that wants an isolatedsession per test -- e.g. one mixing in
OneInstancePerTest, where ScalaTest runs each test in afresh suite instance that executes only
beforeEach/afterEachand neverbeforeAll-- has no wayto tell the inherited hooks which session to operate on; those hooks read the private shared field,
which is
nullin the per-test instance. MakingclassicSparkoverridable removes that limitationwhile leaving the default (shared-session) behavior unchanged.
Does this PR introduce any user-facing change?
No. This is test-only; the default behavior of
SparkSessionBinderBaseand the connect binder isunchanged.
How was this patch tested?
New
PerTestSessionBinderSuiteexercises the per-test binding path and passes. Existing binder-basedsuites compile and run unchanged (the connect binder change is a
private def->override protected defwith the same body).Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)