Skip to content

[SPARK-56505][SQL][TESTS][FOLLOWUP] Make the classic session in SparkSessionBinderBase an overridable seam#56963

Open
cloud-fan wants to merge 1 commit into
apache:masterfrom
cloud-fan:cloud-fan/session-binder-classic-accessor
Open

[SPARK-56505][SQL][TESTS][FOLLOWUP] Make the classic session in SparkSessionBinderBase an overridable seam#56963
cloud-fan wants to merge 1 commit into
apache:masterfrom
cloud-fan:cloud-fan/session-binder-classic-accessor

Conversation

@cloud-fan

@cloud-fan cloud-fan commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This is a follow-up to #56190 (which added SessionQueryTest and the SparkSessionBinder /
SparkSessionBinderBase traits).

This PR makes the classic SparkSession used by the test-session lifecycle hooks in
SparkSessionBinderBase an overridable seam, so a suite can bind its session per test instead of
only through the single shared beforeAll session.

SparkSessionBinderBase exposes the bound session through the abstract spark (typed as the
session-agnostic SparkSession, so the connect binder can narrow it to a connect session). Its
per-test afterEach hook, however, needs the classic session (sharedState.cacheManager), and
today reads the trait's private var _spark directly. That is fine for the common case where the
session is created in beforeAll, but it hard-codes the assumption that the shared field is the
session the hooks should act on.

Changes:

  • Add protected def classicSpark: classic.SparkSession = _spark to SparkSessionBinderBase and
    route the per-test hook through it (afterEach's cacheManager.clearCache()).
  • connect.SparkSessionBinder already computed the same value as a private def classicSpark;
    change it to override protected def classicSpark of the promoted accessor (no behavior change).
  • Add PerTestSessionBinderSuite, a small example suite that runs each test in a fresh instance
    (OneInstancePerTest) bound to a per-test cloned session, by overriding classicSpark. This both
    demonstrates the new flexibility and guards the seam.

Why are the changes needed?

The current binder can only bind the session created in beforeAll. A suite that wants an isolated
session per test -- e.g. one mixing in OneInstancePerTest, where ScalaTest runs each test in a
fresh suite instance that executes only beforeEach/afterEach and never beforeAll -- has no way
to tell the inherited hooks which session to operate on; those hooks read the private shared field,
which is null in the per-test instance. Making classicSpark overridable removes that limitation
while leaving the default (shared-session) behavior unchanged.

Does this PR introduce any user-facing change?

No. This is test-only; the default behavior of SparkSessionBinderBase and the connect binder is
unchanged.

How was this patch tested?

New PerTestSessionBinderSuite exercises the per-test binding path and passes. Existing binder-based
suites compile and run unchanged (the connect binder change is a private def -> override protected def with the same body).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

@cloud-fan cloud-fan changed the title [SPARK-56505][SQL][TESTS] Make the classic session in SparkSessionBinderBase an overridable seam [SPARK-56505][SQL][TESTS][FOLLOWUP] Make the classic session in SparkSessionBinderBase an overridable seam Jul 2, 2026
@cloud-fan

cloud-fan commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

cc @fwc @MaxGekk @zhengruifeng

@cloud-fan cloud-fan force-pushed the cloud-fan/session-binder-classic-accessor branch from f186973 to ae7ee1e Compare July 2, 2026 16:14
…derBase an overridable seam

Follow-up to the SessionQueryTest refactor. SparkSessionBinderBase exposes the
bound session via the abstract `spark`, but its per-test afterEach hook needs
the classic session (sharedState.cacheManager) and reads the trait's private
_spark field directly. That hard-codes the assumption that the shared beforeAll
session is the one the hooks should act on, so a suite cannot bind a per-test
session (e.g. under OneInstancePerTest, where beforeAll never runs on the
per-test instance and _spark stays null).

Promote the classic session to an overridable `protected def classicSpark`
(default = _spark) and route afterEach through it. connect.SparkSessionBinder's
existing private classicSpark becomes an override of it (no behavior change). Add
PerTestSessionBinderSuite demonstrating a per-test cloned session via the
override.

No user-facing change; test-only.
@cloud-fan cloud-fan force-pushed the cloud-fan/session-binder-classic-accessor branch from ae7ee1e to 5a21790 Compare July 2, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant