Java/Kotlin: regression in 2.24.1+ — analysis time 12 min → 3-5 hours (or hang), bisected to CodeQL CLI version

[dustinlewis]

Summary

CodeQL Java/Kotlin analysis on a large multi-module Android Kotlin codebase regressed sharply between CodeQL CLI 2.24.0 and 2.24.1. The same workflow, codebase, runner, and query suite that completed in ~12-15 minutes on 2.24.0 now takes 3-5 hours when it converges at all (~37.5% success rate); 60% of runs hang and hit the GitHub-hosted runner 6-hour cap.

The regression is reproducible by version bisection — pinning the CLI back to 2.24.0 via the tools: parameter restores the historical ~12-minute behavior with 100% convergence on the same codebase and configuration.

Bisection result

Three parallel runs with tools: codeql-bundle-v2.24.0:

Run	Duration	Conclusion
1	12m 56s	Success
2	15m 29s	Success
3	13m 32s	Success

Versus the same workflow without the pin (uses current @v4 bundle, CodeQL CLI 2.25.4):

Suite	Runs	Successes	Failures	Success duration	Failure
security-and-quality	5	2 (40%)	3 (60%)	3h12m, 5h15m	6h cap
security-extended	3	1 (33%)	2 (67%)	4h52m	6h cap
Combined post-2.24.1	8	3 (37.5%)	5 (62.5%)	3-5 hours	6h cap

Only the CLI version differs between the pinned-2.24.0 set and the unpinned set. Same codebase, same workflow, same runner, same suite, sometimes within minutes of each other.

Historical context

Pre-regression, this scan ran nightly for ~3 months with 100% green:

Window	Outcome
Nov 2025 (30 runs)	All green
Dec 2025 (31 runs)	All green
Jan 2026 (31 runs)	All green
Feb 1-6, 2026 (6 runs)	All green
Feb 7, 2026 onward (~150 runs)	Nearly all fail at the 6h cap

Historical successful run durations (sampled across the green period): 12-18 minutes consistently.

Feb 7, 2026 is the day after github/codeql-action v4.32.2 shipped (Feb 5, 2026), which bundled CodeQL CLI 2.24.1.

Suspected cause

Per the 2.24.1 changelog:

The SummarizedCallable.propagatesFlow predicate has been extended with columns Provenance p and boolean isExact. The predicates SummarizedCallable.hasProvenance and SummarizedCallable.hasExactModel have been removed as a consequence. This change affects C/C++, C#, Golang, Java/Kotlin, JavaScript/TypeScript, Python, Ruby, Swift, and Rust libraries.

SummarizedCallable.propagatesFlow is in the same data-flow framework where our hung runs' evaluator logs show repeated 13.5 GB tuple-table evictions on TypeFlow<Location, TypeFlow::Input>::UnivFlow::ForAll<...> predicates. The eviction itself is normal memory management — but on 2.25.x it sometimes thrashes indefinitely; on 2.24.0 it doesn't trigger or recovers quickly.

Environment

• CodeQL CLI: 2.24.0 (green) vs. 2.25.4 (regressed) — bisected, regression starts in 2.24.1
• codeql-action: @v4 (unchanged across both)
• Build mode: manual (Gradle build, ./gradlew --no-daemon --no-build-cache clean assembleDebug)
• Runner: Linux 16-core (GHAS large runner), 64 GB RAM
• Codebase: ~22 Gradle modules, Kotlin-only source (Java only in compiled deps: Android SDK, AndroidX, Compose, Retrofit, OkHttp). Extracted database has 3,645,995 expressions in TypeFlow::FlowStepsInput::TExpr.

Symptom on regressed versions

After the build phase (~12 min consistently), CodeQL init runs (~1 min), first ~146 syntactic queries finish (~5 min). Then:

Created relation ListOfConstantsSanitizer::FlowNode.asExpr/0#dispred#... with 3,640,831 rows
Starting to evaluate predicate TypeFlowImpl::TypeFlow<Location::Location,TypeFlow::Input>::UnivFlow::ForAll<FlowScc,...>::flowJoin/3 (iteration 1)
... delta sizes grow across iterations 1-3 ...
Pausing evaluation to evict 795.21MiB ARRAYS at sequence stamp o+0
Unpausing evaluation: 13.51GiB forgotten: 13.51GiB UNREACHABLE
Starting to evaluate predicate _TypeFlowImpl::TypeFlow<...>::antijoin_rhs#1/4@i3 (iteration 3)
... silence for the remaining 5+ hours ...
Job killed at 6-hour cap

The eviction happens within ~14 seconds of TypeFlow<Location, Input> starting on every regressed run. Sometimes the evaluator continues past it and converges in 3-5 hours; sometimes it never produces another Evaluation done event.

Workaround

Pinning to 2.24.0 via tools: restores reliable nightly scanning. The cost is being frozen on the Jan 2026 QL pack (no new queries, no Kotlin 2.3+ support) until the regression is fixed.

Related

• Java CodeQL hangs for hazelcast repository #10765 (hazelcast, Oct 2022, acknowledged): similar Java taint-tracking hang pattern. Their workaround (per-query exclusion) doesn't help us because the trigger is in the shared framework, not in any specific query.
• codeql-action#2378: reports of Java/Kotlin analysis taking 8× build time on large codebases. May be the same regression.

[hvitved]

Hi 👋

Thanks a lot for the very detailed description. Will it be possible for you to share the entire evaluator log, or (even better) a CodeQL DB (I suspect the latter may not be possible, if this is closed source)?

[dustinlewis]

Hello @hvitved.

Here are the evaluator logs. I'm attaching two:

• execute-queries-20260513.194008.721.log (~11 MB) — a run on CodeQL CLI 2.25.4 that hit the hang. Truncated when the job was killed at the 6-hour cap.
• execute-queries-20260520.155805.989.log (~100 MB) — a run on CodeQL CLI 2.24.0 (pinned via tools:) that completed all 244 queries in 12m 56s. Same codebase, same workflow, same runner.

codeql-eval-logs-2.24.0-vs-2.25.4.zip

[hvitved]

@dustinlewis Thanks for sharing.

As you wrote in the original description, it appears that evaluation gets stuck during evaluation of _#TypeFlow::Input::getAnAncestor/1#dc2d3f6bMerge_TypeFlowImpl::TypeFlow<Location::Location,TypeFlow:__#antijoin_rhs#3. There have been some changes to TypeFlow.qll from 2.24.0 to 2.25.0; @aschackmull do you think that could be the cause?

@dustinlewis : To rule out differences in extracted databases, could you compare the sizes of the QL databases between the two runs (there have also been some internal changes to how Java code gets extracted between 2.24.0 and 2.25.0)? Can you also share how the code gets extracted (i.e. build-mode: none, auto-build, or manual build)?

[aschackmull]

There have been some changes to TypeFlow.qll from 2.24.0 to 2.25.0; @aschackmull do you think that could be the cause?

No, that looks unlikely. Plus, it was noted above that the regression was bisected to occur already in 2.24.1, and when I compared the compiled QL from 2.24.0 to 2.24.1 then there were no changes (caveat: I used a recent cli, so technically I haven't completely ruled out optimiser changes, but I'd be surprised if that was the cause).
However, looking at the changes in 2.24.1 I see that the Kotlin extractor had changes, so maybe that could be the cause? So comparing differences in the extracted databases between 2.24.0 and 2.24.1 sounds like a prudent thing to check.

[dustinlewis]

Build mode: manual (Gradle build, ./gradlew --no-daemon --no-build-cache clean assembleDebug).

Extracted DB comparison between the two runs we ran the eval logs from:

Relation	2.24.0 (working run)	2.25.4 (hung run)	Delta
exprs.rel	16.7 MB	26.1 MB	+55%
hasLocation.rel	11.7 MB	18.8 MB	+60%
callableEnclosingExpr.rel	5.5 MB	10.4 MB	+89%
stmts.rel	3.9 MB	7.5 MB	+92%
statementEnclosingExpr.rel	3.8 MB	6.8 MB	+78%
exprsKotlinType.rel	2.8 MB	4.2 MB	+50%
Total .rel size	73 MB	107 MB	+46%
Relation count	113	113	same

Caveat: the two runs are from different codebase commits this month that are ~1 week apart, but the original regression cliff (Feb 6 last green → Feb 7 first red) had essentially zero codebase change between those days.

[hvitved]

Thanks. Looks like it is then caused by changes in the extractor, which means we are generating a larger DB. I fear it will be quite difficult to debug without access to the DB, what do you think @aschackmull ?

[aschackmull]

I fear it will be quite difficult to debug without access to the DB, what do you think @aschackmull ?

I agree. The diverging code has been quite stable, so it's likely some very specific code pattern that triggers the problem.

[dustinlewis]

@hvitved If you're open to it, we have an MNDA process I can spin up on our side. Once that's in place, I'd be able to share the DB through a private channel.

[hvitved]

@hvitved If you're open to it, we have an MNDA process I can spin up on our side. Once that's in place, I'd be able to share the DB through a private channel.

Since you are using it on a private repo, I assume that means you are a paying customer, and then perhaps you can open a customer issue for this?

[dustinlewis]

Yes. Sounds good. I will go that route and post the case ID back here for cross-reference once it's filed.

[dustinlewis]

@hvitved I created this support ticket, but the DBs are larger than the attachment limit. Waiting for a response from support on how I can get these to you.
https://support.github.com/ticket/personal/0/4443077

[hvitved]

A fix has now been merged; thanks for your patience.