gh-138122: Make the tachyon profiler opcode-aware #142394
Conversation
Introduces LocationInfo struct sequence with end_lineno, col_offset, and end_col_offset fields. Adds opcodes parameter to RemoteUnwinder that extracts the currently executing opcode alongside its source span. Refactors linetable parsing to correctly accumulate line numbers separately from output values, fixing edge cases in computed_line.
New opcode_utils.py maps opcode numbers to names and detects specialized variants using opcode module metadata. Adds normalize_location() and extract_lineno() helpers to collector base for uniform location handling. CLI gains --opcodes flag, validated against compatible formats (gecko, flamegraph, heatmap, live).
Stores per-node opcode counts in the tree structure. Exports opcode mapping (names and deopt relationships) in JSON so the JS renderer can show instruction names and distinguish specialized variants.
Tracks opcode state transitions per thread and emits interval markers when the executing opcode changes. Markers include opcode name, line, column, and duration. Adds Opcodes category to marker schema.
Expandable panel per hot line shows instruction-level sample breakdown with opcode names and specialization percentage. Converts call graph data structures from lists to sets for O(1) deduplication.
New widget displays instruction-level stats for selected function when --opcodes is enabled. Navigation via j/k keys with scroll support. Adds per-thread opcode tracking. Updates pstats collector for new frame format.
Frame location is now a 4-tuple (lineno, end_lineno, col_offset, end_col_offset). MockFrameInfo wraps locations in LocationInfo struct. Updates assertions throughout and adds opcode_utils coverage.
# Conflicts: # Lib/test/test_profiling/test_sampling_profiler/test_integration.py
savannahostrowski
left a comment
There was a problem hiding this comment.
First pass on most of the files (mainly the JS/CSS)! I ended up playing around with this quite a bit (flamegraph, heatmap, live mode, etc) and it looks awesome. You're on a roll 💫 !
A couple of comments here and then I also opened pablogsal#110 to move a bunch of the CSS into proper classes instead of inlining all the styles, which fixes some dark mode theming bugs (figured that'd be simpler than commenting in a bunch of places)!
CSS fixes for opcode-aware Tachyon - pythongh-138122
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
savannahostrowski
left a comment
There was a problem hiding this comment.
Two small comments (and another CSS fix PR) but this looks good to me otherwise!
| python -m profiling.sampling run --opcodes --flamegraph script.py | ||
|
|
||
| This feature provides visibility into Python's bytecode execution, including | ||
| adaptive specialization optimizations. When a generic instruction like |
There was a problem hiding this comment.
Wondering if it's worth linking back to https://docs.python.org/3/whatsnew/3.11.html#whatsnew311-pep659, in case people want to learn more about the specializing adaptive interpreter?
Co-authored-by: Savannah Ostrowski <savannah@python.org>
|
🤖 New build scheduled with the buildbot fleet by @pablogsal for commit ede0f79 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F142394%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
StanFromIreland
left a comment
There was a problem hiding this comment.
Some final notes.
Also https://github.com/pablogsal/cpython/compare/tachyon-opcodes...StanFromIreland:cpython:tachyon-opcodes-jump?expand=1, but that can be done after.
Lib/test/test_external_inspection.py
Outdated
| (taskgroups.__file__, 121, "TaskGroup._aexit", None), | ||
| (taskgroups.__file__, 72, "TaskGroup.__aexit__", None), | ||
| (script_name, 26, "main", None), |
There was a problem hiding this comment.
The line numbers are subject to change, why did you set them instead of using ANY like before?
There was a problem hiding this comment.
Yeah, let me change the ones that are not profiling specific. Good catch
Lib/test/test_external_inspection.py
Outdated
| (taskgroups.__file__, 121, "TaskGroup._aexit", None), | ||
| (taskgroups.__file__, 72, "TaskGroup.__aexit__", None), | ||
| (script_name, 26, "main", None), |
Lib/test/test_external_inspection.py
Outdated
| (taskgroups.__file__, 121, "TaskGroup._aexit", None), | ||
| (taskgroups.__file__, 72, "TaskGroup.__aexit__", None), | ||
| (script_name, 26, "main", None), |
Fix shift when selecting
Merged! Thanks a lot for that! |
StanFromIreland
left a comment
There was a problem hiding this comment.
LGTM
I have a docs suggestion but feel free to disregard it.
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
When profiling Python code, knowing which source line is hot often isn't enough—a single line can contain multiple operations with very different performance characteristics. For example, result = obj.attr + expensive_call() has an attribute load, a function call, and an addition, but traditional line-level profiling treats them identically. This becomes especially important with the adaptive specialization, where understanding whether the interpreter has specialized an instruction (e.g.,
LOAD_ATTR→LOAD_ATTR_INSTANCE_VALUE) provides crucial optimization insights that were previously only visible through manual dis inspection.This PR threads bytecode information through the existing sampling profiler infrastructure by extending RemoteUnwinder to optionally extract the executing opcode and its precise source span (including column offsets) from the remote process's frame state. The column-level location data flows through to the visualization layers, where the heatmap can highlight the exact expression within a line, the gecko format emits interval markers for Firefox Profiler's timeline, and the live TUI displays real-time instruction breakdowns. The feature is opt-in via
--opcodesto avoid the overhead when not needed, and the opcode utilities handle the mapping between specialized instruction variants and their base forms so users can see both what's actually executing and what it was specialized from.screenrecording-2025-12-02_21-12-36.mp4