Add new type of executable to support external JITs #142598
Description
Feature or enhancement
Proposal:
#105727 introduced the ability to have different types of executables. During the discussion of that there was mention of potentially adding support for JITs to be able to plug in as well. There's a similar disucssion of the idea over here: faster-cpython/ideas#575
This extends the existing set of executors so that PEP-523 JITs can hook into the interpreter frames and create their own "lightweight" frames which live on the stack with some minimal amount of initialization.
Currently the initialization required is probably more than idea:
in order for frames to appear complete they need to eagerly initialize instr_ptr
Some things assume f_funcobj is initialized
Because the frames are currently FRAME_OWNED_BY_THREAD various things look at stackpointer.
Presumably the next step after this PR would be to add a new frame owner which is complete but doesn't require these additional fields to be initialized.
The JIT executors are created with a fixed object that includes a callback to reify the frame, a code object, and an optional piece of state. The reifier is called in various places where the JIT needs to populate the frame - for example accessing globals, builtins, or instr_ptr. Currently the API just provides a single callback with no information on what's required but you could imagine we could have a finer grained control if it was desired - but reification of the frame is generally going to happen in the slow path so simplicity seems the best here.
The code object isn't strictly necessary but because profilers and other external introspection tools frequently want a minimal number of hops to determine a functions identity it's tracked here instead of making them head back to the function object and then read the code from there.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
#105727
faster-cpython/ideas#575