BUG: default BLAS to LP64 and add Alpine preload smoke test #30299

t-kalinowski · 2025-11-25T02:35:40Z

Hello,

We recently hit a segfault on Alpine when importing NumPy in a process that had already loaded the system OpenBLAS. After some investigation, it seems Alpine’s system OpenBLAS is LP64 (32-bit BLAS ints) while the musllinux wheels default to ILP64 (64-bit BLAS ints), and mixing the two in one process can crash (seen via a reticulate workflow, rstudio/reticulate#1858 (comment)). From what I can tell, most distro OpenBLAS packages (including Alpine) and MKL defaults are LP64, while manylinux wheels use ILP64 for large-array support, so this change tries to align musl builds with the system default, but please let me know if I’m missing a better precedent.

What changed

Add a Meson option allow-ilp64-on-musl (default false).
Detect musl at configure time and force LP64 BLAS/LAPACK unless the new flag is set; ILP64 stays opt-in. glibc/macOS/Windows unchanged.
Add a manual workflow (alpine_musl.yml + Dockerfile) to smoke-test on Alpine with and without LD_PRELOAD=/usr/lib/libopenblas.so.3.

Testing

macOS host: built from source, ran numpy.test(verbose=0, extra_argv=['-q']) from /tmp (pass).
Alpine 3.20 (docker, aarch64 and x86_64) with system OpenBLAS: built from source and ran numpy.test(verbose=0, extra_argv=['-q', '-k', 'not test_to_ctypes']) both normally and with LD_PRELOAD=/usr/lib/libopenblas.so.3; all passed.

Happy to adjust anything. Thanks!

andyfaff · 2025-11-25T02:47:15Z

I can't speak to the meson changes, or the default option, but I will to the CI job.
Can you write it along the lines of https://github.com/scipy/scipy/blob/main/.github/workflows/musllinux.yml, i.e. not with a dockerfile?

mattip · 2025-11-25T06:30:13Z

This should not be needed. The official wheels use a modified OpenBLAS from scipy-openblas that mangles the exported function names, there should be no overlap with the system OpenBLAS. This is done by using the --with-scipy-openblas -- -Dallow-noblas=false -Dpkg_config_path=${PWD}/.openblas arguments to spin build.

We recently hit a segfault on Alpine when importing NumPy in a process that had already loaded the system OpenBLAS.

Please submit an issue documenting the problem, preferably with a reproducer, so we can try to figure out which symbols are not being mangled. Be sure to specify how you installed NumPy.

mattip · 2025-11-25T06:35:43Z

The reproducer in the R issue is

LD_PRELOAD=/usr/lib/libopenblas.so.3 venv/bin/python -c 'import numpy; print(numpy)'

I tried the equivalent on Ubuntu24.04 x86_64 (glibc) and it did not segfault

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 python -c 'import numpy; print(numpy.__version__)'
2.3.5

mattip · 2025-11-25T06:52:25Z

OK, I can reproduce in a docker with musllinux

$ docker run --rm -it quay.io/pypa/musllinux_1_2_x86_64 /bin/bash
...
# /opt/python/cp312-cp312/bin/python -m venv /tmp/venv312
# source /tmp/venv312/bin/activate
# pip install numpy
# apk add openblas gdb          # Edit: libopenblas -> openblas
# LD_PRELOAD=/usr/lib/libopenblas.so.3 gdb --args python -c 'import numpy; print(numpy)'
...
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x0000721ce6703f69 in scipy_cblas_sdot64_ () from \
    /tmp/venv312/lib/python3.12/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-55393b74.so
(gdb) bt
#0  0x0000721ce6703f69 in scipy_cblas_sdot64_ () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-55393b74.so
#1  0x0000721ce7ec5bbd in FLOAT_dot () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#2  0x0000721ce811ea5a in cblas_matrixproduct () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#3  0x0000721ce805ef4a in PyArray_MatrixProduct2 () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#4  0x0000721ce8050c0c in array_dot () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so

I wonder how this differs from glibc?

mattip · 2025-11-25T07:07:34Z

This seems like another case of #21643. I wonder why compilation for alpine is different.

mattip · 2025-11-25T07:56:42Z

When I run under ltrace, I see a call to sdot_k_HASWELL, which is not being mangled. On glibc the call goes to the numpy openblas, even with LD_PRELOAD.

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 ltrace -x "*sdot*" -L python -c 'import numpy; print(numpy.__version__)'
scipy_cblas_sdot64_@libscipy_openblas64_-fdde5778.so(2, 0x392344f0, 1, 0x392344f0 <unfinished ...>
sdot_k_HASWELL@libscipy_openblas64_-fdde5778.so(2, 0x392344f0, 1, 0x392344f0)                                                         = 0x787724fc7d40
<... scipy_cblas_sdot64_ resumed> )                                                                                                   = 0x787724fc7d40
2.3.5

But on alpine linux it does seem to go to the preloaded libopenblas instead.

mattip · 2025-11-25T13:39:13Z

Another difference I see, using ltrace -x "*openblas*" -L <program> is that even when <program> does not use openblas, i.e. LD_PRELOAD... ltrace -x "*openblas*" -L ls /tmp/abc,
on alpine linux there are calls to the _init function from the preloaded shared object.

tracing on alpine

# LD_PRELOAD=/usr/lib/libopenblas.so.3 ltrace -x "@*openblas*" -L ls /tmp/noexistant
_init@libopenblas.so.3(0x7ed358475860, 128, 0x7ed3564f7000, 0)                                                                      = 0
gotoblas_init@libopenblas.so.3(0x7ed3583b3648, 0, 3, 0x7ed3583b36a0 <unfinished ...>
openblas_fork_handler@libopenblas.so.3(0x7ed3583b3648, 0, 3, 0)                                                                     = 0
openblas_read_env@libopenblas.so.3(0x7ed35847600c, 0x7ed3583d0c1c, 4, 0)                                                            = 0
gotoblas_dynamic_init@libopenblas.so.3(0, 0, 11, 1 <unfinished ...>
support_avx@libopenblas.so.3(7, 8, 0x7ed3581ab510, 0x444d4163)                                                                      = 1
gotoblas_corename@libopenblas.so.3(7, 0, 0x7ed3583adc40, 0x7ed3583adc40)                                                            = 0x7ed35819bd64
openblas_warning@libopenblas.so.3(2, 0x7ffc2dd59e50, 0xffffffff, 0 <unfinished ...>
openblas_verbose@libopenblas.so.3(2, 0x7ffc2dd59e50, 0xffffffff, 0)                                                                 = 0
<... openblas_warning resumed> )                                                                                                    = 0
<... gotoblas_dynamic_init resumed> )                                                                                               = 0
blas_get_cpu_number@libopenblas.so.3(0x3fff, 0x7f00000, 0x7ef0, 0 <unfinished ...>
get_num_procs@libopenblas.so.3(0x3fff, 0x7f00000, 0x7ef0, 0)                                                                        = 24
openblas_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127)                                                   = 0
openblas_goto_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127)                                              = 0
openblas_omp_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127)                                               = 0
<... blas_get_cpu_number resumed> )                                                                                                 = 24
blas_thread_init@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127 <unfinished ...>
blas_memory_alloc@libopenblas.so.3(2, 0x7ffc2dd59dc0, 24, 127 <unfinished ...>
gotoblas_dynamic_init@libopenblas.so.3(0x7ed3583be340, 0x7ffc2dd59dc0, 16, 127)                                                     = 0
<... blas_memory_alloc resumed> )                                                                                                   = 0x7ed34e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed346000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed33e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed336000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed32e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed326000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed31e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed316000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed30e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed306000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2fe000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2f6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2ee000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2e6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2de000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2d6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2ce000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2c6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2be000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2b6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2ae000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed2a6000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed29e000000
blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77)                                                                       = 0x7ed296000000
openblas_thread_timeout@libopenblas.so.3(0, 128, 0x7ed3583c9dc8, 0x7ed358426f77)                                                    = 0
<... blas_thread_init resumed> )                                                                                                    = 0
<... gotoblas_init resumed> )                                                                                                       = 3
ls: /tmp/noexistant: No such file or directory
gotoblas_quit@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...>
blas_shutdown@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...>
blas_thread_shutdown_@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...>
blas_memory_free@libopenblas.so.3(0x7ed34e000000, 1, 16, 0)                                                                         = 0
blas_memory_free@libopenblas.so.3(0x7ed346000000, 128, 0x7ed3583b6320, 0x7ed34e000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed33e000000, 128, 0x7ed3583b6320, 0x7ed346000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed336000000, 128, 0x7ed3583b6320, 0x7ed33e000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed32e000000, 128, 0x7ed3583b6320, 0x7ed336000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed326000000, 128, 0x7ed3583b6320, 0x7ed32e000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed31e000000, 128, 0x7ed3583b6320, 0x7ed326000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed316000000, 128, 0x7ed3583b6320, 0x7ed31e000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed30e000000, 128, 0x7ed3583b6320, 0x7ed316000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed306000000, 128, 0x7ed3583b6320, 0x7ed30e000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2fe000000, 128, 0x7ed3583b6320, 0x7ed306000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2f6000000, 128, 0x7ed3583b6320, 0x7ed2fe000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2ee000000, 128, 0x7ed3583b6320, 0x7ed2f6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2e6000000, 128, 0x7ed3583b6320, 0x7ed2ee000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2de000000, 128, 0x7ed3583b6320, 0x7ed2e6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2d6000000, 128, 0x7ed3583b6320, 0x7ed2de000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2ce000000, 128, 0x7ed3583b6320, 0x7ed2d6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2c6000000, 128, 0x7ed3583b6320, 0x7ed2ce000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2be000000, 128, 0x7ed3583b6320, 0x7ed2c6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2b6000000, 128, 0x7ed3583b6320, 0x7ed2be000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2ae000000, 128, 0x7ed3583b6320, 0x7ed2b6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed2a6000000, 128, 0x7ed3583b6320, 0x7ed2ae000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed29e000000, 128, 0x7ed3583b6320, 0x7ed2a6000000)                                              = 0
blas_memory_free@libopenblas.so.3(0x7ed296000000, 128, 0x7ed3583b6320, 0x7ed29e000000)                                              = 0
<... blas_thread_shutdown_ resumed> )                                                                                               = 0
<... blas_shutdown resumed> )                                                                                                       = 0
gotoblas_dynamic_quit@libopenblas.so.3(0x7ed3583be340, 128, 512, 0x7ed3583b6320)                                                    = 0x7ed3583b62f0
<... gotoblas_quit resumed> )                                                                                                       = 0x7ed3583b62f0
_fini@libopenblas.so.3(0x7ed3560da9e0, 3, 0x7ed35819a855, 0x7ed3560daa38)                                                           = 0
+++ exited (status 1) +++

On glibc the _init is not called, but the cleanup functions are called

tracing on glibc

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 ltrace -x "@*openblas*" -L ls /tmp/noexistant
ls: cannot access '/tmp/noexistant': No such file or directory
gotoblas_quit@libopenblas.so.0(0x729250803170, 1, 1, 1 <unfinished ...>
blas_shutdown@libopenblas.so.0(0x729250803170, 1, 1, 1 <unfinished ...>
blas_thread_shutdown_@libopenblas.so.0(0x729250803170, 1, 1, 1)                                                                     = 0
<... blas_shutdown resumed> )                                                                                                       = 0
gotoblas_dynamic_quit@libopenblas.so.0(0x7292507f3180, 0, 0, 0x7292507f1160)                                                        = 0x7292507f1130
<... gotoblas_quit resumed> )                                                                                                       = 0
+++ exited (status 2) +++

mattip · 2025-11-25T17:20:37Z

Maybe connected to lazy bindings?

mattip · 2025-11-29T21:30:01Z

@rgommers pointed out (in private communication) that this has to do with the RTLD_* flags:

>>> import ctypes, sys, os
>>> openblas = ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_LOCAL)
>>> import numpy
# Success

vs.

>>> import ctypes, sys, os
>>> openblas = ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_GLOBAL)
>>> import numpy
Segmentation fault (core dumped)

On glibc both of these succeed.

Symbols like sdot_k_HASWELL are exported and visible similarly from both the glibc and musl shared object, even though we use the -fvisibility=protected flag when compiling. Even more worrisome, when using ltrace, the blas_init functions are called from the libopenblas shared object and not the libscipy_openbas one, so all the buffers and threads are set up in the wrong shared object.

rgommers · 2025-11-30T14:24:31Z

This is a bit of a mess in OpenBLAS unfortunately. There was one aborted attempt at cleaning this up in OpenMathLib/OpenBLAS#3658. The last comment there seems on point to me.

The other way to deal with this is with a linker script when building scipy-openblas. Passing -Wl,--version-script=openblas.version to GCC, with openblas.version something like:

OPENBLAS {
  global:
    /* BLAS Level 1-3 functions */
    s*_; d*_; c*_; z*_;
    
    /* CBLAS interface */
    cblas_*;
    
    /* LAPACK functions */
    *gesv_; *getrf_; *potrf_; *heev_; *syev_; *geqrf_; *gelqf_; 
    *gesdd_; *geev_; *posv_; *ppsv_; *pbsv_; *gbsv_; *sysv_; *hesv_;
    ...
    
    /* OpenBLAS public configuration API only */
    openblas_get_num_threads;
    openblas_set_num_threads;
    openblas_get_num_procs;
    openblas_get_parallel;
    openblas_get_config;
    openblas_get_corename;
    openblas_get_device_config;
    
  local:
    *;
};

With some tweaking of that and adding scipy_* and *_64_ it may be possible to fix this in a maintainable and low-effort way.

mattip · 2025-12-02T14:57:24Z

I played with versioning but always get the preloaded symbols like gotoblas_init and blas_thread_init@libopenblas.so.3. Even when I version gotoblas_init, when called from the _init of libscipyopenblas the "wrong" one is called. I wonder if statically linking to libscipyopenblas.a (for our wheels, not for conda) will change anything

rgommers · 2025-12-02T15:26:01Z

Static linking isn't worth trying I think, it isn't well-supported by OpenBLAS and my previous attempt at statically linking libopenblas.a gave me weird issues and I gave up.

mattip · 2025-12-02T16:20:27Z

yeah, I see. When the DLL is opened void gotoblas_init(void) is called to set stuff up. It would have to be called manually somehow.

mattip · 2025-12-03T17:26:51Z

Let's go back to the start. @t-kalinowski could you expand on

We recently hit a segfault on Alpine when importing NumPy in a process that had already loaded the system OpenBLAS.

How exactly did the process load the system OpenBLAS? Is there a way to change that to use the equivalent of ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_LOCAL)?

t-kalinowski · 2025-12-03T19:50:19Z

Is there a way to change that to use the equivalent of ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_LOCAL)?

I don’t think so in the Alpine + R + reticulate case. Alpine’s R is built/linked against the system OpenBLAS, so libopenblas is already loaded by the time any R code runs and reticulate embeds Python. The RTLD_LOCAL/RTLD_GLOBAL distinction only applies when a library is loaded via dlopen(), not for a link-time dependency that’s present from process start.

How exactly did the process load the system OpenBLAS?

Alpine’s R package metadata lists openblas as a dependency: pkgs.alpinelinux.org/package/edge/community/x86/R
The Alpine build log for R shows configure probing BLAS via -lopenblas (e.g. checking for dgemm_ in -lopenblas) and reporting BLAS(OpenBLAS) in the summary: build log
Alpine ships ILP64 as a separate package (openblas-ilp64); the default openblas package provides libopenblas.so.3:
- openblas
- openblas-ilp64

So there isn’t a supported knob in R to “re-open” BLAS with RTLD_LOCAL. Avoiding the conflict would require that the globally loaded system OpenBLAS cannot take precedence for BLAS symbols used by NumPy. In practice that means either building R (or the host program) without linking to the system OpenBLAS, or modifying the NumPy/OpenBLAS build so that its BLAS symbols remain isolated from any globally loaded OpenBLAS.

rgommers · 2025-12-04T09:19:37Z

RTLD_LOCAL does not exist at all on musl-based distros. I don't think there are any other solutions possible here aside from ensuring scipy_openblas doesn't leak private symbols (with either a version script or by properly hiding symbols in OpenBLAS itself).

This PR also doesn't do much to help - it may happen to work if R and NumPy both load the same version of LP64 OpenBLAS, but when they're using different versions and the public functions in one library end up calling routines from the other loaded libopenblas you can get weird bugs.

rgommers · 2025-12-04T09:27:01Z

I'm inclined to say we should aim for the proper solution (OpenMathLib/OpenBLAS#3658 (comment)), since that is just the right thing to do and in the end "just" plumbing. I will open a new OpenBLAS issue for this now.

rgommers · 2025-12-04T09:32:31Z

Done in OpenMathLib/OpenBLAS#5560

t-kalinowski added 2 commits November 24, 2025 20:55

musl: default BLAS to LP64 on musl builds

703f96b

CI: add Alpine musl preload smoke test

8b03859

t-kalinowski mentioned this pull request Nov 25, 2025

segfault from py_initialize rstudio/reticulate#1858

Open

CI: run Alpine musl smoke in musllinux container

fe22c05

mattip added the 27 - OpenBLAS label Nov 25, 2025

rgommers marked this pull request as draft November 25, 2025 21:32

charris changed the title ~~musl: default BLAS to LP64 and add Alpine preload smoke test~~ BUG: default BLAS to LP64 and add Alpine preload smoke test Nov 26, 2025

rgommers mentioned this pull request Dec 4, 2025

Ensure only public API has default symbol visibility, hide private symbols OpenMathLib/OpenBLAS#5560

Open

mattip mentioned this pull request Dec 10, 2025

patch to __attribute__ ((visibility ("default"))) on exported functions MacPython/openblas-libs#241

Open

1 task

Uh oh!

BUG: default BLAS to LP64 and add Alpine preload smoke test #30299

Are you sure you want to change the base?

BUG: default BLAS to LP64 and add Alpine preload smoke test #30299

Conversation

t-kalinowski commented Nov 25, 2025

Uh oh!

andyfaff commented Nov 25, 2025

Uh oh!

mattip commented Nov 25, 2025

Uh oh!

mattip commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Nov 25, 2025

Uh oh!

mattip commented Nov 25, 2025

Uh oh!

mattip commented Nov 25, 2025

Uh oh!

mattip commented Nov 25, 2025

Uh oh!

mattip commented Nov 29, 2025

Uh oh!

rgommers commented Nov 30, 2025

Uh oh!

mattip commented Dec 2, 2025

Uh oh!

rgommers commented Dec 2, 2025

Uh oh!

mattip commented Dec 2, 2025

Uh oh!

mattip commented Dec 3, 2025

Uh oh!

t-kalinowski commented Dec 3, 2025

Uh oh!

rgommers commented Dec 4, 2025

Uh oh!

rgommers commented Dec 4, 2025

Uh oh!

rgommers commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattip commented Nov 25, 2025 •

edited

Loading

mattip commented Nov 25, 2025 •

edited

Loading