Python dlopen()s musl libc

Open

Details

4 participants

Josselin Poiret
Lars-Dominik Braun
Ludovic Courtès
Athena Martin

Owner: unassigned

Submitted by: Athena Martin

Severity: normal

Athena Martin wrote on 27 Jun 2023 02:42

Recipients:(address . bug-guix@gnu.org)

Message-ID:20230626204230.7b3b773a@spock.hosts.alm.website

I've had experiences now with multiple Guix packages, including gajim

(bug 60235) and now python-neovim-remote, which have an issue where

Python tries to dlopen() libc, but finds the system libc instead of

Guix's, resulting on Alpine Linux hosts in a crash with this message:

ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

There are a variety of tracebacks that lead up to this, depending on

the package in question.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSaMHYACgkQS8DcQuG8

sXNzwwf/QSs5nNKbTPCDx+ucfoiR1bYTS40WHqBfPvccKoXJ01sGgh332OpRlyMb

ggXdFOjtcTrZqnTNzYOpU5JvE6zWLWNj2Za5Uz4SXWWTX/0E+t93jyQdrDqF0NNr

MheYJ+cSHO5SclaezT60lom3FS9KttQHDaS6IUfDFnMdCtNpnUvFE+hUH55XJmSc

My/kFy1Rv8OgeB5jPj+Kf6I1Fn2n0m68dWenzyjn0GMJl4Wu8qvUlvTz1erDEARE

vKAZdFXmkUKaZEGYL9gC9uf3k1Vf2gUJu0nG/hNhfbb9c6CQbGPiMJNk8xy78DwO

TlDn8/GS/yHryjyeHrS7DbtYPQWz+Q==

=Bjwl

-----END PGP SIGNATURE-----

Ludovic Courtès wrote on 7 Jul 2023 16:00

Recipients:(name . Athena Martin)(address . secure@alm.website)(address . 64309@debbugs.gnu.org)

Message-ID:87mt07kej7.fsf@gnu.org

Hi,

Athena Martin <secure@alm.website> skribis:

Toggle quote (10 lines)> I've had experiences now with multiple Guix packages, including gajim
> (bug 60235) and now python-neovim-remote, which have an issue where
> Python tries to dlopen() libc, but finds the system libc instead of
> Guix's, resulting on Alpine Linux hosts in a crash with this message:
>
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
>
> There are a variety of tracebacks that lead up to this, depending on
> the package in question.

Could you provide a command to reproduce this?

Thanks in advance,

Ludo’.

Ludovic Courtès wrote on 7 Jul 2023 16:00

control message for bug #64309

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87lefrkeiv.fsf@gnu.org

tags 64309 + moreinfo

quit

Athena Martin wrote on 8 Jul 2023 22:16

Re: bug#64309: Python dlopen()s musl libc

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 64309@debbugs.gnu.org)

Message-ID:20230708161658.25be509c@spock.hosts.alm.website

Toggle quote (5 lines)

> > ImportError: libc.musl-x86_64.so.1: cannot open shared object file:

> > No such file or directory

> Could you provide a command to reproduce this?

On an Alpine Linux Edge host, I reproduce with:

$ guix install python-neovim-remote

$ nvr

It's possible that there's some quirk of my specific environment.

(I'm currently in the process of switching to Guix System so I may not

be able to reproduce this forever.)

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSpxDoACgkQS8DcQuG8

sXN83wf/V7A5dV7nXevxTWjWXnMU5SvzC0zdjhXExa1rVn9xMKKbai8+SSAHzAkF

EGFvq6TUc3oTNG7ErrPpsV/AyIVli1tCcLJhreBt1M75ZHi1vX0ccRveJjrk5vk3

eHhYY903rnFfY9kihCMQqMnBnNynhovr3MJc2gdM3khQgFy8VwruCf23zPQahrdT

C6GEfO76w7dQ10SorJuh1U4gsslxfOZe1bLCprZaOgqaHHO+RAxdKVEgr7o4FVMb

SjSPjWvSwuBocRE8VzkZsRdZpZZTHB0m9OuPPNufIO+joYyhu6EECG1gwxajKNix

yr4oXdPH3vVwV/vcO6tTrV2ByKEjrg==

=/syo

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 9 Jul 2023 10:35

Recipients:(address . 64309@debbugs.gnu.org)

Message-ID:87v8et8oug.fsf@jpoiret.xyz

Hi Athena,

Athena Martin via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

Toggle quote (10 lines)> On an Alpine Linux Edge host, I reproduce with:
>
> $ guix install python-neovim-remote
> $ nvr
>
> It's possible that there's some quirk of my specific environment.
>
> (I'm currently in the process of switching to Guix System so I may not
> be able to reproduce this forever.)

Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's trying
to load?

Best,
-- 
Josselin Poiret

Athena Martin wrote on 9 Jul 2023 22:22

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230709162221.39fb0e66@spock.hosts.alm.website

Toggle quote (3 lines)

> Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's

> trying to load?

I've attached the full log. The first mention of musl is on line 200 and
everything seems to happen pretty fast, here's the most relevant
portion:

     16405:	find library=libc.musl-x86_64.so.1 [0]; searching
     16405:	 search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache
     16405:	 search path=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib		(system search path)
     16405:	  trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.musl-x86_64.so.1
     16405:	
Traceback (most recent call last):
  File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
    import nvr.nvr as mod
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
    from .nvr import main
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
    import psutil
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
    from . import _pslinux as _psplatform
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

After that it starts calling fini()s and terminates.

Attachment: stderr.log

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSrFv0ACgkQS8DcQuG8

sXOxAAgAjot+pPtMO79F5afeokuVMiERpmPlw0uDSLsQ/+ivt2WaupOAm7Ag9sdg

MgeecEF1qWRVlVzLa5kxp9t+gb+AfWt4Ykh+eguTQSGnpecwVChuyW6BezkW1/iP

F5udkxgS6gaPk0DOoCsxiapSpwUKUW4DuUdf7GAXYRCJZCOZ1PVxdjMEjukEWE4v

bSF07LbF5Gy/5/8/rygEoxEgj/pVM5ARwPWHaPjUztLXxBnML22UHbN+kudZqLUH

MuDw1HxEJOE0SHLOWa5jbwAiO912tIaRP4CiVTyt0fsWekRhXbFUH2VbiLEeb77v

YI2DjpSG/NP/xJLz/gjADA8NAUHyEA==

=BNXr

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 10 Jul 2023 09:13

Recipients:(name . Athena Martin)(address . secure@alm.website)

Message-ID:87pm508cir.fsf@jpoiret.xyz

Hi Athena,

So it's not the LD_DEBUG output that hold a clue, but rather the Python

traceback.

Athena Martin <secure@alm.website> writes:

Toggle quote (12 lines)>   File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
>     import nvr.nvr as mod
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
>     from .nvr import main
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
>     import psutil
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
>     from . import _pslinux as _psplatform
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
>     from . import _psutil_linux as cext
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

The nvr package in ~/.local seems to be used instead of a Guix package.
That locally installed nvr package expects to use the host's libc, but
since the python interpreter being used has a fixed RPATH and system
search path it won't find it.

.nvr-real should definitely be using the Python code inside the store, I
wonder why that isn't being done.  Maybe our sitecustomize.py is
misbehaving?  Can you do `guix shell python-neovim-remote python --
python3` then type `import sys.path; sys.path`?

Best,
-- 
Josselin Poiret

Athena Martin wrote on 10 Jul 2023 21:47

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230710154734.2b71b56b@spock.hosts.alm.website

Toggle quote (8 lines)

> So it's not the LD_DEBUG output that hold a clue, but rather the

> Python traceback.

> The nvr package in ~/.local seems to be used instead of a Guix

> package. That locally installed nvr package expects to use the host's

> libc, but since the python interpreter being used has a fixed RPATH

> and system search path it won't find it.

Ah, I see.

Toggle quote (6 lines)

> .nvr-real should definitely be using the Python code inside the

> store, I wonder why that isn't being done. Maybe our

> sitecustomize.py is misbehaving? Can you do `guix shell

> python-neovim-remote python -- python3` then type `import sys.path;

> sys.path`?

$ guix shell python-neovim-remote python -- python3

Toggle quote (1 lines)

>>> sys.path

['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']

(It's not from the environment:)

$ guix shell python-neovim-remote python --pure -- python3

Toggle quote (1 lines)

>>> sys.path

Toggle quote (1 lines)

>>> os.environ

environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSsYFYACgkQS8DcQuG8

sXPssAf9EpHPxtQE5Z6rrhegm0NzQhRde1l10Ygdl6S1fEwv7nfi5sMc5HoZqmFJ

0bsPdV7zH28ja3HdTzwVGim4ZlwGQoU6JDzpJRzfb9mT6CYIwo/2/qHExWl233Vk

9AqPg+4Z08x2oYRDJ1l2+pPCJXs5Es58M/2cpj29Y7MIxgkUgPJGsKuwhev2Akd5

XqY4URLFWRUswFws4xFDwyr7I0wy00uNbWxZJVQU38F7rP3n4zT/+fuRNalDk2/A

53SU8E8vegFlY2zWbTxcOAhgitRvMtmFz1DEK1TqIBQIc+0s5xpktnhXN0p4uJbK

+itMeRwHilBS+GbJqXFL05z5HylJ6Q==

=kHyC

-----END PGP SIGNATURE-----

Athena Martin wrote on 10 Jul 2023 22:21

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230710162122.190051b8@spock.hosts.alm.website

Toggle quote (5 lines)

> The nvr package in ~/.local seems to be used instead of a Guix

> package. That locally installed nvr package expects to use the

> host's libc, but since the python interpreter being used has a

> fixed RPATH and system search path it won't find it.

I've just checked and temporarily removing .local/lib/python3.10 makes

Guix's nvr work, and takes the .local site-packages off sys.path.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSsaEIACgkQS8DcQuG8

sXP/uwgAvwU4yWcpnff4eX3I5rhCXgDEoOly3UMZpovPF26dqW9RpFQxgrJF5Aar

6+vGASglCZpSSvwbsDcUhNaL0oeWvXUvmiYrc70mmkR2j0TRLdI8PDD5q7tWr0CZ

rfiOL/i4GS0wso+qD0L4mEWPdDNe8Vx2ob2DDS3JS7VLKXAYo8wu/m5Bu9mVjf6n

SQ+/SVxAl9MDtFM2YzlAJnEMhjB7AfMtRFdrKxEtwURyjSZfAWm3QZqiuSUgbndP

c8CtP+YhD0gRMfRTfdQB07TyCF66Kl2zkairo0XkOh36s4DwlIZ1gN77GcedDY+I

35IOv6PxY0bqQ4IILukHTdpPAxkJMg==

=vq0j

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 11 Jul 2023 10:34

Recipients:(name . Athena Martin)(address . secure@alm.website)

Message-ID:87jzv6979d.fsf@jpoiret.xyz

Hi Athena,

Athena Martin <secure@alm.website> writes:

Toggle quote (12 lines)> $ guix shell python-neovim-remote python -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>
> (It's not from the environment:)
>
> $ guix shell python-neovim-remote python --pure -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>>>> os.environ
> environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

So, looks like the default Python behavior is to load the usercustomize
after the sitecustomize [1], which leads to exactly the behavior you're
experiencing.  I don't know what we should do here, maybe pass `-s` to
the shebang line of Python to disable loading the usercustomize?  That
would probably be a world-rebuild though.  CC'ing the Python team to see
what they think.

[1] https://docs.python.org/3/library/site.html

Best,
-- 
Josselin Poiret

Lars-Dominik Braun wrote on 11 Jul 2023 14:43

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:ZK1OeNPPIGHDZGpw@noor.fritz.box

Hi,

there’s a similar issue #63912 and a thread on the guix-devel

mailinglist at

https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00066.html

Toggle quote (7 lines)

> So, looks like the default Python behavior is to load the usercustomize

> after the sitecustomize [1], which leads to exactly the behavior you're

> experiencing. I don't know what we should do here, maybe pass `-s` to

> the shebang line of Python to disable loading the usercustomize? That

> would probably be a world-rebuild though. CC'ing the Python team to see

> what they think.

I think the problem is bigger than usercustomize. Any custom PYTHONPATH
also slips through and causes this issue, as well as any custom
GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
`guix shell` invokations with different versions of a library for
an example where this could go wrong).

Guix-managed Python packages (libraries nor applications) should
generally not pick up dependencies from random paths – only those
from their package description, so we can keep Guix’ promise of being
self-contained.

I have experimented with customizing Python’s importing mechanism
through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
variable to every Python package’s __init__.py file and modifying the
module loader’s search path accordingly if such a variable exists. It
would provide exactly that guarantee, but it’s just a PoC at this
point – see attached file.

Apart from that I don’t see a good short-term solution right now. It’s
just how Python works.

Cheers,
Lars

# Credits to
# https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/
# for the very good explanation of how Python’s import statement works.

import sys, os
from importlib.abc import MetaPathFinder
from importlib.machinery import SourceFileLoader, ModuleSpec, PathFinder

class GuixPythonFinder (MetaPathFinder):
    def find_spec (self, fullname, path, target=None):
        # Short-circuit for non-top-level imports, which already have a path.
        if path:
            return None

        attrname = '__guix_pythonpath__'
        searchPath = None
        # Search for our caller.
        frame = sys._getframe ()
        while frame:
            # If he has a search path, use it. This is mainly for executable
            # scripts with `__name__ == '__main__'`.
            searchPath = frame.f_globals.get (attrname, None)
            if not searchPath:
                # Otherwise check the top-level package for search paths
                # declared in __init__.py
                package = frame.f_globals.get ('__package__')
                if package:
                    module = sys.modules.get (package)
                    if module:
                        searchPath = getattr (module, attrname, None)
            if searchPath is not None:
                break
            frame = frame.f_back

        # If we have a caller…
        if searchPath is not None:
            return PathFinder.find_spec (fullname, searchPath, target=target)
        else:
            # Otherwise we’re not responsible for this module.
            return None

sys.meta_path.insert (0, GuixPythonFinder ())

Josselin Poiret wrote on 12 Jul 2023 10:50

Recipients:(name . Lars-Dominik Braun)(address . lars@6xq.net)

Message-ID:87h6q98qe7.fsf@jpoiret.xyz

Hi Lars,

Lars-Dominik Braun <lars@6xq.net> writes:

Toggle quote (18 lines)> I think the problem is bigger than usercustomize. Any custom PYTHONPATH
> also slips through and causes this issue, as well as any custom
> GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
> `guix shell` invokations with different versions of a library for
> an example where this could go wrong).
>
> Guix-managed Python packages (libraries nor applications) should
> generally not pick up dependencies from random paths – only those
> from their package description, so we can keep Guix’ promise of being
> self-contained.
>
> I have experimented with customizing Python’s importing mechanism
> through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
> variable to every Python package’s __init__.py file and modifying the
> module loader’s search path accordingly if such a variable exists. It
> would provide exactly that guarantee, but it’s just a PoC at this
> point – see attached file.

Woah, looks like a neat solution. Do you think it would scale for all

our Python packages without manual intervention? If so, this would

definitely be the way forward.

Toggle quote (3 lines)

> Apart from that I don’t see a good short-term solution right now. It’s

> just how Python works.

I mostly agree with you, but for this rather common case of having also
a usercustomize it would be nice to circumvent it.  In general, I don't
think we ever want a Guix-produced Python script to load the
usercustomize, hence my suggestion.  The other case of PYTHONPATH is
also annoying but can be tamed by modifying the env variable
temporarily.

Best,
-- 
Josselin Poiret

Lars-Dominik Braun wrote on 16 Jul 2023 10:35

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:ZLOr5K4bTkZceqyS@noor.fritz.box

Hi Josselin,

Toggle quote (4 lines)

> Woah, looks like a neat solution. Do you think it would scale for all

> our Python packages without manual intervention? If so, this would

> definitely be the way forward.

I hope so, but haven’t tried it yet.

Toggle quote (7 lines)

> I mostly agree with you, but for this rather common case of having also

> a usercustomize it would be nice to circumvent it. In general, I don't

> think we ever want a Guix-produced Python script to load the

> usercustomize, hence my suggestion. The other case of PYTHONPATH is

> also annoying but can be tamed by modifying the env variable

> temporarily.

True, PYTHONPATH can be unset more easily than moving the user site

dir. I’ll have a look at #64573, which should work around this issue.

Cheers,

Lars

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 64309@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date