core-updates: python-minimal@2.7.13 fails due to missing zlib

python-minimal@2.7.13 is failing to build on core-updates:

I saw this near the beginning of the install phase:

------

configure: error: cannot run /bin/sh

/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Modules/_ctypes/libffi/config.sub

Failed to configure _ctypes module

Python build finished, but the necessary bits to build these modules

were not found:

_bsddb _curses _curses_panel

_sqlite3 _ssl _tkinter

bsddb185 dbm dl

gdbm imageop readline

sunaudiodev zlib

To find the necessary bits, look in setup.py in detect_modules() for the

module's name.

------

And at the end, it fails:

------

if test "xinstall" != "xno" ; then \

case install in \

upgrade) ensurepip="--upgrade" ;; \

install|*) ensurepip="" ;; \

esac; \

LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \

$ensurepip --root=/ ; \

fi

Traceback (most recent call last):

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main

"__main__", fname, loader, pkg_name)

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code

exec code in run_globals

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>

ensurepip._main()

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main

default_pip=args.default_pip,

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap

_run_pip(args + [p[0] for p in _PROJECTS], additional_paths)

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip

import pip

zipimport.ZipImportError: can't decompress data; zlib not available

make: *** [Makefile:927: install] Error 1

phase `install' failed after 29.1 seconds

------

Hmmm... Python 2.7.13 seems to bundle libffi-3.1. Do we want that?

Also, I checked libffi/configure: If CONFIG_SHELL was set, it would have picked it up.

Also, why does Python 2.7.13 try to configure things in the *install* phase ?

On Fri, Mar 17, 2017 at 03:08:51PM +0100, Danny Milosavljevic wrote:

The python-minimal package *should* use that libffi. I'm not sure why

it's failing now.

Do you mean if CONFIG_SHELL is not set, so the build process can't find

libffi?

There is also the question about zlib and the bundled pip: why is it

failing now?

I haven't yet found the changes between 2.7.12 and 2.7.13 that are

causing these issues.

Hi Leo,

Yeah, sorry. I checked the package definition now and it explicitly disables the system libffi.

Yes, a workaround for the first problem is:

Setting the environment variables

CPPFLAGS="-I/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/include"

LDFLAGS="-L/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/lib"

before running configure makes it work.

(I've also printed inc_dirs in setup.py without that - and zlib didn't appear)

So I guess the general question is how does Python set up inc_dirs ?

And is it normal that C_INCLUDE_PATH and LIBRARY_PATH doesn't contain zlib ? Apparently, since guix-master python2-minimal 2.7.12 doesn't contain it either - and that one works...

So where would it get the other entries?

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:

I think this is the better solution to the libffi / ctypes issue. It

also fixes python-minimal@3 to use the bundled libffi. We added libffi

to python-minimal@3's inputs in

d0b73960db051baa468fc1a0fbab21fa9f89f885 to work around a similar issue.

In Python 3.7, the bundled libffi will be removed, and we will have to

build our minimal Python-3 variant with an external libffi package:

Here's a patch that fixes this problem for python-minimal@2. I don't

want to change python-minimal@3 this late in the core-updates cycle; we

can do it in a python-updates branch after core-updates.

From 9d7eef243fca8e53fb89bb17a999f0678d191caf Mon Sep 17 00:00:00 2001

libffi.

(python2-minimal)[inputs]: Add zlib.

---

gnu/packages/python.scm | 17 +++++++++++++++--

1 file changed, 15 insertions(+), 2 deletions(-)

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:

Ideally, the minimal Python variants could use the bundled zlib found at

'Modules/zlib'. We sort of gave up on this with python-minimal@3 the

last time we had this problem, but maybe we can solve it now :)

On Sun, Mar 19, 2017 at 03:26:36PM -0400, Leo Famulari wrote:

Here are some build logs showing differences between master and

core-updates.

For 2.7.12, on the master branch, there is this:

------

if test "xinstall" != "xno" ; then \

case install in \

upgrade) ensurepip="--upgrade" ;; \

install|*) ensurepip="" ;; \

esac; \

LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.12.drv-0/Python-2.7.12 ./python -E -m ensurepip \

$ensurepip --root=/ ; \

fi

Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS

phase `install' succeeded after 31.5 seconds

------

I guess we didn't notice the lack of TLS support because the bundled pip

is never able to reach the internet when used to build packages.

For 2.7.13 without zlib, on core-updates, it does this:

------

if test "xinstall" != "xno" ; then \

case install in \

upgrade) ensurepip="--upgrade" ;; \

install|*) ensurepip="" ;; \

esac; \

LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \

$ensurepip --root=/ ; \

fi

Traceback (most recent call last):

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main

"__main__", fname, loader, pkg_name)

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code

exec code in run_globals

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>

ensurepip._main()

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main

default_pip=args.default_pip,

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap

_run_pip(args + [p[0] for p in _PROJECTS], additional_paths)

File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip

import pip

zipimport.ZipImportError: can't decompress data; zlib not available

make: *** [Makefile:927: install] Error 1

phase `install' failed after 10.8 seconds

------

If you build 2.7.13 with zlib on core-updates (log not *not* attached):

------

if test "xinstall" != "xno" ; then \ case install in \

upgrade) ensurepip="--upgrade" ;; \

install|*) ensurepip="" ;; \

esac; \ LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \ $ensurepip --root=/ ; \ fi

Collecting setuptools Collecting pip Installing collected packages: setuptools, pip Successfully installed pip-9.0.1 setuptools-28.8.0

phase `install' succeeded after 21.6 seconds

------

Hi!

Adding libffi and zlib solves the problem, similar to python-minimal@3.

Go for it? :-)

Ludo’.

ludo@gnu.org (Ludovic Courtès) skribis:

Pushed as 5927a843b4a91416b5be9e65c69616447f82e162 in core-updates,

following our discussion on IRC.

Thanks,

Ludo’.

tags 26136 fixed

close 26136