[PATCH 0/8] Misc Python build system improvements

  • Open
  • quality assurance status badge
Details
4 participants
  • Lars-Dominik Braun
  • Marius Bakke
  • Ricardo Wurmus
  • Sharlatan Hellseher
Owner
unassigned
Submitted by
Lars-Dominik Braun
Severity
normal
L
L
Lars-Dominik Braun wrote on 2 Aug 2023 12:37
(address . guix-patches@gnu.org)
cover.1690972374.git.lars@6xq.net
Hi,

the attached patches contain various Python build system improvements
including an original TOML parser (which may be used by other build
systems and importers in the future too) and fixes for #62781 and #63044.
I’m also pushing the patches to the pyproject-toml branch. As always,
a world rebuild is required.

Cheers,
Lars

Lars-Dominik Braun (8):
build-system/pyproject: Use python-sans-pip-wrapper as default-python.
build-system/python: Ignore symlinks when changing mtime.
gnu: criu: Change file mtimes to fix build.
gnu: sssd: Change file mtimes to fix build.
guix: toml: Add TOML parser.
build-system/pyproject: Use TOML parser.
gnu: python-pytest-xdist: Disable failing test.
guix: pyproject-build-system: Default configure-flags to empty
dictionary.

Makefile.am | 2 +
gnu/packages/admin.scm | 2 +-
gnu/packages/astronomy.scm | 81 +++--
gnu/packages/bioinformatics.scm | 79 +++--
gnu/packages/bootloaders.scm | 3 +
gnu/packages/build-tools.scm | 10 +-
gnu/packages/check.scm | 14 +-
gnu/packages/databases.scm | 12 +-
gnu/packages/digest.scm | 2 +-
gnu/packages/engineering.scm | 2 +-
gnu/packages/fontutils.scm | 9 +-
gnu/packages/geo.scm | 10 +-
gnu/packages/graph.scm | 7 +-
gnu/packages/graphviz.scm | 4 +-
gnu/packages/license.scm | 5 +-
gnu/packages/machine-learning.scm | 32 +-
gnu/packages/maths.scm | 4 +-
gnu/packages/monitoring.scm | 2 +-
gnu/packages/package-management.scm | 6 +-
gnu/packages/potassco.scm | 10 +-
gnu/packages/protobuf.scm | 4 +-
gnu/packages/python-build.scm | 33 +-
gnu/packages/python-check.scm | 30 +-
gnu/packages/python-compression.scm | 20 +-
gnu/packages/python-crypto.scm | 15 +-
gnu/packages/python-science.scm | 24 +-
gnu/packages/python-web.scm | 57 ++-
gnu/packages/python-xyz.scm | 260 +++++++++-----
gnu/packages/qt.scm | 3 +-
gnu/packages/rpc.scm | 2 +-
gnu/packages/serialization.scm | 4 +-
gnu/packages/sphinx.scm | 4 +-
gnu/packages/sssd.scm | 9 +-
gnu/packages/statistics.scm | 14 +-
gnu/packages/terminals.scm | 5 +-
gnu/packages/time.scm | 3 +-
gnu/packages/tor.scm | 4 +-
gnu/packages/tree-sitter.scm | 4 +-
gnu/packages/video.scm | 2 +-
gnu/packages/vim.scm | 4 +-
gnu/packages/virtualization.scm | 20 +-
guix/build-system/pyproject.scm | 12 +-
guix/build/pyproject-build-system.scm | 56 +--
guix/build/python-build-system.scm | 4 +-
guix/build/toml.scm | 478 ++++++++++++++++++++++++++
tests/toml.scm | 442 ++++++++++++++++++++++++
46 files changed, 1503 insertions(+), 306 deletions(-)
create mode 100644 guix/build/toml.scm
create mode 100644 tests/toml.scm


base-commit: 5aceacac65784bd2e1fd12304f965ae6026de49d
--
2.41.0
Attachment: 0001-build-system-pyproject-Use-python-sans-pip-wrapper-a.patch
From f4697d0da0018e66ae759a9495a82f364cad5ccd Mon Sep 17 00:00:00 2001
Message-ID: <f4697d0da0018e66ae759a9495a82f364cad5ccd.1690972374.git.lars@6xq.net>
In-Reply-To: <cover.1690972374.git.lars@6xq.net>
References: <cover.1690972374.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sat, 13 May 2023 15:31:06 +0200
Subject: [PATCH 2/8] build-system/python: Ignore symlinks when changing mtime.

* guix/build/python-build-system.scm (ensure-no-mtimes-pre-1980): Ignore
'symlink.
---
guix/build/python-build-system.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

Toggle diff (24 lines)
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index aa04664b25..8e18d6d0df 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -37,6 +37,7 @@ (define-module (guix build python-build-system)
#:use-module (srfi srfi-26)
#:export (%standard-phases
add-installed-pythonpath
+ ensure-no-mtimes-pre-1980
site-packages
python-version
python-build))
@@ -270,7 +271,8 @@ (define* (ensure-no-mtimes-pre-1980 #:rest _)
;; timestamps before 1980.
(let ((early-1980 315619200)) ; 1980-01-02 UTC
(ftw "." (lambda (file stat flag)
- (unless (<= early-1980 (stat:mtime stat))
+ (unless (or (<= early-1980 (stat:mtime stat))
+ (eq? (stat:type stat) 'symlink))
(utime file early-1980 early-1980))
#t))))
--
2.41.0
From a0d4c891e6cf3522c6769e82d888d906445363f5 Mon Sep 17 00:00:00 2001
Message-ID: <a0d4c891e6cf3522c6769e82d888d906445363f5.1690972374.git.lars@6xq.net>
In-Reply-To: <cover.1690972374.git.lars@6xq.net>
References: <cover.1690972374.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sat, 13 May 2023 15:31:22 +0200
Subject: [PATCH 3/8] gnu: criu: Change file mtimes to fix build.

* gnu/packages/virtualization.scm (criu)[arguments]: Add
python-build-system to #:modules and #:imported modules, add phase
'ensure-no-mtimes-pre-1980.
---
gnu/packages/virtualization.scm | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)

Toggle diff (48 lines)
diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm
index 9556fbc61e..04059fc73e 100644
--- a/gnu/packages/virtualization.scm
+++ b/gnu/packages/virtualization.scm
@@ -143,6 +143,7 @@ (define-module (gnu packages virtualization)
#:use-module (guix git-download)
#:use-module (guix gexp)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (guix modules)
#:use-module (guix packages)
#:use-module (guix utils)
#:use-module (srfi srfi-1)
@@ -1750,8 +1751,12 @@ (define-public criu
(string-append "XMLTO="
(search-input-file %build-inputs
"/bin/xmlto")))
- #:modules ((ice-9 ftw)
- ,@%gnu-build-system-modules)
+ #:modules ((guix build gnu-build-system)
+ (guix build utils)
+ ((guix build python-build-system)
+ #:select (ensure-no-mtimes-pre-1980)))
+ #:imported-modules ,(append %gnu-build-system-modules
+ %python-build-system-modules)
#:phases
(modify-phases %standard-phases
(delete 'configure) ; no configure script
@@ -1774,17 +1779,8 @@ (define-public criu
(substitute* "criu/include/plugin.h"
(("/var") (string-append (assoc-ref outputs "out"))))
))
- ;; TODO: use
- ;; (@@ (guix build python-build-system) ensure-no-mtimes-pre-1980)
- ;; when it no longer throws due to trying to call UTIME on symlinks.
(add-after 'unpack 'ensure-no-mtimes-pre-1980
- (lambda _
- (let ((early-1980 315619200)) ; 1980-01-02 UTC
- (ftw "." (lambda (file stat flag)
- (unless (or (<= early-1980 (stat:mtime stat))
- (eq? (stat:type stat) 'symlink))
- (utime file early-1980 early-1980))
- #t)))))
+ ensure-no-mtimes-pre-1980)
(add-before 'build 'fix-symlink
(lambda* (#:key inputs #:allow-other-keys)
;; The file 'images/google/protobuf/descriptor.proto' points to
--
2.41.0
From 824c6ea30573c3a02f33058bd1739be4cd980da2 Mon Sep 17 00:00:00 2001
Message-ID: <824c6ea30573c3a02f33058bd1739be4cd980da2.1690972374.git.lars@6xq.net>
In-Reply-To: <cover.1690972374.git.lars@6xq.net>
References: <cover.1690972374.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sat, 13 May 2023 16:20:23 +0200
Subject: [PATCH 4/8] gnu: sssd: Change file mtimes to fix build.

* gnu/packages/sssd.scm (sssd)[arguments]: Add
python-build-system to #:modules and #:imported modules, add phase
'ensure-no-mtimes-pre-1980.
---
gnu/packages/sssd.scm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Toggle diff (32 lines)
diff --git a/gnu/packages/sssd.scm b/gnu/packages/sssd.scm
index 251275312f..85c6875f4f 100644
--- a/gnu/packages/sssd.scm
+++ b/gnu/packages/sssd.scm
@@ -31,6 +31,7 @@ (define-module (gnu packages sssd)
#:use-module (guix utils)
#:use-module (guix build utils)
#:use-module (guix build-system gnu)
+ #:use-module (guix build-system python)
#:use-module (gnu packages)
#:use-module (gnu packages)
#:use-module (gnu packages adns)
@@ -186,10 +187,16 @@ (define-public sssd
(string-append "--with-xml-catalog-path="
#$(this-package-native-input "docbook-xml")
"/xml/dtd/docbook/catalog.xml"))
+ #:modules '((guix build gnu-build-system)
+ (guix build utils)
+ ((guix build python-build-system)
+ #:select (ensure-no-mtimes-pre-1980)))
+ #:imported-modules (append %gnu-build-system-modules
+ %python-build-system-modules)
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'ensure-no-mtimes-pre-1980
- (@@ (guix build python-build-system) ensure-no-mtimes-pre-1980))
+ ensure-no-mtimes-pre-1980)
(add-after 'patch-source-shebangs 'patch-more-shebangs
(lambda _
(substitute* '("src/tools/analyzer/sss_analyze"
--
2.41.0
Attachment: 0005-guix-toml-Add-TOML-parser.patch
Attachment: 0006-build-system-pyproject-Use-TOML-parser.patch
From e987a9e28ec0d8d1b8ecdb2486eb12eae270a49d Mon Sep 17 00:00:00 2001
Message-ID: <e987a9e28ec0d8d1b8ecdb2486eb12eae270a49d.1690972374.git.lars@6xq.net>
In-Reply-To: <cover.1690972374.git.lars@6xq.net>
References: <cover.1690972374.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Tue, 25 Jul 2023 18:26:58 +0200
Subject: [PATCH 7/8] gnu: python-pytest-xdist: Disable failing test.

* gnu/packages/check.scm (python-pytest-xdist)[arguments]: Skip failing
test.
---
gnu/packages/check.scm | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

Toggle diff (18 lines)
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 691fe67605..2e1cda27b1 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -1631,7 +1631,10 @@ (define-public python-pytest-xdist
(lambda* (#:key tests? #:allow-other-keys)
(when tests?
(invoke "pytest" "-vv"
- "-n" (number->string (parallel-job-count)))))))))
+ "-n" (number->string (parallel-job-count))
+ ;; Fails with OSError: cannot send to <Channel id=1 closed>
+ ;; on foreign distribution.
+ "-k" "not test_internal_errors_propagate_to_controller")))))))
(native-inputs (list python-setuptools-scm python-filelock python-pytest))
(propagated-inputs (list python-execnet python-pytest-forked))
(home-page "https://github.com/pytest-dev/pytest-xdist")
--
2.41.0
From b3726639df72aa3943d8e403e3c2b9a6cde05421 Mon Sep 17 00:00:00 2001
Message-ID: <b3726639df72aa3943d8e403e3c2b9a6cde05421.1690972374.git.lars@6xq.net>
In-Reply-To: <cover.1690972374.git.lars@6xq.net>
References: <cover.1690972374.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 30 Jul 2023 13:36:37 +0200
Subject: [PATCH 8/8] guix: pyproject-build-system: Default configure-flags to
empty dictionary.

PEP 517 specifies it should be a dictionary and thus meson-python cannot
handle an empty list.

---
gnu/packages/build-tools.scm | 10 ++--------
gnu/packages/python-science.scm | 5 -----
guix/build-system/pyproject.scm | 2 +-
3 files changed, 3 insertions(+), 14 deletions(-)

Toggle diff (59 lines)
diff --git a/gnu/packages/build-tools.scm b/gnu/packages/build-tools.scm
index a365cca849..09a8a175e8 100644
--- a/gnu/packages/build-tools.scm
+++ b/gnu/packages/build-tools.scm
@@ -355,10 +355,7 @@ (define-public meson-python
"1hpjw9qj6ff8ixjs0pz7qysc8v57jxgaf5n1p6bqm9bh3mc3wnrx"))))
(build-system pyproject-build-system)
(arguments
- ;; The project is configured to use itself to build ('mesonpy') and fails;
- ;; use another PEP 517 build system.
- (list #:build-backend "setuptools.build_meta"
- #:test-flags #~(list "tests"
+ (list #:test-flags #~(list "tests"
;; The test_pep518 tries to install
;; dependencies from the network using pip.
"-k" "not test_pep518")))
@@ -370,10 +367,7 @@ (define-public meson-python
python-tomli
python-wheel))
(native-inputs
- (list python-pypa-build
- python-wheel
-
- ;; For tests.
+ (list ;; For tests.
git-minimal/pinned
patchelf
pkg-config
diff --git a/gnu/packages/python-science.scm b/gnu/packages/python-science.scm
index 538a3c2f0e..47890389b5 100644
--- a/gnu/packages/python-science.scm
+++ b/gnu/packages/python-science.scm
@@ -93,11 +93,6 @@ (define-public python-scipy
(build-system pyproject-build-system)
(arguments
(list
- ;; FIXME: The default 'mesonpy' build system doesn't seem to work with
- ;; our pyproject-build-system, errors with: AttributeError: 'list'
- ;; object has no attribute 'items' (see:
- ;; https://issues.guix.gnu.org/62781).
- #:build-backend "setuptools.build_meta"
#:phases
#~(modify-phases %standard-phases
(replace 'check
diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm
index 585117cbf0..c0e089eac7 100644
--- a/guix/build-system/pyproject.scm
+++ b/guix/build-system/pyproject.scm
@@ -93,7 +93,7 @@ (define* (lower name
(define* (pyproject-build name inputs
#:key source
(tests? #t)
- (configure-flags ''())
+ (configure-flags ''(@))
(backend-path #f)
(build-backend #f)
(test-backend #f)
--
2.41.0
S
S
Sharlatan Hellseher wrote on 7 Sep 2023 23:55
(address . 65010@debbugs.gnu.org)
CAO+9K5q_ZFZahgcirSxH7Z7XiXqtSoBjE_SiUGEDLm1O2zJw_Q@mail.gmail.com
Hi,
Based on a cover letter it provides more patches in series, was everything sent?
--
… ??? ????? - ???????????? ?????????????? ?????? ??????? ????????
????? ????? ????? ? ??? ??????, ??????????? ????? ???????, ??
?????????? ?? ? ????????? ??????? ????? ? ?????????????????.
M
M
Marius Bakke wrote on 22 Sep 2023 06:01
(address . jgart@dismail.de)
874jjm97pm.fsf@gnu.org
Lars-Dominik Braun <lars@6xq.net> skriver:

Toggle quote (8 lines)
> Hi,
>
> the attached patches contain various Python build system improvements
> including an original TOML parser (which may be used by other build
> systems and importers in the future too) and fixes for #62781 and #63044.
> I’m also pushing the patches to the pyproject-toml branch. As always,
> a world rebuild is required.

Hi Lars!

Toggle quote (11 lines)
> Lars-Dominik Braun (8):
> build-system/pyproject: Use python-sans-pip-wrapper as default-python.
> build-system/python: Ignore symlinks when changing mtime.
> gnu: criu: Change file mtimes to fix build.
> gnu: sssd: Change file mtimes to fix build.
> guix: toml: Add TOML parser.
> build-system/pyproject: Use TOML parser.
> gnu: python-pytest-xdist: Disable failing test.
> guix: pyproject-build-system: Default configure-flags to empty
> dictionary.

I read through these patches and they LGTM overall. Some individual
comments follow.

Toggle quote (16 lines)
>>From 70baccd8068f35ff8dce0955eaea60f7ea9c240e Mon Sep 17 00:00:00 2001
> Message-ID: <70baccd8068f35ff8dce0955eaea60f7ea9c240e.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Thu, 11 May 2023 08:33:02 +0200
> Subject: [PATCH 1/8] build-system/pyproject: Use python-sans-pip-wrapper as
> default-python.
>
> Also adds python-setuptools and python-wheel to relevant packages,
> either to native-inputs or to propagated inputs if the pkg_resources
> Python module is loaded at runtime.
>
> * guix/build-system/pyproject.scm (default-python): Default to
> python-sans-pip-wrapper.

This change makes sense. Using a 'python-toolchain' by default was a
mistake, a quite expensive one by the diff. Sorry about that, and
thanks for fixing it.

As you noted elsewhere (and for those following along at home): this is
necessary to solve a cycle at the root of the Python package graph. It
also is more "correct" in that there are no surprising or hidden inputs.

[...thinking while typing...]

About 'wheel': that package *is* actually required by the build system
(as opposed to pypa-build!): I think we should make it available by
default (e.g. with a #:wheel argument), or propagate it from the build
systems. It makes no sense to add it to _all_ pyproject-build-system
consumers.

WDYT?

[...]

Toggle quote (11 lines)
>>From f4697d0da0018e66ae759a9495a82f364cad5ccd Mon Sep 17 00:00:00 2001
> Message-ID: <f4697d0da0018e66ae759a9495a82f364cad5ccd.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sat, 13 May 2023 15:31:06 +0200
> Subject: [PATCH 2/8] build-system/python: Ignore symlinks when changing mtime.
>
> * guix/build/python-build-system.scm (ensure-no-mtimes-pre-1980): Ignore
> 'symlink.

LGTM.

Toggle quote (8 lines)
>>From a0d4c891e6cf3522c6769e82d888d906445363f5 Mon Sep 17 00:00:00 2001
> Message-ID: <a0d4c891e6cf3522c6769e82d888d906445363f5.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sat, 13 May 2023 15:31:22 +0200
> Subject: [PATCH 3/8] gnu: criu: Change file mtimes to fix build.

This commit title is weird. The build is already working, but using a
workaround that is no longer needed.

Toggle quote (4 lines)
> * gnu/packages/virtualization.scm (criu)[arguments]: Add
> python-build-system to #:modules and #:imported modules, add phase
> 'ensure-no-mtimes-pre-1980.

LGTM.

Toggle quote (12 lines)
>>From 824c6ea30573c3a02f33058bd1739be4cd980da2 Mon Sep 17 00:00:00 2001
> Message-ID: <824c6ea30573c3a02f33058bd1739be4cd980da2.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sat, 13 May 2023 16:20:23 +0200
> Subject: [PATCH 4/8] gnu: sssd: Change file mtimes to fix build.
>
> * gnu/packages/sssd.scm (sssd)[arguments]: Add
> python-build-system to #:modules and #:imported modules, add phase
> 'ensure-no-mtimes-pre-1980.

Ditto.

Toggle quote (16 lines)
>>From 72d66e838ce9d3d2182c570ee3088063e372fcdd Mon Sep 17 00:00:00 2001
> Message-ID: <72d66e838ce9d3d2182c570ee3088063e372fcdd.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sun, 23 Jul 2023 11:20:03 +0200
> Subject: [PATCH 5/8] guix: toml: Add TOML parser.
>
> * guix/build/toml.scm: New file.
> * tests/toml.scm: New file.
> * Makefile.am: Register new files.
> ---
> Makefile.am | 2 +
> guix/build/toml.scm | 478 ++++++++++++++++++++++++++++++++++++++++++++
> tests/toml.scm | 442 ++++++++++++++++++++++++++++++++++++++++

Woow, amazing work, and I'm surprised it's less than 500 lines!

I haven't studied it in detail, but on the surface LGTM.

Toggle quote (25 lines)
>>From 8e079b48b7c07c07360db8eb6305e8044d86dd87 Mon Sep 17 00:00:00 2001
> Message-ID: <8e079b48b7c07c07360db8eb6305e8044d86dd87.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sun, 23 Jul 2023 11:22:03 +0200
> Subject: [PATCH 6/8] build-system/pyproject: Use TOML parser.
>
> More reliable than regular expressions.
>
> * guix/build-system/pyproject.scm (%pyproject-build-system-modules): Add (guix build toml).
> (pyproject-build): Add argument #:backend-path.
> * guix/build/pyproject-build-system.scm (build): Add support for
> auto-detected and override backend-path.
> * gnu/packages/python-build.scm (python-tomli)[arguments]: Remove
> 'add-self-to-path, because it is not necessary any more.
> (python-poetry-core): Same.
> (python-hatchling): Same.
> (python-pdm-backend): Same.
> ---
> gnu/packages/python-build.scm | 30 ++------------
> guix/build-system/pyproject.scm | 3 ++
> guix/build/pyproject-build-system.scm | 56 +++++++++++++++------------
> 3 files changed, 39 insertions(+), 50 deletions(-)

LGTM...

[...]

Toggle quote (44 lines)
> diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm
> index 94b9d79692..585117cbf0 100644
> --- a/guix/build-system/pyproject.scm
> +++ b/guix/build-system/pyproject.scm
> @@ -46,6 +46,7 @@ (define %pyproject-build-system-modules
> ;; Build-side modules imported by default.
> `((guix build pyproject-build-system)
> (guix build json)
> + (guix build toml)
> ,@%python-build-system-modules))
>
> (define (default-python)
> @@ -93,6 +94,7 @@ (define* (pyproject-build name inputs
> #:key source
> (tests? #t)
> (configure-flags ''())
> + (backend-path #f)
> (build-backend #f)
> (test-backend #f)
> (test-flags ''())
> @@ -116,6 +118,7 @@ (define* (pyproject-build name inputs
> #:source #+source
> #:configure-flags #$configure-flags
> #:system #$system
> + #:backend-path #$backend-path
> #:build-backend #$build-backend
> #:test-backend #$test-backend
> #:test-flags #$test-flags
> diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
> index c69ccc9d64..a1919eacf6 100644
> --- a/guix/build/pyproject-build-system.scm
> +++ b/guix/build/pyproject-build-system.scm
> @@ -21,11 +21,13 @@ (define-module (guix build pyproject-build-system)
> #:use-module ((guix build python-build-system) #:prefix python:)
> #:use-module (guix build utils)
> #:use-module (guix build json)
> + #:use-module (guix build toml)
> #:use-module (ice-9 match)
> #:use-module (ice-9 ftw)
> #:use-module (ice-9 format)
> #:use-module (ice-9 rdelim)
> #:use-module (ice-9 regex)
> + #:use-module (ice-9 textual-ports)

This import seems unused?

Toggle quote (12 lines)
> #:use-module (srfi srfi-1)
> #:use-module (srfi srfi-26)
> #:use-module (srfi srfi-34)
> @@ -60,8 +62,8 @@ (define-module (guix build pyproject-build-system)
> ;;; wheel and expected to be created by the installing utility.
> ;;; TODO: Add support for PEP-621 entry points.
> ;;;
> -;;; Caveats:
> -;;; - There is no support for in-tree build backends.
> +;;; This module also supports in-tree build backends, which can be
> +;;; overridden by #:backend-path.

Perhaps this should also be documented in the manual.

Toggle quote (11 lines)
>>From e987a9e28ec0d8d1b8ecdb2486eb12eae270a49d Mon Sep 17 00:00:00 2001
> Message-ID: <e987a9e28ec0d8d1b8ecdb2486eb12eae270a49d.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Tue, 25 Jul 2023 18:26:58 +0200
> Subject: [PATCH 7/8] gnu: python-pytest-xdist: Disable failing test.
>
> * gnu/packages/check.scm (python-pytest-xdist)[arguments]: Skip failing
> test.

OK.

Toggle quote (19 lines)
>>From b3726639df72aa3943d8e403e3c2b9a6cde05421 Mon Sep 17 00:00:00 2001
> Message-ID: <b3726639df72aa3943d8e403e3c2b9a6cde05421.1690972374.git.lars@6xq.net>
> In-Reply-To: <cover.1690972374.git.lars@6xq.net>
> References: <cover.1690972374.git.lars@6xq.net>
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sun, 30 Jul 2023 13:36:37 +0200
> Subject: [PATCH 8/8] guix: pyproject-build-system: Default configure-flags to
> empty dictionary.
>
> PEP 517 specifies it should be a dictionary and thus meson-python cannot
> handle an empty list.
>
> Fixes: <https://issues.guix.gnu.org/62781>
> ---
> gnu/packages/build-tools.scm | 10 ++--------
> gnu/packages/python-science.scm | 5 -----
> guix/build-system/pyproject.scm | 2 +-
> 3 files changed, 3 insertions(+), 14 deletions(-)

The commit message lacks a mention of the changed variables.

[...]

Toggle quote (11 lines)
> diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm
> index 585117cbf0..c0e089eac7 100644
> --- a/guix/build-system/pyproject.scm
> +++ b/guix/build-system/pyproject.scm
> @@ -93,7 +93,7 @@ (define* (lower name
> (define* (pyproject-build name inputs
> #:key source
> (tests? #t)
> - (configure-flags ''())
> + (configure-flags ''(@))

I don't understand how the @ makes it a dictionary. Can you enlighten
me? Either here, or in a comment? :-)

LGTM anyway!

Let's get these merged in the 'python-team' branch. WDYT jgart?
-----BEGIN PGP SIGNATURE-----

iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCZQ0RtQ8cbWFyaXVzQGdu
dS5vcmcACgkQ6HGLpZEUEHcaegD5AaG2ZPX/J0WwaoyYbgtU4FmQKxcpgwGsQ7jg
JUZfoCkA/iN3VSpkcOEJqfzoNnzXxE3m+Wg5clrgZKItz7TY3gAN
=WgZc
-----END PGP SIGNATURE-----

L
L
Lars-Dominik Braun wrote on 23 Sep 2023 08:31
(name . Marius Bakke)(address . marius@gnu.org)
ZQ6GU1EfsoaMCXyk@noor.fritz.box
Hi Marius,

Toggle quote (6 lines)
> About 'wheel': that package *is* actually required by the build system
> (as opposed to pypa-build!): I think we should make it available by
> default (e.g. with a #:wheel argument), or propagate it from the build
> systems. It makes no sense to add it to _all_ pyproject-build-system
> consumers.

it’s not added to all of them. After submitting this patch I realized
it’s actually a dependency of setuptools (bdist_wheel), which otherwise
cannot create wheels. Other backends afaik have their own method of
building wheels or depend on a wheel-generating library. So it might
make sense to add python-wheel as a propagated input to
python-setuptools. I didn’t have time to try it yet though.

Toggle quote (3 lines)
> This commit title is weird. The build is already working, but using a
> workaround that is no longer needed.

Probably a leftover from all the rebasing.

Toggle quote (3 lines)
> > + #:use-module (ice-9 textual-ports)
> This import seems unused?

Yeah, possibly a leftover from previous experiments.

Toggle quote (2 lines)
> Perhaps this should also be documented in the manual.

Indeed. I didn’t know we had separate documentation in the manual.

Toggle quote (6 lines)
> > - (configure-flags ''())
> > + (configure-flags ''(@))
>
> I don't understand how the @ makes it a dictionary. Can you enlighten
> me? Either here, or in a comment? :-)

It’s an implementation detail of our JSON library. The configure-flags
are directly passed to it.

Lars
R
R
Ricardo Wurmus wrote on 10 Feb 13:15 +0100
[PATCH 0/8] Misc Python build system improvements
(address . 65010@debbugs.gnu.org)(name . Lars-Dominik Braun)(address . lars@6xq.net)
87h6igsecw.fsf@elephly.net
I’m currently in the process of reviving the python-team branch. I’d
like to pull in a few upgrades to Python science packages, and I think
this patch series should also be part of the branch.

Lars, do you have a final version of this patch series that we could
merge into the python-team branch? It looks like apart from very minor
issues this is ready.

--
Ricardo
L
L
Lars-Dominik Braun wrote on 16 Feb 07:50 +0100
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 65010@debbugs.gnu.org)
Zc8FmWempkAj43z8@noor.fritz.box
Hi,

Toggle quote (4 lines)
> Lars, do you have a final version of this patch series that we could
> merge into the python-team branch? It looks like apart from very minor
> issues this is ready.

I’m rebasing the changes right now, will look through Marius’ comments
and send an updated patch series.

Lars
L
L
Lars-Dominik Braun wrote on 16 Feb 13:38 +0100
(name . Ricardo Wurmus)(address . rekado@elephly.net)
Zc9XW1fJ3LiNLNtR@noor.fritz.box
Hi again,

Toggle quote (4 lines)
> Lars, do you have a final version of this patch series that we could
> merge into the python-team branch? It looks like apart from very minor
> issues this is ready.

as you suggested I pushed the updated version to the python-team branch,
deleting my own pyproject-toml branch.

Unfortunately the python-setuptools/python-wheel situation is not
easily resolvable, since – contrary to my initial assumption –
python-setuptools does not depend on python-wheel, but rather the latter
extends the former and is only required when using setuptools, but not
for other build backends like flit or poetry. Therefore I kept my
changes as-is, adding python-setuptools and python-wheel to
native-inputs/propagated-inputs where needed. Not pretty, but that’s
PEP 517 in a nutshell.

Since it’s been a while and more packages have been migrated to use
pyproject-build-system this will probably break a few packages, which
do not have these two packages added yet. We’ll have to wait for
https://ci.guix.gnu.org/eval/1123219to finish to assess the damage
done. There are two possible failure modes:

1) Build-time failure: Usually the build will complain about setuptools
not being found. That’s an easy fix, just add setuptools and wheel to
the native-inputs.
2) Run-time failure: The deprecated but still heavily used pkg_resources
is missing during runtime, because it’s part of setuptools. This one
is harder to detect, but once all packages have been built we should
`grep` for pkg_resources and figure out which of the consumers don’t
propagate python-setuptools.

Cheers,
Lars
?