[PATCH 0/2] Python: Ignore unwanted development inputs.

  • Open
  • quality assurance status badge
Details
2 participants
  • Lars-Dominik Braun
  • Nicolas Graves
Owner
unassigned
Submitted by
Nicolas Graves
Severity
normal
N
N
Nicolas Graves wrote on 25 Apr 17:42 +0200
(address . guix-patches@gnu.org)(address . ngraves@ngraves.fr)
20240425155529.30747-1-ngraves@ngraves.fr
This patch series implements a way to robustly and with minimal burden on maintainers to remove some python packages that are often put in native inputs. This allows these packages to be removed almost seemlessly in most cases from native-inputs where they are of no use, enabling to cut down the dependent rebuilds upon update, and more easily update them and their real dependents.

It is for the python-team branch. It will be followed by a more consequential patch series upon the python-team branch to actually remove these unwanted developement inputs, but is send before for reviews.

The patch series handles :
- guix lint warnings against the inclusion of these packages
- guix import pypi will automatically not include them when read in the package result
- ignoring pytest arguments defined in pytest.ini or setup.cfg regarding those packages

The patch series doesn't handle:
- ignoring these packages when they are explicitely listed as required for installation (which means that sometimes a snippet or a patch are necessary, but this is not the general case).

Nicolas Graves (2):
guix: import: pypi: Ignore pypi-ignored-inputs.
guix: pyproject-build-system: Ignore unwanted pytest flags.

guix/build/pyproject-build-system.scm | 88 ++++++++++++++++++++++++++-
guix/import/pypi.scm | 20 +++++-
guix/lint.scm | 11 ++--
tests/pypi.scm | 3 +-
4 files changed, 111 insertions(+), 11 deletions(-)

--
2.41.0
N
N
Nicolas Graves wrote on 25 Apr 17:59 +0200
[PATCH 1/2] guix: import: pypi: Ignore pypi-ignored-inputs.
(address . 70570@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240425160010.6243-1-ngraves@ngraves.fr
* guix/import/pypi.scm (pypi-ignored-inputs): New variable.
(compute-inputs): Use it.

* tests/pypi.scm (parse-requires.txt): Add ignored input to test the
feature.

* guix/lint.scm (check-inputs-should-be-native): Adapt list.
(check-inputs-should-not-be-an-input-at-all): Use pypi-ignored-list.

Change-Id: I774e526c5a090026e778ee44049637174a1dca95
---
guix/import/pypi.scm | 20 +++++++++++++++++---
guix/lint.scm | 11 ++++++-----
tests/pypi.scm | 3 ++-
3 files changed, 25 insertions(+), 9 deletions(-)

Toggle diff (117 lines)
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 6719fde330..a43d4eca1d 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2022 Vivien Kraus <vivien@planete-kraus.eu>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -61,6 +62,7 @@ (define-module (guix import pypi)
#:use-module (guix upstream)
#:use-module ((guix licenses) #:prefix license:)
#:export (%pypi-base-url
+ pypi-ignored-inputs
parse-requires.txt
parse-wheel-metadata
specification->requirement-name
@@ -77,6 +79,17 @@ (define %pypi-base-url
;; Base URL of the PyPI API.
(make-parameter "https://pypi.org/pypi/"))
+(define pypi-ignored-inputs
+ ;; This list contains packages that are useful for development or quality
+ ;; testing, but that most of the time are not necessary to have as an input.
+ (list "argparse" ; native
+ "codecov" "coverage" ; coverage
+ "black" "isort" "pycodestyle" "pep8" ; style
+ "pyflakes" "flake8" "pylint" "mypy" ; style+lint
+ "coveralls" "twine" ; upload integration tools
+ "pytest-isort" "pytest-flake8" "pytest-cov" "pytest-black"
+ "pytest-pep8" "pytest-mypy" "pytest-pep8" "pre-commit")) ; variants
+
(define non-empty-string-or-false
(match-lambda
("" #f)
@@ -424,9 +437,10 @@ (define (compute-inputs source-url wheel-url archive)
"Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, return
the corresponding list of <upstream-input> records."
(define (requirements->upstream-inputs deps type)
- (filter-map (match-lambda
- ("argparse" #f)
- (name (upstream-input
+ (filter-map (lambda (name)
+ (if (member name pypi-ignored-inputs)
+ #f
+ (upstream-input
(name name)
(downstream-name (python->package-name name))
(type type))))
diff --git a/guix/lint.scm b/guix/lint.scm
index 68d532968d..9867b71cfd 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -71,6 +71,7 @@ (define-module (guix lint)
hg-reference-url)
#:autoload (guix bzr-download) (bzr-reference?
bzr-reference-url)
+ #:use-module ((guix import pypi) #:select (pypi-ignored-inputs))
#:use-module (guix import stackage)
#:use-module (ice-9 match)
#:use-module (ice-9 regex)
@@ -557,14 +558,12 @@ (define (check-inputs-should-be-native package)
"m4"
"qttools-5"
"yasm" "nasm" "fasm"
- "python-coverage"
"python-cython"
"python-docutils"
"python-mock"
"python-nose"
"python-pbr"
"python-pytest"
- "python-pytest-cov"
"python-setuptools-scm"
"python-sphinx"
"scdoc"
@@ -586,9 +585,11 @@ (define (check-inputs-should-be-native package)
(define (check-inputs-should-not-be-an-input-at-all package)
;; Emit a warning if some inputs of PACKAGE are likely to should not be
;; an input at all.
- (let ((input-names '("python-setuptools"
- "python-pip"
- "python-pre-commit")))
+ (let ((input-names (append
+ '("python-setuptools"
+ "python-pip"
+ "python-pre-commit")
+ pypi-ignored-inputs)))
(map (lambda (input)
(make-warning
package
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 42b39cde73..fe01ab3beb 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -97,6 +97,7 @@ (define test-requires.txt "\
[test]
pytest (>=2.5.0)
+pytest-cov # read but ignored
")
;; Beaker contains only optional dependencies.
@@ -244,7 +245,7 @@ (define-syntax-rule (with-pypi responses body ...)
(map specification->requirement-name test-specifications))
(test-equal "parse-requires.txt"
- (list '("foo" "bar") '("pytest"))
+ (list '("foo" "bar") '("pytest" "pytest-cov"))
(mock ((ice-9 ports) call-with-input-file
call-with-input-string)
(parse-requires.txt test-requires.txt)))
--
2.41.0
N
N
Nicolas Graves wrote on 25 Apr 17:59 +0200
[PATCH 2/2] guix: pyproject-build-system: Ignore unwanted pytest flags.
(address . 70570@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240425160010.6243-2-ngraves@ngraves.fr
* guix/build/pyproject-build-system.scm : Ignore unwanted pytest flags.

Change-Id: Ib9f1602e5af11227e5b7ce124f0f9be4fa2b78e4
---
guix/build/pyproject-build-system.scm | 88 ++++++++++++++++++++++++++-
1 file changed, 86 insertions(+), 2 deletions(-)

Toggle diff (119 lines)
diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
index 947d240114..3ca3c76c1c 100644
--- a/guix/build/pyproject-build-system.scm
+++ b/guix/build/pyproject-build-system.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -142,7 +143,86 @@ (define* (build #:key outputs build-backend backend-path configure-flags #:allow
wheel-dir
config-settings)))
-(define* (check #:key tests? test-backend test-flags #:allow-other-keys)
+;; Pytest plugin to filter out arguments to ignore.
+(define pytest-default-ignore-alist
+ '(("cov" . ("--cov" "--cov-reset" "--cov-report" "--cov-config"
+ "--no-cov-on-fail" "--no-cov" "--cov-fail-under"
+ "--cov-append" "--cov-branch" "--cov-context"))
+ ("mypy" . ("--mypy" "--mypy-config-file" "--mypy-ignore-missing-imports"))
+ ("isort" . ("--isort"))
+ ("flake8" . ("--flake8"))
+ ("black" . ("--black"))
+ ("flakes" . ("--flakes"))
+ ("pep8" . ("--pep8"))))
+
+;; Allow guix to ignore these options when underlying pytest package is not
+;; an input. These flags are not necessary to properly run tests.
+(define (pytest-ignore-options-plugin flags)
+ "This function converts an list of flags in a string that can
+ be instantiated as a python pytest plugin."
+ (format #f "\
+import pytest
+
+def pytest_addoption(parser):
+ group = parser.getgroup('guix','Guix ignored options')
+ options = [~{~s, ~}]
+ for option in options:
+ group.addoption(option, action='append', nargs='*')"
+ flags))
+
+(define (call-with-guix-pytest-plugin inputs thunk)
+ (let* ((former-path (getenv "PYTHONPATH"))
+ (input-names
+ (filter (match-lambda
+ (((name . _) ...)
+ (if (string-prefix? "python-pytest-" name)
+ name
+ #f))
+ ( _ #f))
+ inputs))
+ (filtered-flags
+ (filter identity
+ (append-map
+ (match-lambda
+ ((group . flags)
+ (if (member (string-append "python-pytest-" group)
+ input-names)
+ (list #f)
+ flags))
+ (_ (list #f)))
+ pytest-default-ignore-alist))))
+ (dynamic-wind
+ (lambda ()
+ (setenv "PYTHONPATH"
+ (string-append
+ (if former-path
+ (string-append former-path ":")
+ "")
+ ".guix-pytest"))
+ (setenv "PYTEST_PLUGINS"
+ (string-append
+ (if (getenv "PYTEST_PLUGINS")
+ (string-append former-path ",")
+ "")
+ "pytest_guix_plugin"))
+ (mkdir-p ".guix-pytest")
+ (with-output-to-file ".guix-pytest/__init__.py"
+ (lambda _ (display "")))
+ (with-output-to-file ".guix-pytest/pytest_guix_plugin.py"
+ (lambda _
+ (display (pytest-ignore-options-plugin filtered-flags)))))
+ thunk
+ (lambda ()
+ (setenv "PYTHONPATH" former-path)
+ (unsetenv "PYTEST_PLUGINS")
+ (when (file-exists? ".guix-pytest")
+ (delete-file-recursively ".guix-pytest"))))))
+
+(define-syntax-rule (with-guix-pytest-plugin inputs exp ...)
+ "Evaluate EXP in a context where the Guix pytest plugin is added."
+ (call-with-guix-pytest-plugin inputs (lambda () exp ...)))
+
+(define* (check #:key inputs tests? test-backend test-flags #:allow-other-keys)
"Run the test suite of a given Python package."
(if tests?
;; Unfortunately with PEP 517 there is no common method to specify test
@@ -165,7 +245,8 @@ (define* (check #:key tests? test-backend test-flags #:allow-other-keys)
(format #t "Using ~a~%" use-test-backend)
(match use-test-backend
('pytest
- (apply invoke pytest "-vv" test-flags))
+ (with-guix-pytest-plugin inputs
+ (apply invoke pytest "-vv" test-flags)))
('nose
(apply invoke nosetests "-v" test-flags))
('nose2
@@ -386,3 +467,6 @@ (define* (pyproject-build #:key inputs (phases %standard-phases)
(apply python:python-build #:inputs inputs #:phases phases args))
;;; pyproject-build-system.scm ends here
+;;; Local Variables:
+;;; eval: (put 'with-guix-pytest-plugin 'scheme-indent-function 1)
+;;; End:
--
2.41.0
L
L
Lars-Dominik Braun wrote on 26 Apr 10:26 +0200
Re: [PATCH 1/2] guix: import: pypi: Ignore pypi-ignored-inputs.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)(address . 70570@debbugs.gnu.org)
ZitlNNs1jvQiF-BY@noor.fritz.box
Hi,

Toggle quote (17 lines)
> +(define pypi-ignored-inputs
> + ;; This list contains packages that are useful for development or quality
> + ;; testing, but that most of the time are not necessary to have as an input.
> + (list "argparse" ; native
> + "codecov" "coverage" ; coverage
> + "black" "isort" "pycodestyle" "pep8" ; style
> + "pyflakes" "flake8" "pylint" "mypy" ; style+lint
> + "coveralls" "twine" ; upload integration tools
> + "pytest-isort" "pytest-flake8" "pytest-cov" "pytest-black"
> + "pytest-pep8" "pytest-mypy" "pytest-pep8" "pre-commit")) ; variants

> + (let ((input-names (append
> + '("python-setuptools"
> + "python-pip"
> + "python-pre-commit")
> + pypi-ignored-inputs)))

we should remove python-setuptools from this list now (since it actually
should be an input on the python-team branch), python-pre-commit is
also part of pypi-ignored-inputs and maybe we can just add pip to
pypi-ignored-inputs and use only that list?

Also note that check-inputs-should-not-be-an-input-at-all expects Guix
package names (with python- prefix), whereas pypi-ignored-inputs uses
Python package names, so you probably want to (map (cut (string-append
"python-" <>) …) (or similar) here.

Lars
L
L
Lars-Dominik Braun wrote on 26 Apr 10:47 +0200
Re: [PATCH 2/2] guix: pyproject-build-system: Ignore unwanted pytest flags.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)(address . 70570@debbugs.gnu.org)
ZitqM7eUAWMzKZQ0@noor.fritz.box
Hi,

pretty smart idea to use a pytest plugin :)

Toggle quote (3 lines)
> +;; Pytest plugin to filter out arguments to ignore.
> +(define pytest-default-ignore-alist

From the comment it’s not entirely clear to me what this list
does. It’s a map from pytest plugin name to it’s pytest command line
options, right?

Toggle quote (5 lines)
> +;; Allow guix to ignore these options when underlying pytest package is not
> +;; an input. These flags are not necessary to properly run tests.
> +(define (pytest-ignore-options-plugin flags)
> + "This function converts an list of flags in a string that can
> + be instantiated as a python pytest plugin."
…
Toggle quote (2 lines)
> +(define (call-with-guix-pytest-plugin inputs thunk)

Same here. As far as I understand you want to emulate command line
options provided by pytest plugins, so pytest won’t fail if the plugin
is not present. And we only do that if the plugin is not an input to
avoid clashing command line options, right?

+ for option in options:
+ group.addoption(option, action='append', nargs='*')"

Not sure nargs='*' is a good idea, since it might consume positional
arguments intended for pytest. '?' would be a more conservative option,
especially since we cannot override this easily per-package.

Toggle quote (1 lines)
> + (let* ((former-path (getenv "PYTHONPATH"))
…
Toggle quote (3 lines)
> + (dynamic-wind
> + (lambda ()
> + (setenv "PYTHONPATH"
…
Toggle quote (3 lines)
> + (lambda ()
> + (setenv "PYTHONPATH" former-path)

Isn’t it GUIX_PYTHONPATH?

Lars
N
N
Nicolas Graves wrote on 26 Apr 12:14 +0200
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 70570@debbugs.gnu.org)
87mspgv3a7.fsf@ngraves.fr
On 2024-04-26 10:47, Lars-Dominik Braun wrote:

Toggle quote (11 lines)
> Hi,
>
> pretty smart idea to use a pytest plugin :)
>
>> +;; Pytest plugin to filter out arguments to ignore.
>> +(define pytest-default-ignore-alist
>
> From the comment it’s not entirely clear to me what this list
> does. It’s a map from pytest plugin name to it’s pytest command line
> options, right?

Right, I'll update the command.

Toggle quote (14 lines)
>
>> +;; Allow guix to ignore these options when underlying pytest package is not
>> +;; an input. These flags are not necessary to properly run tests.
>> +(define (pytest-ignore-options-plugin flags)
>> + "This function converts an list of flags in a string that can
>> + be instantiated as a python pytest plugin."
> …
>> +(define (call-with-guix-pytest-plugin inputs thunk)
>
> Same here. As far as I understand you want to emulate command line
> options provided by pytest plugins, so pytest won’t fail if the plugin
> is not present. And we only do that if the plugin is not an input to
> avoid clashing command line options, right?

Will do.
Toggle quote (8 lines)
>
> + for option in options:
> + group.addoption(option, action='append', nargs='*')"
>
> Not sure nargs='*' is a good idea, since it might consume positional
> arguments intended for pytest. '?' would be a more conservative option,
> especially since we cannot override this easily per-package.

It works. I'll try with '?' then!

Toggle quote (12 lines)
>
>> + (let* ((former-path (getenv "PYTHONPATH"))
> …
>> + (dynamic-wind
>> + (lambda ()
>> + (setenv "PYTHONPATH"
> …
>> + (lambda ()
>> + (setenv "PYTHONPATH" former-path)
>
> Isn’t it GUIX_PYTHONPATH?
>
I know it works this way. Could be GUIX_PYTHONPATH too, I'm not sure I
properly get the difference. Can test with GUIX_PYTHONPATH too, but in
any case it's just for the tests, so it should be cleared out
too. Should I?

Toggle quote (3 lines)
> Lars
>

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 26 Apr 12:23 +0200
Re: [PATCH 1/2] guix: import: pypi: Ignore pypi-ignored-inputs.
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 70570@debbugs.gnu.org)
87jzkkv2vx.fsf@ngraves.fr
On 2024-04-26 10:26, Lars-Dominik Braun wrote:

Toggle quote (24 lines)
> Hi,
>
>> +(define pypi-ignored-inputs
>> + ;; This list contains packages that are useful for development or quality
>> + ;; testing, but that most of the time are not necessary to have as an input.
>> + (list "argparse" ; native
>> + "codecov" "coverage" ; coverage
>> + "black" "isort" "pycodestyle" "pep8" ; style
>> + "pyflakes" "flake8" "pylint" "mypy" ; style+lint
>> + "coveralls" "twine" ; upload integration tools
>> + "pytest-isort" "pytest-flake8" "pytest-cov" "pytest-black"
>> + "pytest-pep8" "pytest-mypy" "pytest-pep8" "pre-commit")) ; variants
>
>> + (let ((input-names (append
>> + '("python-setuptools"
>> + "python-pip"
>> + "python-pre-commit")
>> + pypi-ignored-inputs)))
>
> we should remove python-setuptools from this list now (since it actually
> should be an input on the python-team branch), python-pre-commit is
> also part of pypi-ignored-inputs and maybe we can just add pip to
> pypi-ignored-inputs and use only that list?

Yes, but then it should be done in an independent commit IMO to keep
this one focused on native-inputs and pytest, and then add one for your
other changes of the build system.

But yes, I think they ought to be in the same list.

Toggle quote (6 lines)
>
> Also note that check-inputs-should-not-be-an-input-at-all expects Guix
> package names (with python- prefix), whereas pypi-ignored-inputs uses
> Python package names, so you probably want to (map (cut (string-append
> "python-" <>) …) (or similar) here.

Perfectly right, thanks for noticing.

About pre-commit : if we add the python- prefix, then we use the
outdated variable. Maybe we should remove if pypi-ignored-inputs here,
and add "pre-commit" idependently.


Toggle quote (3 lines)
> Lars
>

Another quick note : The plugin could also be several plugins (one per
package) in the store added in the PYTHONPATH, which would enable us to
not regenerate the file everytime, but it doesn't compile with (guix
gexp) module. It might be more efficient / guixy this way, but I'm not
able to do it.

If you know how to, don't hesitate ;)

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 27 Apr 18:09 +0200
Re: [PATCH 2/2] guix: pyproject-build-system: Ignore unwanted pytest flags.
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 70570@debbugs.gnu.org)
877cgivlbw.fsf@ngraves.fr
On 2024-04-26 12:14, Nicolas Graves wrote:

Toggle quote (58 lines)
> On 2024-04-26 10:47, Lars-Dominik Braun wrote:
>
>> Hi,
>>
>> pretty smart idea to use a pytest plugin :)
>>
>>> +;; Pytest plugin to filter out arguments to ignore.
>>> +(define pytest-default-ignore-alist
>>
>> From the comment it’s not entirely clear to me what this list
>> does. It’s a map from pytest plugin name to it’s pytest command line
>> options, right?
>
> Right, I'll update the command.
>
>>
>>> +;; Allow guix to ignore these options when underlying pytest package is not
>>> +;; an input. These flags are not necessary to properly run tests.
>>> +(define (pytest-ignore-options-plugin flags)
>>> + "This function converts an list of flags in a string that can
>>> + be instantiated as a python pytest plugin."
>> …
>>> +(define (call-with-guix-pytest-plugin inputs thunk)
>>
>> Same here. As far as I understand you want to emulate command line
>> options provided by pytest plugins, so pytest won’t fail if the plugin
>> is not present. And we only do that if the plugin is not an input to
>> avoid clashing command line options, right?
>
> Will do.
>>
>> + for option in options:
>> + group.addoption(option, action='append', nargs='*')"
>>
>> Not sure nargs='*' is a good idea, since it might consume positional
>> arguments intended for pytest. '?' would be a more conservative option,
>> especially since we cannot override this easily per-package.
>
> It works. I'll try with '?' then!
>
>>
>>> + (let* ((former-path (getenv "PYTHONPATH"))
>> …
>>> + (dynamic-wind
>>> + (lambda ()
>>> + (setenv "PYTHONPATH"
>> …
>>> + (lambda ()
>>> + (setenv "PYTHONPATH" former-path)
>>
>> Isn’t it GUIX_PYTHONPATH?
>>
>
> I know it works this way. Could be GUIX_PYTHONPATH too, I'm not sure I
> properly get the difference. Can test with GUIX_PYTHONPATH too, but in
> any case it's just for the tests, so it should be cleared out
> too. Should I?

Actually, it doesn't seem to work with a replacement from PYTHONPATH to
GUIX_PYTHONPATH, so I would rather keep it this way.

Toggle quote (4 lines)
>
>> Lars
>>

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 27 Apr 18:54 +0200
[PATCH v2 1/2] guix: import: pypi: Ignore pypi-ignored-inputs.
(address . 70570@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240427165504.8843-1-ngraves@ngraves.fr
* guix/import/pypi.scm (pypi-ignored-inputs): New variable.
(compute-inputs): Use it.

* tests/pypi.scm (parse-requires.txt): Add ignored input to test the
feature.

* guix/lint.scm (check-inputs-should-be-native): Adapt list.
(check-inputs-should-not-be-an-input-at-all): Use pypi-ignored-list.

Change-Id: I774e526c5a090026e778ee44049637174a1dca95
---
guix/import/pypi.scm | 20 +++++++++++++++++---
guix/lint.scm | 12 +++++++-----
tests/pypi.scm | 3 ++-
3 files changed, 26 insertions(+), 9 deletions(-)

Toggle diff (118 lines)
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 6719fde330..a43d4eca1d 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2022 Vivien Kraus <vivien@planete-kraus.eu>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -61,6 +62,7 @@ (define-module (guix import pypi)
#:use-module (guix upstream)
#:use-module ((guix licenses) #:prefix license:)
#:export (%pypi-base-url
+ pypi-ignored-inputs
parse-requires.txt
parse-wheel-metadata
specification->requirement-name
@@ -77,6 +79,17 @@ (define %pypi-base-url
;; Base URL of the PyPI API.
(make-parameter "https://pypi.org/pypi/"))
+(define pypi-ignored-inputs
+ ;; This list contains packages that are useful for development or quality
+ ;; testing, but that most of the time are not necessary to have as an input.
+ (list "argparse" ; native
+ "codecov" "coverage" ; coverage
+ "black" "isort" "pycodestyle" "pep8" ; style
+ "pyflakes" "flake8" "pylint" "mypy" ; style+lint
+ "coveralls" "twine" ; upload integration tools
+ "pytest-isort" "pytest-flake8" "pytest-cov" "pytest-black"
+ "pytest-pep8" "pytest-mypy" "pytest-pep8" "pre-commit")) ; variants
+
(define non-empty-string-or-false
(match-lambda
("" #f)
@@ -424,9 +437,10 @@ (define (compute-inputs source-url wheel-url archive)
"Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, return
the corresponding list of <upstream-input> records."
(define (requirements->upstream-inputs deps type)
- (filter-map (match-lambda
- ("argparse" #f)
- (name (upstream-input
+ (filter-map (lambda (name)
+ (if (member name pypi-ignored-inputs)
+ #f
+ (upstream-input
(name name)
(downstream-name (python->package-name name))
(type type))))
diff --git a/guix/lint.scm b/guix/lint.scm
index 68d532968d..dd9bf0fb46 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -71,6 +71,7 @@ (define-module (guix lint)
hg-reference-url)
#:autoload (guix bzr-download) (bzr-reference?
bzr-reference-url)
+ #:use-module ((guix import pypi) #:select (pypi-ignored-inputs))
#:use-module (guix import stackage)
#:use-module (ice-9 match)
#:use-module (ice-9 regex)
@@ -557,14 +558,12 @@ (define (check-inputs-should-be-native package)
"m4"
"qttools-5"
"yasm" "nasm" "fasm"
- "python-coverage"
"python-cython"
"python-docutils"
"python-mock"
"python-nose"
"python-pbr"
"python-pytest"
- "python-pytest-cov"
"python-setuptools-scm"
"python-sphinx"
"scdoc"
@@ -586,9 +585,12 @@ (define (check-inputs-should-be-native package)
(define (check-inputs-should-not-be-an-input-at-all package)
;; Emit a warning if some inputs of PACKAGE are likely to should not be
;; an input at all.
- (let ((input-names '("python-setuptools"
- "python-pip"
- "python-pre-commit")))
+ (let ((input-names (append
+ '("python-setuptools"
+ "python-pip"
+ "pre-commit")
+ (map (cut (string-append "python-" <>))
+ pypi-ignored-inputs))))
(map (lambda (input)
(make-warning
package
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 42b39cde73..fe01ab3beb 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -97,6 +97,7 @@ (define test-requires.txt "\
[test]
pytest (>=2.5.0)
+pytest-cov # read but ignored
")
;; Beaker contains only optional dependencies.
@@ -244,7 +245,7 @@ (define-syntax-rule (with-pypi responses body ...)
(map specification->requirement-name test-specifications))
(test-equal "parse-requires.txt"
- (list '("foo" "bar") '("pytest"))
+ (list '("foo" "bar") '("pytest" "pytest-cov"))
(mock ((ice-9 ports) call-with-input-file
call-with-input-string)
(parse-requires.txt test-requires.txt)))
--
2.41.0
N
N
Nicolas Graves wrote on 27 Apr 18:54 +0200
[PATCH v2 2/2] guix: pyproject-build-system: Ignore unwanted pytest flags.
(address . 70570@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240427165504.8843-2-ngraves@ngraves.fr
* guix/build/pyproject-build-system.scm : Ignore unwanted pytest flags.

Change-Id: Ib9f1602e5af11227e5b7ce124f0f9be4fa2b78e4
---
guix/build/pyproject-build-system.scm | 91 ++++++++++++++++++++++++++-
1 file changed, 89 insertions(+), 2 deletions(-)

Toggle diff (122 lines)
diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
index 947d240114..ebe4e1941d 100644
--- a/guix/build/pyproject-build-system.scm
+++ b/guix/build/pyproject-build-system.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -142,7 +143,89 @@ (define* (build #:key outputs build-backend backend-path configure-flags #:allow
wheel-dir
config-settings)))
-(define* (check #:key tests? test-backend test-flags #:allow-other-keys)
+(define pytest-default-ignore-alist
+ '(("cov" . ("--cov" "--cov-reset" "--cov-report" "--cov-config"
+ "--no-cov-on-fail" "--no-cov" "--cov-fail-under"
+ "--cov-append" "--cov-branch" "--cov-context"))
+ ("mypy" . ("--mypy" "--mypy-config-file" "--mypy-ignore-missing-imports"))
+ ("isort" . ("--isort"))
+ ("flake8" . ("--flake8"))
+ ("black" . ("--black"))
+ ("flakes" . ("--flakes"))
+ ("pep8" . ("--pep8"))))
+
+(define (pytest-ignore-flags-plugin flags)
+ "This function converts an list of flags into a string that can
+ be instantiated as a python pytest plugin."
+ (format #f "\
+import pytest
+
+def pytest_addoption(parser):
+ group = parser.getgroup('guix','Guix ignored options')
+ options = [~{~s, ~}]
+ for option in options:
+ group.addoption(option, action='append', nargs='?')"
+ flags))
+
+(define (call-with-guix-pytest-plugin inputs thunk)
+ "This function emulates command line options provided by pytest plugins in
+the absence of the plugins defining these options.
+
+This is done by selecting absent plugins, gettings their flags defined in
+PYTEST-DEFAULT-IGNORE-ALIST, and generating the plugin from there with
+PYTEST-IGNORE-FLAGS-PLUGIN."
+ (let* ((former-path (getenv "PYTHONPATH"))
+ (input-names
+ (filter (match-lambda
+ (((name . _) ...)
+ (if (string-prefix? "python-pytest-" name)
+ name
+ #f))
+ ( _ #f))
+ inputs))
+ (filtered-flags
+ (filter identity
+ (append-map
+ (match-lambda
+ ((group . flags)
+ (if (member (string-append "python-pytest-" group)
+ input-names)
+ (list #f)
+ flags))
+ (_ (list #f)))
+ pytest-default-ignore-alist))))
+ (dynamic-wind
+ (lambda ()
+ (setenv "PYTHONPATH"
+ (string-append
+ (if former-path
+ (string-append former-path ":")
+ "")
+ ".guix-pytest"))
+ (setenv "PYTEST_PLUGINS"
+ (string-append
+ (if (getenv "PYTEST_PLUGINS")
+ (string-append former-path ",")
+ "")
+ "pytest_guix_plugin"))
+ (mkdir-p ".guix-pytest")
+ (with-output-to-file ".guix-pytest/__init__.py"
+ (lambda _ (display "")))
+ (with-output-to-file ".guix-pytest/pytest_guix_plugin.py"
+ (lambda _
+ (display (pytest-ignore-flags-plugin filtered-flags)))))
+ thunk
+ (lambda ()
+ (setenv "PYTHONPATH" former-path)
+ (unsetenv "PYTEST_PLUGINS")
+ (when (file-exists? ".guix-pytest")
+ (delete-file-recursively ".guix-pytest"))))))
+
+(define-syntax-rule (with-guix-pytest-plugin inputs exp ...)
+ "Evaluate EXP in a context where the Guix pytest plugin is added."
+ (call-with-guix-pytest-plugin inputs (lambda () exp ...)))
+
+(define* (check #:key inputs tests? test-backend test-flags #:allow-other-keys)
"Run the test suite of a given Python package."
(if tests?
;; Unfortunately with PEP 517 there is no common method to specify test
@@ -165,7 +248,8 @@ (define* (check #:key tests? test-backend test-flags #:allow-other-keys)
(format #t "Using ~a~%" use-test-backend)
(match use-test-backend
('pytest
- (apply invoke pytest "-vv" test-flags))
+ (with-guix-pytest-plugin inputs
+ (apply invoke pytest "-vv" test-flags)))
('nose
(apply invoke nosetests "-v" test-flags))
('nose2
@@ -386,3 +470,6 @@ (define* (pyproject-build #:key inputs (phases %standard-phases)
(apply python:python-build #:inputs inputs #:phases phases args))
;;; pyproject-build-system.scm ends here
+;;; Local Variables:
+;;; eval: (put 'with-guix-pytest-plugin 'scheme-indent-function 1)
+;;; End:
--
2.41.0
N
N
Nicolas Graves wrote on 28 Apr 12:14 +0200
(address . 70570@debbugs.gnu.org)
874jblvlnx.fsf@ngraves.fr
On 2024-04-27 18:54, Nicolas Graves wrote:

Toggle quote (65 lines)
> * guix/build/pyproject-build-system.scm : Ignore unwanted pytest flags.
>
> Change-Id: Ib9f1602e5af11227e5b7ce124f0f9be4fa2b78e4
> ---
> guix/build/pyproject-build-system.scm | 91 ++++++++++++++++++++++++++-
> 1 file changed, 89 insertions(+), 2 deletions(-)
>
> diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
> index 947d240114..ebe4e1941d 100644
> --- a/guix/build/pyproject-build-system.scm
> +++ b/guix/build/pyproject-build-system.scm
> @@ -1,6 +1,7 @@
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
> ;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
> +;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -142,7 +143,89 @@ (define* (build #:key outputs build-backend backend-path configure-flags #:allow
> wheel-dir
> config-settings)))
>
> -(define* (check #:key tests? test-backend test-flags #:allow-other-keys)
> +(define pytest-default-ignore-alist
> + '(("cov" . ("--cov" "--cov-reset" "--cov-report" "--cov-config"
> + "--no-cov-on-fail" "--no-cov" "--cov-fail-under"
> + "--cov-append" "--cov-branch" "--cov-context"))
> + ("mypy" . ("--mypy" "--mypy-config-file" "--mypy-ignore-missing-imports"))
> + ("isort" . ("--isort"))
> + ("flake8" . ("--flake8"))
> + ("black" . ("--black"))
> + ("flakes" . ("--flakes"))
> + ("pep8" . ("--pep8"))))
> +
> +(define (pytest-ignore-flags-plugin flags)
> + "This function converts an list of flags into a string that can
> + be instantiated as a python pytest plugin."
> + (format #f "\
> +import pytest
> +
> +def pytest_addoption(parser):
> + group = parser.getgroup('guix','Guix ignored options')
> + options = [~{~s, ~}]
> + for option in options:
> + group.addoption(option, action='append', nargs='?')"
> + flags))
> +
> +(define (call-with-guix-pytest-plugin inputs thunk)
> + "This function emulates command line options provided by pytest plugins in
> +the absence of the plugins defining these options.
> +
> +This is done by selecting absent plugins, gettings their flags defined in
> +PYTEST-DEFAULT-IGNORE-ALIST, and generating the plugin from there with
> +PYTEST-IGNORE-FLAGS-PLUGIN."
> + (let* ((former-path (getenv "PYTHONPATH"))
> + (input-names
> + (filter (match-lambda
> + (((name . _) ...)
> + (if (string-prefix? "python-pytest-" name)
> + name
> + #f))
> + ( _ #f))
> + inputs))

This filter is not working properly as it doesn't output names but
pairs. Will be changed in the next revision with a map car.

Toggle quote (20 lines)
> + (filtered-flags
> + (filter identity
> + (append-map
> + (match-lambda
> + ((group . flags)
> + (if (member (string-append "python-pytest-" group)
> + input-names)
> + (list #f)
> + flags))
> + (_ (list #f)))
> + pytest-default-ignore-alist))))
> + (dynamic-wind
> + (lambda ()
> + (setenv "PYTHONPATH"
> + (string-append
> + (if former-path
> + (string-append former-path ":")
> + "")
> + ".guix-pytest"))

I found that in some edge cases, a hidden directory in the current dir
can cause tests to fail. It only happened for one package which was
failing on the __init__.py from .guix-pytest, meaning that tests could
consider this directory.

So it's better to put that out-of-source in ../.guix-pytest. Will be
done in the next revision.

Toggle quote (45 lines)
> + (setenv "PYTEST_PLUGINS"
> + (string-append
> + (if (getenv "PYTEST_PLUGINS")
> + (string-append former-path ",")
> + "")
> + "pytest_guix_plugin"))
> + (mkdir-p ".guix-pytest")
> + (with-output-to-file ".guix-pytest/__init__.py"
> + (lambda _ (display "")))
> + (with-output-to-file ".guix-pytest/pytest_guix_plugin.py"
> + (lambda _
> + (display (pytest-ignore-flags-plugin filtered-flags)))))
> + thunk
> + (lambda ()
> + (setenv "PYTHONPATH" former-path)
> + (unsetenv "PYTEST_PLUGINS")
> + (when (file-exists? ".guix-pytest")
> + (delete-file-recursively ".guix-pytest"))))))
> +
> +(define-syntax-rule (with-guix-pytest-plugin inputs exp ...)
> + "Evaluate EXP in a context where the Guix pytest plugin is added."
> + (call-with-guix-pytest-plugin inputs (lambda () exp ...)))
> +
> +(define* (check #:key inputs tests? test-backend test-flags #:allow-other-keys)
> "Run the test suite of a given Python package."
> (if tests?
> ;; Unfortunately with PEP 517 there is no common method to specify test
> @@ -165,7 +248,8 @@ (define* (check #:key tests? test-backend test-flags #:allow-other-keys)
> (format #t "Using ~a~%" use-test-backend)
> (match use-test-backend
> ('pytest
> - (apply invoke pytest "-vv" test-flags))
> + (with-guix-pytest-plugin inputs
> + (apply invoke pytest "-vv" test-flags)))
> ('nose
> (apply invoke nosetests "-v" test-flags))
> ('nose2
> @@ -386,3 +470,6 @@ (define* (pyproject-build #:key inputs (phases %standard-phases)
> (apply python:python-build #:inputs inputs #:phases phases args))
>
> ;;; pyproject-build-system.scm ends here
> +;;; Local Variables:
> +;;; eval: (put 'with-guix-pytest-plugin 'scheme-indent-function 1)
> +;;; End:

--
Best regards,
Nicolas Graves
?