[PATCH 0/6] Update Scrapy.

  • Done
  • quality assurance status badge
Details
2 participants
  • Christopher Baines
  • Vinicius Monego
Owner
unassigned
Submitted by
Vinicius Monego
Severity
normal
V
V
Vinicius Monego wrote on 6 Aug 2023 17:31
(address . guix-patches@gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153159.285234-1-monego@posteo.net
Vinicius Monego (6):
gnu: python-w3lib: Update to 2.1.2.
gnu: python-cssselect: Update to 1.2.0.
gnu: python-parsel: Update to 1.8.1.
gnu: python-itemadapter: Update to 0.8.0.
gnu: python-itemloaders: Update to 1.1.0.
gnu: python-scrapy: Update to 2.10.0.

gnu/local.mk | 1 -
.../python-w3lib-fix-test-failure.patch | 60 --------------
gnu/packages/python-web.scm | 80 +++++++------------
gnu/packages/python-xyz.scm | 13 +--
4 files changed, 38 insertions(+), 116 deletions(-)
delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 1/6] gnu: python-w3lib: Update to 2.1.2.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-1-monego@posteo.net
* gnu/packages/python-web.scm (python-w3lib): Update to 2.1.2.
[source]: Remove patch.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
[native-inputs]: Remove python-six.
* gnu/packages/patches/python-w3lib-fix-test-failure.patch: Delete file.
* gnu/local.mk (dist_patch_DATA): Remove it.
---
gnu/local.mk | 1 -
.../python-w3lib-fix-test-failure.patch | 60 -------------------
gnu/packages/python-web.scm | 16 ++---
3 files changed, 4 insertions(+), 73 deletions(-)
delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

Toggle diff (60 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 77707127a7..e091b310c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1730,7 +1730,6 @@ dist_patch_DATA = \
%D%/packages/patches/python-shiboken-2-compat.patch \
%D%/packages/patches/python-telingo-fix-comparison.patch \
%D%/packages/patches/python-typeguard-python3.10.patch \
- %D%/packages/patches/python-w3lib-fix-test-failure.patch \
%D%/packages/patches/python-wxwidgets-type-errors.patch \
%D%/packages/patches/quodlibet-fix-invalid-glob.patch \
%D%/packages/patches/quodlibet-fix-mtime-tests.patch \
diff --git a/gnu/packages/patches/python-w3lib-fix-test-failure.patch b/gnu/packages/patches/python-w3lib-fix-test-failure.patch
deleted file mode 100644
index f38fc749df..0000000000
--- a/gnu/packages/patches/python-w3lib-fix-test-failure.patch
+++ /dev/null
@@ -1,60 +0,0 @@
-From fae6cc40e112cd13697cb0e8d79976f32c72491d Mon Sep 17 00:00:00 2001
-From: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
-Date: Wed, 10 Mar 2021 12:31:05 -0300
-Subject: [PATCH] [CI] Mark single add_or_replace_parameter test as xfail
-
----
-This patch is based on upstream commit
-fae6cc40e112cd13697cb0e8d79976f32c72491d which is part of
-https://github.com/scrapy/w3lib/pull/166.
-
- .gitignore | 3 ++-
- tests/test_url.py | 25 ++++++++++++++-----------
- 2 files changed, 16 insertions(+), 12 deletions(-)
-
-diff --git a/tests/test_url.py b/tests/test_url.py
-index 8b07c00..0f7458e 100644
---- a/tests/test_url.py
-+++ b/tests/test_url.py
-@@ -1,12 +1,14 @@
--# -*- coding: utf-8 -*-
- from __future__ import absolute_import
- import os
- import unittest
-+
-+import pytest
-+from six.moves.urllib.parse import urlparse
-+
- from w3lib.url import (is_url, safe_url_string, safe_download_url,
- url_query_parameter, add_or_replace_parameter, url_query_cleaner,
- file_uri_to_path, parse_data_uri, path_to_file_uri, any_to_uri,
- urljoin_rfc, canonicalize_url, parse_url, add_or_replace_parameters)
--from six.moves.urllib.parse import urlparse
-
-
- class UrlTests(unittest.TestCase):
-@@ -310,10 +311,6 @@ def test_add_or_replace_parameter(self):
- self.assertEqual(add_or_replace_parameter(url, 'arg3', 'nv3'),
- 'http://domain/test?arg1=v1&arg2=v2&arg3=nv3')
-
-- url = 'http://domain/test?arg1=v1;arg2=v2'
-- self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
-- 'http://domain/test?arg1=v3&arg2=v2')
--
- self.assertEqual(add_or_replace_parameter("http://domain/moreInfo.asp?prodID=",'prodID', '20'),
-@@ -338,6 +335,13 @@ def test_add_or_replace_parameter(self):
- self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
-
-+ @pytest.mark.xfail(reason="https://github.com/scrapy/w3lib/issues/164")
-+ def test_add_or_replace_parameter_fail(self):
-+ self.assertEqual(
-+ add_or_replace_parameter('http://domain/test?arg1=v1;arg2=v2','arg1', 'v3'),
-+ )
-+
- def test_add_or_replace_parameters(self):
- self.assertEqual(add_or_replace_parameters(url, {'arg': 'v'}),
Toggle diff (36 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 17c9563cfd..24071a8553 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -7752,25 +7752,17 @@ (define-public python-smart-open
(define-public python-w3lib
(package
(name "python-w3lib")
- (version "1.22.0")
+ (version "2.1.2")
(source
(origin
(method url-fetch)
(uri (pypi-uri "w3lib" version))
- (patches (search-patches "python-w3lib-fix-test-failure.patch"))
(sha256
(base32
- "1pv02lvvmgz2qb61vz1jkjc04fgm4hpfvaj5zm4i3mjp64hd1mha"))))
- (build-system python-build-system)
- (arguments
- `(#:phases
- (modify-phases %standard-phases
- (replace 'check
- (lambda* (#:key tests? #:allow-other-keys)
- (when tests?
- (invoke "pytest")))))))
+ "1cd4b3w5g3pfccsg79kjj27fwi216ip927rjq7isp8pfjzlp8nzd"))))
+ (build-system pyproject-build-system)
(native-inputs
- (list python-pytest python-six))
+ (list python-pytest))
(home-page "https://github.com/scrapy/w3lib")
(synopsis "Python library of web-related functions")
(description
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 2/6] gnu: python-cssselect: Update to 1.2.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-2-monego@posteo.net
* gnu/packages/python-web.scm (python-cssselect): Update to 1.2.0.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
---
gnu/packages/python-web.scm | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)

Toggle diff (31 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 24071a8553..9ecd9f35a3 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2095,7 +2095,7 @@ (define-public python-css-parser
(define-public python-cssselect
(package
(name "python-cssselect")
- (version "1.1.0")
+ (version "1.2.0")
(source (origin
;; The PyPI release does not contain tests.
(method git-fetch)
@@ -2105,13 +2105,8 @@ (define-public python-cssselect
(file-name (git-file-name name version))
(sha256
(base32
- "0xslrnhbrmgakp4xg6k26qffay3kqffp3a2z2sk27c65rwxa79kc"))))
- (build-system python-build-system)
- (arguments
- `(#:phases (modify-phases %standard-phases
- (replace 'check
- (lambda _
- (invoke "pytest" "-vv"))))))
+ "1x4nrvb1p1byi1whmspik7lbh303akdlh762dayfxam3hycsh5kk"))))
+ (build-system pyproject-build-system)
(native-inputs
(list python-lxml python-pytest))
(home-page "https://github.com/scrapy/cssselect")
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 3/6] gnu: python-parsel: Update to 1.8.1.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-3-monego@posteo.net
* gnu/packages/python-web.scm (python-parsel): Update to 1.8.1.
[build-system]: Use pyproject-build-system.
[propagated-inputs]: Add python-jmespath, python-typing-extensions. Remove
python-six.
[native-inputs]: Add python-psutil. Remove python-pytest-runner.
---
gnu/packages/python-web.scm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)

Toggle diff (34 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 9ecd9f35a3..70a1ab082f 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8039,21 +8039,22 @@ (define-public python-protego
(define-public python-parsel
(package
(name "python-parsel")
- (version "1.6.0")
+ (version "1.8.1")
(source
(origin
(method url-fetch)
(uri (pypi-uri "parsel" version))
(sha256
- (base32 "0yawf9r3r863lwxj0n89i7h3n8xjbsl5b7n6xg76r68scl5yzvvh"))))
- (build-system python-build-system)
+ (base32 "0f8yh30y3961a7kqwcnp4j3s7044ilakykiavc0skwdkr5l8xwmg"))))
+ (build-system pyproject-build-system)
(propagated-inputs
(list python-cssselect
+ python-jmespath
python-lxml
- python-six
+ python-typing-extensions
python-w3lib))
(native-inputs
- (list python-pytest python-pytest-runner))
+ (list python-psutil python-pytest))
(home-page "https://github.com/scrapy/parsel")
(synopsis "Extract data from HTML and XML using XPath and CSS selectors")
(description "Parsel is a library to extract and remove data from
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 4/6] gnu: python-itemadapter: Update to 0.8.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-4-monego@posteo.net
* gnu/packages/python-xyz.scm (python-itemadapter): Update to 0.8.0.
---
gnu/packages/python-xyz.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Toggle diff (22 lines)
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 884c37e975..e6cbe7d333 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32504,13 +32504,13 @@ (define-public python-posix-ipc
(define-public python-itemadapter
(package
(name "python-itemadapter")
- (version "0.5.0")
+ (version "0.8.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "itemadapter" version))
(sha256
- (base32 "083wp3h2brh8x19jbdr8rz3biqwp3jlqd0rfzcyrjyhssffsgdh5"))))
+ (base32 "1aa898gjgwy3axxfrgsh4kdvhp6n6wz3ccdishq0gh8azf2q8xbp"))))
(build-system python-build-system)
(home-page "https://github.com/scrapy/itemadapter")
(synopsis "Common interface for data container classes")
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 5/6] gnu: python-itemloaders: Update to 1.1.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-5-monego@posteo.net
* gnu/packages/python-xyz.scm (python-itemloaders): Update to 1.1.0.
[build-system]: Use pyproject-build-system.
[native-inputs]: Add python-pytest.
---
gnu/packages/python-xyz.scm | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)

Toggle diff (28 lines)
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index e6cbe7d333..c7bce1b135 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32534,16 +32534,17 @@ (define-public python-itemadapter
(define-public python-itemloaders
(package
(name "python-itemloaders")
- (version "1.0.4")
+ (version "1.1.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "itemloaders" version))
(sha256
- (base32 "15hc78h90qhwass1bga1c3xar2dd6j8sxg61zg6jvh74lf6csxqj"))))
- (build-system python-build-system)
+ (base32 "0j2aw4ipalj208594x80blpgkh1i63gqqa4nb67b823av9hirn11"))))
+ (build-system pyproject-build-system)
+ (native-inputs (list python-pytest))
(propagated-inputs
- (list python-itemadapter python-jmespath python-parsel python-w3lib))
+ (list python-itemadapter python-jmespath python-parsel python-w3lib))
(home-page "https://github.com/scrapy/itemloaders")
(synopsis "Base library for scrapy's ItemLoader")
(description "Itemloaders is a library that helps you collect data
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 6/6] gnu: python-scrapy: Update to 2.10.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-6-monego@posteo.net
* gnu/packages/python-web.scm (python-scrapy): Update to 2.10.0.
[build-system]: Use pyproject-build-system.
[arguments]: Use G-Expressions. Do not override the check phase, skip tests
in #:test-flags instead. Run tests sequentially and skip fewer tests.
---
gnu/packages/python-web.scm | 42 ++++++++++++++++---------------------
1 file changed, 18 insertions(+), 24 deletions(-)

Toggle diff (61 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 70a1ab082f..c12740b587 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8065,36 +8065,30 @@ (define-public python-parsel
(define-public python-scrapy
(package
(name "python-scrapy")
- (version "2.7.1")
+ (version "2.10.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "Scrapy" version))
(sha256
- (base32 "0kpi3hg2ycs6s8cg41r2zc1axd0rpnps8bnzg7wisjyjaf1l1yih"))))
- (build-system python-build-system)
+ (base32 "0fdf1kx39zy7b0v5gkwwqvm58ww63z5bvm78yjymhfn0r0zs65jf"))))
+ (build-system pyproject-build-system)
(arguments
- `(#:phases
- (modify-phases %standard-phases
- (replace 'check
- (lambda* (#:key tests? #:allow-other-keys)
- (when tests?
- (invoke "pytest"
- "-n" (number->string (parallel-job-count))
- ;; These tests fail when run in parallel (see:
- ;; https://github.com/scrapy/scrapy/issues/5502).
- "--ignore" "tests/test_engine.py"
- "--ignore" "tests/test_engine_stop_download_bytes.py"
- "--ignore" "tests/test_engine_stop_download_headers.py"
- ;; This test require network access.
- "--ignore" "tests/test_command_check.py"
- "-k"
- (string-append
- ;; The followin tests fail for unknown reasons.
- "not test_server_set_cookie_domain_suffix_public_private"
- " and not test_user_set_cookie_domain_suffix_public_private"
- " and not test_pformat")
- "tests")))))))
+ (list #:test-flags
+ ;; Tests fail with DNS lookup or need a display.
+ #~(list "-k" (string-append
+ "not test_SCRAPY_CHECK_set"
+ " and not "
+ (string-join
+ (list "test_check_all_default_contracts"
+ "test_check_cb_kwargs_contract"
+ "test_check_returns_items_contract"
+ "test_check_returns_requests_contract"
+ "test_check_scrapes_contract"
+ "test_pformat"
+ "test_pformat_old_windows"
+ "test_pformat_windows")
+ " and not ")))))
(propagated-inputs
(list python-botocore ; Optional: For S3FeedStorage class.
python-cryptography
--
2.34.1
C
C
Christopher Baines wrote on 27 Aug 2023 20:14
Re: [bug#65114] [PATCH 0/6] Update Scrapy.
(name . Vinicius Monego)(address . monego@posteo.net)
87h6okz70v.fsf@cbaines.net
Vinicius Monego <monego@posteo.net> writes:

Toggle quote (15 lines)
> Vinicius Monego (6):
> gnu: python-w3lib: Update to 2.1.2.
> gnu: python-cssselect: Update to 1.2.0.
> gnu: python-parsel: Update to 1.8.1.
> gnu: python-itemadapter: Update to 0.8.0.
> gnu: python-itemloaders: Update to 1.1.0.
> gnu: python-scrapy: Update to 2.10.0.
>
> gnu/local.mk | 1 -
> .../python-w3lib-fix-test-failure.patch | 60 --------------
> gnu/packages/python-web.scm | 80 +++++++------------
> gnu/packages/python-xyz.scm | 13 +--
> 4 files changed, 38 insertions(+), 116 deletions(-)
> delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

These changes look good to me +1

Thanks,

Chris
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmTrkrBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdYKQ/7B6MDvufDf29Ihn44D3wUgz6R/SN97/a4
cosAFyyW+9ZQ++P3BvgjceIG2b0y5KMeR1fH1TEOJfN/gF/ZCwlWuY9KVBVWtBIl
whyhAjHysxlA4ys79FOY152OEnQUvSlg0prCBLShVfgbyp0xmV/DpXZ9DW4OEOCH
XVjfiYvtJfP0ykeZi3ugkwATJQxWOF0pO4FN0hI55Oemz90TI3/WRwE02QgHCfKk
6tGCO/8flpQ2NtVD8owYM+5QKWkn5a/t5vJZRHFEgheYSFXUDsP1dCLXDFO2H2Yt
r9Cxsx+41Vo7WGKtsSrRVRPr4WsxnjwdD7wo5W35L5qPW8DpXmzOvm8Wlc0SnVmw
Jx5Cx40SzIfcFcRNDRfcuSVIO0VdkjduEF3ezPmA6HJ/e6dbmoyAdN7mnmsR2rwK
iaViiyhvjeHi6/GjPVEtGmOHF9pxd8z6DInELiJNu0vJ2FbAuPV9VPjKmzoQfEsA
HBW6lLPhkp7XAjdhODrQ7qj6MDEkpA7YgUPmmClVKLIIPA0tcO0sD/ktlsqvHPOq
pfrzVOyD77cKiy3GNTCEx0A0qXv+JlfoxbJiswLOnm975P9iXTMmkT6NE4KLEu12
cFtFIKM8i/Q54HxB9esl2DOph6D5kqryzi5Old1bR+QwWRq4DrNCzX1o4osLr3+l
pFT/GEN/IaQ=
=k50B
-----END PGP SIGNATURE-----

V
V
Vinicius Monego wrote on 2 Sep 2023 19:59
(name . Christopher Baines)(address . mail@cbaines.net)(address . 65114-done@debbugs.gnu.org)
8f624fb52caf9169f71db63a228555a87304ceda.camel@posteo.net
Em dom, 2023-08-27 às 19:14 +0100, Christopher Baines escreveu:


[...]

Toggle quote (3 lines)
> These changes look good to me +1
>

[...]

Applied with a minor update to Scrapy, thanks.

Vinicius
Closed
?