OpenSSL 1.1.1n test failures due to expired certificates (time bomb)

  • Open
  • quality assurance status badge
Details
5 participants
  • Ludovic Courtès
  • Maxim Cournoyer
  • Maxime Devos
  • Sjors Provoost
  • zimoun
Owner
unassigned
Submitted by
Sjors Provoost
Severity
important
Merged with
S
S
Sjors Provoost wrote on 19 Oct 2022 21:46
build of /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv failed
(address . bug-guix@gnu.org)
1DA1DE12-F6F8-466C-A81F-1823B6626F65@sprovoost.nl
Sorry if this is a duplicate or has already been fixed in a more recent commit.

/builder for `/gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv' failed with exit code 1
build of /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv failed
View build log at '/var/log/guix/drvs/mw/6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv.gz'.
cannot build derivation `/gnu/store/236k6ncjl0nf7bqv4j0hni8i4yib3la4-git-minimal-2.36.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/gd577lh9007s0687m56fn65n8hrsjiqf-mallard-ducktype-1.0.2-checkout.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/rvj5bx06w2kjlxm3fg5p88dkxb6n8v9p-openjpeg-data-2020.11.30-checkout.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/00p96drllzndfp7zr63y26n1d64bdjwl-mallard-ducktype-1.0.2.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kz4g88f4jv0w75qibq74q5lmpkgpl894-openjpeg-data-2020.11.30.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/izf75k3gvz0x6399qiks1drps445ykpg-openjpeg-2.4.0.drv': 1 dependencies couldn't be built
Backtrace:
14 (primitive-load "/gnu/store/wkw084zcvkyj53acs1gkchnvp0m7bvbl-compute-guix-derivation")
In ice-9/eval.scm:
155:9 13 (_ _)
159:9 12 (_ #(#(#(#(#(#(#(#(#(#(#(#(#(#(#(#(#<directory (guile-u?> ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?))
In ice-9/boot-9.scm:
152:2 11 (with-fluid* _ _ _)
152:2 10 (with-fluid* _ _ _)
In ./guix/store.scm:
2129:24 9 (run-with-store #<store-connection 256.99 7fbb6af39140> #<procedure 7fbb55577a50 at ./guix/self.scm:12?> ?)
1966:8 8 (_ #<store-connection 256.99 7fbb6af39140>)
In ./guix/gexp.scm:
300:22 7 (_ #<store-connection 256.99 7fbb6af39140>)
1181:2 6 (_ #<store-connection 256.99 7fbb6a984690>)
1047:2 5 (_ #<store-connection 256.99 7fbb6a984690>)
893:4 4 (_ #<store-connection 256.99 7fbb6a984690>)
In ./guix/store.scm:
2014:12 3 (_ #<store-connection 256.99 7fbb6a984690>)
1406:5 2 (map/accumulate-builds #<store-connection 256.99 7fbb6a984690> #<procedure 7fbb5d369580 at ./guix/stor?> ?)
1421:15 1 (_ #<store-connection 256.99 7fbb6a984690> ("/gnu/store/gcvv1i5shqmkd6x1pjwjdrvr7z4lb5ss-guile-ssh-?" ?) ?)
1421:15 0 (loop #f)

./guix/store.scm:1421:15: In procedure loop:
ERROR:
1. &store-protocol-error:
message: "build of `/gnu/store/gwqx9mq7ll5ic97zvz22j9irlx2922wx-graphviz-2.49.0.drv' failed"
status: 100
guix pull: error: You found a bug: the program '/gnu/store/wkw084zcvkyj53acs1gkchnvp0m7bvbl-compute-guix-derivation'
failed to compute the derivation for Guix (version: "998eda3067c7d21e0d9bb3310d2f5a14b8f1c681"; system: "x86_64-linux";
host version: "1.3.0.18313-998eda"; pull-version: 1).

- Sjors
Attachment: 6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv.gz

Z
Z
zimoun wrote on 3 Nov 2022 11:03
(name . Sjors Provoost)(address . sjors@sprovoost.nl)(address . 58650@debbugs.gnu.org)
86iljwbbj3.fsf@gmail.com
Hi,

Thanks for the report.

On Wed, 19 Oct 2022 at 21:46, Sjors Provoost <sjors@sprovoost.nl> wrote:
Toggle quote (42 lines)
> Sorry if this is a duplicate or has already been fixed in a more recent commit.
>
> /builder for `/gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv' failed with exit code 1
> build of /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv failed
> View build log at '/var/log/guix/drvs/mw/6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv.gz'.
> cannot build derivation `/gnu/store/236k6ncjl0nf7bqv4j0hni8i4yib3la4-git-minimal-2.36.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/gd577lh9007s0687m56fn65n8hrsjiqf-mallard-ducktype-1.0.2-checkout.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/rvj5bx06w2kjlxm3fg5p88dkxb6n8v9p-openjpeg-data-2020.11.30-checkout.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/00p96drllzndfp7zr63y26n1d64bdjwl-mallard-ducktype-1.0.2.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/kz4g88f4jv0w75qibq74q5lmpkgpl894-openjpeg-data-2020.11.30.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/izf75k3gvz0x6399qiks1drps445ykpg-openjpeg-2.4.0.drv': 1 dependencies couldn't be built
> Backtrace:
> 14 (primitive-load "/gnu/store/wkw084zcvkyj53acs1gkchnvp0m7bvbl-compute-guix-derivation")
> In ice-9/eval.scm:
> 155:9 13 (_ _)
> 159:9 12 (_ #(#(#(#(#(#(#(#(#(#(#(#(#(#(#(#(#<directory (guile-u?> ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?))
> In ice-9/boot-9.scm:
> 152:2 11 (with-fluid* _ _ _)
> 152:2 10 (with-fluid* _ _ _)
> In ./guix/store.scm:
> 2129:24 9 (run-with-store #<store-connection 256.99 7fbb6af39140> #<procedure 7fbb55577a50 at ./guix/self.scm:12?> ?)
> 1966:8 8 (_ #<store-connection 256.99 7fbb6af39140>)
> In ./guix/gexp.scm:
> 300:22 7 (_ #<store-connection 256.99 7fbb6af39140>)
> 1181:2 6 (_ #<store-connection 256.99 7fbb6a984690>)
> 1047:2 5 (_ #<store-connection 256.99 7fbb6a984690>)
> 893:4 4 (_ #<store-connection 256.99 7fbb6a984690>)
> In ./guix/store.scm:
> 2014:12 3 (_ #<store-connection 256.99 7fbb6a984690>)
> 1406:5 2 (map/accumulate-builds #<store-connection 256.99 7fbb6a984690> #<procedure 7fbb5d369580 at ./guix/stor?> ?)
> 1421:15 1 (_ #<store-connection 256.99 7fbb6a984690> ("/gnu/store/gcvv1i5shqmkd6x1pjwjdrvr7z4lb5ss-guile-ssh-?" ?) ?)
> 1421:15 0 (loop #f)
>
> ./guix/store.scm:1421:15: In procedure loop:
> ERROR:
> 1. &store-protocol-error:
> message: "build of `/gnu/store/gwqx9mq7ll5ic97zvz22j9irlx2922wx-graphviz-2.49.0.drv' failed"
> status: 100
> guix pull: error: You found a bug: the program '/gnu/store/wkw084zcvkyj53acs1gkchnvp0m7bvbl-compute-guix-derivation'
> failed to compute the derivation for Guix (version: "998eda3067c7d21e0d9bb3310d2f5a14b8f1c681"; system: "x86_64-linux";
> host version: "1.3.0.18313-998eda"; pull-version: 1).

It seems an error with the store. Do you use the offload mechanism?
And have you allowed the substitutes?


Cheers,
simon
M
M
Maxime Devos wrote on 3 Nov 2022 11:32
(address . 58650@debbugs.gnu.org)
bfdb1591-d922-93d6-b2f8-12cd500925ca@telenet.be
On 03-11-2022 11:03, zimoun wrote:
Toggle quote (16 lines)
> Hi,
>
> Thanks for the report.
>
> On Wed, 19 Oct 2022 at 21:46, Sjors Provoost <sjors@sprovoost.nl> wrote:
>> Sorry if this is a duplicate or has already been fixed in a more recent commit.
>>
>> /builder for `/gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv' failed with exit code 1
>> build of /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv failed
>> View build log at '/var/log/guix/drvs/mw/6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv.gz'.
>> [...]
>>
>> ./guix/store.scm:1421:15: In procedure loop: [...]1).
>
> It seems an error with the store. Do you use the offload mechanism?
> And have you allowed the substitutes?
Looking at the attached build log, it is a build failure, not some store
error:
Test Summary Report
-------------------
../test/recipes/80-test_ssl_new.t (Wstat: 256 Tests: 29
Failed: 1)
Failed test: 12
Non-zero exit status: 1
Files=158, Tests=2640, 66 wallclock secs ( 0.87 usr 0.07 sys + 56.47
cusr 7.90 csys = 65.31 CPU)
Result: FAIL
make[1]: *** [Makefile:208: _tests] Error 1
make[1]: Leaving directory
'/tmp/guix-build-openssl-1.1.1n.drv-0/openssl-1.1.1n'
make: *** [Makefile:205: tests] Error 2
Except for the different version number IIRC, I've noticed that one
before (on core-updates). That was without offloading and with
substitutes, though the substitute servers didn't have a substitute
available.
As the backtrace is a distraction, I propose merging something like
Greetings,
Maxime
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
S
S
Sjors Provoost wrote on 3 Nov 2022 12:07
(address . 58650@debbugs.gnu.org)
93EB0CFC-82DA-4858-A477-EA7480BD29FD@sprovoost.nl
I built using --no-substitutes and no offloading.
Z
Z
zimoun wrote on 3 Nov 2022 12:03
(address . 58650@debbugs.gnu.org)
86cza4b8q7.fsf@gmail.com
Hi,

On Thu, 03 Nov 2022 at 11:32, Maxime Devos <maximedevos@telenet.be> wrote:

Toggle quote (17 lines)
> Looking at the attached build log, it is a build failure, not some store
> error:
>
> Test Summary Report
> -------------------
> ../test/recipes/80-test_ssl_new.t (Wstat: 256 Tests: 29
> Failed: 1)
> Failed test: 12
> Non-zero exit status: 1
> Files=158, Tests=2640, 66 wallclock secs ( 0.87 usr 0.07 sys + 56.47
> cusr 7.90 csys = 65.31 CPU)
> Result: FAIL
> make[1]: *** [Makefile:208: _tests] Error 1
> make[1]: Leaving directory
> '/tmp/guix-build-openssl-1.1.1n.drv-0/openssl-1.1.1n'
> make: *** [Makefile:205: tests] Error 2

Indeed. My bad, I have missed the attachment.

Well, looking closer, I am confused by:

Toggle snippet (4 lines)
failed to compute the derivation for Guix (version: "998eda3067c7d21e0d9bb3310d2f5a14b8f1c681"; system:
"x86_64-linux"; host version: "1.3.0.18313-998eda"; pull-version: 1).

What is this host version?


Toggle quote (3 lines)
> As the backtrace is a distraction, I propose merging something like
> <https://issues.guix.gnu.org/50238>.

Well, I do not know if it is related, although patch#50238 would help
for sure.

Cheers,
simon
S
S
Sjors Provoost wrote on 3 Nov 2022 12:25
(address . 58650@debbugs.gnu.org)
BFBDEDEF-9EFA-4625-A773-A9A00DFA5CD6@sprovoost.nl
I tried building again using:
guix build --cores=1 /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv

This made it more clear that the error was an expired certificate:

../test/recipes/80-test_ssl_new.t ..................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/29 subtests

I was able to work around that by adjusting the machine time:

sudo timedatectl set-ntp no
sudo date --set "28 may 2022 15:00:00"
guix build ....
sudo timedatectl set-ntp yes
M
M
Maxime Devos wrote on 3 Nov 2022 12:32
7ea63efb-e8a3-f94f-b24d-0fb1493e3a69@telenet.be
reopen 56137
merge 56137 58650
thanks
On 03-11-2022 12:25, Sjors Provoost wrote:
Toggle quote (15 lines)
> I tried building again using:
> guix build --cores=1 /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv
>
> This made it more clear that the error was an expired certificate:
>
> ../test/recipes/80-test_ssl_new.t ..................
> Dubious, test returned 1 (wstat 256, 0x100)
> Failed 1/29 subtests
>
> I was able to work around that by adjusting the machine time:
>
> sudo timedatectl set-ntp no
> sudo date --set "28 may 2022 15:00:00"
> guix build ....
> sudo timedatectl set-ntp yes
In that case, this appears to be an instance
https://issues.guix.gnu.org/56137 (‘OpenSSL 3.0.3/1.1.1n includes a
time-dependent test’), this time for different test case.
I propose to implement https://issues.guix.gnu.org/56137#3 to solve
this more permanently.
Greetings,
Maxime.
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
Maxime Devos wrote on 8 Nov 2022 02:59
(name . GNU bug tracker automated control server)(address . control@debbugs.gnu.org)(address . tracker@debbugs.gnu.org)
8b080433-e0b1-b8dd-64b4-fa9f4b18599f@telenet.be
severity 58650 important
merge 56137 58650
thanks
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
L
L
Ludovic Courtès wrote on 15 Nov 2022 17:15
control message for bug #58650
(address . control@debbugs.gnu.org)
87k03wkxdg.fsf@gnu.org
retitle 58650 OpenSSL 1.1.1n test failures due to expired certificates (time bomb)
quit
M
M
Maxim Cournoyer wrote on 27 Feb 2023 05:03
Re: bug#58650: OpenSSL 1.1.1n test failures due to expired certificates (time bomb)
(address . 56137@debbugs.gnu.org)
871qmbdaae.fsf_-_@gmail.com
Hi,

I also tried with libfaketime, which seemed more complete and easy to
setup globally via environment variables:

Toggle snippet (77 lines)
modified gnu/packages/tls.scm
@@ -491,11 +491,47 @@ (define (target->openssl-target target)
(error "unsupported openssl target architecture")))))
(string-append kernel "-" arch))))
+;;; A minimal version of libfaketime that should remain private. Its only
+;;; purpose is to avoid introducing a cycle with openssl due to libfaketime's
+;;; git-fetch origin, which pulls git (which requires openssl).
+(define libfaketime-minimal
+ (package
+ (name "libfaketime")
+ (version "0.9.10")
+ (home-page "https://github.com/wolfcw/libfaketime")
+ (source (origin
+ (method url-fetch)
+ ;; XXX: We cheat and use a dynamically generated archive GitHub
+ ;; link here, since we can't fetch from git.
+ (uri (string-append "https://github.com/wolfcw/" name
+ "/archive/refs/tags/v" version ".tar.gz"))
+ (sha256
+ (base32
+ "0zwlwxpya3scayf8b3ans6pp82k8k42bk5wfqvcm02kmkhxx76kj"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list
+ #:make-flags #~(list "all")
+ #:tests? #f
+ #:phases
+ #~(modify-phases %standard-phases
+ (replace 'configure
+ (lambda* (#:key outputs #:allow-other-keys)
+ (setenv "CC" #$(cc-for-target))
+ (setenv "PREFIX" #$output))))))
+ (synopsis "Fake the system time for single applications")
+ (description
+ "The libfaketime library allows users to modify the system time that an
+application \"sees\". It is meant to be loaded using the dynamic linker's
+@code{LD_PRELOAD} environment variable. The @command{faketime} command
+provides a simple way to achieve this.")
+ (license license:gpl2)))
+
(define-public openssl-1.1
;; Note to maintainers: when updating this package, make sure to update the
;; RELEASE-DATE variable below. It is used by datefudge to avoid time bombs
;; in the test suite.
- (let ((release-date "2021-08-24 00:00"))
+ (let ((release-date "@2021-08-24 00:00:00"))
(package
(name "openssl")
(version "1.1.1l")
@@ -517,7 +553,7 @@ (define-public openssl-1.1
(outputs '("out"
"doc" ;6.8 MiB of man3 pages and full HTML documentation
"static")) ;6.4 MiB of .a files
- (native-inputs (list datefudge perl))
+ (native-inputs (list libfaketime-minimal perl))
(arguments
(list
#:modules '((guix build gnu-build-system)
@@ -537,6 +573,15 @@ (define-public openssl-1.1
#:disallowed-references (list (canonical-package perl))
#:phases
#~(modify-phases %standard-phases
+ (add-before 'unpack 'setup-libfaketime
+ (lambda* (#:key native-inputs inputs #:allow-other-keys)
+ (let ((libfaketime.so.1 (search-input-file
+ (or native-inputs inputs)
+ "lib/faketime/libfaketime.so.1")))
+ (setenv "LD_PRELOAD" libfaketime.so.1)
+ (setenv "NO_FAKE_STAT" "1")
+ (setenv "FAKETIME_DONT_RESET" "1")
+ (setenv "FAKETIME" #$release-date))))
#$@(if (%current-target-system)
#~((add-before 'configure 'set-cross-compile


But I still get the same error:

Toggle snippet (48 lines)
../../util/shlib_wrap.sh /gnu/store/hy6abswwv4d89zp464fw52z65fkzr7h5-perl-5.34.0/bin/perl -I ../../util/perl ../generate_ssl_tests.pl ../ssl-tests/12-ct.conf.in > 12-ct.conf.30543.tmp => 0
ok 1 - Getting output from generate_ssl_tests.pl.
ok 2 - Comparing generated sources.
# Subtest: ../ssl_test
1..1
# Subtest: test_handshake
1..6
ok 1 - iteration 1
# ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
# [2] compared to [0]
# INFO: @ test/ssl_test.c:34
# ExpectedResult mismatch: expected Success, got ClientFail.
# 140450700142400:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
not ok 2 - iteration 2
ok 3 - iteration 3
# ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
# [2] compared to [0]
# INFO: @ test/ssl_test.c:34
# ExpectedResult mismatch: expected Success, got ClientFail.
# 140450700142400:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
not ok 4 - iteration 4
# ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
# [4] compared to [0]
# INFO: @ test/ssl_test.c:34
# ExpectedResult mismatch: expected Success, got FirstHandshakeFailed.
# 140450700142400:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
not ok 5 - iteration 5
# ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
# [4] compared to [0]
# INFO: @ test/ssl_test.c:34
# ExpectedResult mismatch: expected Success, got FirstHandshakeFailed.
# 140450700142400:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
not ok 6 - iteration 6
not ok 1 - test_handshake
../../util/shlib_wrap.sh ../ssl_test 12-ct.conf.30543.tmp => 1
not ok 3 - running ssl_test 12-ct.conf

# Failed test 'running ssl_test 12-ct.conf'
# at ../test/recipes/80-test_ssl_new.t line 148.
# Looks like you failed 1 test of 3.
not ok 12 - Test configuration 12-ct.conf

# Failed test 'Test configuration 12-ct.conf'
# at
# /tmp/guix-build-openssl-1.1.1l.drv-0/openssl-1.1.1l/test/../util/perl/OpenSSL/Test.pm
# line 1212.

When attempting to build with

Toggle snippet (3 lines)
./pre-inst-env guix build --no-grafts -e '(@@ (gnu packages tls) openssl-1.1)'

Upstream seems to have moved to give very large expiry dates on their
test certs (100 years), so perhaps we can simply remove this test and
hope the problem doesn't come back to haunt us...

--
Thanks,
Maxim
L
L
Ludovic Courtès wrote on 17 Apr 2023 15:23
control message for bug #58650
(address . control@debbugs.gnu.org)
871qkifxqv.fsf@gnu.org
merge 58650 60821
quit
?