low performance when building profile of already built packages

OpenSubmitted by Ricardo Wurmus.
Details
2 participants
  • Ludovic Courtès
  • Ricardo Wurmus
Owner
unassigned
Severity
important
Merged with
R
R
Ricardo Wurmus wrote on 24 Apr 2021 22:38
(address . bug-guix@gnu.org)
87zgxnpgd9.fsf@elephly.net
Here is a manifest that isn’t unusual for a bioinformatics
workflow:

Toggle snippet (34 lines)
(define packages
(list "bash-minimal"
"fastqc"
"gzip"
"htseq"
"multiqc"
"r-corrplot"
"r-crosstalk"
"r-data-table"
"r-deseq2"
"r-dt"
"r-genomicalignments"
"r-genomicranges"
"r-ggplot2"
"r-ggrepel"
"r-gprofiler"
"r-knitr"
"r-pheatmap"
"r-plotly"
"r-reshape2"
"r-rmarkdown"
"r-rsamtools"
"r-rtracklayer"
"r-s4vectors"
"r-scales"
"r-summarizedexperiment"
"r-tximport"
"samtools"
"sed"
"trim-galore"))

(specifications->manifest packages)

When all these packages already exist and nothing is to be done
when Guix is asked to instantiate this manifest, more than 13
seconds pass.

Toggle snippet (59 lines)
$ [env] export GUIX_PROFILING="object-cache
add-data-to-store-cache rpc"
$ [env] time guix package -m manifest-test.scm -p /tmp/foo
The following packages will be installed:
bash-minimal 5.0.16
fastqc 0.11.5
gzip 1.10
htseq 0.9.1
multiqc 1.5
r-corrplot 0.84
r-crosstalk 1.1.1
r-data-table 1.14.0
r-deseq2 1.30.1
r-dt 0.17
r-genomicalignments 1.26.0
r-genomicranges 1.42.0
r-ggplot2 3.3.3
r-ggrepel 0.9.1
r-gprofiler 0.7.0
r-knitr 1.31
r-pheatmap 1.0.12
r-plotly 4.9.3
r-reshape2 1.4.4
r-rmarkdown 2.7
r-rsamtools 2.6.0
r-rtracklayer 1.50.0
r-s4vectors 0.28.1
r-scales 1.1.1
r-summarizedexperiment 1.20.0
r-tximport 1.18.0
samtools 1.11
sed 4.8
trim-galore 0.6.6
[[most of the time passes here]]

nothing to be done
Store object cache:
fresh caches: 2
lookups: 45493
hits: 43943 (96.6%)
cache size: 1549 entries

'add-data-to-store' cache:
lookups: 7755
hits: 3411 (44.0%)
.drv files: 4357 (56.2%)
Scheme files: 3343 (43.1%)
Remote procedure call summary: 5283 RPCs
built-in-builders ... 1
add-to-store/tree ... 27
add-to-store ... 213
query-references ... 698
add-text-to-store ... 4344

real 0m13.379s
user 0m16.414s
sys 0m0.253s

Some of these R packages have a lot of transitive inputs, such as
r-deseq2 with 81 transitive propagated inputs. So there’s a lot
of propagation here, which I suspect contributes to things being
so unexpectedly slow.

This report may be related to performance issues reported for
“guix environment” in https://issues.guix.gnu.org/41702,which
also involves R packages (such as r-learnr with 46 transitive
propagated inputs).

--
Ricardo
L
L
Ludovic Courtès wrote on 30 Apr 2021 17:45
control message for bug #48005
(address . control@debbugs.gnu.org)
87lf8zbwsv.fsf@gnu.org
severity 48005 important
quit
L
L
Ludovic Courtès wrote on 27 May 2021 18:32
control message for bug #41702
(address . control@debbugs.gnu.org)
87wnrkjfwv.fsf@gnu.org
merge 41702 48005
quit
?