From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 14:22:26 2022 Received: (at 52555) by debbugs.gnu.org; 25 Jan 2022 19:22:26 +0000 Received: from localhost ([127.0.0.1]:50138 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCROa-0001CQ-9r for submit@debbugs.gnu.org; Tue, 25 Jan 2022 14:22:26 -0500 Received: from mout02.posteo.de ([185.67.36.66]:55307) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCROU-0001C0-RN for 52555@debbugs.gnu.org; Tue, 25 Jan 2022 14:22:23 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id B595C240103 for <52555@debbugs.gnu.org>; Tue, 25 Jan 2022 20:22:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1643138532; bh=d5WCY24/0Il2IHG4p/6YJQnYqcD4WOhnAqlw7escmoI=; h=From:To:Cc:Subject:Date:From; b=cFaGdg3TV4AcmYezCcN3Sno/LguMEEP14HRAeQQvdbzmzbAEYHeHPor+jpVvhVing 4vo3ejKuSip3vCpYAZmDdC6cGGxQM1V9fuElLSW73+S1zPNoZboPyBPjtWrWuHL+45 JYTjqKJGHUX4TUFH4eJQDqkJm6RzntlHGpCd6y8DspHtLKw5VeJY3l2izWAmwg1IjD y3QARnXu97N+RhBBwkwXtkZzOHV+P4TtdTh2ZnYzFG5eX167Ish7PS8bVZlZa8OwBm Flh9yMADL+aezW4bmqwB4vvA9BfI/3UpA0oGhc/72+Kglcd1SNdVOg0Vya0i/Oc25h wLyl3v2vXo14Q== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JjxYv1gq0z9rxM; Tue, 25 Jan 2022 20:22:10 +0100 (CET) From: pukkamustard To: 52555@debbugs.gnu.org Subject: [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS Date: Tue, 25 Jan 2022 19:21:56 +0000 Message-Id: <20220125192201.7582-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 52555 Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Guix, Here comes the V2 of a proposal towards decentralizing substitute distribution with ERIS. A quick summary (as this has become quite long): - This adds support for publishing and getting substitutes over IPFS. - By using the ERIS encoding we are not limited to using IPFS as transport. We can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support for these can be added in (guix eris). - These patches are still very rough and we need better logic for when to use IPFS et. al. and when to fallback to HTTP. - There might be performance issues when using IPFS via the IPFS daemon HTTP API. I found the setup for testing this a bit tricky. I will try and describe how I have been testing it. Please let me know how this can be improved! ** Authorize local substitutes We will be running a local substitute server so we need to add the local signing key to the list of authorized keys. In the system configurations: #+BEGIN_SRC scheme (modify-services %base-services (guix-service-type config => (guix-configuration (inherit config) (authorized-keys (cons* ;; allow substitutes from ourselves for testing purposes (local-file "/etc/signing-key.pub") %default-authorized-guix-keys))))) #+END_SRC ** Configure the local Guix checkout #+BEGIN_SRC shell ./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc #+END_SRC The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~. ** Start the IPFS daemon #+BEGIN_SRC shell guix shell go-ipfs -- ipfs daemon #+END_SRC Start a local substitute server: #+BEGIN_SRC shell sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19 #+END_SRC We use port 8081 as IPFS is running on 8080. We use the temporary cache directory ~/tmp/guix-publish-cache~. ** Build some package locally First we build some package: #+BEGIN_SRC shell ./pre-inst-env guix build hello --no-substitutes --no-offload #+END_SRC #+RESULTS: : /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 ** Trigger the substitute server to "bake" a susbtitute #+BEGIN_SRC shell curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo #+END_SRC --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 Compression: zstd NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm NarSize: 190480 References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg== --8<---------------cut here---------------end--------------->8--- If you do this again after a few seconds you will get a different response that has the ERIS URN and the FileSizes. The reason for this is that Guix publish bakes the nars asyncrhonisly in the background: #+BEGIN_SRC shell curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo #+END_SRC #+RESULTS: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 Compression: zstd FileSize: 57691 NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm NarSize: 190480 ERISFormat: application/x-nix-archive+zstd-19 ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg== --8<---------------cut here---------------end--------------->8--- These patches have added the ERIS and ERISFormat fields. Eventually we would have figured out what the best format for use over ERIS is, for now we encode it in the ERISFormat field. ** Removing a package from the store This is necessary in order to make guix look for a substitute. #+BEGIN_SRC shell ./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 #+END_SRC ** Start the Guix daemon from the repository #+BEGIN_SRC shell sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --debug --substitute-urls=http://localhost:8081/ #+END_SRC Note this will probably stop your system Guix daemon. Run ~sudo herd restart guix-daemon~ to restart it. #+BEGIN_SRC shell ./pre-inst-env guix build hello #+END_SRC --8<---------------cut here---------------start------------->8--- substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11... downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ... urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q 502KiB/s 00:00 | 56KiB transferred /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11 --8<---------------cut here---------------end--------------->8--- We have just retreived the substitute for the hello package from IPFS. Hello decentralized substitutes! I have only tested this for fairly small packages (up to a few MB). One issue with IPFS might be that we have to create a new HTTP connection to the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem to support HTTP connection re-use and neither does the Guile (web client). I fear this might become a performance issue. It seems possible to use IPFS more directly by exposing the Go code as a C library and then using that with the Guile FFI [1]. This is however a bit complicated and adds a lot of dependencies. In particular, this should not become a dependency of Guix itself. The performance of IPFS itself also needs to be evaluated, maybe the IPFS HTTP API will not be the bottle-neck. As mentioned in previous mail a simple HTTP transport for blocks would be a good fallback. This would allow users to get missing blocks (things that somehow got dropped from IPFS) directly from a substitute server. This is different then getting the entire NAR from a substitute server. A user might be missing a single 32KiB block and should be able to get only that. However, such a HTTP fallback would also suffer from the one-connection-per-block issue. As part of general ERIS research we are investigating CoAP as a better fallback transport. In any case, it would be necessary for the substitute server to store encoded blocks of the NAR. For this I think it makes sense to use a small database. We have bindings to use ERIS with GDBM [2]. It might also make sense to use SQLite, especially if there are other use-cases for such a database. I will be looking into the HTTP fallback and also using BitTorrent and GNUNet as transports. Thanks for making it so far and happy hacking! -pukkamustard [1] https://github.com/scala-network/libipfs/ [2] https://codeberg.org/eris/guile-eris/src/branch/main/eris/blocks/gdbm.scm pukkamustard (5): WIP: gnu: guile-eris: Update to unreleased git version. publish: Add ERIS URN to narinfo Add (guix eris). publish: Add support for storing ERIS encoded blocks to IPFS. substitute: Fetch substitutes using ERIS. Makefile.am | 1 + configure.ac | 5 +++ gnu/packages/guile-xyz.scm | 10 ++--- gnu/packages/package-management.scm | 1 + guix/eris.scm | 60 +++++++++++++++++++++++++++++ guix/narinfo.scm | 14 +++++-- guix/scripts/publish.scm | 32 +++++++++++++-- guix/scripts/substitute.scm | 21 +++++++--- 8 files changed, 126 insertions(+), 18 deletions(-) create mode 100644 guix/eris.scm -- 2.34.0