Hi! pukkamustard skribis: > Ludovic Courtès writes: > >>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >>> Compression: gzip >>> FileSize: 67363 >>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE >>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 >>> Compression: zstd >>> FileSize: 64917 >>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM >> >> Do we really need one URN per compression method? Couldn’t we leave >> compression (of individual chunks, possibly) as a “detail” handled by >> the encoding or the transport layer? >> > > I agree that it would be nice to leave this to the encoding layer as > that would allow certain optimizations (e.g. de-duplication). > > Unfortunately, we haven't figured out yet what the most suitable > compression/format would be. Something like EROSFS seems good (as it > aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for > just an archive format and there do not seem to be any libraries that we > could use to neatly integrate. It seems possible to block-align a Tar > archive, but that seems a bit hackey [2]. Other things to look into > might be Tarlz [3] and ZPAQ [4]. Yeah. Though it may be that deduplication at the block level doesn’t buy us much. That was the conclusion I reached a long time ago[a], and also seems to be supported by the recent guix-daemon deduplication improvements[b]. [a] https://hal.inria.fr/hal-00187069/en [b] https://issues.guix.gnu.org/24937#20-lineno0 > To get started I suggest just using one of the compressions/formats > already in Guix. zstd seems to be a reasonable choice (for the same > reasons why it makes sense to use zstd with `--discover` [5]). > > Does that sound like a plan? Sure! > I meant to send in a V2 that does this before going on holidays, but I'm > afraid I won't make it. V2 will come in early January! Alright, we’ll see! :-) Until then, enjoy your holidays! Ludo’.