From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 21 20:34:34 2021 Received: (at 51787) by debbugs.gnu.org; 22 Dec 2021 01:34:34 +0000 Received: from localhost ([127.0.0.1]:56282 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzqWY-0008DB-AA for submit@debbugs.gnu.org; Tue, 21 Dec 2021 20:34:34 -0500 Received: from mx.kolabnow.com ([212.103.80.153]:24362) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzqWV-0008Cr-UY for 51787@debbugs.gnu.org; Tue, 21 Dec 2021 20:34:32 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id B7888410ED; Wed, 22 Dec 2021 02:34:25 +0100 (CET) Authentication-Results: ext-mx-out003.mykolab.com (amavisd-new); dkim=pass (4096-bit key) reason="pass (just generated, assumed good)" header.d=kolabnow.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:content-type:content-type:mime-version :message-id:in-reply-to:date:date:subject:subject:from:from :references:received:received:received; s=dkim20160331; t= 1640136863; x=1641951264; bh=3i63tiFp+CoNc/GuP9QPl1lU2yVYTFDvyF2 sJxmBOu0=; b=y6lWu7Pkj1WxW9gmZFp0PWS5LXAjz9+gnSFUifbB8n0eaGZuMIU S2MwHeis1SdzMI9j8rcS7hV39ZyjFo9J4FHLq5fN1xQ2E8fuSBEUYKjEFLx4/1d4 IVMx++PCYa4fHPhGp9g3OXJe7gDxaymob7V+89dnE/9BH7+ZdmOyKtlRFIMIFfT+ /0JLAudAG91n30PXfeSznSDt3F8A5XaYxCvW0ROcjP9DqkgTGM9nG6QvMjJ3um6h UiglYVFnx87HF85A9BUZrFHRroC+Fj98TBhZqgxDMFFTMiOXzwKPTQaQTCscWNfI xOFoHt3rvoUuooYNGG80vx7Qes2wwdRurU6egwHB9VXw9F/O/nNmdv1ne3IIRJZV vkxI0nxNPHFIwxla1Ky5On7U9MlaG38Qy6p8GG3uOOAA4qk/kArGBytOezn71yby E0B7iH3Fqnv7FGKlhHikR4zkVbfoknO4ZOdHlnlVfLI2CrcAW4hTGLDzmaIa+on3 fMA3QTpeOE0Ar807JaKpzU1wVq4iabbq+BqRf4lBk/lOd6s3gONxwKL7UzKfh5M0 dh+J1QpT4gec+vHONtawFuR+BsWDoBotVUnchApK1jL/Ja28pTtLCbRN3ACqi8hT cag4fUmeuCcjU31/7S8q2R2FZLWqqMrXE9kaR1IgoqwPL4vLvknfVtmg= X-Virus-Scanned: amavisd-new at mykolab.com X-Spam-Flag: NO X-Spam-Score: -1.899 X-Spam-Level: X-Spam-Status: No, score=-1.899 tagged_above=-10 required=5 tests=[BAYES_00=-1.9, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out003.mykolab.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HB6ko0HtvW3e; Wed, 22 Dec 2021 02:34:23 +0100 (CET) Received: from int-mx001.mykolab.com (unknown [10.9.13.1]) by mx.kolabnow.com (Postfix) with ESMTPS id 56A15408FC; Wed, 22 Dec 2021 02:34:22 +0100 (CET) Received: from ext-subm003.mykolab.com (unknown [10.9.6.3]) by int-mx001.mykolab.com (Postfix) with ESMTPS id D796921AC; Wed, 22 Dec 2021 02:34:20 +0100 (CET) References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> <87v8zhn9m1.fsf@elephly.net> From: Thiago Jung Bauermann To: 51787@debbugs.gnu.org Subject: Re: bug#51787: Disk performance on ci.guix.gnu.org Date: Tue, 21 Dec 2021 21:27:46 -0300 In-reply-to: <87v8zhn9m1.fsf@elephly.net> Message-ID: <87lf0dzalt.fsf@kolabnow.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 51787 Cc: Ricardo Wurmus , Mathieu Othacehe , Bengt Richter , Leo Famulari X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Ricardo Wurmus writes: > Today we discovered a few more things and discussed them on IRC. Here=E2= =80=99s > a summary. > > /var/cache sits on the same storage as /gnu. We mounted the 5TB ext4 > file system that=E2=80=99s hosted by the SAN at /mnt_test and started cop= ying > over /var/cache to /mnt_test/var/cache. Transfer speed was considerably > faster (not *great*, but reasonably fast) than the copy of > /gnu/store/trash to the same target. > > This confirmed our suspicions that the problem is not with the storage > array but due to the fact that /gnu/store/trash (and also /gnu/store) > is an extremely large, flat directory. /var/cache is not. There was an interesting thread in the Linux kernel mailing lists about this very issue earlier this year: https://lore.kernel.org/linux-fsdevel/206078.1621264018@warthog.procyon.org= .uk/ I=E2=80=99m not sure I completely understood all of the concerns discussed = there, but my understanding of it is that for workloads which don=E2=80=99t concurrent= ly modify the huge directory, it=E2=80=99s size isn=E2=80=99t a problem for btrfs and= XFS and in fact it=E2=80=99s even more efficient to have one big directory rather than subdirectories=C2=B9. It=E2=80=99s should also be well handled even by ext4= , IIUC=C2=B2. The problem for all filesystems is concurrently modifying the directory (e.g., adding or removing files), because the kernel serializes directory operations at the VFS layer. Also in that case XFS can also have allocation issues when adding new files if one isn=E2=80=99t careful.=C2=B3 --=20 Thanks Thiago =C2=B9 https://lore.kernel.org/linux-fsdevel/20210517232237.GE2893@dread.di= saster.area/ =C2=B2 https://lore.kernel.org/linux-fsdevel/6E4DE257-4220-4B5B-B3D0-B67C7B= C69BB5@dilger.ca/ =C2=B3 https://lore.kernel.org/linux-fsdevel/20210519125743.GP2893@dread.di= saster.area/