From debbugs-submit-bounces@debbugs.gnu.org Thu Apr 16 09:27:43 2020 Received: (at 24937) by debbugs.gnu.org; 16 Apr 2020 13:27:43 +0000 Received: from localhost ([127.0.0.1]:37743 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jP4YQ-0008AH-Qq for submit@debbugs.gnu.org; Thu, 16 Apr 2020 09:27:43 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21335) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jP4YO-0008A8-3V for 24937@debbugs.gnu.org; Thu, 16 Apr 2020 09:27:41 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1587043654; cv=none; d=zohomail.com; s=zohoarc; b=gjN/fdRn5klYUMhJRP97MIgV4+WAtR/CaZQ2/2yhOAa6P2RPZ+0wZ6k8Q2GZTg7763GtZBmeoDbBa4W54uvRVxbbGk3/KdnCr3ppVU8WU/0xzkNA7AdHPta4Ylhy3VpddwZV7O42g2U0sRZikl6Q9g3PN+GmGXDfJOzW7FDgOyY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1587043654; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=CetexPByQizFw7YwqtgKpHwuPuDD5g2j6jbh87dcg8M=; b=AvmFJPJOMF63AWdEtAVLdjkUJEoIXsyrXTO//Zm3lxMWYynInn0jDuoBoleGDdZfrQH7NSGmuXbdI7Lk5kY3p5daPdwfDLt1S5i1RZR2aBQRtGVsRiqDK5y4dBCU3y4pmyE/iuZK6t5dapzmC/g4WJ/GcKIGwEiL6X8qdtJ59qg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1587043654; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=CetexPByQizFw7YwqtgKpHwuPuDD5g2j6jbh87dcg8M=; b=MMQAYyVGh1bN/4EQ5+y0e1EPX/H4ZpAm0ssLY+9J0Bi2PM3smeXfLnPT2Zt7A1uD I0UKJIB0UIcYEXaSWXu/XUNBxLPyTVIi+dZh+6bgaKsDe/oiu7yEs34qrBiS5i3Gurx +aVXq6Sj8Q+2zSzcELfCKZRV2Szfl4xClBnMmOyk= Received: from localhost (p54AD4D27.dip0.t-ipconnect.de [84.173.77.39]) by mx.zohomail.com with SMTPS id 1587043621480388.8881527687968; Thu, 16 Apr 2020 06:27:01 -0700 (PDT) References: <87wpg7ffbm.fsf@gnu.org> <87lgvm4lzu.fsf@gnu.org> <87twaaa6j9.fsf@netris.org> <87twaa2vjx.fsf@gnu.org> <87lgvm9sgq.fsf@netris.org> <87d1gwvgu0.fsf@gnu.org> <87wpf4yoz0.fsf@netris.org> <87fulrsqxx.fsf@gnu.org> <87vaunbvcu.fsf@mdc-berlin.de> User-agent: mu4e 1.2.0; emacs 26.3 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#24937: "deleting unused links" GC phase is too slow In-reply-to: <87vaunbvcu.fsf@mdc-berlin.de> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Thu, 16 Apr 2020 15:26:58 +0200 Message-ID: <87ftd3muhp.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24937 Cc: 24937@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > Ludovic Court=C3=A8s writes: > >> Ricardo, Roel: would you be able to run that links-traversal.c from >> >> on a machine with a big store, as described at >> ? > > I just ran this on my workstation in the office where I regularly build > packages. Here=E2=80=99s the output of =E2=80=9Cdf -i /gnu=E2=80=9D > > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/mapper/fedora-root 3301376 1098852 2202524 34% / > > Probably not large enough to derive conclusions about hydra=E2=80=99s beh= aviour. > > [I can=E2=80=99t run it on the shared store at the MDC because NFS perfor= mance is > too poor. I recently ran =E2=80=9Cguix gc --optimize=E2=80=9D to dedupe = the shared > store (post-build deduplication is disabled since a few weeks) and it=E2= =80=99s > at 3,197,489 used inodes.] > > Here are the results of running the link-traversal code on my > workstation: > > --8<---------------cut here---------------start------------->8--- > rwurmus in ~: gcc -std=3Dgnu99 -Wall links-traversal.c -DMODE=3D3 > rwurmus in ~: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > rwurmus in ~: time ./a.out > 412825 dir_entries, 107 seconds > stat took 0 seconds > > real 1m47.264s > user 0m0.214s > sys 0m1.314s > > rwurmus in ~: gcc -std=3Dgnu99 -Wall links-traversal.c -DMODE=3D2 > rwurmus in ~: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > rwurmus in ~: time ./a.out > 12821 dir_entries, 107 seconds (including stat) > > real 1m46.475s > user 0m0.201s > sys 0m1.309s > --8<---------------cut here---------------end--------------->8--- I ran this for the first time on ci.guix.gnu.org, which has a very big store (currently at around 29TB). df -i /gnu: Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 610021376 132350406 477670970 22% /gnu I had to increase the number of MAX_ENTRIES to 135000000. I forgot to drop caches initially. This is the first run: --8<---------------cut here---------------start------------->8--- root@berlin ~ [env]# gcc links-traversal.c -DMODE=3D3 -o links-traversal root@berlin ~ [env]# time ./links-traversal 57079502 dir_entries, 3906 seconds stat took 136 seconds real 67m48.145s user 0m59.575s sys 2m30.065s --8<---------------cut here---------------end--------------->8--- I aborted the run after I dropped caches after 67 minutes. I=E2=80=99m going to continue testing on one of the build nodes, and I=E2= =80=99ll try using statx. -- Ricardo