libgccjit is unusable

  • Done
  • quality assurance status badge
Details
5 participants
  • John Kehayias
  • Liliana Marie Prikler
  • Tobias Geerinckx-Rice
  • Remco van 't Veer
  • Andrew Whatson
Owner
unassigned
Submitted by
Liliana Marie Prikler
Severity
normal
L
L
Liliana Marie Prikler wrote on 26 May 2022 15:07
(address . bug-guix@gnu.org)
b4c6419964da360a33377d373f034f0061619cfd.camel@gmail.com
Hi Guix,

with the release of Emacs 28.1, there has been some demand to enable
native-compilation. While trying to set that up, I've come to realize
that no matter how I slice it, I can't make libgccjit usable.

My test:

$ cd /tmp
$ # fetch the "Hello World" [1] and insert it into gccjit-test.c
$ guix shell --pure gcc-toolchain@9 libgccjit@9 -- \
gcc gccjit-test.c -o gccjit-test -lgccjit
$ ./gccjit-test
x86_64-unknown-linux-gnu-gcc-9.4.0: fatal error: cannot execute 'as':
execvp: No such file or directory
compilation terminated.

Welp, okay, so maybe testing gccjit outside of its installation does
not work. What if we try it inside the shell (we can always propagate
libgccjit, no?)

guix shell gcc-toolchain@9 libgccjit@9 -- ./gccjit-test
ld: cannot find crtbeginS.o: Datei oder Verzeichnis nicht gefunden
ld: cannot find -lgcc
ld: cannot find -lgcc
collect2: Fehler: ld gab 1 als Ende-Status zurück
libgccjit.so: error: error invoking gcc driver
NULL result

For the record, Emacs 28 fails with a different error during
configuration, though interestingly they use a smaller program and
appear to omit -lgccjit.
./conftest: error while loading shared libraries: libgccjit.so.0:
cannot open shared object file: No such file or directory

I'm at a loss. Is there any way to make libgccjit actually usable?

R
R
Remco van 't Veer wrote on 4 Jun 2022 16:07
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)(address . 55657@debbugs.gnu.org)
87pmjomsio.fsf@remworks.net
2022/05/26 15:07, Liliana Marie Prikler:

Toggle quote (6 lines)
> Hi Guix,
>
> with the release of Emacs 28.1, there has been some demand to enable
> native-compilation. While trying to set that up, I've come to realize
> that no matter how I slice it, I can't make libgccjit usable.

The guixrus channel has an emacs which seems to support gccjit. I have
not tried it myself.


Especially:


Maybe you can borrow some knowledge there? ;-)

Cheers,
Remco
T
T
L
L
Liliana Marie Prikler wrote on 4 Jun 2022 17:14
0f103f8a872fc5b884b734fe71cc4ca5258e39ac.camel@gmail.com
Am Samstag, dem 04.06.2022 um 14:25 +0000 schrieb Tobias Geerinckx-
Rice:
Toggle quote (3 lines)
> Meant to include link to
> https://github.com/flatwhatson/guix-channel/blob/657da22f0229b978b7bf4e4d476f59f17f6a175f/flat/packages/gcc.scm#L27
>  which is where the majic happens
I'm not personally using that channel, but I noticed this issue by
trying to adapt that recipe towards master. Now there's a chance that
I've made an error in my copypasta game, but I'm pretty sure the
current state of libgccjit is not a good one regardless.

Cheers
J
J
John Kehayias wrote on 28 Jun 2022 02:53
Re: libgccjit is unusable
(name . 55657@debbugs.gnu.org)(address . 55657@debbugs.gnu.org)
-WzfXRgBunfV6CTG4v5TA24Vk7Vty4mGqGpQdTTU2OkJpt-i1gM2iOOEDl9ODBZd1xuARNui4gnp1gZfkPtmHwlSwMmdagPc2fEXJyCWRJI=@protonmail.com
Hello,

I currently use the flatwhatson channel (cc'ed on this email, hope that is okay Andrew!) and it has worked well for Emacs, though never tried anything else with libgccjit. It would be great to get this upstreamed in Guix, especially since libgccjit is a beast to build (more than emacs if I remember correctly).

Happy to help test and work on this, let me know anything I can do.

Thanks,
John
L
L
Liliana Marie Prikler wrote on 28 Jun 2022 06:17
67ca146d031c320b484979c6aa5a89b9c9b8472d.camel@gmail.com
Hi,

Am Dienstag, dem 28.06.2022 um 00:53 +0000 schrieb John Kehayias:
Toggle quote (7 lines)
> Hello,
>
> I currently use the flatwhatson channel (cc'ed on this email, hope
> that is okay Andrew!) and it has worked well for Emacs, though never
> tried anything else with libgccjit. It would be great to get this
> upstreamed in Guix, especially since libgccjit is a beast to build
> (more than emacs if I remember correctly).
Keyword here is "has worked for emacs". I've tried porting the logic
from flatwhatson's channel over, but regardless of what I do, it
already fails in the configure step of Emacs (in a manner that's
reproducible outside as well). Thus, I think this is a bug in
libgccjit (or perhaps our packaging of it) that simply happened to be
ignored during development of Emacs 28, but no longer in the release.

Cheers
J
J
John Kehayias wrote on 28 Jun 2022 07:16
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
g7erfFF7ErV3F2PgmisSVSUcmk4fyChDnpWBykGnRoUS6sI4frkev0XxPlpqJOACTPN1PT6Yee3miMavYf5g61TDXxdt9AgKgGtcVX3mhvc=@protonmail.com
Hi,

------- Original Message -------
On Tuesday, June 28th, 2022 at 12:17 AM, Liliana Marie Prikler wrote:

Toggle quote (8 lines)
> Keyword here is "has worked for emacs". I've tried porting the logic
> from flatwhatson's channel over, but regardless of what I do, it
> already fails in the configure step of Emacs (in a manner that's
> reproducible outside as well). Thus, I think this is a bug in
> libgccjit (or perhaps our packaging of it) that simply happened to be
> ignored during development of Emacs 28, but no longer in the release.
>

Sorry, I should be extra clear that I mean has in the past and continues to work for Emacs. I've been using emacs-pgtk-native-comp through the flatwhatson channel from well before v28 was released. Currently I'm using emacs-pgtk-native-comp-28.1.50-223.3ddccb5. Everything has built, installed, and run fine for as long as I have been using it. Just in case that was in question, and as a point of reference.

Anyway, I'll try to reproduce when I can (tomorrow likely) what you reported in the first message using this setup, if that is of use.

Appreciate the efforts from everyone working on this!
John
J
J
John Kehayias wrote on 3 Aug 2022 23:13
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
QBzbwGsoJAPXLEIoj0pztUgjvlWLfxW4UxA2NoD7I4eMnZ0SeFxrlEjPn30QWo-lsMi-bQfJPRp-rpIM1S-JX4cphCaRWSbnXvilsyRPyXI=@protonmail.com
Hi everyone,

Found out some useful info and a work around for the original reported issue of the simple "hello world" of gccjit not working.


------- Original Message -------
On Tuesday, June 28th, 2022 at 1:16 AM, John Kehayias wrote:

Toggle quote (18 lines)
> Hi,
>
> ------- Original Message -------
> On Tuesday, June 28th, 2022 at 12:17 AM, Liliana Marie Prikler wrote:
>
> > Keyword here is "has worked for emacs". I've tried porting the logic
> > from flatwhatson's channel over, but regardless of what I do, it
> > already fails in the configure step of Emacs (in a manner that's
> > reproducible outside as well). Thus, I think this is a bug in
> > libgccjit (or perhaps our packaging of it) that simply happened to be
> > ignored during development of Emacs 28, but no longer in the release.
>
>
> Sorry, I should be extra clear that I mean has in the past and continues to work for Emacs. I've been using emacs-pgtk-native-comp through the flatwhatson channel from well before v28 was released. Currently I'm using emacs-pgtk-native-comp-28.1.50-223.3ddccb5. Everything has built, installed, and run fine for as long as I have been using it. Just in case that was in question, and as a point of reference.
>
> Anyway, I'll try to reproduce when I can (tomorrow likely) what you reported in the first message using this setup, if that is of use.
>

I was able to reproduce the original error, though I used the libgccjit package from the flatwhatson channel, at v11.3.0 (along with GCC at that version). For good measure, I also used the tutorial at that version, just in case https://gcc.gnu.org/onlinedocs/gcc-11.3.0/jit/intro/tutorial01.html I chose this version since that is what emacs-native-comp from that channel is built with.

Searching for these error messages of missing libraries/files, I found



I didn't dive into the details and I'm not expert here, but it gave me the clues to work around it. Seems that where gccjit looks for things has some assumptions (bugs?) which we can fix at runtime with:

LIBRARY_PATH=$GUIX_ENVIRONMENT/lib/gcc/x86_64-unknown-linux-gnu/11.3.0:$LIBRARY_PATH ./gccjittest

The errors reported before were solved with this LIBRARY_PATH addition of the lib/gcc subdirectory. So, the test program runs in

guix shell gcc-toolchain@11 libgccjit@11 --pure

where I compiled to gccjittest following the tutorial directions (no change to LIBRARY_PATH).

So, looking at the emacs-native-comp definition in flatwhatson, we can see that a phase is used to set LIBRARY_PATH before configure just as I did here: https://github.com/flatwhatson/guix-channel/blob/master/flat/packages/emacs.scm#L65

Hope this is helpful and unblocks libgccjit and emacs-native-comp for Guix!

John
L
L
Liliana Marie Prikler wrote on 4 Aug 2022 06:26
(name . John Kehayias)(address . john.kehayias@protonmail.com)
e520cab43357f150b792fbf48e38fa3f6785afe8.camel@gmail.com
Hi John,

Am Mittwoch, dem 03.08.2022 um 21:13 +0000 schrieb John Kehayias:
Toggle quote (14 lines)
> I didn't dive into the details and I'm not expert here, but it gave
> me the clues to work around it. Seems that where gccjit looks for
> things has some assumptions (bugs?) which we can fix at runtime with:
>
> LIBRARY_PATH=$GUIX_ENVIRONMENT/lib/gcc/x86_64-unknown-linux-
> gnu/11.3.0:$LIBRARY_PATH ./gccjittest
>
> The errors reported before were solved with this LIBRARY_PATH
> addition of the lib/gcc subdirectory. So, the test program runs in
>
> guix shell gcc-toolchain@11 libgccjit@11 --pure
>
> where I compiled to gccjittest following the tutorial directions (no
> change to LIBRARY_PATH).
while this does help insofar as I now know which snippet I forgot to
copy, I do still think that this leaves us with two unreasonable
options if we want to use emacs to compile other packages:

1. Propagate gcc-toolchain from emacs.
2. Patch LIBRARY_PATH not just before configuration, but also via a
wrapper.

At the very least I don't see how Emacs would be able to compile other
packages to native code without either of the above.

WDYT?
A
A
Andrew Whatson wrote on 4 Aug 2022 06:48
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
CAPE069cr2axfF2oH=6wosvN3-e-7_WTezho=NKiGJYf80C1-6Q@mail.gmail.com
Hi John, Liliana,

Sorry I haven't jumped in before now, I appreciate your efforts to
bring emacs native-comp to guix!

On Thu, 4 Aug 2022 at 04:26, Liliana Marie Prikler
<liliana.prikler@gmail.com> wrote:
Toggle quote (14 lines)
>
> while this does help insofar as I now know which snippet I forgot to
> copy, I do still think that this leaves us with two unreasonable
> options if we want to use emacs to compile other packages:
>
> 1. Propagate gcc-toolchain from emacs.
> 2. Patch LIBRARY_PATH not just before configuration, but also via a
> wrapper.
>
> At the very least I don't see how Emacs would be able to compile other
> packages to native code without either of the above.
>
> WDYT?

The solution used in the package-definition in my channel is to patch
`comp.el` to directly reference the necessary gcc/glibc paths instead
of relying on the environment. This occurs in the
"patch-driver-options" step immediately after the "set-libgccjit-path"
step mentioned earlier. This makes gcc-toolchain part of emacs
closure, without requiring it to be propagated into the profile.

If I understand the problem correctly, that should suffice?

Cheers,
Andrew
L
L
Liliana Marie Prikler wrote on 4 Aug 2022 18:52
(name . Andrew Whatson)(address . whatson@gmail.com)
e584cb2d93cb424e77b2d66a9450daee096d465c.camel@gmail.com
Am Donnerstag, dem 04.08.2022 um 04:48 +0000 schrieb Andrew Whatson:
Toggle quote (31 lines)
> Hi John, Liliana,
>
> Sorry I haven't jumped in before now, I appreciate your efforts to
> bring emacs native-comp to guix!
>
> On Thu, 4 Aug 2022 at 04:26, Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote:
> >
> > while this does help insofar as I now know which snippet I forgot
> > to
> > copy, I do still think that this leaves us with two unreasonable
> > options if we want to use emacs to compile other packages:
> >
> > 1. Propagate gcc-toolchain from emacs.
> > 2. Patch LIBRARY_PATH not just before configuration, but also via a
> > wrapper.
> >
> > At the very least I don't see how Emacs would be able to compile
> > other
> > packages to native code without either of the above.
> >
> > WDYT?
>
> The solution used in the package-definition in my channel is to patch
> `comp.el` to directly reference the necessary gcc/glibc paths instead
> of relying on the environment.  This occurs in the
> "patch-driver-options" step immediately after the "set-libgccjit-
> path" step mentioned earlier.  This makes gcc-toolchain part of emacs
> closure, without requiring it to be propagated into the profile.
>
> If I understand the problem correctly, that should suffice?
But if I read your recipe correctly, you're not resolving %host-type in
those options. Does that really suffice?
A
A
Andrew Whatson wrote on 5 Aug 2022 02:59
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
CAPE069cyYY-uVDC_rW6OfEOHnPO6ea08ckAtzs6sysoD6tvTpw@mail.gmail.com
On Thu, 4 Aug 2022 at 16:52, Liliana Marie Prikler
<liliana.prikler@gmail.com> wrote:
Toggle quote (13 lines)
>
> > The solution used in the package-definition in my channel is to patch
> > `comp.el` to directly reference the necessary gcc/glibc paths instead
> > of relying on the environment. This occurs in the
> > "patch-driver-options" step immediately after the "set-libgccjit-
> > path" step mentioned earlier. This makes gcc-toolchain part of emacs
> > closure, without requiring it to be propagated into the profile.
> >
> > If I understand the problem correctly, that should suffice?
>
> But if I read your recipe correctly, you're not resolving %host-type in
> those options. Does that really suffice?

Ah, yes that is a little confusing. This is a quirk of the different
behaviour of the LIBRARY_PATH environment variable and the -B flag to
gcc.

I recommend reading about "-Bprefix" in `man gcc`, but in short it
tries those paths with and without "machine/version" appended for the
target machine and compiler version. We *could* hard-code those, but
it isn't necessary, and it seemed like that might cause problems if
someone's brave enough to attempt cross-compilation of native-comp
emacs.

A major benefit of patching "comp.el" directly is that it avoids
leaking gcc into the user's environment. It's possible that someone
is running emacs in a profile with a specific version of gcc, maybe
without libgccjit support (eg. while hacking on some legacy code), so
having emacs insist on a libgccjit-compatible gcc present in the
environment at runtime would cause lots of problems.

The other important bit is the libgccjit package. The one in guix
mainline works fine, it should be possible to get a working libgccjit
as-is. I updated the package definition on my channel for the
following reasons:

a) Support newer gcc versions

While developing native-comp support for emacs, Andrea found and fixed
some libgccjit bugs which made their way into subsequent releases of
gcc. The native-comp library includes work-arounds for these
problems, but produces faster/smaller code with a fresher libgccjit.

b) Reduce compilation time

The definition in guix is basically the standard gcc build, but with
libgccjit also enabled. This results in an arduous double-bootstrap
and building a bunch of compilers and libraries that are completely
unnecessary and unused by libgccjit. I've disabled all the
unnecessary stuff and depend on the main gcc package to build
libgccjit, relying on the fact that it's already properly
bootstrapped. It's much quicker to build, which is important if you
don't have substitutes.

Hope this helps,
Andrew
L
L
Liliana Marie Prikler wrote on 5 Aug 2022 20:44
(name . Andrew Whatson)(address . whatson@gmail.com)
65497ff154c8bed606d30e5a367e8f17b1a53cc7.camel@gmail.com
Am Freitag, dem 05.08.2022 um 00:59 +0000 schrieb Andrew Whatson:
Toggle quote (15 lines)
> On Thu, 4 Aug 2022 at 16:52, Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote
> > But if I read your recipe correctly, you're not resolving %host-
> > type in those options.  Does that really suffice?
>
> Ah, yes that is a little confusing.  This is a quirk of the different
> behaviour of the LIBRARY_PATH environment variable and the -B flag to
> gcc.
>
> I recommend reading about "-Bprefix" in `man gcc`, but in short it
> tries those paths with and without "machine/version" appended for the
> target machine and compiler version.  We *could* hard-code those, but
> it isn't necessary, and it seemed like that might cause problems if
> someone's brave enough to attempt cross-compilation of native-comp
> emacs.
Fair enough. The compiler driver does not appear to be the failing
part anyway, though given the status quo I can't exactly claim it is
not failing.

Toggle quote (3 lines)
> The other important bit is the libgccjit package.  The one in guix
> mainline works fine, it should be possible to get a working libgccjit
> as-is.  
From my own experience, I doubt that. Read also the thread title.

Toggle quote (24 lines)
> I updated the package definition on my channel for the following
> reasons:
>
> a) Support newer gcc versions
>
> While developing native-comp support for emacs, Andrea found and
> fixed some libgccjit bugs which made their way into subsequent
> releases of gcc.  The native-comp library includes work-arounds for
> these problems, but produces faster/smaller code with a fresher
> libgccjit.
>
> b) Reduce compilation time
>
> The definition in guix is basically the standard gcc build, but with
> libgccjit also enabled.  This results in an arduous double-bootstrap
> and building a bunch of compilers and libraries that are completely
> unnecessary and unused by libgccjit.  I've disabled all the
> unnecessary stuff and depend on the main gcc package to build
> libgccjit, relying on the fact that it's already properly
> bootstrapped.  It's much quicker to build, which is important if you
> don't have substitutes.
>
> Hope this helps,
> Andrew
I actually am at a loss, so instead of referring to your code, I shall
share mine. Note that, style aside, it should actually perform the
same steps as yours, but fails to do so. I also tried adding the
unversioned lib directory to LIBRARY_PATH and setting LD_LIBRARY_PATH
to little fanfare. Whatever I do, it seems configure wants to smoke a
different joint.

Cheers

;; gcc.scm

(define-public (make-libgccjit gcc)
(package
(inherit gcc)
(name "libgccjit")
(outputs (delete "lib" (package-outputs gcc)))
(properties (alist-delete 'hidden? (package-properties gcc)))
(arguments
(substitute-keyword-arguments (package-arguments gcc)
((#:modules _ '())
'((guix build gnu-build-system)
(guix build utils)
(ice-9 regex)
(srfi srfi-1)
(srfi srfi-26)))
((#:configure-flags flags)
#~(cons* "--disable-bootstrap"
"--disable-libatomic"
"--disable-libgomp"
"--disable-libquadmath"
"--disable-libssp"
"--enable-host-shared"
"--enable-languages=jit"
(remove (cut string-match "--enable-languages.*" <>)
#$flags)))
((#:phases phases)
#~(modify-phases #$phases
(add-after 'install 'remove-broken-or-conflicting-files
(lambda* (#:key outputs #:allow-other-keys)
(for-each delete-file
(find-files
(string-append (assoc-ref outputs "out")
"/bin")
".*(c\\+\\+|cpp|g\\+\\+|gcov|gcc|gcc-
.*)"))))))))
(inputs (modify-inputs (package-inputs gcc)
(delete "libstdc++")))
(native-inputs (modify-inputs (package-native-inputs gcc)
(prepend gcc)))
(synopsis "GCC library generating machine code on-the-fly at
runtime")
(description
"This package is part of the GNU Compiler Collection and provides
an
embeddable library for generating machine code on-the-fly at runtime.
This
shared library can then be dynamically-linked into bytecode
interpreters and
other such programs that want to generate machine code on-the-fly at
run-time.
It can also be used for ahead-of-time code generation for building
standalone
compilers. The just-in-time (jit) part of the name is now something of
a
misnomer.")))

(define-public libgccjit-9 (make-libgccjit gcc-9))
(define-public libgccjit-10 (make-libgccjit gcc-10))
(define-public libgccjit-11 (make-libgccjit gcc-11))
(define-public libgccjit-12 (make-libgccjit gcc-12))

(define-public libgccjit libgccjit-10)

;; emacs.scm

(define-public emacs
(package
(name "emacs")
(version "28.1")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/emacs/emacs-"
version ".tar.xz"))
(sha256
(base32
"1qbmmmhnjhn4lvzsnyk7l5ganbi6wzbm38jc1a7hhyh3k78b7c98"))
(patches (search-patches "emacs-exec-path.patch"
"emacs-fix-scheme-indent-
function.patch"
"emacs-source-date-
epoch.patch"))
(modules '((guix build utils)))
(snippet
'(with-directory-excursion "lisp"
;; Delete the bundled byte-compiled elisp files and
generated
;; autoloads.
(for-each delete-file
(append (find-files "." "\\.elc$")
(find-files "." "loaddefs\\.el$")
(find-files "eshell" "^esh-
groups\\.el$")))

;; Make sure Tramp looks for binaries in the right
places on
;; remote Guix System machines, where 'getconf PATH'
returns
;; something bogus.
(substitute* "net/tramp.el"
;; Patch the line after "(defcustom tramp-remote-
path".
(("\\(tramp-default-remote-path")
(format #f "(tramp-default-remote-path ~s ~s ~s ~s
"
"~/.guix-profile/bin" "~/.guix-
profile/sbin"
"/run/current-system/profile/bin"
"/run/current-system/profile/sbin")))

;; Make sure Man looks for C header files in the
right
;; places.
(substitute* "man.el"
(("\"/usr/local/include\"" line)
(string-join
(list line
"\"~/.guix-profile/include\""
"\"/var/guix/profiles/system/profile/include\"")
" ")))))))
(build-system glib-or-gtk-build-system)
(arguments
(list
#:tests? #f ; no check target
#:modules `((guix build glib-or-gtk-build-system)
(guix build utils)
(srfi srfi-1)
(ice-9 ftw))
#:configure-flags #~(list "--with-modules"
"--with-cairo"
"--with-native-compilation"
"--disable-build-details")
#:make-flags #~(list "NATIVE_FULL_AOT=1")
#:phases
#~(modify-phases %standard-phases
(add-after 'set-paths 'set-libgccjit-path
(lambda* (#:key inputs #:allow-other-keys)
(define (first-subdirectory/absolute directory)
(let ((files (scandir
directory
(lambda (file)
(and (not (member file '("." "..")))
(file-is-directory? (string-append
directory "/"
file)))))))
(and (not (null? files))
(string-append directory "/" (car files)))))

(let* ((libgccjit-libdir
(first-subdirectory/absolute ;; version
(first-subdirectory/absolute ;; host type
(search-input-directory inputs "lib/gcc")))))
(setenv "LIBRARY_PATH"
(string-append libgccjit-libdir ":"
(getenv "LIBRARY_PATH"))))))
(add-after 'unpack 'enable-elogind
(lambda _
(substitute* "configure.ac"
(("libsystemd") "libelogind"))
(when (file-exists? "configure")
(delete-file "configure"))))
(add-after 'unpack 'patch-program-file-names
(lambda* (#:key inputs #:allow-other-keys)
(substitute* '("src/callproc.c"
"lisp/term.el"
"lisp/htmlfontify.el"
"lisp/textmodes/artist.el"
"lisp/progmodes/sh-script.el")
(("\"/bin/sh\"")
(format #f "~s" (search-input-file inputs
"/bin/sh"))))
(substitute* "lisp/doc-view.el"
(("\"(gs|dvipdf|ps2pdf|pdftotext)\"" all what)
(let ((replacement (false-if-exception
(search-input-file
inputs
(string-append "/bin/" what)))))
(if replacement
(string-append "\"" replacement "\"")
all))))
;; match ".gvfs-fuse-daemon-real" and ".gvfsd-fuse-real"
;; respectively when looking for GVFS processes.
(substitute* "lisp/net/tramp-gvfs.el"
(("\\(tramp-compat-process-running-p \"(.*)\"\\)" all
process)
(format #f "(or ~a (tramp-compat-process-running-p
~s))"
all (string-append "." process "-real"))))))
(add-after 'unpack 'patch-compilation-driver
(lambda _
(substitute* "lisp/emacs-lisp/comp.el"
(("\\(defcustom native-comp-driver-options nil")
(format
#f "(defcustom native-comp-driver-options '(~s ~s ~s
~s)"
(string-append
"-B" #$(this-package-input "binutils") "/bin/")
(string-append
"-B" #$(this-package-input "glibc") "/lib/")
(string-append
"-B" #$(this-package-input "libgccjit") "/lib/")
(string-append
"-B" #$(this-package-input "libgccjit")
"/lib/gcc/"))))))
(add-before 'configure 'fix-/bin/pwd
(lambda _
;; Use `pwd', not `/bin/pwd'.
(substitute* (find-files "." "^Makefile\\.in$")
(("/bin/pwd")
"pwd"))))
(add-after 'install 'install-site-start
;; Use 'guix-emacs' in "site-start.el", which is used
autoload the
;; Elisp packages found in EMACSLOADPATH.
(lambda* (#:key inputs outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))
(lisp-dir (string-append out "/share/emacs/site-
lisp"))
(emacs (string-append out "/bin/emacs")))

;; This is duplicated from emacs-utils to prevent
coupling.
(define* (emacs-byte-compile-directory dir)
(let ((expr `(progn
(setq byte-compile-debug t)
(byte-recompile-directory
(file-name-as-directory ,dir) 0 1))))
(invoke emacs "--quick" "--batch"
(format #f "--eval=~s" expr))))

(copy-file #$(local-file
(search-auxiliary-file "emacs/guix-
emacs.el"))
(string-append lisp-dir "/guix-emacs.el"))
(with-output-to-file (string-append lisp-dir "/site-
start.el")
(lambda ()
(display
(string-append
"(when (require 'guix-emacs nil t)\n"
" (guix-emacs-autoload-packages)\n"
" (advice-add 'package-load-all-descriptors"
" :after #'guix-emacs-load-package-
descriptors))"))))
;; Remove the extraneous subdirs.el file, as it causes
Emacs to
;; add recursively all the the sub-directories of a
profile's
;; share/emacs/site-lisp union when added to
EMACSLOADPATH,
;; which leads to conflicts.
(delete-file (string-append lisp-dir "/subdirs.el"))
;; Byte compile the site-start files.
(emacs-byte-compile-directory lisp-dir))))
(add-after 'glib-or-gtk-wrap 'restore-emacs-pdmp
;; restore the dump file that Emacs installs somewhere in
;; libexec/ to its original state
(lambda* (#:key outputs target #:allow-other-keys)
(let* ((libexec (string-append (assoc-ref outputs "out")
"/libexec"))
;; each of these ought to only match a single
file,
;; but even if not (find-files) sorts by string<,
;; so the Nth element in one maps to the Nth
element of
;; the other
(pdmp (find-files libexec "\\.pdmp$"))
(pdmp-real (find-files libexec "\\.pdmp-real$")))
(for-each rename-file pdmp-real pdmp))))
(add-after 'glib-or-gtk-wrap 'strip-double-wrap
(lambda* (#:key outputs #:allow-other-keys)
;; Directly copy emacs-X.Y to emacs, so that it is not
wrapped
;; twice. This also fixes a minor issue, where WMs would
not be
;; able to track emacs back to emacs.desktop.
(with-directory-excursion (assoc-ref outputs "out")
(copy-file
(car (find-files "bin" "^emacs-([0-9]+\\.)+[0-9]+$"))
"bin/emacs"))))
(add-after 'strip-double-wrap 'wrap-emacs-paths
(lambda* (#:key inputs outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))
(lisp-dirs (find-files (string-append out
"/share/emacs")
"^lisp$"
#:directories? #t)))
(for-each
(lambda (prog)
(wrap-program prog
;; emacs-next and variants rely on uname being in
PATH for
;; Tramp. Tramp paths can't be hardcoded, because
they
;; need to be portable.
`("PATH" suffix
,(map dirname
(list (search-input-file inputs
"/bin/gzip")
;; for coreutils
(search-input-file inputs
"/bin/yes"))))
`("EMACSLOADPATH" suffix ,lisp-dirs)))
(find-files (string-append out "/bin")
;; Matches versioned and unversioned emacs
binaries.
;; We don't patch emacsclient, because it
takes its
;; environment variables from emacs.
;; Likewise, we don't need to patch helper
binaries
;; like etags, ctags or ebrowse.
"^emacs(-[0-9]+(\\.[0-9]+)*)?$"))))))))
(inputs
(list gnutls
ncurses

;; For native compilation
binutils
glibc
libgccjit

;; Required for "core" functionality, such as dired and
compression.
coreutils
gzip

;; Avoid Emacs's limited movemail substitute that retrieves
POP3
;; email only via insecure channels.
;; This is not needed for (modern) IMAP.
mailutils

gpm
libx11
gtk+
cairo
pango
harfbuzz
libxft
libtiff
giflib
lcms
libjpeg-turbo
libselinux
acl
jansson
gmp
ghostscript
poppler
elogind

;; When looking for libpng `configure' links with `-lpng -
lz', so we
;; must also provide zlib as an input.
libpng
zlib
(if (target-x86-64?)
librsvg-bootstrap
librsvg-2.40)
libxpm
libxml2
libice
libsm
alsa-lib
dbus

;; multilingualization support
libotf
m17n-lib))
(native-inputs
(list autoconf pkg-config texinfo))
(native-search-paths
(list (search-path-specification
(variable "EMACSLOADPATH")
(files '("share/emacs/site-lisp")))
(search-path-specification
(variable "INFOPATH")
(files '("share/info")))))

(synopsis "The extensible, customizable, self-documenting text
editor")
(description
"GNU Emacs is an extensible and highly customizable text editor.
It is
based on an Emacs Lisp interpreter with extensions for text editing.
Emacs
has been extended in essentially all areas of computing, giving rise to
a
vast array of packages supporting, e.g., email, IRC and XMPP messaging,
spreadsheets, remote server editing, and much more. Emacs includes
extensive
documentation on all aspects of the system, from basic editing to
writing
large Lisp programs. It has full Unicode support for nearly all human
languages.")
(license license:gpl3+)))
J
J
John Kehayias wrote on 5 Aug 2022 22:01
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
aFZzdBZYLsLYrGQ5lraauHWZPt_tUA1MnvM6VyKW4oyhxlF86lNNsRgRjpdAg1hrP0FrLppcRMi5UNyP7Wt00zZkVpUGl4Wtwrqt5WlBBC4=@protonmail.com
Hi,

Looks like the code you sent got line wrapped and I couldn't easily untangle it. The difference that pops out to me is how you are locating the lib/gcc directory, is it maybe picking up from the build system rather than libgccjit? (though I think gcc:lib should be the same here, but not gcc-toolchain)

In any event, I just want to reiterate that the libgccjit and emacs-native-comp from Andrew's channel does indeed work: it compiles, runs, and does native compilations successfully. So hopefully we've narrowed down now that there is some difference in the code you tried and from Andrew's that is leading to different behavior.

John
L
L
Liliana Marie Prikler wrote on 5 Aug 2022 23:31
(name . John Kehayias)(address . john.kehayias@protonmail.com)
1df52a98b3eaf5e49bbbef852dd33d188b16f757.camel@gmail.com
Am Freitag, dem 05.08.2022 um 20:01 +0000 schrieb John Kehayias:
Toggle quote (7 lines)
> Hi,
>
> Looks like the code you sent got line wrapped and I couldn't easily
> untangle it. The difference that pops out to me is how you are
> locating the lib/gcc directory, is it maybe picking up from the build
> system rather than libgccjit? (though I think gcc:lib should be the
> same here, but not gcc-toolchain)
No: /gnu/store/640hfpr069hvqv9gcs0ayq0is21zfii6-libgccjit-
10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0

Toggle quote (5 lines)
> In any event, I just want to reiterate that the libgccjit and emacs-
> native-comp from Andrew's channel does indeed work: it compiles,
> runs, and does native compilations successfully. So hopefully we've
> narrowed down now that there is some difference in the code you tried
> and from Andrew's that is leading to different behavior.
The style changes are not the only thing at play here, though. In
particular, I am trying to build Emacs 28.1, whereas Andrew builds
other versions, though notably the newest one ought to include the
smoke test. You could also try other combinations for libgccjit and
gcc to see if those make a difference – last time I tried it did not.

Attached the definitions as a file this time.

Cheers
J
J
John Kehayias wrote on 6 Aug 2022 07:37
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
q2VSuzWAkYniVdF04bfQHL_GK4DZODzOdVqeJQEDvAZXdFXGwKmXQnyZVWnXfeYv5FCIfsPwM1uHpuC0w_P6fld4oRza-cu8cszjC35CnCU=@protonmail.com
Hello,

------- Original Message -------
On Friday, August 5th, 2022 at 5:31 PM, Liliana Marie Prikler wrote:

Toggle quote (7 lines)
> The style changes are not the only thing at play here, though. In
> particular, I am trying to build Emacs 28.1, whereas Andrew builds
> other versions, though notably the newest one ought to include the
> smoke test. You could also try other combinations for libgccjit and
> gcc to see if those make a difference – last time I tried it did not.
>

I'm seeing the same version, 28.1.90, in yours and from Andrew's channel. And just checking that indeed it is 28.1.90 that I've been running locally from that channel, with libgccjit@11.3.0. I have not tried with v29.

Toggle quote (3 lines)
> Attached the definitions as a file this time.
>

Thanks, will play around with it this weekend to see if I have anything useful to add.
L
L
Liliana Marie Prikler wrote on 6 Aug 2022 07:53
(name . John Kehayias)(address . john.kehayias@protonmail.com)
2368e93f4beab4a962a324c33a234786ac45848b.camel@gmail.com
Am Samstag, dem 06.08.2022 um 05:37 +0000 schrieb John Kehayias:
Toggle quote (17 lines)
> Hello,
>
> ------- Original Message -------
> On Friday, August 5th, 2022 at 5:31 PM, Liliana Marie Prikler wrote:
>
> > The style changes are not the only thing at play here, though. In
> > particular, I am trying to build Emacs 28.1, whereas Andrew builds
> > other versions, though notably the newest one ought to include the
> > smoke test. You could also try other combinations for libgccjit and
> > gcc to see if those make a difference – last time I tried it did
> > not.
> >
>
> I'm seeing the same version, 28.1.90, in yours and from Andrew's
> channel. And just checking that indeed it is 28.1.90 that I've been
> running locally from that channel, with libgccjit@11.3.0. I have not
> tried with v29.
Pardon that, it's an artifact from trying to fetch 28.1.90 via url-
fetch and failing. Note that the hash is the one for 28.1.
J
J
John Kehayias wrote on 7 Aug 2022 05:19
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
t9upvjsXJyLxaD7Vu0lDBHVsS8xkXhzsF0yDs5b2b4qKSd8-lMdUu1yyitEj3im4aQrLkKP0jnSCwUQdB7oMsHP1aYMBMPvRSwqJPwI7_6M=@protonmail.com
Hi,

------- Original Message -------
On Saturday, August 6th, 2022 at 1:53 AM, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

Toggle quote (3 lines)
> Pardon that, it's an artifact from trying to fetch 28.1.90 via url-
> fetch and failing. Note that the hash is the one for 28.1.

On Andrew's channel I can change the version/commit/hash to 28.1 and it builds fine. Though this is using a git checkout at the 28.1 tag commit, but I don't think that matters here? (Emacs runs too, but only tried in guix shell and just to see it opens, didn't see if it compiles anything.)

And yet with your code I cannot. I tried using the phase from Andrew instead of your code, tried adding gcc to the native-inputs to match Andrew, used gcc-11 with libgccjit-11, ...still the same configure failure on the libgccjit test program: can't find libgccjit.so

Weird.

Adding just the libgccjit library path to LD_LIBRARY_PATH makes configure find libgccjit, but then fails to find libx11 in that same libgccjit test program. Instead, making LD_LIBRARY_PATH=LIBRARY_PATH fixes the configure and it finally fails on the runpath validation. Maybe cause of messing with LD_LIBRARY_PATH or could use the patch Andrew has? Anyway, wasn't looking into this.

I thought I had a similar problem with the test program you had started with, but in the end just the LIBRARY_PATH tweak was needed. I wish I could remember the combination of things I tried that also had it failing to find libgccjit. Or perhaps it was something libgccjit was linked to, but I hadn't explored?

So. I tried to see what was different between your and Andrew's code and I'm not seeing what it could be. There must be some subtle difference we're missing? Seems we have all the ingredients and a known working example.

John
L
L
Liliana Marie Prikler wrote on 7 Aug 2022 15:59
(name . John Kehayias)(address . john.kehayias@protonmail.com)
462ab8b824dc49f099b81b1e2eb8f0e59a0d7e39.camel@gmail.com
Am Sonntag, dem 07.08.2022 um 03:19 +0000 schrieb John Kehayias:
Toggle quote (4 lines)
> So. I tried to see what was different between your and Andrew's code
> and I'm not seeing what it could be. There must be some subtle
> difference we're missing? Seems we have all the ingredients and a
> known working example.
It's a really stupid one. Basically, the tests and really any
executable you try to build fails to execute without LD_LIBRARY_PATH
set. Now obviously, that's an issue with ld and you know which package
has ld? That's right, it's binutils!

In Andrew's recipe, he sneakily snarfed out binutils from inputs using
assoc-ref, but I'm using the new package style with (this-package-
input) to achieve the same thing, so I had to add binutils. But this
now shadows ld-wrapper. So we have to add ld-wrapper as well. Now
this makes me question whether the driver options are actually sane or
whether we'd have to prepend ld-wrapper to those as well.

WDYT, Andrew?
J
J
John Kehayias wrote on 7 Aug 2022 17:09
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
5SoKwC-pzglh33cUxrCFU0cfNr-6bvU4E_fNwP3eWNBZLC8f38M6b_-dv3Th39qgzxVgXXiBxJpXbGPxq9UOJNY9ulVUXcO6zMXIxr5agnk=@protonmail.com
------- Original Message -------
On Sunday, August 7th, 2022 at 9:59 AM, Liliana Marie Prikler wrote:

Toggle quote (6 lines)
> It's a really stupid one. Basically, the tests and really any
> executable you try to build fails to execute without LD_LIBRARY_PATH
> set. Now obviously, that's an issue with ld and you know which package
> has ld? That's right, it's binutils!
>

Ah! That LD_LIBRARY_PATH was needed was the clue.

Toggle quote (8 lines)
> In Andrew's recipe, he sneakily snarfed out binutils from inputs using
> assoc-ref, but I'm using the new package style with (this-package-
> input) to achieve the same thing, so I had to add binutils. But this
> now shadows ld-wrapper. So we have to add ld-wrapper as well. Now
> this makes me question whether the driver options are actually sane or
> whether we'd have to prepend ld-wrapper to those as well.
>

Is the assoc-ref for binutils (implicit input?) equivalent to just using #$binutils directly? e.g. (string-append "-B" #$binutils "/bin/") with no added binutils to the inputs. Is that not a good idea? As a test, that does indeed work, everything configures, builds, and runs (only tested opening with no configuration).

Toggle quote (1 lines)
> WDYT, Andrew?
L
L
Liliana Marie Prikler wrote on 7 Aug 2022 17:41
(name . John Kehayias)(address . john.kehayias@protonmail.com)
c52f6607e58f41b779a29c2eef40b075e8f5c862.camel@gmail.com
Am Sonntag, dem 07.08.2022 um 15:09 +0000 schrieb John Kehayias:
Toggle quote (5 lines)
> Is the assoc-ref for binutils (implicit input?) equivalent to just
> using #$binutils directly? e.g. (string-append "-B" #$binutils
> "/bin/") with no added binutils to the inputs. Is that not a good
> idea? As a test, that does indeed work, everything configures,
> builds, and runs (only tested opening with no configuration).
Not quite, because this binutils is actually binutils-final from
commencement.scm, which it doesn't seem we can import the normal way.
Also doing this would make it so that binutils isn't configurable,
which with an input you get. The trick is just to also use make-ld-
wrapper from base.scm

Cheers
L
L
Liliana Marie Prikler wrote on 9 Aug 2022 20:37
[PATCH 0/6] Add native compilation to Emacs
(address . guix-patches@gnu.org)
4c232648145659a2c3edca6d32725d8120cc14d3.camel@gmail.com
Hi Guix,

at long last the following patch set should enable native compilation
for both Emacs and emacs-build-system. I tested emacs-dash and at the
very least native code is generated, though I haven't yet checked
whether it is also loaded.

As with any shiny new Emacs feature, please verify that the Emacs
portion of your manifests/home configurations build and report any
related errors *before* I push this and curse your configuration.

Cheers

Liliana Marie Prikler (6):
gnu: Parameterize libgccjit.
gnu: libgccjit: Build with bootstrapped gcc.
gnu: libgccjit: Build multiple versions.
gnu: emacs: Build with native compilation.
guix: emacs-utils: Add emacs-compile-directory.
build-system: emacs: Use native compilation.

gnu/packages/emacs.scm | 64 ++++++++++++++++++++++++++++++-
gnu/packages/gcc.scm | 53 +++++++++++++++++--------
guix/build/emacs-build-system.scm | 5 ++-
guix/build/emacs-utils.scm | 26 +++++++++++++
4 files changed, 128 insertions(+), 20 deletions(-)

--
2.37.0
Closed
J
J
John Kehayias wrote on 9 Aug 2022 22:44
Re: libgccjit is unusable
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)
B8CApiFvNSCTNNxMAiyApTJzI2makyG69nPaGc_TMEBQ-xJHUBwAJcqq4kp3NRXp8L_XArmz9K0lWI8IZA_DVoPF8P16sFXLYckicFxBqGA=@protonmail.com
Thanks for the new patchset Lily, very excited to have emacs with native compiliation in upstream Guix! (At the very least so I don't have to compile libgccjit and emacs)

For everyone following along at home, please see https://issues.guix.gnu.org/57086

As for the original issue here, I guess the LIBRARY_PATH and ld shadowing is the workaround. I don't know if that is something that can/should be incorporated into the libgccjit package definition or should just be handled by any package using it. Currently, that will just be emacs anyway and hopefully the discussion here is useful to anyone trying to use libgccjit.

Thanks again Lily and Andrew for your work here!
A
A
Andrew Whatson wrote on 10 Aug 2022 01:53
(name . John Kehayias)(address . john.kehayias@protonmail.com)
CAPE069dbexuxVo1axHHdp+q9-N0BxpmqD7y7KhvgVKeS92AvhQ@mail.gmail.com
John Kehayias wrote:
Toggle quote (3 lines)
>
> As for the original issue here, I guess the LIBRARY_PATH and ld shadowing is the workaround. I don't know if that is something that can/should be incorporated into the libgccjit package definition or should just be handled by any package using it. Currently, that will just be emacs anyway and hopefully the discussion here is useful to anyone trying to use libgccjit.

This might be possible by adding LIBRARY_PATH to native-search-paths
in the libgccjit package definition?

Toggle quote (2 lines)
> Thanks again Lily and Andrew for your work here!

Thanks John & Lily for persisting with getting this submitted, I'm
very grateful for your efforts.

Cheers,
Andrew
?