Invoking ld-linux.so
A Radical Solution for Distributing ELF to Non-FHS Compliant System
The Curse of NixOS
Poeple seem to have a love-hate relationship with NixOS. The Curse of NixOS is an excellent article by Wesley Aptekar-Cassels, which tells the good and the bad of NixOS, and serves as a great reading for background on this article.
Neither am I a user of NixOS, nor I want to be. Still, I got stormed by issues on GitHub from NixOS users, which can be summarized by a quote from NixOS Wiki:
Downloading and attempting to run a binary on NixOS will almost never work.
Here is a demonstration:
[nix-shell:~]# ./dart --version
bash: ./dart: No such file or directory
[nix-shell:~]# ls -l dart
-rwxr-xr-x 1 root root 5154328 May 31 23:59 dart
Running the dart executable downloaded from dart.dev would fail in a very confusing way on NixOS. At a glance it may look like bash is complaining about file ./dart does not exist. However, checking with ls would reject that idea immediately. This is why many NixOS users get lost, where they have no choice but to seek help from the developer. It is pretty common for developers to have no cue given the limited information that appear to be contradictory. I don’t blame puzzled NixOS users, as I would probably have no idea without my previous experience of dealing with three different libc on a single Linux system.
Executable and Linking Format
Linux executable files are usually in Executable and Linking Format (ELF). For dynamically linked programs, the loading of shared libraries is performed by ld-linux.so, the dynamic linker, whose location is hard coded in PT_INTERP program header. Most of the Linux distributions have the dynamic linker in a common location, so that programs compiled on one distribution can run on other distributions, as long as all dependencies are satisfied.
[nix-shell:~]# ldd dart
linux-vdso.so.1 (0x00007ffd53b6b000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f6b42fc7000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007f6b42fc2000)
libm.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libm.so.6 (0x00007f6b42ee2000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f6b42cd9000)
/lib64/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f6b43677000)
[nix-shell:~]# ls -l /lib64/ld-linux-x86-64.so.2
ls: cannot access '/lib64/ld-linux-x86-64.so.2': No such file or directory
NixOS is different that the ld-linux.so does not exist at its common location hence the “no such file or directory” error. This issue comes from a deliberate decision from Nix to ignore the Filesystem Hierarchy Standard (FHS), and NixOS created a tool called patchelf to rectify the consequences.
[nix-shell:~]# patchelf --set-interpreter /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 ./dart
[nix-shell:~]# ldd dart
linux-vdso.so.1 (0x00007fff717bc000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f523a6ed000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007f523a6e8000)
libm.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libm.so.6 (0x00007f523a608000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f523a3ff000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f523ada6000)
[nix-shell:~]# ./dart --version
Dart SDK version: 2.19.6 (stable) (Tue Mar 28 13:41:04 2023 +0000) on "linux_x64"
Once PT_INTERP program header is modified to the correct location for NixOS, the dart executable now works as expected. However, even with the tools being available, it’s still a trouble for developers who distribute precompiled Linux binaries. On NixOS, the location of ld-linux.so changes every time glibc is updated, therefore distributing already modified ELF is unreliable. Patching during at the runtime is also undependable as patchelf may not be available.
Invoking ld-linux.so
The dynamic linker is a shared library, but it is a special runnable one, which can be explicitly invoked to execute an ELF.
[nix-shell:~]# ./dart --version
bash: ./dart: No such file or directory
[nix-shell:~]# /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 ./dart --version
Dart SDK version: 2.19.6 (stable) (Tue Mar 28 13:41:04 2023 +0000) on "linux_x64"
The unmodified dart executable works on NixOS when explicitly invoked through ld-linux.so. As for finding the uncertain location of the dynamic linker on NixOS, it can be done by reading the program header from /proc/self/exe.
The Radical Solution
Normally, when facing this kind of issues, NixOS users would have to face the challenge of either modifying downloaded programs or building programs from source, which the vast majority of people using NixOS are not familiar with. Even after getting an offical repackaging in Nixpkgs, the compliants from NixOS users would not stop, as many users may still choose to download instead.
A radical solution was born to put a stop:
# Try to execute ELF.
begin
exec(*COMMAND, *ARGV)
# Catch the "no such file or directory" error.
rescue Errno::ENOENT
# Locate `ld-linux.so` by parsing `/proc/self/exe`.
# See: https://github.com/sass-contrib/sass-embedded-host-ruby/blob/main/lib/sass/elf.rb
require_relative 'elf'
# Rethrow if `ld-linux.so` cannot be located.
raise if ELF::INTERPRETER.nil?
# Invoke `ld-linux.so` to execute ELF.
exec(ELF::INTERPRETER, *COMMAND, *ARGV)
end
One More Thing
The solution above was shipped as part of the sass-embedded gem. It has worked for a while, until NixOS managed to break it with another innovative non-standard behavior in 24.05 release:
NixOS now installs a stub ELF loader that prints an informative error message when users attempt to run binaries not made for NixOS.
In the past, there was either a properly working ld-linux.so or none at its standard location. Now in NixOS a stub-ld that does nothing but immediately exits with an error message is installed at the standard location of ld-linux.so, so that ENOENT is no more when calling exec. The exec system call will always succeed in running stub-ld, the stub-ld will always fail, and the actual foreign binary will never execute. To avoid stub-ld altogether, the command must be modified to begin with a real ld-linux.so, before even calling exec.
A more radical solution was born:
# Locate `ld-linux.so` by parsing `/proc/self/exe`.
# See: https://github.com/sass-contrib/sass-embedded-host-ruby/blob/main/lib/sass/elf.rb
require_relative '../../lib/sass/elf'
module Sass
module CLI
# Preparsed `ld-linux.so` of the foreign binary, different for each platform
INTERPRETER = '/lib/ld-linux-aarch64.so.1'
# The last component of the preparsed `ld-linux.so`
INTERPRETER_SUFFIX = '/ld-linux-aarch64.so.1'
# Prepend the `ld-linux.so` of `/proc/self/exe` if it differs from the `ld-linux.so` of the foreign binary
# yet shares the same filename indicating that they are compatible.
COMMAND = [
*(ELF::INTERPRETER if ELF::INTERPRETER != INTERPRETER && ELF::INTERPRETER&.end_with?(INTERPRETER_SUFFIX)),
File.absolute_path('dart-sass/src/dart', __dir__).freeze,
File.absolute_path('dart-sass/src/sass.snapshot', __dir__).freeze
].freeze
end
private_constant :CLI
end
It’s unlikely that NixOS will be able to break it again. I hope.