How to compile my own glibc C standard library from source and use it?
Last tested on Ubuntu 20.04 with glibc development version 2.33.9000 (see glibc/version.h
) on 27 June 2021.
How to download and build glibc, and run its benchmarks
You can manually obtain the glibc source code here: https://www.gnu.org/software/libc/sources.html:
git clone https://sourceware.org/git/glibc.git
cd glibc
git checkout master
3rd party mirror on GitHub: https://github.com/bminor/glibc/tree/master/benchtests
See also:
- https://kazoo.ga/a-simple-tool-to-test-malloc-performance/
If you ever wish to manually build glibc and its benchtests, do so as follows:
# IMPORTANT: begin AT THE SAME DIRECTORY LEVEL as the `glibc` source code
# directory, NOT inside the `glibc` source code dir! In other words, if
# you are in the correct dir, running `ls` will show the `glibc` source
# code dir (that you just cloned) inside the dir you are in.
mkdir -p glibc-build
mkdir -p glibc-install
cd glibc-build
../glibc/configure --prefix="$(realpath "../glibc-install")"
time make -j8 # build with 8 threads (jobs); on a fast laptop this takes ~3 min.
time make install # (optional: install to the `glibc-install` dir you created)
# Build the benchtests (everything inside the `glibc/benchtests` dir) too;
# see the makefile 'glibc/benchtests/Makefile' for more build commands.
time make bench-build -j8
# Now you have this executable file you can use for malloc speed tests, for instance!:
# ../glibc-build/benchtests/bench-malloc-thread
# To build **and run** all glibc benchtests, do:
time make bench
References:
- https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
- https://kazoo.ga/a-simple-tool-to-test-malloc-performance/
- https://github.com/f18m/malloc-benchmarks/blob/master/Makefile#L122-L129 : I learned much of this by studying this makefile target:
$(glibc_install_dir)/lib/libc.so.6: @echo "Building GNU libc... go get a cup of coffee... this will take time!" mkdir -p $(glibc_build_dir) cd $(glibc_build_dir) && \ ../glibc/configure --prefix=$(glibc_install_dir) && \ make $(parallel_flags) && \ make install [ -x $(glibc_build_dir)/benchtests/bench-malloc-thread ] && echo "GNU libc benchmarking utility is ready!" || echo "Cannot find GNU libc benchmarking utility! Cannot collect benchmark results"
- How to compile my own glibc C standard library from source and use it?
Keywords: how to build and run glibc and its benchtests, incl. malloc benchtests, from source; build glibc from source on linux ubuntu
The Makefile
is going to exist in your build-glibc
directory if the configure
script finishes successfully.
If everything seems to have gone smoothly during configure
and still no Makefile
, then you probably missed an idiosyncrasy:
While doing a configure
for glibc, it is expected that you normally provide an alternative --prefix
, because installing to the default location (/usr/local
) can potentially cripple the system. If you don't provide one, then you need to switch on --disable-sanity-checks
.
If this is not the case either, look for a config.log
file, and read its contents.
Setup 1: glibc without dedicated GCC
This setup might work and is quick as it does not recompile the whole GCC toolchain, just glibc.
The only problem I have with this setup is that I haven't found a nice way to use runtime objects such as crt1.o
, crti.o
, and crtn.o
provided by our glibc, and I'm using the host ones for now. This is mentioned at: https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location Those objects do early setup that glibc relies on, so I wouldn't be surprised if things crashed in wonderful and awesomely subtle ways. See attempts at solving this below.
Build glibc and install locally:
git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.32
mkdir build
cd build
export glibc_install="$(pwd)/install"
../configure --prefix "$glibc_install"
make -j `nproc`
make install -j `nproc`
Setup 1: verify the build
test_glibc.c
#define _GNU_SOURCE
#include <assert.h>
#include <gnu/libc-version.h>
#include <stdatomic.h>
#include <stdio.h>
#include <threads.h>
atomic_int acnt;
int cnt;
int f(void* thr_data) {
for(int n = 0; n < 1000; ++n) {
++cnt;
++acnt;
}
return 0;
}
int main(int argc, char **argv) {
/* Basic library version check. */
printf("gnu_get_libc_version() = %s\n", gnu_get_libc_version());
/* Exercise thrd_create from -pthread,
* which is not present in glibc 2.27 in Ubuntu 18.04.
* https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291 */
thrd_t thr[10];
for(int n = 0; n < 10; ++n)
thrd_create(&thr[n], f, NULL);
for(int n = 0; n < 10; ++n)
thrd_join(thr[n], NULL);
printf("The atomic counter is %u\n", acnt);
printf("The non-atomic counter is %u\n", cnt);
}
Compile and run with test_glibc.sh
:
#!/usr/bin/env bash
set -eux
gcc \
-L "${glibc_install}/lib" \
-I "${glibc_install}/include" \
-Wl,--rpath="${glibc_install}/lib" \
-Wl,--dynamic-linker="${glibc_install}/lib/ld-linux-x86-64.so.2" \
-std=c11 \
-o test_glibc.out \
-v \
test_glibc.c \
-pthread \
;
ldd ./test_glibc.out
./test_glibc.out
Command adapted from https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location
The program outputs the expected:
gnu_get_libc_version() = 2.32
The atomic counter is 10000
The non-atomic counter is 8674
ldd
output confirms that the ldd
and libraries that we've just built are actually being used as expected:
+ ldd test_glibc.out
linux-vdso.so.1 (0x00007ffe4bfd3000)
libpthread.so.0 => /home/ciro/glibc/build/install/lib/libpthread.so.0 (0x00007fc12ed92000)
libc.so.6 => /home/ciro/glibc/build/install/lib/libc.so.6 (0x00007fc12e9dc000)
/home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000)
The gcc
compilation debug output shows that my host runtime objects were used, which is bad as mentioned previously, but I don't know how to work around it, e.g. it contains:
COLLECT_GCC_OPTIONS=/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crt1.o
Setup 1: modify glibc
Now let's modify glibc with:
diff --git a/nptl/thrd_create.c b/nptl/thrd_create.c
index 113ba0d93e..b00f088abb 100644
--- a/nptl/thrd_create.c
+++ b/nptl/thrd_create.c
@@ -16,11 +16,14 @@
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
+#include <stdio.h>
+
#include "thrd_priv.h"
int
thrd_create (thrd_t *thr, thrd_start_t func, void *arg)
{
+ puts("hacked");
_Static_assert (sizeof (thr) == sizeof (pthread_t),
"sizeof (thr) != sizeof (pthread_t)");
Then recompile and re-install glibc, and recompile and re-run our program:
cd glibc/build
make -j `nproc`
make -j `nproc` install
./test_glibc.sh
and we see hacked
printed a few times as expected.
This further confirms that we actually used the glibc that we compiled and not the host one.
Tested on Ubuntu 20.10.
Setup 1: attempts at using the correct crt*
objects
https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location suggests adding --sysroot
to the gcc
command but:
- it does not really change the objects to ours according to logs
- and it makes compilation fail with
/usr/bin/ld: cannot find libgcc_s.so.1
presumably because thesysroot
gets used for this GCC provided object, which we don't have at that sysroot because we only built glibc
At https://stackoverflow.com/a/66634184/895245 ZeZNiQ provides a workaround that is likely correct, by passing:
-nostartfiles
followed by all the objects. You just need to extract the correct objects from the full command with -nostartfiles
and pass them manually.
For example, on my amd64 machine, the objects used were different from his 32-bit command, so this is a bit fiddly.
Bibliography:
- How do I change GCC's default search directory for crti.o?
- https://gcc.gnu.org/legacy-ml/gcc-help/2015-02/msg00016.html
- https://gcc.gnu.org/legacy-ml/gcc-help/2001-11/msg00029.html
Setup 2: crosstool-NG pristine setup
This is an alternative to setup 1, and it is the most correct setup I've achieved far: everything is correct as far as I can observe, including the C runtime objects such as crt1.o
, crti.o
, and crtn.o
.
In this setup, we will compile a full dedicated GCC toolchain that uses the glibc that we want.
The only downside to this method is that the build will take longer. But I wouldn't risk a production setup with anything less.
crosstool-NG is a set of scripts that downloads and compiles everything from source for us, including GCC, glibc and binutils.
Yes the GCC build system is so bad that we need a separate project for that.
This setup is only not perfect because crosstool-NG does not support building the executables without extra -Wl
flags, which feels weird since we've built GCC itself. But everything seems to work, so this is only an inconvenience.
Get crosstool-NG, configure and build it:
git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
git checkout a6580b8e8b55345a5a342b5bd96e42c83e640ac5
export CT_PREFIX="$(pwd)/.build/install"
export PATH="/usr/lib/ccache:${PATH}"
./bootstrap
./configure --enable-local
make -j `nproc`
./ct-ng x86_64-unknown-linux-gnu
./ct-ng menuconfig
env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`
The build takes about thirty minutes to two hours.
The only mandatory configuration option that I can see, is making it match your host kernel version to use the correct kernel headers. Find your host kernel version with:
uname -a
which shows me:
4.15.0-34-generic
so in menuconfig
I do:
Operating System
Version of linux
so I select:
4.14.71
which is the first equal or older version. It has to be older since the kernel is backwards compatible.
Setup 2: optional configurations
The .config
that we generated with ./ct-ng x86_64-unknown-linux-gnu
has:
CT_GLIBC_V_2_27=y
To change that, in menuconfig
do:
C-library
Version of glibc
save the .config
, and continue with the build.
Or, if you want to use your own glibc source, e.g. to use glibc from the latest git, proceed like this:
Paths and misc options
Try features marked as EXPERIMENTAL
: set to true
C-library
Source of glibc
Custom location
: say yesCustom location
Custom source location
: point to a directory containing your glibc source
where glibc was cloned as:
git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.28
Setup 2: test it out
Once you have built he toolchain that you want, test it out with:
#!/usr/bin/env bash
set -eux
install_dir="${CT_PREFIX}/x86_64-unknown-linux-gnu"
PATH="${PATH}:${install_dir}/bin" \
x86_64-unknown-linux-gnu-gcc \
-Wl,--dynamic-linker="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib/ld-linux-x86-64.so.2" \
-Wl,--rpath="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib" \
-v \
-o test_glibc.out \
test_glibc.c \
-pthread \
;
ldd test_glibc.out
./test_glibc.out
Everything seems to work as in Setup 1, except that now the correct runtime objects were used:
COLLECT_GCC_OPTIONS=/home/ciro/crosstool-ng/.build/install/x86_64-unknown-linux-gnu/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crt1.o
Setup 2: failed efficient glibc recompilation attempt
It does not seem possible with crosstool-NG, as explained below.
If you just re-build;
env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`
then your changes to the custom glibc source location are taken into account, but it builds everything from scratch, making it unusable for iterative development.
If we do:
./ct-ng list-steps
it gives a nice overview of the build steps:
Available build steps, in order:
- companion_tools_for_build
- companion_libs_for_build
- binutils_for_build
- companion_tools_for_host
- companion_libs_for_host
- binutils_for_host
- cc_core_pass_1
- kernel_headers
- libc_start_files
- cc_core_pass_2
- libc
- cc_for_build
- cc_for_host
- libc_post_cc
- companion_libs_for_target
- binutils_for_target
- debug
- test_suite
- finish
Use "<step>" as action to execute only that step.
Use "+<step>" as action to execute up to that step.
Use "<step>+" as action to execute from that step onward.
therefore, we see that there are glibc steps intertwined with several GCC steps, most notably libc_start_files
comes before cc_core_pass_2
, which is likely the most expensive step together with cc_core_pass_1
.
In order to build just one step, you must first set the "Save intermediate steps" in .config
option for the intial build:
Paths and misc options
Debug crosstool-NG
Save intermediate steps
and then you can try:
env -u LD_LIBRARY_PATH time ./ct-ng libc+ -j`nproc`
but unfortunately, the +
required as mentioned at: https://github.com/crosstool-ng/crosstool-ng/issues/1033#issuecomment-424877536
Note however that restarting at an intermediate step resets the installation directory to the state it had during that step. I.e., you will have a rebuilt libc - but no final compiler built with this libc (and hence, no compiler libraries like libstdc++ either).
and basically still makes the rebuild too slow to be feasible for development, and I don't see how to overcome this without patching crosstool-NG.
Furthermore, starting from the libc
step didn't seem to copy over the source again from Custom source location
, further making this method unusable.
Bonus: stdlibc++
A bonus if you're also interested in the C++ standard library: How to edit and re-build the GCC libstdc++ C++ standard library source?
Adding to Ciro's earlier answer/solution https://stackoverflow.com/a/52454710/4726668 :
@CiroSantilli Editing your answer returns "Suggested edit queue is full". The ldd script you are calling in the test_glibc.sh
script points to the host dynamic linker: /home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000)
. To fix this, in test_glibc.sh
, change ldd
to ${glibc_install}/bin/ldd
. This will require you to add the built crt
*.o files as well to the script:
-nostartfiles \
${glibc_install}/lib/crti.o \
${glibc_install}/lib/crtn.o \
${glibc_install}/lib/crt1.o \
On my GNU/Linux i386/i686 (32-bit x86 arch) machine, following is my working test_glibc.sh
:
#!/usr/bin/env bash
set -eux
gcc \
-L "${glibc_install}/lib" \
-I "${glibc_install}/include" \
-Wl,--rpath="${glibc_install}/lib" \
-Wl,--dynamic-linker="${glibc_install}/lib/ld-linux.so.2" \
-std=c11 \
-nostartfiles \
${glibc_install}/lib/crti.o \
${glibc_install}/lib/crtn.o \
${glibc_install}/lib/crt1.o \
-o test_glibc.out \
-v \
test_glibc.c \
-pthread \
;
${glibc_install}/bin/ldd ./test_glibc.out
./test_glibc.out