Learning to compile things from source (on Unix/Linux/OSX)
Solution 1:
I apologise for directly answering everything, but I don't know any useful tutorials, FAQs, etc. Basically what follows is 8 years of making desktop apps (that I help distribute), frustration and googling:
1. How do I figure out what arguments to pass to ./configure?
Practice really. Autotools is easy enough as it is consistent. But there's plenty of stuff out there using cmake, or custom build scripts. Generally, you shouldn't have to pass anything to configure, it should figure out if your system can build foo-tool or not.
Configure and GNU tools all look in /, /usr and /usr/local for dependencies. If you install anything anywhere else (which makes things painful if the dependency was installed by MacPorts or Fink), you will have to pass a flag to configure or modify the shell's environment to help GNU tools find these dependencies.
2. How shared libraries work under OS X / Linux - where they live on the filesystem, how ./configure && make finds them, what actually happens when they are linked against
On Linux they need to be installed to a path that the dynamic linker can find, this is defined by the LD_LIBRARY_PATH environment variable and the contents of /etc/ld.conf. On Mac it is the same for most open source software almost always (unless it is an Xcode Project). Except the env variable is DYLD_LIBRARY_PATH instead.
There is a default path that the linker searches for libraries. It is /lib:/usr/lib:/usr/local/lib
You can supplement this by using the CPATH variable, or CFLAGS or any number of other environment variables really (conveniently complicated). I suggest CFLAGS like so:
export CFLAGS="$CFLAGS -L/new/path"
The -L parameter adds to the link path.
Modern stuff uses the pkg-config tool. Modern stuff you install also installs a .pc file that describes the library and where it is and how to link to it. This can make life easier. But it doesn't come with OS X 10.5 so you'll have to install that too. Also a lot of basic deps don't support it.
The act of linking is just "resolve this function at runtime", really it's a big string table.
3. What are the actual differences between a shared and a statically linked library? Why can't I just statically link everything (RAM and disk space are cheap these days) and hence avoid weird library version conflicts?
When you link to a static library file the code becomes part of your application. It would be like if there was one giant .c file for that library and you compiled it into your application.
Dynamic libraries have the same code, but when the app is run, the code is loaded into the app at runtime (simplified explanation).
You can statically link to everything, however, sadly hardly any build systems make this easy. You'd have to edit build system files manually (eg. Makefile.am, or CMakeLists.txt). However this is probably worth learning if you regularly install things that require different versions of libraries and you are finding installing dependencies in parallel difficult.
The trick is to change the link line from -lfoo to -l/path/to/static/foo.a
You can probably find and replace. Afterwards check the tool doesn't link to the .so or dylib using ldd foo or otool -L foo
Another problem is not all libraries compile to static libraries. Many do. But then MacPorts or Debian may have decided not to ship it.
4. How can I tell what libraries I have installed, and what versions?
If you have pkg-config files for those libraries it is easy:
pkg-config --list-all
Otherwise you often can't easily. The dylib may have a soname (ie. foo.0.1.dylib, the soname is 0.1) that is the same as the library's version. However this is not required. The soname is a binary computability feature, you have to bump the major part of the soname if you change the format of the functions in the library. So you can get eg. version 14.0.5 soname for a 2.0 library. Although this is not common.
I got frustrated with this sort of thing and have developed a solution for this on Mac, and I'm talking about it next.
5. How can I install more than one version of a library without breaking my normal system?
My solution to this is here: http://github.com/mxcl/homebrew/
I like installing from source, and wanted a tool that made it easy, but with some package management. So with Homebrew I build, eg. wget myself from source, but make sure to install to a special prefix:
/usr/local/Cellar/wget/1.1.4
I then use the homebrew tool to symlink all that into /usr/local, so I still have /usr/local/bin/wget and /usr/local/lib/libwget.dylib
Later if I need a different version of wget I can install it in parallel and just change the version that is linked into the /usr/local tree.
6. If I am installing stuff from source on a system that is otherwise managed using packages, what's the cleanest way of doing so?
I believe the Homebrew way is cleanest, so use it or do the equivalent. Install to /usr/local/pkgs/name/version and symlink or hard link the rest in.
Do use /usr/local. Every build tool that exists searches there for dependencies and headers. Your life will be much easier.
7. Assuming I manage to compile something fiddly from source, how can I then package that up so other people don't have to jump through the same hoops? Particularly on OS X....
If it has no dependencies you can tar up the the build directory and give it to someone else who can then do "make install". However you can only do this reliably for the exact same versions of OS X. On Linux it will probably work for similar Linux (eg. Ubuntu) with the same Kernel version and libc minor version.
The reason it is not easy to distribute binaries on Unix is because of binary compatibility. The GNU people, and everyone else change their binary interfaces often.
Basically don't distribute binaries. Things will probably break in very strange ways.
On Mac, the best option is to make a macports package. Everyone uses macports. On Linux there are so many different build systems and combinations, I don't think there is any better advise than to write a blog entry about how you succeeded building x tool in y strange configuration.
If you make a package description (for macports or homebrew) then anyone can install that package, and it solves the dependency problems too. However this is often not easy, and it also isn't easy to get your macports recipe included in the main macports tree. Also macports doesn't support exotic installation types, they offer one choice for all packages.
One of my future goals with Homebrew is to make it possible to click a link on a website (eg. homebrew://blah and it will download that Ruby script, install the deps for that package and then build the app. But yeah, not yet done, but not too tricky considering the design I chose.
8. What are the command line tools I need to master to get good at this stuff? Stuff like otool, pkg-config etc.
otool is really only useful afterwards. It tells you what the built binary links to. When you are figuring out the dependencies of a tool you have to build, it is useless. The same is true of pkg-config as you will have already installed the dependency before you can use it.
My tool chain is, read the README and INSTALL files, and do a configure --help. Watch the build output to check it is sane. Parse any build errors. Maybe in future, ask on serverfault :)
Solution 2:
This is a huge topic so lets start with shared libraries on Linux (ELF on Linux and Mach-O on OS X), Ulrich Drepper has a good introduction to writing DSOs (dynamic shared objects) which covers some history of shared libraries on Linux available here including why they are important
Ulrich also describes why static linking is considered harmful one of the key points here is security updates. Buffer overflows in a common library (eg zlib) that is extensively linked statically can cause a huge overhead for distributions - this occured with zlib 1.1.3 (Red Hat advisory)
ELF
The linker ld.so manual page
man ld.so
explains the basic paths and files involved in runtime dynamic linking. On modern Linux systems you'll see additional paths added via /etc/ld.so.conf.d/ added usually via a glob include in /etc/ld.so.conf.
If you want to see what is available dynamically via your ld.so configuration you can run
ldconfig -v -N -X
Reading the DSO howto should give you a good basic level of knowledge in order to then go on to understand how those principles apply to Mach-O on OS X.
Mach-O
On OS X the binary format is Mach-O. Local system documentation for the linker is
man dyld
The Mach format documentation is available from Apple
UNIX build tools
The common configure, make, make install process is generally provided by GNU autotools which has an online book that covers some of the history of the configure/build split and the GNU toolchain. Autoconf uses tests to determine feature availability on the target build system, it uses M4 macro language to drive this. Automake is basically a templating method for Makefiles, the template generally being called Makefile.am which outputs a Makefile.in that the output of autoconf (the configure script) converts into a Makefile.
The GNU hello program acts as a good example for understanding the GNU toolchain - and the manual includes autotools documentation.
Solution 3:
Simon! I know how you feel; I struggled with this part of learning Linux, too. Based on my own experiences, I wrote a tutorial about some of the items that you address (mostly as a reference for myself!): http://easyaspy.blogspot.com/2008/12/buildinginstalling-application-from.html. I think you'll appreciate my note about how simple Python applications are to build/install. :)
Hope that this helps! And happy compiling.
Tim Jones
Building/Installing an application from source in Ubuntu Linux
While the Ubuntu repositories are chock full of great applications, at one time or another you are bound to come across that "must-have" tool that isn't in the repositories (or doesn't have a Debian package) or you need a newer version than in the repositories. What do you do? Well, you have to build the application from source! Don't worry, it's really not as complicated as it sounds. Here are some tips, based on my experiences of going from being a rank amateur! (While I'm using Ubuntu for this example, the general concepts should be applicable to most any Unix/Linux distribution, such as Fedora, and even the Cygwin platform on Windows.)
The basic process of building (compiling) most applications from source follows this sequence: configure --> compile --> install. The typical Unix/Linux commands to do these things are: config --> make --> make install. In some cases, you'll even find web pages that show that all of these can be combined into a single command:
$ config && make && make install
Of course, this command assumes that there are no problems in any of these steps. This is where the fun comes in!
Getting Started
If you haven't compiled an application from source on your system before, you will probably need to set it up with some general development tools, such as the gcc compiler suite, some common header files (think of this as code that has already been written by someone else that is used by the program you are installing), and the make tool. Fortunately, in Ubuntu, there is a metapackage called build-essential that will install of this. To install it (or just make sure that you already have it!), run this command in the terminal:
$ sudo apt-get install build-essential
Now that you have the basic setup, download the application source files and save them to a directory for which you have read/write permissions, such as your "home" directory. Typically, these will be in an archive file with file extension of either .tar.gz or .tar.bz2. The .tar simply means that it's a "tape archive", which is a grouping of files that preserves their relative directory structure. The .gz stands for gzip (GNU zip), which is a popular Unix/Linux compression format. Similarly, the .bz2 stands for bzip2, which is a newer compression format that provides higher compression (smaller compressed file size) than gzip.
After you've downloaded the source file, open a terminal window (System Terminal from the Ubuntu menu) and change to the directory where you saved your file. (I'll use ~/download in this example. Here, '~' is a shortcut to your "home" directory.) Use the tar command to extract the files from the downloaded archive file:
If your file is a gzip archive (e.g., ends with .tar.gz), use the command:
$ tar -zxvf filename.tar.gz
If your file is a bzip2 archive (e.g., ends with .tar.bz2), use the command:
$ tar -jxvf filename.tar.gz
Tip: If you don't want to have to remember all of the command line switches for extracting archives, I recommend getting one (or both) of these utilities: dtrx (my favorite!) or deco (more popular). With either of these utilities, you just enter the name of the utility (dtrx or deco) and the filename, it it does all of the rest. Both of these "know" how to handle most any archive format that you are likely to run across and they have great error handling.
When building from source, there are two common types of errors that you are likely to encounter:
- Configuration errors occur when you run the configuration script (usually named config or configure) to create a makefile that is specific to your setup.
- Compiler errors happen when you run the make command (after the makefile has been generated) and the compiler is unable to find some code that it needs.
We'll look at each of these and discuss how to resolve them.
Configuration and Configuration Errors
After you've extracted the source code archive file, in the terminal, you should change to the directory that contains the extracted files. Typically, this directory name will be the same as the name of the file (without the .tar.gz or .tar.bz2 extension). However, sometimes the directory name is just the name of the application, without any version information.
In the source directory look for a README file and/or an INSTALL file (or something with similar names). These files typically contain useful information about how to build/compile the application and to install it, including information about dependencies. "Dependencies" are just a fancy name for other components or libraries that are required to successfully compile.
After you've read the README and/or INSTALL file (and, hopefully looked at any relevant online documentation for the application), look for an executable (has the "x" permission set on the file) file named config or configure. Sometimes the file might have an an extension, such as .sh (e.g., config.sh). This is usually a shell script that runs some other utilities to confirm that you have a "sane" environment for compiling. In other words, it will check to ensure that you have everything installed that you need.
Tip: If this is a Python-based application, instead of a config file, you should find a file named
setup.py. Python applications are typically very simple to install. To install this application, as root (e.g., put sudo in front of the following command under Ubuntu), run this command:$ python setup.py installThat should be all that you need to do. You can skip the remainder of this tutorial and proceed directly to using and enjoying your application.
Run the configuration script in the terminal. Typically, you can (and should!) run your configuration script with your regular user account.
$ ./config
The script will display some messages to give you an idea of what it is doing. Often, the script will give you an indication of whether it succeeded or failed and, if it failed, some information about the cause of the failure. If you don't get any error messages, then you can usually assume that everything went fine.
If you don't find any script that looks like a configuration script, then it typically means that the application is a very simple one and it is platform independent. This means that you can simply skip to the build/compile step below, because the provided Makefile should work on any system.
An Example
In this tutorial, I'm going to use the text-based RSS reader called Newsbeuter as an example for the types of errors that you may encounter when building your application. For Newsbeuter, the name of the configuration script is config.sh. On my system, when I run config.sh, the following errors occur:
tester@sitlabcpu22:~/download/newsbeuter-1.3$ ./config.sh
Checking for package sqlite3... not found
You need package sqlite3 in order to compile this program.
Please make sure it is installed.
Upon doing some research, I found that, in fact, the sqlite3 application was installed. However, since I'm trying to build from source, this is a tip that what config.sh is actually looking for are the development libraries (headers) for sqlite3. In Ubuntu, most packages have an associated development counterpart package that ends in -dev. (Other platforms, such as Fedora, often use a package suffix of -devel for the development packages.)
To find the appropriate package for the sqlite3 development package, we can use the apt-cache utility in Ubuntu (and, similarly, the yum utility in Fedora):
tester@sitlabcpu22:~/download/newsbeuter-1.3$ sudo apt-cache search sqlite
This command returns quite a big list of results, so we have to do a bit of detective work to determine which is the appropriate package. In this case, the appropriate package turns out to be libsqlite3-dev. Notice that sometimes the package we are looking for will have the lib prefix, instead of just the same package name plus -dev. This is because sometimes we are just looking for a shared library that may be used by many different applications. To install libsqlite3-dev, run the typical apt-get install command in the terminal:
tester@sitlabcpu22:~/download/newsbeuter-1.3$ sudo apt-get install libsqlite3-dev
Now, we have to run config.sh again to make sure that we've resolved this dependency problem and that we don't have any more dependency problems. (While I won't show it here, in the case of Newsbeuter, I also had to install the libcurl4-openssl-dev package, as well.) Also, if you install a development package (like libsqlite3-dev) and the associated application package (e.g., sqlite3) is not already installed, most systems will automatically install the associated application package at the same time.
When the configuration runs successfully, the result will be that it will create one or more make files. These files are typically named Makefile (remember that file name case matters in Unix/Linux!). If the build package includes sub-directories, such as src, etc., each of these sub-directories will contain a Makefile, as well.
Building and Compilation Errors
Now, we are ready to actually compile the application. This is often called building and the name is borrowed from the real-world process of constructing something. The various "pieces" of the application, which are typically multiple source code files, are combined together to form the overall application. The make utility manages the build process and calls other applications, such as the compiler and linker, to actually do the work. In most cases, you simply run make (with your regular user account) from directory where you ran the configuration. (In a few cases, such as compiling applications written with the Qt library, you will need to run another "wrapper" application like qmake instead. Again, always check the README and/or INSTALL documents for details.)
As with the configuration script above, when you run make (or the similar utility) in the terminal, it will display some messages about what is executing and any warnings and errors. You can typically ignore warnings, as they are mainly for the developers of the application and are telling them that there are some standard practices that are being violated. Usually, these warnings do not affect the application function. On the other hand, compiler errors must be dealt with. With Newsbeuter, when I ran make, things went fine for a while, but then I got an error:
tester@sitlabcpu22:~/download/newsbeuter-1.3$ make
...
c++ -ggdb -I/sw/include -I./include -I./stfl -I./filter -I. -I./xmlrss -Wall -Wextra -DLOCALEDIR=\"/usr/local/share/locale\" -o src/configparser.o -c src/configparser.cpp
c++ -ggdb -I/sw/include -I./include -I./stfl -I./filter -I. -I./xmlrss -Wall -Wextra -DLOCALEDIR=\"/usr/local/share/locale\" -o src/colormanager.o -c src/colormanager.cpp
In file included from ./include/pb_view.h:5,
from src/colormanager.cpp:4:
./include/stflpp.h:5:18: error: stfl.h: No such file or directory
In file included from ./include/pb_view.h:5,
from src/colormanager.cpp:4:
./include/stflpp.h:33: error: ISO C++ forbids declaration of \u2018stfl_form\u2019 with no type
./include/stflpp.h:33: error: expected \u2018;\u2019 before \u2018*\u2019 token
./include/stflpp.h:34: error: ISO C++ forbids declaration of \u2018stfl_ipool\u2019 with no type
./include/stflpp.h:34: error: expected \u2018;\u2019 before \u2018*\u2019 token
make: *** [src/colormanager.o] Error 1
The make process will stop as soon as the first error is encountered. Handling compiler errors can sometimes be tricky business. You have to look at the errors for some clues about the problem. Typically, the problem is that some header files, which usually have extension of .h or .hpp, are missing. In the case of the error above, it is (or should be!) clear that the problem is that stfl.h header file cannot be found. As this example shows, you want to look at the first lines of the error message and work your way down to find the underlying cause of the problem.
After looking at the Newsbeuter documentation (which I should have done before I started, but then this part of the tutorial wouldn't be very meaningful!), I found that it requires a 3rd-party library called STFL. So what do we do in this case? Well, we essentially repeat this exact same process for that required library: obtain the library and execute the configure-build-install process for it and, then, resume building the desired application. For example, in the case of STFL, I had to install the libncursesw5-dev package for it to build properly. (Usually, it is not necessary to redo the configuration step on our original application after installing another required application, but it never hurts either.)
After successfully installing STFL toolkit, the make process for Newsbeuter ran successfully. The make process typically picks up where it leaves off (at the point of the error). Thus, any files that had already compiled successfully will not be recompiled. If you want to recompile everything, you can run make clean all to remove any compiled objects and then run make again.
Installing
After the build process completes successfully, you are ready to install the application. In most cases, to install the application to the common areas of the file system (e.g., /usr/bin or /usr/share/bin, etc.), you will need to run the installation as root. Installing is really the simplest step in the whole process. To install, in the terminal run:
$ make install
Check the output of this process for any errors. If everything was successful, you should be able to run the command name in the terminal and it will launch. (Append & to the end of the command line, if it's a GUI application, or you won't be able to use the terminal session until the application finishes running.)
When you build an application from source, it won't typically add an icon or shortcut to the GUI menus in Ubuntu. You will need to add this manually.
And that is basically the process, albeit potentially iterative, to building and installing an application from source in Ubuntu. After you've done this just a few times, it'll become second nature to you!
Solution 4:
Well, ./configure --help will give you a lot of information, for GNU autotools generated configure files. Most of it comes down to --with/--without to enable features (these may take an extra parameter, like "shared" to say where to find the library).
Other important ones are --prefix (which defaults to /usr/local/ most of the time) to say where to install to (if you're building packages you usually want this as --prefix=/usr or maybe --prefix=/opt/YourPackage).
On Linux, /lib, /usr/lib and /usr/local/lib are generally searched my gcc, and included in ldconfig's default configuration. Unless you have a good reason, this is where you want your libraries. /etc/ld.so.conf may list extra entries, however.
configure and make find them by just attempting to run "gcc -l" and seeing if it errors. You can add "-L" to your CFLAGS parameter to add extra paths to search.
You can have multiple versions installed, and software linked against an older version will stay linked against it (run ldd to find out the binding on Linux), but new compiles generally target the latest version of a dynamic library on your system.
Most software assumes dynamic libs, especially if it uses libtool, so you may find non-trivial apps don't build correctly statically.
ls -l is your best bet to find libraries installed.
And that's where I'm out of information; how to play well with packages: don't know. When possible, I try and wrap things up into a package to avoid the problem.
Solution 5:
To answer a bit of your question I found a good way the other day to see what libraries you have installed and the versions (This is on Linux Debian so will should work with other versions as well).
dpkg --list
You should get a really long list with some output like this
ii libssl0.9.8 0.9.8c-4etch5 SSL shared libraries
ii libssp0 4.1.1-21 GCC stack smashing protection library
ii libstdc++5 3.3.6-15 The GNU Standard C++ Library v3
ii libstdc++5-3.3 3.3.6-15 The GNU Standard C++ Library v3 (development
ii libstdc++6 4.1.1-21 The GNU Standard C++ Library v3