A program uses functions. These functions are either created by the user or imported from a library. These imported functions that are made available to the user are stored in object code format in library files.
Functions stored in these library files are called library functions or runtime libraries. In Linux, library files are stored in /usr/lib
and sometimes in /usr/local/lib
based on their functionality.
There are mainly two types of libraries in Linux:
- Shared libraries
- Static libraries
Shared libraries
Shared libraries contain object code that can be shared by more than one program. During compile time, objects are not copied to the executable, but instead, a reference to the object is made. You can identify these shared libraries by their extensions, like .so
in Linux or .dll
on Windows.
Shared libraries offer various benefits, such as making the executable smaller and more memory-efficient since multiple programs can share the same copy of the library in memory. Additionally, the library can be modified (given that the API doesn’t change). However, it has its drawbacks, such as the program needing to know where to find these libraries (runtime dependency).
Example: Using Shared Libraries with the ls
Command
Let’s take the example of the ls
command. It relies on shared libraries like glibc. To check which shared libraries the ls
command uses, we can use the ldd
command, which lists the shared library dependencies of an executable:
$ ldd /bin/ls
linux-vdso.so.1 (0x00007ffe183fc000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f8e2c952000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e2c77e000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f8e2c6e6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8e2c6e0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8e2c9ab000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8e2c6be000)
or you can use ldconfig
$ ldconfig -p | grep ls
libsmartcols.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libsmartcols.so.1
libkeyutils.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libkeyutils.so.1
libgnutls.so.30 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgnutls.so.30
libcurl-gnutls.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4
This output shows that ls
depends on shared libraries such as libc.so.6
(GNU C Library) and libselinux.so.1
. These libraries are dynamically loaded at runtime, which allows ls
to share common code across multiple programs that use libc
or libselinux
. When we run the command ls -l
, the system will load the shared libraries dynamically. Since it is a shared library, it helps reduce the size of the ls
binary and allows easy updates of shared libraries without recompiling the ls
binary.
Static libraries
In static libraries, object files or function libraries are directly embedded into your executable during compile time. Once compiled, the executable contains all necessary code from the library, and no additional files are required to run the program. Static libraries offer various benefits, mostly portability and no runtime dependency. However, one disadvantage is that the size of the executable becomes very large.
You can identify whether a library is static or dynamic based on its extension. If a library has .a
on Linux or .lib
on Windows, these are static libraries.
Example: Implementing a Static ls
Command in Rust
Let’s take an example of the simple ls
command. Instead of using the shared library glibc, let's try to write a simple ls
implementation that uses musl and is statically linked to the binary:
use std::env;
use std::fs;
use std::io;
use std::path::Path;
fn main() -> io::Result<()> {
// Get current directory or specified directory
let args: Vec<String> = env::args().collect();
let path = if args.len() > 1 {
&args[1]
} else {
"."
};
// Read and list the directory contents
let entries = fs::read_dir(Path::new(path))?;
for entry in entries {
let entry = entry?;
println!("{}", entry.file_name().into_string().unwrap());
}
Ok(())
}
The above program replicates the behavior of the ls
command by reading the contents of the current directory and printing the filenames.
Before compiling the binary, let’s update Cargo.toml
and specify the target as x86_64-unknown-linux-musl
. This will set the default target to x86_64-unknown-linux-musl
, which is required to statically link with musl instead of the default dynamic glibc. There are many arguments on topics like portability, security, and performance when compiling the binary using musl, but we will explore glibc and musl in more depth in another article, as it is a very interesting topic (Google it, it is actually very interesting, here is link).
Anyways, let’s continue and update our Cargo.toml
file:
[package]
name = "ls_rust"
version = "0.1.0"
edition = "2021"
[dependencies]
[build]
target = "x86_64-unknown-linux-musl"
If you are on Linux, you can now compile the binary. However, I am using macOS, and since it uses clang
as the default compiler, it does not support the flags required to compile the binary. So, instead of compiling on my local machine, I will use Docker.
You can use a muslrust
Docker image, which is preconfigured for compiling Rust projects statically linked with musl:
$ docker run --rm -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder cargo build --release
Let’s see information about the compiled target binary:
$ file target/x86_64-unknown-linux-musl/release/ls_rust
target/x86_64-unknown-linux-musl/release/ls_rust: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=03e5bb44a2900b34a7af560bbedb98f53a0dacea, with debug_info, not stripped
Once compiled, the directory tree looks like this, and you can see the binary in the /target/x86_64-unknown-linux-musl/release
folder:
➜ ls_rust git:(main) ✗ tree
.
├── Cargo.lock
├── Cargo.toml
├── src
│ └── main.rs
└── target
├── CACHEDIR.TAG
├── release
│ ├── build
│ ├── deps
│ ├── examples
│ └── incremental
└── x86_64-unknown-linux-musl
├── CACHEDIR.TAG
└── release
├── build
├── deps
│ ├── ls_rust-7c70c9897f5563c5
│ ├── ls_rust-7c70c9897f5563c5.d
│ ├── ls_rust-f1eda63bd00f6def.39cde77tzb7k5x82i4nzfxj84.rcgu.o
│ ├── ls_rust-f1eda63bd00f6def.d
│ └── ls_rust-f1eda63bd00f6def.ls_rust.d9ffdd3156cf9f3-cgu.0.rcgu.o
├── examples
├── incremental
├── ls_rust
└── ls_rust.d
Let's check the info about the file:
➜ file target/x86_64-unknown-linux-musl/release/ls_rust
target/x86_64-unknown-linux-musl/release/ls_rust: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=03e5bb44a2900b34a7af560bbedb98f53a0dacea, with debug_info, not stripped
As you can see, a lot of information is shown here, including the architecture the binary was built for, the build ID, format, etc. However, we are particularly interested in the section that says ‘statically linked,’ as it indicates that the binary is statically linked, and all necessary libraries and dependencies are included within the binary itself. It does not require any external shared libraries (such as libc.so
) to run. This makes the binary self-contained and portable across compatible systems, though it increases the binary size.
Now, let’s try to use it. First, we will write a Dockerfile and move the built binary into the image. You can run it on macOS as well, but you would need to cross-compile using a cross-compiler. I find Docker easier. The Dockerfile should look like this:
FROM scratch
COPY target/x86_64-unknown-linux-musl/release/ls_rust /ls_rust
ENTRYPOINT ["/ls_rust"]
Let’s build the image:
$ docker build -t ls_rust
STEP 1/3: FROM scratch
STEP 2/3: COPY target/x86_64-unknown-linux-musl/release/ls_rust /ls_rust
--> Using cache 95df9f06e8dfe6cce5abb9f951d7bfd82c2787f6a4f026c1f7efd80f275fc9aa
--> 95df9f06e8df
STEP 3/3: ENTRYPOINT ["/ls_rust"]
--> Using cache 05bae5b499cc0791de40cb04cfe25533237d5fb88463f777df481154037d2ae5
COMMIT ls_rust
--> 05bae5b499cc
Successfully tagged localhost/ls_rust:latest
Successfully tagged localhost/ls_rust_image:latest
05bae5b499cc0791de40cb04cfe25533237d5fb88463f777df481154037d2ae5
Now we have built the image, which consists of our own implementation of ls
. Let's use this ls_rust
binary to list files and folders:
➜ ls_rust git:(main) ✗ docker run --rm ls_rust /etc
mtab
resolv.conf
hosts
hostname
➜ ls_rust git:(main) ✗ docker run --rm ls_rust /
ls_rust
etc
proc
dev
sys
run
As you can see, it is listing the contents of the provided folder inside the container. You can make it list contents on your machine by mounting a volume, as shown below:
➜ ls_rust git:(main) ✗ docker run --rm -v $(pwd):/mnt ls_rust /mnt
Cargo.toml
Dockerfile
target
Cargo.lock
.gitignore
.git
src
Congratulations, you have successfully written your own implementation of ls
in Rust with a statically linked library and used it instead of ls
(glibc).
Comparing Shared and Static Libraries
One question you might have is: How is ls_rust
different from other ls
(which uses glibc)?
This is a good question!
The major difference is that ls
uses a shared library, i.e., glibc (as shown in the previous example in the shared library section), whereas ls_rust
uses musl, is statically linked, and includes the required dependency library code into the binary. You can also see the difference in size between them.
Since ls
is dynamically linked, it is small because the binary depends on external shared libraries, which are loaded at runtime:
➜ ls_rust git:(main) ✗ ls -lh /bin/ls
-rwxr-xr-x 1 root wheel 151K Sep 5 02:17 /bin/ls
Whereas ls_rust
is statically linked, the resulting binary is completely self-contained and doesn't require any shared libraries at runtime, making it larger in size:
➜ ls_rust git:(main) ✗ ls -lh target/x86_64-unknown-linux-musl/release/ls_rust
-rwxr-xr-x 2 pthapa staff 3.6M Oct 19 17:36 target/x86_64-unknown-linux-musl/release/ls_rust
That’s it !!!
We’ve explored the differences between shared and static libraries in Linux. Shared libraries offer memory efficiency and smaller executables, while static libraries provide portability and eliminate runtime dependencies. We’ve also implemented a statically linked ls
command in Rust to demonstrate how static linking works.
I hope you have learned a thing or two about libraries in Linux. Happy coding!!!