how compatible is libreSSL ?

portability

yesterday the “portable” version of libressl was released.
http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.0.0.tar.gz

i set up a package in sabotage linux, and went on a voyage to investigate whether the full set of packages can be used with libressl instead of openssl.

first of all, i had to fight some obstacles to get libressl compiling though…

obstacle 1 – -Werror


../include/openssl/bio.h:622:3: error: '__bounded__' attribute directive ignored [-Werror=attributes]


-Werror is hardcoded in the configure script, which is a very bad idea, and the opposite of portable.
using -Werror is a guarantueed build break whenever the build is tried on a system the original developer had no access to.
it’s sufficient to use a different compiler version, different libc version, etc to make new warnings pop up.

fixed with

sed -i 's/-Werror//' configure


obstacle 2 – unconditional inclusion of internal glibc header


compat/issetugid_linux.c:7:30: fatal error: gnu/libc-version.h: No such file or directory


many people assume linux == glibc, but that is not the reality.
sabotage linux uses musl libc, and there are at least 4 other libcs that could be used instead (uclibc, dietlibc, klibc, bionic).

looking at issetugid_linux.c uncovers a dubious hack:
if glibc 2.19 is detected, getauxval(AT_SECURE) is not used, because there was once a bug (see comment in source code).

however it’s common practice in distros to backport bugfixes, without updating the version number.
so this hack prevents proper usage of getauxval even if your libc version is long fixed.
the mentioned bug is very likely already fixed in any distro using glibc 2.19.

to get the thing out of my way and compilation going on, the quick fix was to cover everything with #ifdef __GLIBC__.
what the code really should do though is to just use the getauxval call unconditionally without the glibc version check.

obstacle 3 – unnecessary unconditional inclusion of sys/sysctl.h

compat/getentropy_linux.c:27:24: fatal error: sys/sysctl.h: No such file or directory


musl does not have sys/sysctl.h, because:
(citing musl’s author Rich Felker)

sysctl does not work, and NEVER worked. using it is bogus.
it was a bogus experimental syscall that was deprecated before it was ever used (basically, a broken binary version of /proc/sys, without any stability between kernel versions for what the binary constants meant).

since the code in question does not use the sysctl function (declared in sys/sysctl.h) and does the syscall() directly,
it was safe and sufficient to just remove the include statement.

still it leaves a bad taste in my mouth that it was used at all…

having fixed these 3 issues, libressl built successfully.
https://github.com/sabotage-linux/sabotage/commit/4f2da253f669a0e2e69e6d5607e56c552b716eff

on the plus side: using 8 cores, libressl builds in about 1 minute, while openssl requires 1:45.
also openssl depends on perl, which takes an additional 2 minutes buildtime.
so if nothing else depends on perl, it’s about 3x faster.

compatibility

with libressl in place, a “world” metapackage (contains almost all packages) build was started.
the results:

wget failed to build due to lack of RAND_egd() function.
fixed by using a patch from openbsd.
https://github.com/sabotage-linux/sabotage/commit/234185c0dd408a9eeb6a14a31719760841f97be5

stunnel failed to build due to lack of RAND_egd() function.
fixed by using a custom patch conceptually equivalent to the wget one.
https://github.com/sabotage-linux/sabotage/commit/9b47cbbf3ce903dee042c45c8197db066e8e0053

cryptsetup and others failed to detect openssl due to lack of pkgconfig files.
i modified my package build script to create these .pc files (copies from openssl).
https://github.com/sabotage-linux/sabotage/commit/156a36253b9e19f83103135113451209796c39cf

php, xorg-server and others failed to build subtly due to an ugly hack used in libressl’s libcompat.a, linked into libcrypto.so:

$ gcc test.c -lcrypto -fvisibility=hidden
/bin/ld: a.out: hidden symbol `main' in /tmp/ccobhDjc.o is referenced by DSO
/bin/ld: final link failed: Bad value

$ readelf -a /lib/libcrypto.so | grep main
000000345708 000a00000006 R_X86_64_GLOB_DAT 0000000000000000 main + 0
10: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND main
2146: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND main


in getentropy_linux.c:

extern int main(int, char *argv[]);
#define HD(x) (SHA512_Update(&ctx, (char *)&(x), sizeof (x)))
HD(main); /* an addr in program */


the address of main() is used to gather entropy… very smart… NOT.

most of the methods used in this file to gather entropy are very dubious.
the crypto experts from OpenBSD should know better and just use /dev/urandom and/or getauxval(AT_RANDOM)
instead of all these hacks.

https://github.com/sabotage-linux/sabotage/commit/1a8113699311c95a42c0cce5d09746ca290852ac

with that fixed, most programs expecting openssl seem to compile and work correctly.

mysql’s CMAKE build system fails to detect SSL libraries.

-- OPENSSL_INCLUDE_DIR = /usr/include
-- OPENSSL_LIBRARY = /usr/lib/libssl.so
-- CRYPTO_LIBRARY = /usr/lib/libcrypto.so
-- OPENSSL_MAJOR_VERSION = 2
-- Looking for SHA512_DIGEST_LENGTH
-- Looking for SHA512_DIGEST_LENGTH - found
CMake Error at cmake/ssl.cmake:231 (MESSAGE):
Cannot find appropriate system libraries for SSL. Use WITH_SSL=bundled to
enable SSL support

not patched yet.

the last build error was in apache:
ssl_engine_init.c:445:28: error: ‘ENGINE_CTRL_CHIL_SET_FORKCHECK’ undeclared

this is a macro which is available in openssl’s engine.h, and was removed from libressl for unknown reasons.
not patched yet.

apart from these two, everything seems to be usable without big effort.
so if the libressl developers rip out all their dubious entropy generation methods in favor of /dev/urandom on linux it might be well worth switching to it.

the whole adventure is documented in the libressl_replaces_openssl branch.
https://github.com/sabotage-linux/sabotage/commits/libressl_replaces_openssl

Update 07/13

OpenBSD released an updated version 2.0.1 earlier today.
the new release fixes the following problems
– reference to main() which breaks packages using -fvisibility=hidden
– usage of -Werror
– generation of pkg-config files
– unconditional inclusion of sys/sysctl.h

so the portability concerns have largely been addressed.
the only portability issue not fixed is the glibc-specific stuff in issetugid_linux.c.
instead, a patch containing an issetugid implementation for inclusion in musl
was sent to the musl maillist.

on the application compatibility side nothing seems to have changed.
RAND_egd() is still missing, as well as the macros used by apache.

the dubious fallbacks for getentropy (obsolete sysctl syscall, function addresses) are still present.

this blog about similar testing done on gentoo (recommended read) has a link to a patch for the apache build.
there is also a patch for a segfault in openssh.

sqlite’s anal gamation

sqlite’s slogan: “Small. Fast. Reliable. Choose any three.”

i always wondered though, how such a small or “lite” package can take such a considerable amount of time to build.

as the main author of the sabotage linux distribution, building software is my daily bread, so i own a pretty fast build box.
it’s an 8 core machine with 3.1 GHz, which builds a complete 3.11 linux kernel in less than 5 minutes, making use of all 8 cores via the nice parallel build feature of GNU make.

make -j8

when invoking make like this, it first determines the dependencies between the translation units,
and then runs up to 8 build processes, one per cpu core, each one building a different .c file.

GCC 3.4.6, a C compiler with full C99 support builds in 43 sec:

$ time butch rebuild gcc3
2013.09.25 12:13:50 building gcc3 (/src/build/build_gcc3.sh) -> /src/logs/build_gcc3.log
2013.09.25 12:14:33 done.
real 0m 43.97s
user 1m 36.66s
sys 0m 13.74s

however, for sqlite, a supposedly small package, build times are comparatively huge:

$ time butch rebuild sqlite
2013.09.25 12:18:27 building sqlite (/src/build/build_sqlite.sh) -> /src/logs/build_sqlite.log
2013.09.25 12:19:21 done.
real 0m 54.03s
user 0m 52.02s
sys 0m 1.51s

nearly one minute, a fifth of the time used to build the linux kernel and 10 seconds more than the gcc compiler.

the full-blown postgresql database server package, takes less time to build as well:

$ time butch rebuild postgresql
2013.09.25 12:19:21 building postgresql (/src/build/build_postgresql.sh) -> /src/logs/build_postgresql.log
2013.09.25 12:19:57 done.
real 0m 36.63s
user 1m 53.34s
sys 0m 12.03s

how is it possible that postgresql, shipping 16 MB of compressed sources, as opposed to 1.8MB of sqlite, builds 33% faster ?

if you look at the user times above, you start getting an idea.
the user time (i.e. the entire cpu time burnt in userspace) for postgresql is 1m53, while the total time that actually passed was only 36s.

that means that the total work of 113 seconds was distributed among multiple cpu cores.
dividing the user time through the real time gives us a concurrency factor of 3.13.
not perfect, given that make was invoked with -j8, but much better than sqlite, which apparently only used a single core.

let’s take a look at sqlite’s builddir

$ find . -name '*.c'
./sqlite3.c
./shell.c
./tea/generic/tclsqlite3.c
./tea/win/nmakehlp.c

ah, funny. there are only 4 C files total. that partially explains why 8 cores didn’t help.
the 2 files in tea/ are not even used, which leaves us with

$ ls -la *.c
-rw-r--r-- 1 root root 91925 Jan 16 2012 shell.c
-rw-r--r-- 1 root root 4711082 Jan 16 2012 sqlite3.c

so in the top level builddir, there are just 2 C files, one being 90 KB, and the other roughly 5MB.
the 90KB version is built in less than 1 second, so after that the entire time spent is waiting for the single cpu core building the huge sqlite3.c.

so why on earth would somebody stuff all source code into a single translation unit and thereby defeat
makefile parallellism ?

after all, the IT industry’s mantra of the last 10 years was “parallellism, parallellism, and even more parallellism”.

here’s the explanation:
https://www.sqlite.org/amalgamation.html

it’s a “feature”, which they call amalgamation.

i call it anal gamation.

In addition to making SQLite easier to incorporate into other projects, the amalgamation also makes it run faster.
Many compilers are able to do additional optimizations on code when it is contained with in a single translation unit such as it is in the amalgamation.

so they have 2 reasons for wasting our time:

• reason 1: easier to incorporate
• reason 2: generated code is better as the compiler sees all code at once.

let’s look at reason 1:
what they mean with incorporation is embedding the sqlite source code into another projects source tree.

it is usually considered bad practice to embed third-party source code into your own source tree, for multiple reasons:

• every program that uses its own embedded copy of library X does not benefit from security updates when the default library install is updated.
• we have multiple different versions on the harddrive and loaded in RAM, wasting system resources
• having multiple incompatible versions can lead to a lot of breakage when it’s used from another lib:
  for example application X uses lib Y and lib Z, and lib Z uses a “incorporated” version of lib Y.
  so we have a nice clash of 2 different lib Y versions. if lib Y has global state, it will get even worse.
• if the library in question is using some unportable constructs, wrong ifdefs etc., it needs to be patched to build.
  having to apply and maintain different sets of patches against multiple different versions “incorporated” into other packages, represents a big burden for the packager.

instead, the installed version of libraries should be used.

pkg-config can be used to query existence, as well as CFLAGS and LDFLAGS needed to build against the installed version of the library. if the required library is not installed or too old, just throw an error at configure time and tell the user to install it via apt-get or whatever.

conclusion: “incorporation” of source code is a bad idea to begin with.

now let’s look at reason 2 (better optimized code):
it possibly sometimes made sense to help the compiler do its job in the 70ies, when everything started.
however, it’s 2013 now.
compilers do a great job optimizing, and they get better at it every day.

since GCC 4.5 was released in 2010, it ships with a feature called LTO
it builds object files together with metadata that allows it to strip off unneeded functions and variables, inline functions that are only called once or twice, etc at link time – pretty much anything the sqlite devs want to achieve, and probably even more than that.

conclusion: pseudo-optimizing C code by stuffing everything into a big file is obsolete since LTO is widely available.
LTO does a better job anyway – not that it matters much, as sqlite spends most time waiting for I/O.
every user who wants to make sqlite run faster, can simply add -flto to his CFLAGS.
there’s no need to dictate him which optimization he wants to apply.
following this logic, they could as well just ship generated assembly code…

but hey – we have the choice !
here ‘s actually a tarball containing the ORIGINAL, UN-ANAL-GAMATED SOURCE CODE…

… just that it’s not a tarball.

it’s a fscking ZIP file.
yes, you heard right.
they distribute their source as ZIP files, treating UNIX users as second-class citizens.

additionally they say that you should not use it:

sqlite-src-3080002.zip (5.12 MiB)
A ZIP archive of the complete source tree for SQLite version 3.8.0.2 as extracted from the version control system.
The Makefile and configure script in this tarball are not supported.
Their use is not recommended.
The SQLite developers do not use them.
You should not use them either.
If you want a configure script and an automated build, use either the amalgamation tarball or TEA tarball instead of this one.
To build from this tarball, hand-edit one of the template Makefiles in the root directory of the tarball and build using your own customized Makefile.

Note how the text talks about “this tarball” despite it being a ZIP file.

Fun. there’s only a single TARball on the entire site, so that’s what you naturally pick for your build system.
and that one contains the ANAL version.
Note that my distro’s build system does not even support zip files, as i don’t have a single package in my repo that’s not building from a tarball.
should i change it and write special case code for one single package which doesn’t play by the rules ?
i really don’t think so.

funny fact: they even distribute LINUX BINARY downloads as .zip.
i wonder in which world they live in.

why do i care so much about build time ? it’s just a minute after all.
because the distribution gets built over and over again. and it’s not just me building it, but a lot of other people as well – so the cumulated time spent waiting for sqlite to finish building its 5 MB file gets bigger and bigger each day.
in the past i built sqlite more than 200 times, so my personal cumulated wasted time on it already exceeds the amount of time i needed to write this blog post.

so what i hope to see is sqlite

1. using tarballs for all their sources (and eventually distribute an additional .zip for windows lusers)
2. using tarballs for their precompiled linux downloads
3. either getting rid of the anal version entirely, now that they learned about LTO, or offer the anal version as an additional download and do not discourage users from using the sane version.

Update: I just upgraded sqlite from 3071000 to 3080002

2013.09.27 02:23:24 building sqlite (/src/build/build_sqlite.sh) -> /src/logs/build_sqlite.log
2013.09.27 02:25:31 done.

it now takes more than 2 minutes.