Create lightweight, hardened containers with Kubler

cross-posted from: https://lemmy.srcfiles.zip/post/32334

What is Kubler?

Kubler is a generic, extendable build orchestrator, written in Bash. It can be used to take advantage of Portage’s features to build lightweight Docker or Podman images without needing to mess with crossdev, or as a tool to assist with ebuild development.

Why should you use it?

You like lightweight, easy-to-create, containers

You want to reduce the attack surface by including only what’s required

You want to take advantage of USE flags to manage package features

You want the awesome package library offered by the Gentoo ebuild repository (and other ebuild repos)

You want up-to-date containers

You want a containerised environment for building and testing ebuilds

A real-world example

I recently needed to integrate a containerised application with a vendor-managed openldap instance that uses mTLS authentication. Unfortunately the containerised application does not work with mTLS and the vendor managed openldap instance can’t be easily configured to use anything else.

I came up with the solution of using openldap’s lloadd LDAP load balancer daemon to proxy connections from an encrypted internal network to the LDAP server but was left with the issue that I didn’t have a working openldap container that contained lloadd - of the existing containers that I tried the only one that actually had an lloadd bin didn’t actually include required dependencies!

Glossing over a recent ebuild update to openldap to enable the building of lloadd, enter Kubler - It’s turned out to be an incredibly flexible and hands-off tool compared to trying to accomplish the same thing with (e.g.) Dockerfiles.

Kubler in action

This (lightly sanitised) real-world example creates create a new namespace called ‘larry’ which may contain multiple images.

Use the new command to take care of the boilerplate; choose ‘multi’ when asked for the namespace type:`
$ kubler new namespace larry
»»»
»»»  to accept default value
»»»
»»» Working dir type? Choices:
»»»   single - You can't add further namespaces to the created working dir, it only holds images
»»»   multi  - Creates a working dir that can hold multiple namespaces
»[?]» Type (single): multi
»»»
»»» Top level directory name for new namespace 'larry'? The directory is created at /data/development/gentoo-containers/
»[?]» Namespaces Dir (kubler-images):
»»»
»»»»» Initial image tag, a.k.a. version?
»[?]» Image Tag (20230706):
»»»
»[!]» New namespace location:  /data/development/gentoo-containers/kubler-images/larry
»»»
»»»»» Who maintains the new namespace?
»[?]» Name (Your Name): Larry the Cow
»[?]» EMail (your@mail.org): Larry.the.Cow@gentoo.zip
»»»
»»»»» Default build engine?
»[?]» Engine (docker):
»»»
»[✔]» Successfully created "larry" namespace at /data/development/gentoo-containers/kubler-images
»»»
»[!]» Configuration file: /data/development/gentoo-containers/kubler-images/larry/kubler.conf
»»»
»[!]» To manage the new namespace with GIT you may want to run:
»»»
»»» $ git init /data/development/gentoo-containers/kubler-images/larry
»»»
»[!]» To create images in the new namespace run:
»»»
»»» $ cd /data/development/gentoo-containers/kubler-images/larry
    $ kubler new image larry/
Although not strictly required, installing Kubler’s example images is a good idea.
$ cd larry/
$ kubler update
It is worthwhile to begin tracking this new namespace with Git so that images can be tracked as they are created and updated. Kubler has already placed a prepopulated a .gitignore file for convenience.
pushd /data/development/gentoo-containers/kubler-images/larry
git init .
git add .
git commit -m "Initial commit"
popd
Create the new ‘openldap’ within the existing ‘larry’ namespace, based on the ‘kubler/busybox’ image.
kubler new image larry/openldap
»»»
»»»  to accept default value
»»»
»»» Extend an existing Kubler managed image? Fully qualified image id (i.e. kubler/busybox) or scratch
»[?]» Parent Image (scratch): kubler/busybox
»»»
»»» Add test template(s)? Possible choices:
»»»   hc  - Add a stub for Docker's HEALTH-CHECK, recommended for images that run daemons
»»»   bt  - Add a stub for a custom build-test.sh script, a good choice if HEALTH-CHECK is not suitable
»»»   yes - Add stubs for both test types
»»»   no  - Fck it, we'll do it live!
»[?]» Tests (hc): yes
»»»
»[✔]» Successfully created new image at /data/development/gentoo-containers/kubler-images/larry/images/openldap
»»»
Note: This step is ‘‘not’’ required; it is possible to directly edit the build.sh file if you are familiar with Portage.

Kubler brings a unique feature to the table when constructing an container image: the --interactive build argument. As the name implies, this launches the build container in an interactive manner, enabling users to investigate the current / inherited configuration.
$ kubler build larry/openldap -i
This will build any missing parent images/builders; the first run may take quite a bit of time - once the local binary package cache and build containers are seeded future runs will be much faster. Once the prerequisite images are ready the build container will present a shell.

For first-time users it may be convenient to search for the openldap package to ensure that the correct atom is selected and investigate any USE flags that are of interest:
# eix openldap
* net-nds/openldap
     Available versions:  2.4.59-r2^t 2.5.14(0/2.5)^t 2.6.3-r7(0/2.6)^t ~2.6.4-r1(0/2.6)^t ~2.6.4-r2(0/2.6)^t {argon2 autoca +berkdb +cleartext crypt cxx debug experimental gnutls iodbc ipv6 kerberos kinit minimal odbc overlays pbkdf2 perl samba sasl selinux sha2 smbkrb5passwd ssl static-libs +syslog systemd tcpd test ABI_MIPS="n32 n64 o32" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Homepage:            https://www.openldap.org/
     Description:         LDAP suite of application and development tools
Edit the image’s build script:
nano /config/build.sh
Note: The /config directory in the build container is the host mounted image directory at larry/images/openldap/. Feel free to use a local IDE/editor to edit build.sh instead.

Add the net-nds/openldap and net-misc/curl packages to the _packages variable in build.sh, update cURL USE flags, enable the ~arch (~amd64 - the Gentoo ‘testing’ keyword) for packages we care about:
_packages="net-nds/openldap net-misc/curl"
...
configure_rootfs_build()
{
    # Update a Gentoo package use flag.
    update_use 'net-misc/curl' '+ldap'
    # ..or a Gentoo package keyword
    update_keywords 'net-misc/curl' '+~amd64'
    update_keywords 'net-nds/openldap' '+~amd64'
...
}
Note: If using the busybox image as a parent, unset the su USE flag from sys-apps/util-linux in the build.sh.

Perform a test run of the first build phase:
$ kubler-build-root
Once this completes successfully exit the interactive builder using exit.

Building the image

Assuming that build.sh has been configured as described above, it should be safe to attempt to build the image.
$ kubler build larry/openldap -nF
»[✘]»[larry/openldap]» fatal: build-test.sh for image larry/openldap:20230704 failed with exit signal: 1
Note: The arguments are short hand for --no-deps and --force-full-image-build, omitting -n would also rebuild all parent images, which is waste of time in this case.

The build will fail, as expected, due to the build-test.sh script not being implemented. This is a good time to implement the build-test.sh script, which will be used to verify that the image is functional.

Note: pipefail will cause build-test.sh to fail on busybox-based images
#!/usr/bin/env sh

set -eo

# Do some tests and exit with either 0 for healthy or 1 for unhealthy
# Check that the openldap bin launches and provides some expected output
/usr/lib/openldap/lloadd -VV  2>&amp;1 | grep "OpenLDAP" || exit 1

exit 0
Unfortunately this image is not suitable for a build-time docker health check via the docker-healthcheck.sh mechanism, so must be disabled in larry/images/openldap/build.conf:
POST_BUILD_HC=false
A health check suitable for your environment should be provided using standard docker syntax in the image’s Dockerfile.template instead. Ensure that the provided docker-healthcheck.sh script iS updated (or commented out of the dockerfile) as the default will fail.

Modify the image’s Dockerfile.template to add any finishing touches, such as the ENTRYPOINT or CMD directives. In this example the container will act as an LDAP proxy via lloadd; additional configuration will be provided at runtime by mounting the configuration into the container.
FROM ${IMAGE_PARENT}
LABEL maintainer="${MAINTAINER}"

ADD rootfs.tar /

COPY docker-healthcheck.sh /usr/bin/docker-healthcheck
HEALTHCHECK --interval=60s --timeout=5s --start-period=5s --retries=3 CMD ["docker-healthcheck"]

CMD ["/usr/lib/openldap/lloadd"]
Re-run the build:
$ kubler build larry/openldap -nF
»[✔]»[larry/openldap]» done.
At this point the image should exist in the local Docker/Podman registry and be ready for use:
docker images
REPOSITORY                                              TAG                       IMAGE ID       CREATED          SIZE
larry/openldap                                          20230704                  09347c55282b   2 minutes ago    56.4MB
larry/openldap                                          latest                    09347c55282b   2 minutes ago    56.4MB
Hopefully this has been useful and you are now ready to build your own images! I’ve been incredibly impressed with how easy it is to use the tool (and it it’ll run from any distro with a recent version of Docker/Podman), and the quality of the resulting images. I’m a recent convert, but have updated the Gentoo Wiki with the above information (and some extra info on using it for ebuild development) and will be using Kubler in future wherever I need to create images.

Happy containering!