Cloud-based Continuous Integration for QtQuickVcp

6 minutes read

This blog post is about the continuous integration of Qt applications with the example of the QtQuickVcp project.

It demonstrates how to build Qt applications and libraries for Linux and Android using Travis CI and Docker.

Furthermore, I mention how to build Qt applications for Windows and OS X.

My Motivation

At the beginning of October, I visited the QtCon conference in Berlin. There I attended a talk about the CI of the MuseScore project.

From this talk, I got a lot of new ideas of how to implement CI for QtQuickVcp using Travis CI, Docker and AppVeyor.

This talk was the reason for me to think about retiring my existing Buildbot-based CI setup for QtQuickVcp.

Travis CI vs. Buildbot

As mentioned, I previously used a Buildbot-based setup for continuously integrating QtQuickVcp.

Buildbot is Python-based CI system which uses a master-slave configuration. You have to have one build master to monitor Git repositories and to trigger builds.

The build master manages multiple build slaves. These build slaves are Python client applications executing commands send by the master.

In my QtQuickVcp scenario, I host the master on my virtual server. The build slaves are physical machines.

The build configuration itself is written in Python as a set of build steps. The whole setup is rather complicated and time intensive.

However, the biggest reason for me to retire my build farm is, that I don't want to maintain multiple physical devices.

Travis CI and other similar CI systems on the other hand offer script based solutions to run builds entirely in the cloud.

Which leads me to another advantage of Travis CI over Buildbot -> The build configuration is hosted in the same Git repository as the project itself.

Setup Overview

Since I don't want to maintain a build farm, I decided to use Travis CI and AppVeyor for continuous integration.

qtquickvcp_ci

For building the Linux and Android version, I decided to use the same setup as MuseScore using Docker containers.

Travis CI uses a Linux, or a Mac OS X build host which would be suitable for compiling the Linux and Android versions of QtQuickVcp.

However, using Docker containers in combination with Travis CI allows full control of the build environment and easier local testing of the build script.

Furthermore, using Docker containers allows us to create a build environment which we can "pollute" as much as we want during building.

Docker Containers

Creating custom Docker containers is very easy:

  • Select a base image
  • Write a build script
  • Build the container

It all boils down to the following Dockerfile:

FROM library/debian:jessie
MAINTAINER Alexander Rössler <mail@roessler.systems>
COPY init.sh /tmp/init.sh
RUN sed -i "s/httpredir.debian.org/ftp.us.debian.org/" /etc/apt/sources.list
RUN /tmp/init.sh

and the build command:

docker build --no-cache=true -t machinekoder/qtquickvcp:linux-x64 .

The fourth line of the Dockerfile probably needs some explanation:

Per default, Docker containers come with the httpredir.debian.org mirror enabled. However, it turned out that the HTTP redirection causes the Docker builds to be unreliable. Using the US mirror produces consistent results.

You can find my Qt build Dockerfiles on GitHub.

Additionally, I use the Docker Hub cloud build service to automatically build the Docker images whenever the Git repository is updated. You can find the Docker Hub repository here.

Installing Qt-SDK in Docker build

One of the challenges I faced when creating the Docker build script was how to install the Qt-SDK without the installer.

The answer is surprisingly simple:

  • Install the Qt-SDK on your development machine.
  • Zip the contents of the Qt platform folder (e.g. ~/Qt/5.7/gcc_64/).
  • Upload the zipped file somewhere accessible from the web.
  • Download and unzip it in your Docker container.

Here are the commands in short.

On your development machine:

cd ~/Qt/5.7/gcc_64/
tar -cjSf Qt-5.7-Linux-x64.tar.bz2 *
scp Qt-5.7-Linux-x64.tar.bz2 youruser@yourhost.com:~/webcontent/qt-bin/

In the Docker build script:

mkdir -p qt5 && wget -q -O qt5.tar.bz2 http://yourhost.com/files/qt-bin/Qt-5.7-Linux-x64.tar.bz2
tar xjf qt5.tar.bz2 -C qt5
rm qt5.tar.bz2

During the Qt build:

export PATH="${PWD}/qt5/bin:$PATH"
export LD_LIBRARY_PATH="${PWD}/qt5/lib:$LD_LIBRARY_PATH"
export QT_PLUGIN_PATH="${PWD}/qt5/plugins"
export QML_IMPORT_PATH="${PWD}/qt5/qml"
export QML2_IMPORT_PATH="${PWD}/qt5/qml"
export QT_QPA_PLATFORM_PLUGIN_PATH="${PWD}/qt5/plugins/platforms"
export QT_INSTALL_PREFIX="${PWD}/qt5"

That's it.

If you wonder how to install the Qt-SDK on the Windows, build host. This answer is also simple: AppVeyor already comes with a recent Qt-SDK preinstalled.

Qt-SDK Linux runtime dependencies

Of course installing Qt-SDK alone does not build anything. We also have to install the runtime dependencies.

I figured out that the Qt-SDK needs the following packages installed on Debian Jessie:

apt-get install -y build-essential gdb dh-autoreconf libgl1-mesa-dev libxslt1.1 git
# dependencies of qmlplugindump
apt-get install -y libfontconfig1 libxrender1 libdbus-1-3 libegl1-mesa

Qt for Android Docker build

Automating the Qt for Android build was the hardest part of setting up the CI system.

Summarized, Qt for Android has the following requirements:

  • Qt-SDK for Android installed
  • Java JDK installed
  • Android SDK installed
  • Android NDK installed

Installing the Qt-SDK and the Java JDK are the easiest part. We already know how to install the Qt-SDK.

The Java JDK and other Android SDK dependencies can be installed from the Debian repositories:

apt install -y make default-jdk ant lib32z1 lib32ncurses5 lib32stdc++6

Note that the Android SDK requires some 32bit libraries.

Installing the Android-NDK is also straightforward. You can download the zipped NDK from the Google servers:

mkdir -p android-ndk && wget -q -O android-ndk.zip https://dl.google.com/android/repository/android-ndk-r12b-linux-x86_64.zip
unzip -qq android-ndk.zip -d android-ndk
rm android-ndk.zip
cd android-ndk
mv */* .

Now the tricky part: Installing the Android SDK from the command line.

The Android SDK does not come with any platforms and build tools installed. Therefore, you need to run the Android SDK tool before doing anything with the SDK.

I figured out that it easiest to install the SDK on your development machine. Then run the Android SDK and install a target platform. And finally to Zip the resulting folder.

Unfortunately, I wasted some time on trying to use the latest Android build tools version (25). It turns out that they do not play well with the current Qt-SDK. So ended up using android-19 (4.4) and build tools 23 as for the CI.

Last but not least, you need to set up an Android toolchain. This can be done with the following command:

./android-ndk/build/tools/make-standalone-toolchain.sh --install-dir=/opt/android-toolchain --arch=arm
export PATH=/opt/android-toolchain/bin:$PATH

In addition to the Qt environment variables I use following Android specific variables in my build setup:

export ANDROID_NDK_ROOT="${PWD}/android-ndk"
export ANDROID_SDK_ROOT="${PWD}/android-sdk"
export ANDROID_HOME="${PWD}/android-sdk"

Versioning using Git

When using a continuous integration system for continuous deployment as well, it also necessary to do versioning automatically.

I figured out that this can be easily done using a few git commands.

# create a full clone
git fetch --unshallow
# find out version number
release=1
git describe --exact-match HEAD 2> /dev/null || release=0
if [ $release -eq 0 ]; then
    date="$(date -u +%Y%m%d%H%M)"

    branch="$TRAVIS_BRANCH"
    [ "$branch" ] || branch="$(git rev-parse --abbrev-ref HEAD)"

    revision="$(echo "$TRAVIS_COMMIT" | cut -c 1-7)"
    [ "$revision" ] || revision="$(git rev-parse --short HEAD)"
    version="${date}-${branch}-${revision}"
else
    version="$(git describe --tags)"
fi

First of all, you need to create a full clone from the shallow Travis CI clone. If you skip this command, you are not able to count the number to commits on the repository.

Next, check if the current HEAD of the Git repository matches any tag. If so this is a release build. If it does not match a Git tag, it is a development build.

For versioning QtQuickVcp using Git tags I use the following schema v0.9.5.

The prefix v is recommended for versioning releases in Git to prevent aliasing with branch names.

Android

Android requires a new release code for every upload to Google Play Store. However, I don't like to update the version code in the AndroidManifest.xml every time I change something in code.

Therefore, I use the number of commits in the Git repository, or exactly in the current branch, to generate the version code.

This method ensures that the apk file has a new release code with every build as long as you do not change history.

Rebasing inside the master branch is not recommend anyway, so this is a safe assumption.

The following commands do the trick:

version_name="$(git describe --tags --abbrev=0)"
version_code="$(git rev-list --first-parent --count HEAD)"
manifest="${PWD}/apps/MachinekitClient/android/AndroidManifest.xml"
sed -i -E "s/(android:versionName=\")([^ ]+)(\")/\1${version_name}\3/" $manifest
sed -i -E "s/(android:versionCode=\")([^ ]+)(\")/\1${version_code}0\3/" $manifest

Note that the --first-parent argument of the git rev-list command does skip any squashes and merges.

Also, note that I had to append a 0 to the version code since I already polluted my version number space during testing.

Deployment

One great argument for me to setup a CI for QtQuickVcp is continuous deployment.

Deployment for small one-shot apps does not take much time. However, if you plan to work on a project for a long time the amount of time required to package and upload binaries to app stores and storage servers quickly adds up.

In my Buildbot setup, I uploaded the files to my own web space.

In the new cloud-based configuration, I deploy the files to Bintray and the Google Play Store Alpha track.

However, not only where to deploy the applications but also how to deploy the applications matter.

Qt does come with packaging applications for Windows, OS X and Android. However, there is no deployment helper for Linux so far.

Therefore, I wrote the Qt-Deployment-Scripts some time ago.

These scripts provide the qt-deploy tool which helps to deploy Qt applications for all platforms.

For Windows it used the windeployqt tool, on OS X the macdeployqt and on Android the androiddeployqt tool. On Linux, it uses a list of libraries which should be deployed within the application.

Additionally, for bundling the Linux packages, I chose AppImage.

I used the MuseScore build scripts as an example of how to deploy Qt applications with AppImage.

Summing up

Summarized, in this blog post, I explained the new continuous integration setup for QtQuickVcp.

Besides informing you about the status of QtQuickVcp, it may also serve as a source of inspiration for setting up CI for your Qt project.

You can find the complete setup on GitHub in the QtQuickVcp repository in the build subfolder.

I hope you enjoyed reading this article and I would like to hear your feedback.

Your
Machine Koder

Spread the love

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.