Cross-compilation

Table of Contents

cross_compile tool
Legacy tool instructions

For background information on cross-compilation, see the conceptual article.

This document provides you with details on how to cross-compile the ROS 2 software stack as well as provide examples for cross-compiling to systems based on the Arm cores.

cross_compile tool 

Instructions to use the tool are in the cross_compile package.

Legacy tool instructions 

Nota

Follow the steps below only if you are using the old version (release 0.0.1) of the cross-compile tool. For all other purposes, follow the cross_compile package documentation.

Although ROS 2 is a rich software stack with a number of dependencies, it primarily uses two different types of packages:

Python based software, which requires no cross-compilation.
CMake based software, which provides a mechanism to do cross-compilation.

Furthermore, the ROS 2 software stack is built with Colcon which provides a mechanism to forward parameters to the CMake instance used for the individual build of each package/library that is part of the ROS 2 distribution.

When building ROS 2 natively, the developer is required to download all the dependencies (e.g. Python and other libraries) before compiling the packages that are part of the ROS 2 distribution. When cross-compiling, the same approach is required. The developer must first have the target system’s filesystem with all dependencies already installed.

The next sections of this document explain in detail the use of cmake-toolchains and the CMAKE_SYSROOT feature to cross-compile ROS 2.

CMake toolchain-file 

A CMake toolchain-file is a file which defines variables to configure CMake for cross-compilation. The basic entries are:

CMAKE_SYSTEM_NAME: the target platform, e.g. linux

CMAKE_SYSTEM_PROCESSOR: the target architecture, e.g. aarch64 or arm

CMAKE_SYSROOT: the path to the target file-system

CMAKE_C_COMPILER: the C cross-compiler, e.g. aarch64-linux-gnu-gcc

CMAKE_CXX_COMPILER: the C++ cross-compiler, e.g. aarch64-linux-gnu-g++

CMAKE_FIND_ROOT_PATH: an alternative path used by the find_* command to find the file-system

When cross-compiling ROS 2, the following options are required to be set:

CMAKE_FIND_ROOT_PATH: the alternative path used by the find_* command, use it to specify the path to ROS 2 /install folder

CMAKE_FIND_ROOT_PATH_MODE_*: the search strategy for program,package,library, and include, usually: NEVER (look on the host-fs), ONLY (look on sysroot), and BOTH (look on both sysroot and host-fs)

PYTHON_SOABI: the index name of the python libraries generated by ROS 2, e.g. cpython-36m-aarch64-linux-gnu

THREADS_PTHREAD_ARG "0" CACHE STRING "Result from TRY_RUN" FORCE: Force the result of the TRY_RUN cmd to 0 (success) because binaries can not run on the host system.

The toolchain-file is provided to CMake with the -DCMAKE_TOOLCHAIN_FILE=path/to/file parameter. This will also set the CMAKE_CROSSCOMPILING variable to true which can be used by the software being built.

The CMAKE_SYSROOT is particularly important for ROS 2 as the packages need many dependencies (e.g. python, openssl, opencv, poco, eigen3, …). Setting CMAKE_SYSROOT to a target file-system with all the dependencies installed on it will allow CMake to find them during the cross-compilation.

Nota

You can find more information on the CMake documentation page.

When downloading the ROS 2 source code, a generic toolchain-file is available in the repository ros-tooling/cross_compile/cmake-toolchains which can be downloaded separately. Further examples on using it can be found on the Cross-compiling examples for Arm section.

Target file-system 

As mentioned previously, ROS 2 requires different libraries which needs to be provided to cross-compile.

There are a number of ways to obtain the file-system:

downloading a pre-built image
installing the dependencies on the target and exporting the file-system (e.g. with sshfs)
using qemu + docker (or chroot) to generate the file-system on the host machine.

Nota

You can find information on how to use Docker + qemu on the next Cross-compiling examples for Arm section.

Build process 

The build process is similar to native compilation. The only difference is an extra argument to Colcon to specify the toolchain-file:

colcon build --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_TOOLCHAIN_FILE="<path_to_toolchain/toolchainfile.cmake>"

The toolchain-file provide to CMake the information of the cross-compiler and the target file-system. Colcon will call CMake with the given toolchain-file on every package of ROS 2.

Cross-compiling examples for Arm 

After downloading the ROS 2 source code, you can add cross-compilation assets to the workspace via git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1 src/ros2/cross_compile. These are working examples on how to cross-compile for Arm cores.

The following targets are supported:

Ubuntu-arm64: To be used with any ARMv8-A based system.
Ubuntu-armhf: To be used with any modern ARMv7-A based system.

These are the main steps:

Installing development tools
Downloading ROS 2 source code
Downloading the ROS 2 cross-compilation assets
Preparing the sysroot
Cross-compiling the ROS 2 software stack

The next sections explains in detail each of these steps. For a quick-setup, have a look at the Automated Cross-compilation.

Nota

These steps were tested on an Ubuntu 18.04 (Bionic)

1. Install development tools

This step is similar to when building natively. The difference is that some of the libraries and tools are not required because they will be in the sysroot instead. The following packages are required

sudo apt update && sudo apt install -y \
    cmake \
    git \
    wget \
    python3-pip \
    qemu-user-static \
    g++-aarch64-linux-gnu \
    g++-arm-linux-gnueabihf \
    pkg-config-aarch64-linux-gnu

python3 -m pip install -U \
    vcstool \
    colcon-common-extensions

Nota

You can install vcstool and colcon-common-extensions via pip. This means you are not required to add extra apt repositories.

Docker is used to build the target environment. Follow the official documentation for the installation.

2. Download ROS 2 source code

Then create a workspace and download the ROS 2 source code:

mkdir -p ~/cc_ws/ros2_ws/src
cd ~/cc_ws/ros2_ws
wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos
vcs-import src < ros2.repos
git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1 src/ros2/cross_compile
cd ..

3. Prepare the sysroot

Build an arm Ubuntu image with all the ROS 2 dependencies using Docker and qemu: Copy the qemu-static binary to the workspace. It will be used to install the ROS 2 dependencies on the target file-system with docker.

mkdir qemu-user-static
cp /usr/bin/qemu-*-static qemu-user-static

The standard setup process of ROS 2 is run inside an arm docker. This is possible thanks to qemu-static, which will emulate an arm machine. The base image used is an Ubuntu Bionic from Docker Hub.

docker build -t arm_ros2:latest -f ros2_ws/src/ros2/cross_compile/sysroot/Dockerfile_ubuntu_arm .
docker run --name arm_sysroot arm_ros2:latest

Export the resulting container to a tarball and extract it:

docker container export -o sysroot_docker.tar arm_sysroot
mkdir sysroot_docker
tar -C sysroot_docker -xf sysroot_docker.tar lib usr opt etc
docker rm arm_sysroot

This container can be used later as virtual target to run the created file-system and run the demo code.

4. Build

Set the variables used by the generic toolchain-file

export TARGET_ARCH=aarch64
export TARGET_TRIPLE=aarch64-linux-gnu
export CC=/usr/bin/$TARGET_TRIPLE-gcc
export CXX=/usr/bin/$TARGET_TRIPLE-g++
export CROSS_COMPILE=/usr/bin/$TARGET_TRIPLE-
export SYSROOT=~/cc_ws/sysroot_docker
export ROS2_INSTALL_PATH=~/cc_ws/ros2_ws/install
export PYTHON_SOABI=cpython-36m-$TARGET_TRIPLE

The following packages still cause errors during the cross-compilation (under investigation) and must be disabled for now.

touch \
    ros2_ws/src/ros2/rviz/COLCON_IGNORE \
    ros2_ws/src/ros-visualization/COLCON_IGNORE

The Poco pre-built has a known issue where it is searching for libz and libpcre on the host system instead of SYSROOT. As a workaround for the moment, please link both libraries into the the host’s file-system.

mkdir -p /usr/lib/$TARGET_TRIPLE
ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libz.so.1 /usr/lib/$TARGET_TRIPLE/libz.so
ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libpcre.so.3 /usr/lib/$TARGET_TRIPLE/libpcre.so

Then, start a build with colcon specifying the toolchain-file:

cd ros2_ws

colcon build --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
        -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/ros2/cross_compile/cmake-toolchains/generic_linux.cmake" \
        -DSECURITY=ON

Done! The install and build directories will contain the cross-compiled assets.

Automated Cross-compilation 

All the steps above are also included into a Dockerfile and can be used for automation/CI.

First, download the dockerfile and build the image:

wget https://raw.githubusercontent.com/ros-tooling/cross_compile/master/Dockerfile_cc_for_arm
docker build -t ros2-crosscompiler:latest - < Dockerfile_cc_for_arm

Now run the image with: (it will take a while !)

docker run -it --name ros2_cc \
    -v /var/run/docker.sock:/var/run/docker.sock \
    ros2-crosscompiler:latest

..note:: The -v /var/run/docker.sock allow us to use Docker inside Docker.

The result of the build will be inside the ros2_ws directory, which can be exported with:

docker cp ros2_cc:/root/cc_ws/ros2_ws .

Cross-compiling against a pre-built ROS 2 

It is possible to cross-compile your packages against a pre-built ROS 2. The steps are similar to the previous Cross-compiling examples for Arm section, with the following modifications:

Instead of downloading the ROS 2 stack, just populate your workspace with your package (ros2 examples on this case) and the cross-compilation assets:

mkdir -p ~/cc_ws/ros2_ws/src
cd ~/cc_ws/ros2_ws/src
git clone https://github.com/ros2/examples.git
git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1
cd ..

Generate and export the file-system as described in 3. Prepare the sysroot, but with the provided Dockerfile_ubuntu_arm64_prebuilt. These _prebuilt Dockerfile will use the binary packages to install ROS 2 instead of building from source.

Modify the environment variable ROS2_INSTALL_PATH to point to the installation directory:

export ROS2_INSTALL_PATH=~/cc_ws/sysroot_docker/opt/ros/crystal

Source the setup.bash script on the target file-system:

source $ROS2_INSTALL_PATH/setup.bash

Then, start a build with Colcon specifying the toolchain-file:

colcon build \
    --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
        -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/cross_compile/cmake-toolchains/generic_linux.cmake"

Run on the target 

Copy the file-system on your target or use the previously built docker image:

docker run -it --rm -v `pwd`/ros2_ws:/ros2_ws arm_ros2:latest

Source the environment:

source /ros2_ws/install/local_setup.bash

Run some of the C++ or python examples:

ros2 run demo_nodes_cpp listener &
ros2 run demo_nodes_py talker