Compile an IETF QUIC stack for Android and test it on a cellular radio access network context using OpenAirInterface

Quic is expected to be the successor of the old TCP as Transport Protocol. Perhaps in a short next future, we should say that internet thanks to QUIC/IP instead of current TCP/IP stack. Quic has been proposed by Google in 2013 and it is now standardized by IETF as RFC9000 since June 2021. Quic is a promising technology to increase throughput, privacy and ubiquity while correcting some TCP limitations.

In this context many open source stacks has been proposed and had the advantage to be implemented in user space instead of kernel space with TCP.
By this, it is simple to test and experiment with Quic, easier than with TCP kernel module.

In this short article, I give my procedure to test a Quic HTTP connection using a testbed cellular radio access network.

# Compile an open-source Quic HTTP client for an Android phone

In this experiment I chose lsquic developed by CDN provider LiteSpeed as Quic stack. I made this choice because:
- It is written in C for high-performances
- It provides a library to link with an Android application for example
- It implements last drafts and features
- It is well maintained by a specialist internet company
- There is a straight way to compile a simple http client for android

However, I also could recommend to use picoquic developped by a french researcher (Christian Huitema) which is also written in a high-quality C.

My android phone is a chinese generic phone based on an ARM snapdragon CPU. It is important to be root on this phone to run untrusted binaries. I installed LineageOS but it should work also for a rooted android. Android 9 is installed with an android SDK level 28.

First, you have to get the Android cross compile toolchain (In the following, I guess you use Linux):

# Download android studio
wget https://redirector.gvt1.com/edgedl/android/studio/ide-zips/4.2.2.0/android-studio-ide-202.7486908-linux.tar.gz
# Extract it && run it
tar -xf android-studio-ide-2020.7486908-linux.tar.gz && cd android-studio/bin && ./studio.sh

On “Configure”, add android 28 SDK. After that, you should have your toolchain installed under :

less ~/Android/Sdk/ndk/22.1.7171670/build/cmake/android.toolchain.cmake

Now, let’s build and install lsquic.

The first required components to build is BoringSSL for crypto :

# Clone repository and checkout to a stable commit
git clone https://boringssl.googlesource.com/boringssl
cd boringssl
git checkout a2278d4d2cabe73f6663e3299ea7808edfa306b9
# cmake configure
~/Android/Sdk/cmake/3.18.1/bin/cmake -DCMAKE_TOOLCHAIN_FILE=~/Android/Sdk/ndk/22.1.7171670/build/cmake/android.toolchain.cmake -DANDROID_NDK=~/Android/Sdk/ndk/22.1.7171670/ -DCMAKE_BUILD_TYPE=Release -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 -DBUILD_SHARED_LIBS=1 .
# cmake build
~/Android/Sdk/cmake/3.18.1/bin/cmake --build .
# check library generated
ls crypto/libcrypto.so

By default, android should not embed the libevent library that is required for lsquic binaries (not for lsquic libraries).

# Download a stable version of libevent
wget https://github.com/libevent/libevent/releases/download/release-2.1.12-stable/libevent-2.1.12-stable.tar.gz
tar -zxvf libevent-2.1.12-stable.tar.gz
# build for android
cd libevent-2.1.12-stable
mkdir build && cd build
~/Android/Sdk/cmake/3.18.1/bin/cmake -DCMAKE_TOOLCHAIN_FILE=~/Android/Sdk/ndk/22.1.7171670/build/cmake/android.toolchain.cmake -DANDROID_NDK=~/Android/Sdk/ndk/22.1.7171670/ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 -DASSIMP_BUILD_TESTS=OFF -DASSIMP_NO_EXPORT=TRUE -DANDROID_TOOLCHAIN=clang -DEVENT_DISABLE_OPENSSL=ON ..
~/Android/Sdk/cmake/3.18.1/bin/cmake --build ..
# check library generated
ls build/lib/libevent.so

Now build lsquic executables:

# Clone lsquic
git clone https://github.com/litespeedtech/lsquic.git
cd lsquic && git checkout 077ef87880929db37c5b97cac86c488a08b1e275
git submodule init
git submodule update
# Configure and build with cmake
~/Android/Sdk/cmake/3.18.1/bin/cmake -DTESTS=NO -DANDROID_PLATFORM=android-28 -DBORINGSSL_DIR=../boringssl -DCMAKE_TOOLCHAIN_FILE=~/Android/Sdk/ndk/22.1.7171670/build/cmake/android.toolchain.cmake  -DANDROID_ABI=arm64-v8a -DEVENT_INCLUDE_DIR=../libevent/include -DEVENT_LIB=../libevent/build/lib/libevent.so -DLSQUIC_SHARED_LIB=1 .
~/Android/Sdk/cmake/3.18.1/bin/cmake --build .
# check for executables
ls bin/http_*
# Push to phone
sudo adb push bin/http_client /bin/

# Compile lsquic http server for a linux machine

Almost the same as android but easier. Quickly:

# BoringSSL
sudo apt install zlib1g-dev libevent-dev
git clone https://boringssl.googlesource.com/boringssl
cd boringssl
git checkout a2278d4d2cabe73f6663e3299ea7808edfa306b9
rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz
cmake -DBUILD_SHARED_LIBS=1 .
make
# lsquic
git clone https://github.com/litespeedtech/lsquic.git
cd lsquic && git checkout 077ef87880929db37c5b97cac86c488a08b1e275
git submodule init
git submodule update
cmake -DLSQUIC_SHARED_LIB=1 -DBORINGSSL_DIR=../boringssl .
make
make test

# Generate stuffs for Quic server crypto

Quic is a deeply encrypted protocol intrinsically designed around TLS1.3.

TCP packet encryption vs Quic encryption

You can easily create keys and certificate for the HTTP Quic server using openssl:

openssl req -x509 -nodes -new sha256 -days 1024 -newkey rsa:2048 -keyout key.pem -out server.pem -subj "/C=FR/CN=your-domain"
openssl x509 -outform pem -in server.pem -out server.crt

You have all you need to run a Quic-based transfer between a linux HTTP server and an Android HTTP client. An example of Quic transfer is given in the section after the following.

# Install and run OpenAirInterface LTE cellular network

OpenAirInterface is an initiative to develop an open-source LTE and 5G cellular network. Codes for base station (for radio access network) and for core are available here and here.

The testbed should look like as follow:

With a server PC for base station and Core, an USRP SDR for radio interface and your android phone.

The official tutorial to configure and build OAI is available at this wiki page. Also, I would recommend to take a look at the resources provided by open-cells which is an individual company owned by an active developer of OAI.

You will have to buy a empty SIM card, burn it and add it to the HSS (a SIM card database).

At the end, to run your testbed:

# OAI Core
cd openair-cn/scripts/
./run_hss
./run_mme
./run_spgw
# OAI eNB
cd oai/
sudo -E ./cmake_targets/ran_build/build/lte-softmodem -O ci-scripts/conf_files/mono.enb.band7.tm1.50PRB.usrpb210.conf
# android UE configured with ADB, deactive airplane mode
adb shell settings put global airplane_mode_on 0
adb shell am broadcast -a android.intent.action.AIRPLANE_MODE

# Run a Quic-based HTTP transfer

Everything is now configured to observe how works Quic on a cellular network.

# Server side
./bin/http_server -c your-domain,server.crt,key.pem -s 0.0.0.0:12345 -L info -G .
sudo tcpdump -i eno1 -p -s 0 -w quic_transfert.pcap 'udp && port 12345'
# UE side
sudo adb shell
su -
cd /bin
./http_client -p /file-1M -H your-domain -s X.X.X.X:12345 -t -a -L info -o version=h3-29  # Or whatever version available on both client and server

Congratulation, you run your (probably not) first Quic transfer on a cellular network !

With the tcpdump pcap and the SSL keys exported by your http_server (lsquic/*.keys), you can analyse your transfer using wireshark.
In Preferences-> Protocols -> TLS, add your key in “Master-Secret log filename” field. You should get something like that

beginning of a QUIC connection

The first corresponds to the establishment of crypto and Quic context, after that, in a direction, you have a data transfer using a STREAM frame and in the other, ACK frame for data acknowledgement. Without keys, it is impossible with QUIC to know what is the purpose of a packet (if we expect guessing using packet size and long header vs short header). I let you explore further.

Finally, Quic as TCP and other transport protocols uses a Congestion Control algorithm to transfer data without saturing the link. By default, lsquic uses Cubic like linux TCP. You can change it for a most recent CC like BBR and see the difference in term of achieved throughput.
For me, I observe this profile of transfer with bunch of transmission followed by period of pause due to a lack of data acknowledgment.

Quic data transfer cumulative volume

The achieved throughput is around 11Mbps (far less than 37Mbps effective capacity on my testbed). The cause here is an early exit from slow start mechanism.

I will not going deeper into details since my objective with this article is to demonstrate a (relatively) simple possibility to test the new Quic transport protocol on a every-people-smartphone.

Thank you for reading, I hope it helps you or interests you to setup OpenAirInterface and/or lsquic.

— Octavien dM. PhD student.