Running Docker / Alpine Linux in an IPv6-only environment

This month, my virtual private server started offering IPv6-only as a general service. 

At the OS level, both servers and clients support IPv4 and IPv6 network environments using dual-stack but there are quite a few HTTP/HTTPS servers that do not yet have AAAA records. By default, the Docker server configures container networks for IPv4-only, so I had a hard time running it in this environment.

To help others facing similar scenarios, I wanted to share my experience running Docker/Alpine Linux in the IPv6-only environment of Rocky Linux 8.5.

Installing Docker

First, install Docker using the dnf command.

sudo dnf update -y
sudo dnf install -y dnf-utils device-mapper-persistent-data lvm2
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf makecache
sudo dnf install -y docker-ce
sudo systemctl start docker
sudo systemctl enable docker

I used the dnf command because yum is not recommended on recent CentOS Linux.

Permission settings for users running Docker

If you want to run Docker somewhere other than root, set your user to belong to the Docker group.

sudo groupadd docker
#groupadd: group'docker' already exists
sudo usermod -aG docker $ USER

Create a daemon .json for IPv6

Add IPv6-compatible settings to the Docker daemon startup options at /etc/docker/daemon.json.

sudo vi /etc/docker/daemon.json

The contents of the created file are as follows:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd11:2233:4455:6677::/64",
  "registry-mirrors": ["https://mirror.gcr.io"]
}

Docker’s default registry registry-1.docker.io only returns IPv4 addresses, so you can’t communicate in an IPv6-only environment. In this case, set Google’s registry mirror that supports IPv6. The fixed-cidr-v6 setting is a Unique Local Address (ULA) that you set yourself and is equivalent to an IPv6 private address. To use it, specify any address starting with ‘fd.’, for example fd11:2233:4455:6677::/64

Restart Docker

Restart Docker for these settings to take effect.

sudo systemctl restart docker

If you get an error, review the daemon.json configuration file and fix it.

Add NAT with ip6tables

Set NAT in fixed-cidr-v6 of daemon.json so that you can communicate outwards from the internal network.

sudo ip6tables -t nat -A POSTROUTING -s fd11:2233:4455:6677::/64 ! -o docker0 -j MASQUERADE

Check if the NAT table has been added correctly by specifying -t nat as an argument.

sudo ip6tables -t nat -L

MASQUERADE all fd11:2233:4455:6677::/64 anywhere 

It’s okay if the following has the line MASQUERADE all fd11:2233:4455:6677::/64 anywhere in the output.

Chain PREROUTING (policy ACCEPT)
target prot opt ​​source destination

Chain INPUT (policy ACCEPT)
target prot opt ​​source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt ​​source destination
MASQUERADE all fd11:2233:4455:6677::/64 anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt ​​source destination

Rc.local settings on reboot

Set rc.local so that the NAT setting is valid even after rebooting.

sudo vi /etc/rc.d/rc.local

The rc.local is a backwards compatibility file but you should add the following line at the end.

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
#It is highly advisable to create own systemd services or udev rules
#to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
#this script will NOT be run after all other services.
#
# Please note that you must run'chmod +x /etc/rc.d/rc.local' to ensure
#that this script will be executed during boot.

touch/var/lock/subsys/local

ip6tables -t nat -A POSTROUTING -s fd11:2233:4455:6677::/64 ! -o docker0 -j MASQUERADE

If this is your first time setting up rc.local, run the following command to make it run on reboot:

sudo chmod +x /etc/rc.d/rc.local
sudo systemctl start rc-local

Check the current settings with Docker info

Run docker info to see the current settings. You should get an output like this:

Client: Client:
 Context: default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
~ Omitted ~
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false

It’s not a problem if the output has https://mirror.gcr.io/ in Registry Mirrors:.

Creating a Dockerfile for Alpine Linux

Create a Dockerfile in a directory of your choice with vi Dockerfile. The contents of the Dockerfile are as follows:

FROM alpine: 3.15.4
RUN apk update && apk add bind-tools curl
CMD ["/bin/sh"]

After pulling the image of alpine:3.15.4, I wanted to use the dig command and curl command, so I installed the bind tools and curl apk package and start the shell by default.

Build Docker

The tag name can be whatever you like, but you must build Docker with ipv6/alpine.

docker build -t ipv6/alpine

If you can download the apk package from the IPv6 server and get the following result at build time, it has been successful.

Sending build context to Docker daemon 3.072kB
Step 1/3: FROM alpine: 3.15.4
3.15.4: Pulling from library/alpine
df9b9388f04a: Pull complete
Digest: sha256: 4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
Status: Downloaded newer image for alpine: 3.15.4
 ---> 0ac33e5f5afa
Step 2/3: RUN apk update && apk add bind-tools curl
 ---> Running in a1ebb762c17e
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
v3.15.4-78-ge239b26b75 [https://dl-cdn.alpinelinux.org/alpine/v3.15/main]
v3.15.4-79-gaf675e239c [https://dl-cdn.alpinelinux.org/alpine/v3.15/community]
OK: 15855 distinct packages available
(1/18) Installing fstrm (0.6.1-r0)
(2/18) Installing krb5-conf (1.0-r2)
(3/18) Installing libcom_err (1.46.4-r0)
(4/18) Installing keyutils-libs (1.6.3-r0)
(5/18) Installing libverto (0.3.2-r0)
(6/18) Installing krb5-libs (1.19.3-r0)
(7/18) Installing json-c (0.15-r1)
(8/18) Installing protobuf-c (1.4.0-r0)
(9/18) Installing libuv (1.42.0-r0)
(10/18) Installing xz-libs (5.2.5-r1)
(11/18) Installing libxml2 (2.9.13-r0)
(12/18) Installing bind-libs (9.16.27-r0)
(13/18) Installing bind-tools (9.16.27-r0)
(14/18) Installing ca-certificates (20211220-r0)
(15/18) Installing brotli-libs (1.0.9-r5)
(16/18) Installing nghttp2-libs (1.46.0-r0)
(17/18) Installing libcurl (7.80.0-r1)
(18/18) Installing curl (7.80.0-r1)
Executing busybox-1.34.1-r5.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 15 MiB in 32 packages
Removing intermediate container a1ebb762c17e
 ---> 18d7e9a3b51b
Step 3/3: CMD ["/bin/sh"]
 ---> Running in ccda19d240b3
Removing intermediate container ccda19d240b3
 ---> ff8bbe4b67c8
Successfully built ff8bbe4b67c8
Successfully tagged ipv6/alpine: latest

If there are no errors, it has been successful. Congratulations!

Errors in registry-1.docker.io

If you get the following error when building Docker, it has failed.

Sending build context to Docker daemon 4.096kB
Step 1/3: FROM alpine: 3.15.4
Get "https://registry-1.docker.io/v2/": dial tcp 34.203.135.183:443: connect: network is unreachable

This means registry-1.docker.io does not have an IPv6 AAAA record and you are trying to communicate with an IPv4 address.

In this case, you’ll need to specify "registry-mirrors": ["https://mirror.gcr.io"] in the above daemon.json so the mirror server can communicate with IPv6. If that fails, it may be that the download URL is mis-cached by Docker. Try changing the setting and downloading another version such as FROM alpine: 3.15.4 or FROM alpine: 3.15.2.

If you get an error on dl-cdn .alpinelinux.org

If you get the following error when building Docker, it is failing because you cannot communicate from outwards from Docker in an IPv6-only environment.

fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.15/main: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.15/main: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.15/community: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.15/community: No such file or directory

The ip6tables NAT is necessary to communicate from the internal network and is specified with "fixed-cidr-v6": "fd11:2233:4455:6677::/64" in the daemon.json mentioned above. You need to check if it is enabled.

Also, as of 3 May 2022, dl-cdn.alpinelinux.org is dual-stacked and has IPv6 AAAA records, so you should be able to communicate in an IPv6-only environment.

$ dig dl-cdn.alpinelinux.org. A
; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> dl-cdn.alpinelinux.org. A
~ Omitted ~
;; QUESTION SECTION:
; dl-cdn.alpinelinux.org. IN A

;; ANSWER SECTION:
dl-cdn.alpinelinux.org. 1420 IN CNAME dualstack.d.sni.global.fastly.net.
dualstack.d.sni.global.fastly.net. 30 IN A 151.101.110.133

$ dig dl-cdn.alpinelinux.org. AAAA

; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> dl-cdn.alpinelinux.org. AAAA
~ Omitted ~
;; QUESTION SECTION:
; dl-cdn.alpinelinux.org. IN AAAA

;; ANSWER SECTION:
dl-cdn.alpinelinux.org. 1484 IN CNAME dualstack.d.sni.global.fastly.net.
dualstack.d.sni.global.fastly.net. 30 IN AAAA 2a04:4e42:1a::645

If the server does not return AAAA records, it will fail when trying to communicate with an IPv4 address. To fix this, specify a mirror server that can communicate with IPv6, such as by executing with Dockerfile:

RUN sed -r "s;//dl-cdn.alpinelinux.org/alpine;//ftp.udx.icscoe.jp/Linux/alpine;g" -i /etc/apk/repositories

You can check the list of official mirror servers at Official Alpine Linux mirrors.

Run Docker

Docker runs with the tag name ipv6/alpine specified at build time. You can run the shell interactively with the -i option.

docker container run -it ipv6/alpine

After successfully entering the shell, use the curl command to check if HTTP communication with the IPv6 server is possible.

/ # cd / tmp
/ tmp # curl http://google.com/
<HTML> <HEAD> <meta http-equiv = "content-type" content = "text / html; charset = utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
/tmp #

A 301 response indicates success, showing it was possible to communicate with IPv6-only. Congratulations.

BusyBox wget isn’t completely compatible with IPv6

BusyBox’s wget is not fully IPv6 compliant, so it’s a good idea to replace the wget command with apk add wget or use the curl command.

IPv6 only環境でbusybox wgetで名前解決する際にIPv4アドレスを優先してしまってダウンロードできない問題：ラッパー関数xhost2sockaddrの中でstr2sockaddrに(sa_family_t)AF_UNSPECを指定してしまっていて、getaddrinfoで複数のIPアドレス(v4/v6)が返ってきたときに最初のIPをそのまま使ってしまう罠.

— takesako (@takesako) May 3, 2022

It would be great if the Internet could be used more in an IPv6-only environment.

Translated and adapted from the original post on Hatena Blog.

Yoshinori Takesako is the founder of SECCON.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.