Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

issues with swarm encrypted overlay network and iptables-nft on RHEL 8+ #43382

Closed
luca-heitmann opened this issue Mar 15, 2022 · 7 comments
Closed

issues with swarm encrypted overlay network and iptables-nft on RHEL 8+ #43382

luca-heitmann opened this issue Mar 15, 2022 · 7 comments
Labels
area/networking area/security kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.

Comments

@luca-heitmann
Copy link

luca-heitmann commented Mar 15, 2022 •
edited by corhere

Description

We see errors in the journal log when creating an encrypted overlay network with RHEL 8.5 and Docker 20.10. I created an test environment with two Rocky Linux 8.5 VMs where the issue is reproducible (see below). As Mirantis does not support RHEL 8.5 for Docker 20.10 yet, I also verified that the issue is reproducible with Rocky 8.4.

could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1048832 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))

The errors in the log show that some iptables rules cannot be created. The reason seems to be u32 extension was deprecated in RHEL 8: https://bugzilla.redhat.com/show_bug.cgi?id=2061288

The encrypted overlay network seem to use u32:

moby/libnetwork/drivers/overlay/encryption.go

Line 211 in 2b70006

rule = []string{"-p", "udp", "--dport", p, "-m", "u32", "--u32", c, "-j", "MARK", "--set-mark", m}

I am not an expert and don't fully understand the consequences of the missing iptables/nftables rules. In the test environment, I cannot detect any misbehavior of the routing mesh or other components.

Steps to reproduce the issue:

  1. Apply the following compose file by running docker stack deploy -c stack.yml test-stack on a system which uses nftables:
version: '3.8'

networks:
  default:
    driver_opts:
      encrypted: "true"
    name: test-net

services:
  test-srv-1:
    deploy:
      placement:
        constraints:
          - node.hostname==node1
    image: nginx
    ports:
    - published: 30001
      target: 80
  test-srv-2:
    deploy:
      placement:
        constraints:
          - node.hostname==node2
    image: nginx
    ports:
    - published: 30002
      target: 80
  1. Execute journalctl

Describe the results you received:

Found these errors when executing journalctl:

Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.640857615Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.640885838Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.652697451Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:23 node1 dockerd[5991]: time="2022-03-16T10:20:23.664394572Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.041791548Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.042004393Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.052837063Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:20:24 node1 dockerd[5991]: time="2022-03-16T10:20:24.064097404Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP:
 iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.492985295Z" level=warning msg="could not install mangle rule:  (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.493022369Z" level=warning msg=" (iptables failed: iptables --wait -t mangle -A OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))"
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.508195851Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry
iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."
Mar 16 10:26:46 node1 dockerd[5991]: time="2022-03-16T10:26:46.521236295Z" level=error msg="could not add input rule:  (iptables failed: iptables --wait -t filter -A INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP: iptables v1.8.4 (nf_tables): Couldn't load match `u32':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)). Please do it manually."

Describe the results you expected:

not having these errors

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

[root@node1 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:07:38 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:05:59 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

[root@node1 ~]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: x0ntpiod2v7v86o3m0iu6gnq9
  Is Manager: true
  ClusterID: 6nciqolcj6zogy3sj79d5n10c
  Managers: 1
  Nodes: 2
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 172.16.16.101
  Manager Addresses:
   172.16.16.101:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-305.19.1.el8_4.x86_64
 Operating System: Rocky Linux 8.4 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.775GiB
 Name: node1
 ID: H5J2:TIN2:BZYH:EGX5:3F4K:HX5A:RTVT:WJO7:437D:NK2S:WROP:P55O
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

I use the following Vagrantfile in my test environment:

# -*- mode: ruby -*-
# vi: set ft=ruby :

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'
ENV['VAGRANT_NO_PARALLEL'] = 'yes'

Vagrant.configure(2) do |config|

  NodeCount = 2

  (1..NodeCount).each do |i|

    config.vm.define "node#{i}" do |node|

      node.vm.box               = "bento/rockylinux-8.4"
      node.vm.box_check_update  = false
      node.vm.box_version       = "202110.26.0"
      node.vm.hostname          = "node#{i}"

      node.vm.network "private_network", ip: "172.16.16.10#{i}"

      node.vm.provider :virtualbox do |v|
        v.gui     = true
        v.name    = "node#{i}"
        v.memory  = 2048
        v.cpus    = 2
      end

      node.vm.provision "shell", path: "bootstrap.sh"
      if i == 1
        node.vm.provision "shell", inline: "docker swarm init --advertise-addr 172.16.16.101"
        node.vm.provision "shell", inline: "docker swarm join-token -q worker > /vagrant/token"
      else
        node.vm.provision "shell", inline: "docker swarm join --advertise-addr 172.16.16.10#{i} --listen-addr 172.16.16.10#{i}:2377 --token `cat /vagrant/token` 172.16.16.101:2377"
      end
    end
  end
end

this bootstrap.sh file is used to provision the VMs:

#!/bin/bash

# Enable ssh password authentication
echo "Enable ssh password authentication"
sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/.*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl reload sshd

# Set Root password
echo "Set root password"
echo -e "admin\nadmin" | passwd root >/dev/null 2>&1

# Install Docker
echo "Install Docker"
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y docker-ce-20.10.13-3.el8 docker-ce-cli-20.10.13-3.el8 containerd.io
systemctl enable docker
systemctl start docker

If you have vagrant with VirtualBox set up and want to use this configuration:

  1. Create the Vagrantfile and bootstrap.sh in a new directory with the content from above
  2. Create the file "stack.yml" with the content from above
  3. Execute "vagrant up"
  4. Execute "vagrant ssh node1"
  5. Execute "sudo docker stack deploy -c /vagrant/stack.yml test-stack"
  6. Execute "sudo journalctl | grep iptables"

Edit: I just noticed that Mirantis does not support RHEL 8.5 but 8.4. So I changed the test environment to Rocky 8.4 and verified that the issue is also reproducible there.
@johnbizokk
Copy link

@luca-heitmann Hello! After a kernel rolled back (5.17.3-> 4.18.0) during a system update, I had the same issue. It all works like a clockwork after reverting to the "5. X" kernel.

@xinfengliu
Copy link
Contributor

xinfengliu commented Jun 1, 2022 •
edited

This is because RHEL 8.x and other variants (Rocky, CentOS) do not include xt_u32 kernel module by default. You can install it by sudo dnf install kernel-modules-extra then restart docker daemon.

@MarcelGeo
Copy link

MarcelGeo commented Jul 7, 2022 •
edited

@xinfengliu we have the same error messages with iptables after sudo dnf install kernel-modules-extra. RHEL 8

@johnbizokk
Copy link

@xinfengliu we have the same error messages with iptables after sudo dnf install kernel-modules-extra. RHEL 8

I would suggest upgrading to the latest kernel.

@MarcelGeo
Copy link

Thanks for answer johnbizokk and xinfengliu. Our version is:

4.18.0-372.9.1.el8.x86_64

@yoheiueda yoheiueda mentioned this issue Sep 22, 2022
Open
@corhere corhere changed the title issues with swarm encrypted overlay network and nftables issues with swarm encrypted overlay network and iptables-nft on RHEL 8+ Mar 8, 2023
@corhere
Copy link
Contributor

corhere commented Mar 8, 2023

RedHat decided to deprecate the xt_u32 kernel module in RHEL 8 and remove it in RHEL 9 as a way to nudge users towards migrating to native nftables. This is an arbitrary limitation imposed by RedHat; iptables-nft is compatible with all legacy packet matches which are available for the running kernel. iptables-nft is deprecated as of RHEL 9 but is still available...for now.

@neersighted
Copy link
Member

This issue became, and was fixed during the remediation of:

@neersighted neersighted added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. area/security area/networking labels Apr 4, 2023
This was referenced Apr 4, 2023
Closed
Closed
This was referenced Oct 3, 2023
Merged
Merged
@github-actions github-actions bot mentioned this issue Oct 10, 2023
Merged
1 task
This was referenced Oct 23, 2023
Merged
Merged
Merged
Merged
Merged
This was referenced Oct 30, 2023
Merged
Merged
Merged
@github-actions github-actions bot mentioned this issue Nov 28, 2023
Merged
1 task
This was referenced Dec 7, 2023
Merged
Merged
@github-actions github-actions bot mentioned this issue Dec 21, 2023
Closed
This was referenced Dec 22, 2023
Merged
Merged
This was referenced Dec 30, 2023
Merged
Merged
Merged
Merged
Merged
Merged
Merged
This was referenced Jan 7, 2024
Merged
Merged
Merged
@github-actions github-actions bot mentioned this issue Jan 16, 2024
Merged
1 task
This was referenced Jan 23, 2024
Merged
Merged
@demian711 demian711 mentioned this issue Jan 25, 2024
Closed
@github-actions github-actions bot mentioned this issue Feb 2, 2024
Merged
1 task
@novda novda mentioned this issue Apr 4, 2024
Merged
7 tasks
@github-actions github-actions bot mentioned this issue Apr 7, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking area/security kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@corhere @xinfengliu @neersighted @johnbizokk @MarcelGeo @luca-heitmann