anfine
← All posts

Tailscale dropped Aliyun DNS replies on an ECS instance

· 5 min read

DNS requests reached Aliyun, but Tailscale's ts-input rule dropped replies from its 100.64.0.0/10 address space.

Status
Resolved
#Linux#DNS#Tailscale

Summary

After I started Tailscale on an Aliyun ECS instance, a Docker-hosted agent could no longer call the DeepSeek API. It reported Temporary failure in name resolution. DNS worked again when Tailscale was stopped.

The overlap between Tailscale’s 100.64.0.0/10 address space and Aliyun’s internal DNS addresses initially suggested a routing problem. The route was correct. DNS replies arrived on eth0, then matched a Tailscale iptables rule that dropped traffic sourced from 100.64.0.0/10 unless it came through tailscale0.

Impact

The agent could not resolve the API hostname and therefore could not call DeepSeek. Direct lookups through Aliyun DNS also timed out during the incident; I did not establish the impact on other services.

Investigation

The instance used systemd-resolved. /etc/resolv.conf pointed to its local stub resolver:

nameserver 127.0.0.53

resolvectl status showed the upstream servers on eth0:

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute
Current DNS Server: 100.100.2.138
       DNS Servers: 100.100.2.136 100.100.2.138

Tailscale DNS management was already off (Tailscale DNS: disabled.). Both resolvectl query api.deepseek.com and resolvectl query www.baidu.com failed. Bypassing the local resolver did not help:

nslookup www.baidu.com 100.100.2.136
communications error to 100.100.2.136#53: timed out
no servers could be reached

Aliyun’s 100.100.2.136 and 100.100.2.138 sit inside 100.64.0.0/10, which Tailscale also uses. I checked whether Tailscale had taken the outbound route:

ip route get 100.100.2.136
100.100.2.136 via 172.31.207.253 dev eth0 src 172.31.194.159

The main table also had routes for both DNS servers via eth0. ip route show table 52 showed only 100.100.100.100 dev tailscale0, not a 100.64.0.0/10 dev tailscale0 route. The routing evidence did not support the original hypothesis.

I then captured traffic on the physical interface while repeating the lookup:

tcpdump -ni eth0 host 100.100.2.136
nslookup www.baidu.com 100.100.2.136

The capture contained both the query and its reply:

172.31.194.159.58418 > 100.100.2.136.53:
A? www.baidu.com.

100.100.2.136.53 > 172.31.194.159.58418:
CNAME www.a.shifen.com.,
A 180.101.51.73,
A 180.101.49.44

nslookup still timed out. The reply had reached eth0, but it had not reached the process.

Root cause

iptables -S showed that INPUT jumped to ts-input, where Tailscale had installed this rule:

-A INPUT -j ts-input
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP

The DNS reply had source address 100.100.2.136 and arrived on eth0. It therefore matched the drop rule. The same applied to replies from 100.100.2.138.

This was a firewall decision on an inbound packet, not a competition between an Aliyun /32 route and a Tailscale /10 route. The outbound /32 route sent the query through eth0; the INPUT rule later discarded the returning reply. That explains why tcpdump saw a response while nslookup saw a timeout.

Resolution

I added a narrow exception for DNS replies from the two Aliyun servers, ahead of the jump to ts-input:

iptables -N ALIYUN-DNS
iptables -A ALIYUN-DNS -i eth0 -s 100.100.2.136/32 -p udp --sport 53 -j ACCEPT
iptables -A ALIYUN-DNS -i eth0 -s 100.100.2.138/32 -p udp --sport 53 -j ACCEPT
iptables -A ALIYUN-DNS -i eth0 -s 100.100.2.136/32 -p tcp --sport 53 -j ACCEPT
iptables -A ALIYUN-DNS -i eth0 -s 100.100.2.138/32 -p tcp --sport 53 -j ACCEPT
iptables -A ALIYUN-DNS -j RETURN
iptables -I INPUT 1 -j ALIYUN-DNS

Afterward, nslookup www.baidu.com 100.100.2.136 returned addresses, and resolvectl query api.deepseek.com succeeded. curl -I https://api.deepseek.com returned HTTP/2 401: the unauthenticated request reached the API server and received an HTTP response.

I also added a systemd oneshot service to rebuild the rule after startup, ordered after network-online.target and tailscaled.service. Its script recreates the chain contents and inserts the jump if needed:

#!/bin/bash
set -e

CHAIN="ALIYUN-DNS"

iptables -N "$CHAIN" 2>/dev/null || true
iptables -F "$CHAIN"

iptables -A "$CHAIN" -i eth0 -s 100.100.2.136/32 -p udp --sport 53 -j ACCEPT
iptables -A "$CHAIN" -i eth0 -s 100.100.2.138/32 -p udp --sport 53 -j ACCEPT
iptables -A "$CHAIN" -i eth0 -s 100.100.2.136/32 -p tcp --sport 53 -j ACCEPT
iptables -A "$CHAIN" -i eth0 -s 100.100.2.138/32 -p tcp --sport 53 -j ACCEPT
iptables -A "$CHAIN" -j RETURN

iptables -C INPUT -j "$CHAIN" 2>/dev/null || iptables -I INPUT 1 -j "$CHAIN"

What I learned

ip route get showed where the system intended to send the query. tcpdump showed that the request left and the reply returned. iptables explained why the application never saw that reply. Keeping routing, observed traffic, and firewall decisions separate was the useful part of this investigation.

Follow-up

If Tailscale is reconfigured while the server is running, I need to confirm that the ALIYUN-DNS jump still precedes ts-input. The startup service handles reboot persistence, but the draft does not establish that it handles every later rewrite of INPUT.