Who is messing with my DNS server? Discovering and managing network daemons

The first thing we need to do is read /etc/resolv.conf, since many programs that modify it include an explanatory message, that usually says we shouldn't modify it cause is going to be overwritten.

For example, this is one of my machines /etc/resolv.conf, which is modified by systemd-resolved:

$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

And this is the /etc/resolv.conf content in case NetworkManager is managing it:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.122.1

But we can also found /etc/resolv.conf without comments:

$ cat /etc/resolv.conf
nameserver 192.168.122.1

In this last case, I discovered that was being managed by dhclient.

However, in case we are in doubt, we can apply other techniques.

Alternatively, we can monitor /etc/resolv.conf file for write operations. We can do this with opensnoop, an utility that uses eBPF for tracing file opening operations. In order to use it we must install the bpfcc-tools package and Linux headers (in the case of Debian can be done with sudo apt install linux-headers-$(uname -r)).

Once the tool is installed, we can execute it and await a few minutes to check if any process is modifying some resolv.conf file:

$ sudo opensnoop-bpfcc -f O_WRONLY -f O_RDWR | grep 'resolv.conf'
1857   dhclient-script     3   0 /etc/resolv.conf.dhclient-new.1857
1857   dhclient-script     3   0 /etc/resolv.conf.dhclient-new.1857
1857   dhclient-script     3   0 /etc/resolv.conf

With the previous command, we trace with opensnoop the files open for writing and filter them with grep to just show those whose name includes resolv.conf, not just /etc/resolv.conf.

If you want to learn more about eBPF, check Learn eBPF Tracing: Tutorial and Examples by Brendan Gregg, who shows a lot of tools for tracing based on eBPF, like tcpconnect, one of my favorites, that allows to see the processes network connections in real time.

It is advisable to monitor not just /etc/resolv.conf, but any file that contains resolv.conf since many programs use cache files with similar names to store temporal results. For instance, dhclient uses files like /etc/resolv.conf.dhclient-new.590 to write changes and then checks if those are different from /etc/resolv.conf, if that is not the case, /etc/resolv.conf is not written.

On the other hand, it is possible for programs to just rename the cache file to /etc/resolv.conf instead of writing it (something like mv /etc/resolv.conf.HOIHS2 /etc/resolv.conf, which is done by NetworkManager), so /etc/resolv.conf won't appear in as an open file (since rename syscall is used instead of open and opensnoop won't be able to detect it).

Anyway, after examining /etc/resolv.conf content and monitoring it, we should already have an idea about the daemon that is managing the DNS server, but even if we are not sure, I'm going to explain many of them and see how we can check if they are running.

Let's how we can add a DNS server when dhclient is in charge. First, we need to make sure it is running:

$ ps -ef | grep dhclient
root         469       1  0 21:14 ?        00:00:00 dhclient -4 -v -i -pf /run/dhclient.enp1s0.pid -lf /var/lib/dhcp/dhclient.enp1s0.leases -I -df /var/lib/dhcp/dhclient6.enp1s0.leases enp1s0
root         585       1  0 21:18 ?        00:00:00 dhclient
user         611     554  0 21:25 pts/0    00:00:00 grep dhclient

Once we know for sure that dhclient is running, in order to add a new DNS server we need to modify /etc/dhcp/dhclient.conf, its configuration file, where we can found several DNS options related. Specifically, to add a new DNS server we can add some of these lines:

prepend domain-name-servers 127.0.0.1;
append domain-name-servers 192.168.122.13;

This way we can add our DNS with higher or lower priority, respectively, to the one added by DHCP configuration. Additionally, we can avoid using the DHCP configured by DHCP by removing the domain-name-servers item from the request statement that we can found in /etc/dhcp/dhclient.conf.

dhclient also allows to create specific configurations for each network interface where we can also specify a DNS server for such interface, but I'm not going to discuss that here. In case you want more information you can check dhclient.conf(5).

Therefore, if we add the previously discussed lines to /etc/dhcp/dhclient.conf and we restart the ifup service reexecute dhclient (or wait for a while) we should see the changes applied to /etc/resolv.conf:

$ sudo systemctl restart ifup@enp1s0.service
$ cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 192.168.122.1
nameserver 192.168.122.13;

Be aware that the ifup service, which spawns dhclient is given the network interface as a parameter, which in my case is enp1s0, but yours may differ.

Besides, if we reboot the machine, changes should remain.

As we state before, NetworkManager usually is on charge of managing DNS servers when we found a /etc/resolver.conf file similar to the following:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.122.1

Besides, we can confirm that NetworkManager is running by checking the service:

$ systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendo>
     Active: active (running) since Sun 2024-08-11 10:36:47 CEST; 9h ago
       Docs: man:NetworkManager(8)
   Main PID: 1270 (NetworkManager)
      Tasks: 3 (limit: 37642)
     Memory: 14.4M
        CPU: 5.143s
     CGroup: /system.slice/NetworkManager.service
             └─1270 /usr/sbin/NetworkManager --no-daemon

NetworkManager is more complex that other network managers, since it offers several options to manage the DNS server such as do it itself or delegate the task in third-parties like systemd-resolved or dhclient. You can get more information on the dns section of NetworkManager.conf(5). In this section we are going to assume that the DNS server is going to be managed by NetworkManager itself, since other options are explored in their respective sections.

We can use nmcli to manage NetworkManager. With this tool we can specify a new DNS server for the network connection we want (I didn't found how to specify for all connections). We can list the connections (network interfaces) with nmcli connection show:

$ nmcli connection show
NAME                UUID                                  TYPE      DEVICE
Wired connection 1  56d704b3-e21d-4fba-93b8-c89870296a94  ethernet  eth0
lo                  28786bc1-47ab-4264-bdca-3e25b38361b3  loopback  lo

And afterwards add a DNS server with nmcli connection modify:

$ network_connection="Wired connection 1"
$ sudo nmcli connection modify "$network_connection" ipv4.dns "192.168.122.135"
$ sudo systemctl restart NetworkManager
$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.122.135
nameserver 192.168.122.1

As we can appreciate, our DNS servers were updated after executing the command, and if we reboot the machine, changes should remain. Be aware that your network connection name could be different from mine so you may need to adapt the command.

Additionally, if you don't want to use the DHCP specified DNS you can use the following command:

sudo nmcli con mod "$network_connection" ipv4.ignore-auto-dns yes

Last but not least, we have systemd-resolved. We can verify that is been used by checking that /etc/resolv.conf has a similar content to the following:

$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

If we put attention to the content, we can see that systemd-resolved is pointing to its own local DNS server at 127.0.0.53.

On the other hand, /etc/resolv.conf will be a link to /run/systemd/resolve/stub-resolv.conf:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 ago 19  2022 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

And we cannot forget to verify that systemd-resolved is running:

$ sudo systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-08-13 20:04:55 CEST; 29min ago
...

So, now we know systemd-resolved is on charge of DNS resolutions we can add a DNS server. To do this we need to add a new DNS entry in /etc/systemd/resolved.conf, like the following:

$ cat /etc/systemd/resolved.conf | grep DNS=
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
DNS=192.168.122.135
#FallbackDNS=
#MulticastDNS=no

Then we restart the systemd-resolved service:

sudo systemctl restart systemd-resolved.service

And we can confirm our DNS server is set:

$ resolvectl status
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 192.168.122.135
       DNS Servers: 192.168.122.135

Link 2 (enp1s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.122.1
       DNS Servers: 192.168.122.1

We can verify it by reading /run/systemd/resolve/resolv.conf as well:

$ cat /run/systemd/resolve/resolv.conf | grep nameserver
nameserver 192.168.122.135
nameserver 192.168.122.1

We must note that this time we didn't check /etc/resolv.conf to verify the changes, but resolvectl and /run/systemd/resolve/resolv.conf. This is due to systemd-resolved don't really modify /etc/resolv.conf, but adds its own local DNS server in 127.0.0.53 and then it redirects the DNS requests to the servers we indicate.

Anyway, changes should remain after we reboot the machine.