This guide is intended for OpsRamp environments or any K3s cluster setup that requires custom or internal DNS configuration. It ensures Kubernetes workloads can reliably resolve internal domains and private DNS servers by manually configuring CoreDNS’s DNS settings.

Overview

In environments where Kubernetes workloads must resolve internal domains or use custom DNS servers, the default DNS configuration may not be sufficient. This guide provides step-by-step instructions to:

  • Define a custom resolv.conf file for DNS settings.
  • Configure K3s to use this file.
  • Restart necessary services and verify DNS resolution.

This is especially useful when:

  • K3s doesn’t inherit the correct DNS configuration from the host.
  • Virtual machines lack proper DNS entries.
  • Specific internal name servers or domain suffixes are required.

Prerequisites

  • Access to K3s nodes with root privileges.
  • Familiarity with basic Linux commands and Kubernetes CLI (kubectl).
  • A list of DNS server IP addresses and internal domain suffixes to configure.

Step-by-Step Configuration

Step 1: Create a Custom resolv.conf File

Create a dedicated DNS configuration file specifically for K3s to use, ensuring it doesn’t get overwritten by the host system.

sudo mkdir -p /etc/rancher/k3s
sudo vi /etc/rancher/k3s/resolv.conf

Add your DNS servers and search domains:

nameserver <DNS_IP_1>
nameserver <DNS_IP_2>
search <DOMAIN_1> <DOMAIN_2>
  • Replace <DNS_IP_1>, <DNS_IP_2> with your internal or external DNS server IPs.
  • Replace <DOMAIN_1>, <DOMAIN_2> with your corporate or internal domain suffixes.

Example:

nameserver 10.1.2.3
nameserver 10.1.2.4
search corp.local opsramp.internal

This file directs CoreDNS to use your specified DNS servers and search domains instead of default host settings

Step 2: Configure K3s to Use the Custom DNS File

Edit the K3s configuration file to instruct the kubelet to use the custom DNS resolver.

sudo vi /etc/rancher/k3s/config.yaml

Add or append the following:

kubelet-arg:
- "resolv-conf=/etc/rancher/k3s/resolv.conf"

This step ensures the kubelet is responsible for managing pods, will propagate the custom DNS settings to all pods running on the node.

Step 3: Restart K3s

Restart the K3s service to apply the DNS configuration changes:

sudo systemctl restart k3s

or

sudo service k3s restart

Verify the service restarted without errors:

sudo systemctl status k3s

Step 4: Restart CoreDNS

CoreDNS caches DNS configurations and needs to be restarted to pick up the new resolver config:

kubectl delete pod -n kube-system -l k8s-app=kube-dns

or using the exact pod name:

kubectl delete pod $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}') -n kube-system

Kubernetes will automatically recreate the CoreDNS pod with updated settings.

Step 5: Verify DNS Settings Inside CoreDNS Pod

To confirm CoreDNS is using the new DNS configuration:

kubectl debug -it $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}') \
  -n kube-system \
  --image=busybox:1.28 \
  --target=kube-dns \
  -- cat /etc/resolv.conf

You should see the nameserver and search entries matching your custom resolv.conf.

Step 6: Restart Application Pods

DNS search domain changes only affect new pod DNS configurations. To apply changes, restart relevant application pods:

kubectl delete pod <pod-name> -n <namespace>

Example:

kubectl delete pod nextgen-gw-0 -n opsramp

Consider rolling restarts for production workloads:

kubectl rollout restart deployment/<deployment-name> -n <namespace>

Additional Considerations

DNS Resolution for StatefulSets and DaemonSets

Pods managed by StatefulSets or DaemonSets may require specific restart procedures to avoid downtime or data loss. Plan pod restarts accordingly.

Monitoring DNS Performance

Custom DNS settings might impact query performance or availability:

  • Monitor DNS query latency and errors using Kubernetes monitoring tools.
  • Validate fallback DNS servers are reachable to prevent resolution failures.

Security Implications

  • Ensure that the configured DNS servers are trusted and secure.
  • Avoid exposing internal DNS servers to public or untrusted networks.

Best Practices

  • Use FQDNs (Fully Qualified Domain Names) where possible.
  • Keep DNS server IPs updated if your network changes.
  • Avoid editing /etc/resolv.conf directly on K3s nodes.
  • Document DNS configuration for operational continuity.
  • Test DNS resolution from inside pods after applying changes.