Preparing Linux Nodes and Networking for a Bare-Metal Cluster

Pods, Deployments, Services — the mental model for the system that runs most modern infrastructure.

Core Philosophy: A Kubernetes node is a real machine with a real operating system underneath it, sitting on a network you designed. Prepare those two layers inconsistently and you get failures that appear on one node only, at intervals, for reasons that look like nothing. Uniformity is not tidiness here — it is the thing that makes the cluster diagnosable.

Every node gets the same base

Before any cluster software goes on, each machine's OS must be installed and prepared the same way. An inconsistent base produces the worst class of bug: one that reproduces on a single node, intermittently, and points at the wrong layer.

The standard preparation, in order:

Install a server OS. A server edition with no graphical desktop — lighter, and it matches how real servers run. Anything else is carrying weight you will never use.

Give each machine a clear, unique hostname. node-control, node-worker-1, node-worker-2 make a cluster enormously easier to reason about than three machines all called ubuntu. You will read these names in logs at times when reading is hard.

Set a static address. Every node needs an address that does not change. This is covered properly below, and it is the single most consequential setting on this list.

Update the system. Apply current updates through the package manager so you start from a known, patched state rather than whatever the installer image happened to contain.

A first hardening pass. Log in as a normal user rather than root, prefer SSH keys over passwords, and remove services you do not need. This is a minimum, not a security programme — but the minimum belongs here, not later.

Synchronise the clock. All nodes must agree on the time. Distributed systems rely on consistent clocks, and clocks that drift apart cause failures that are genuinely baffling: certificates that are not yet valid, tokens that expired before they were issued, logs that interleave wrongly.

sudo apt update && sudo apt upgrade -y     # start from a patched state

sudo hostnamectl set-hostname node-worker-1
hostname                                    # confirm it

timedatectl status                          # confirm time sync is active

Repeat those on every machine, identically. The repetition is the lesson.

The network is where the confusing failures come from

On-prem there is no provider network doing this for you. You design it, and networking mistakes cause the most opaque failures in the whole stack — because a broken network looks like broken software.

Your machines sit on a local network (LAN), connected by a router and usually a switch. Four things have to be right.

Static IP addresses. By default, addresses are handed out automatically and can change. A cluster node whose address changes is a node the other nodes can no longer find — and the cluster does not report this as "the address changed", it reports it as components timing out. Every node must have a fixed address. If you fix nothing else on this page, fix this.

Internal name resolution. Machines should be reachable by name, not just by number. Either run a small internal DNS service, or use each machine's hosts file to map names to addresses. Referring to nodes by raw IP everywhere works right up until an address changes, at which point it fails everywhere at once.

A deliberate addressing plan. Decide the scheme and write it down — control plane at one fixed address, workers at the next few, a reserved range for services. Addresses assigned ad hoc and never recorded are how a network becomes something nobody wants to touch.

Connectivity in and out. The cluster needs to reach the internet to pull images and updates, and your users need to reach the applications it serves. The second half of that is the load-balancing problem, which needs its own solution on-prem.

ip addr                          # this machine's current address
hostname -I                      # just the IP

sudo nano /etc/hosts
#   192.168.1.10   node-control
#   192.168.1.11   node-worker-1
#   192.168.1.12   node-worker-2

ping node-worker-1               # confirm reachability BY NAME

Setting the static address itself is done either in the router — reserving a fixed address per machine, which is usually easier to manage — or in the machine's own network configuration. The method varies by setup. The goal does not: each node's address must never change.

A note on doing this on one VM

A single VM still benefits from a fixed address and a clear hostname, and every command above applies unchanged. If you create extra VMs to practise multi-node behaviour, put them on a network where they can reach each other and give each a fixed address. The steps are the real ones; only the hardware is simulated.

Where people get this wrong

Letting node addresses change. The number one on-prem networking failure, and it presents as everything else being broken.

Default or duplicate hostnames. Two machines called ubuntu will cost you an hour eventually.

Skipping time synchronisation. Drifting clocks produce failures that are extremely hard to trace back to their cause, because the symptom is never "the clock is wrong". Confirm sync on every node.

Preparing nodes differently. Bugs that appear on one node only are almost always this. If you did it by hand on three machines, you did it three slightly different ways.

Unstable links. A cluster needs reliable connectivity. Intermittent drops cause failures that look random and are not.

What you should have now

Each node runs a prepared server OS with a clear unique hostname, a static address, current patches, a minimal hardening pass, and a synchronised clock. Every node was prepared identically. The addressing plan is written down somewhere you will find it again.

That is a foundation you can build a cluster on and, more importantly, one you can debug.

Static address: in the router, or on the machine?

Both work, and they fail differently, so it is worth choosing deliberately rather than by accident.

A DHCP reservation in the router ties a fixed address to a machine's MAC address. The machine still asks for an address on boot and still gets the same one every time. The advantage is that every reservation lives in one place, so the router's admin page becomes your addressing plan whether you meant it to or not. The failure mode is that a machine which cannot reach the router on boot gets no address at all.

Static configuration on the machine puts the address in the node's own network config. It survives a router replacement and it comes up whether or not DHCP is answering. The failure mode is that the truth is now spread across every machine, and a duplicate address is entirely possible because nothing is coordinating.

For a small cluster, reservations are usually the better trade — one place to look, and one place to change. For anything you would be unhappy to lose to a router failure, configure it on the machine and keep the plan in a file you back up.

Whichever you pick, do not mix the two for the same machine. A host configured statically with an address the router is also handing out to something else produces an intermittent, address-dependent failure that will take a long evening to find.

Firewall ports, before they bite you

A cluster's nodes talk to each other on specific ports, and an over-zealous firewall silently blocks them. The symptom is a worker that will not join, or one that joins and then goes NotReady for no visible reason.

Full firewall configuration belongs with security hardening, but be aware now that if a join fails and the network is otherwise fine, the firewall is the first thing to check. Test reachability directly rather than assuming — ping proves the machines can see each other, but it says nothing about whether a specific TCP port is open, which is the thing that actually matters here.