servbin.io home

Thoughts

How I provision and harden a new Debian 12 server: the process, the prompt, and why order matters

← All thoughtsHomeAI markdown

In the previous article I talked about how the security of AI-written work comes down to what knowledge you bring to the prompts. This is a concrete example.

Every new server I provision, whether it’s a Proxmox VM, bare-metal host, or VPS, goes through the same documented two-phase process. I used to run through it by hand. Then scripts I’d copy and adapt. Now I send an agent in over SSH and it follows the documented steps exactly. The process hasn’t changed. The execution got more consistent.

Two phases, not one

The biggest mistake in server hardening is trying to lock the door while you’re still getting in. Disable password authentication and enable a strict firewall in the same pass, something goes wrong, and you’re locked out of a machine that now refuses entry.

Two phases. Never combined.

Phase 1 is bootstrap. The server accepts your SSH key, you’ve got root or sudo access, nothing locked down yet. In this phase the agent:

  • Sets hostname and timezone
  • Runs a full dist-upgrade
  • Installs the base packages: sudo, fail2ban, unattended-upgrades, nftables, curl, git, tmux, qemu-guest-agent
  • Creates the non-root admin user, copies the authorized SSH key, sets up passwordless sudo
  • Hardens SSH: PermitRootLogin no, PasswordAuthentication no, key-only auth, keepalive settings
  • Configures fail2ban on the SSH jail with the Tailscale CGNAT range (100.64.0.0/10) and the admin home IP whitelisted
  • Enables unattended-upgrades for security patches
  • Installs Tailscale

Phase 1 ends with a prompt for the operator to manually run tailscale up with the appropriate tags for the host. That step stays manual. The auth flow needs a browser or a pre-generated auth key, and the tag assignment decides what the host can reach in the tailnet ACL.

Phase 2 is lockdown. The agent doesn’t proceed until Tailscale is joined and SSH is verified working as the non-root user. Then it applies the nftables firewall.

The firewall model

The firewall runs in a dedicated inet <host>_guard table rather than a flat ruleset. On hosts that later run Docker, this matters. Docker owns the iptables-nft nat and filter chains, and flushing the ruleset would wipe Docker’s rules. A separate named table coexists cleanly with whatever Docker is doing.

The actual rules are simple:

  • Tailscale interface (tailscale0) is accepted unconditionally. Admin access lives here once the tailnet is up.
  • WAN SSH on port 22 is accepted from the admin home IP only. Break-glass path.
  • Port 80 and 443 from WAN if the server runs a public web service. Nothing else.
  • Everything else from WAN is dropped.

The WAN SSH exception is the one people skip. Some folks remove it after Tailscale is up, thinking they don’t need WAN SSH if they’ve got Tailscale. Tailscale can break. Control plane outage. Bad ACL update that cuts you off. If your only admin path is Tailscale and Tailscale is down, you’re on provider KVM or Proxmox console, which is slower and less reliable for real work.

Keeping WAN SSH from a single known IP costs nothing. Always a second door.

Break-glass, offline

Beyond the per-host SSH exception, there’s fleet-level break-glass that’s entirely offline. Three identical encrypted USB sticks in three separate locations (home, in-laws, parents’ house). Each has a VeraCrypt vault with SSH keys, Kopia repository credentials, Tailscale API keys, and Infisical exports for every production project. Everything needed to recover any host in the fleet with no internet and no running services.

The sticks get a full refresh quarterly or immediately after any major credential rotation. Passphrase lives in Vaultwarden and one paper copy off the sticks. Sticks are labeled. Passphrase never on them.

Sounds like overhead until something goes wrong. Then it’s the only thing that matters.

Secrets stay out of the scripts

The bootstrap scripts have no credentials in them. Anything that needs a secret, Tailscale auth keys for automated provisioning, Infisical tokens for runtime injection, B2 credentials for Kopia, gets pulled at runtime from Infisical using infisical run with the appropriate project and path.

The agent prompt includes which Infisical project and path to draw from. Not the values. The script can live in the git repo, the documented prompt can be shared or reviewed, and the actual credentials stay centralized with full access logging. I wrote more about this pattern in secrets, keys, and why AI agents make this harder to ignore.

Sanitized runbook

In my fleet the full scripts live in a private repo and the agent pulls them through RAG. That doesn’t help you if you’re reading this and trying the pattern yourself.

So here’s a sanitized runbook with placeholder values instead of real hostnames, IPs, or secret paths. Paste it into the same agent session as the prompt below, or open it at /docs/debian-12-two-phase-bootstrap.md.

Phase 1 bootstrap, the operator gate before lockdown, Phase 2 nftables, recovery if something goes wrong. Same process described above.

The agent prompt

Structure of the prompt when provisioning a new server. Include the runbook in the same session (or point the agent at /docs/debian-12-two-phase-bootstrap.md):

Bootstrap a new Debian 12 server.

Role: [what this server does]
Hostname: [hostname]
Public WAN NIC: [interface name, e.g. ens18]
Admin home WAN IP: [your home IP, goes into fail2ban whitelist and break-glass SSH allow]
Tailscale tags: [e.g. tag:edge,tag:uses-infisical]
Public services (ports to allow from WAN): [e.g. 80,443 or "none"]

Follow the two-phase runbook in this session.
Phase 1: run as root over SSH. Stop before applying the firewall.
Confirm Tailscale is joined and SSH works as the non-root user before proceeding to Phase 2.
Phase 2: apply the nftables guard table for this host type.

Secrets come from Infisical project [project] at path [path].
Replace [project] and [path] with your vault location, or omit if this host needs no runtime secrets.

The agent fills in host-specific variables and runs phase 1. Pauses. I verify SSH and Tailscale. Then phase 2.

Same checklist I used to run by hand. Difference is I don’t skip the fail2ban whitelist, forget keepalive settings, or leave password auth enabled because I’m tired at midnight. The agent follows the same steps every time.

Tailscale tags and ACL

Tag assignment during tailscale up isn’t administrative boilerplate. The Tailscale ACL on my network is segmented so public-facing edge hosts can’t reach internal services (Infisical, Gitea, backups) even if compromised. A host tagged tag:edge gets traffic rules for a public-facing server. A host tagged tag:uses-infisical gets the ACL rule that lets it talk to the Infisical instance in the homelab.

Get the tags right at join time and the host has exactly the network access it needs, nothing more. Change tags after the fact and you’re doing an ACL update plus re-auth, annoying enough to be a real incentive to get it right the first time.

Why an agent for this

Manual bootstrap takes maybe twenty minutes. The agent takes about the same. I’m not doing it for speed.

I’m doing it for consistency. A manual process across a hundred servers over several years accumulates drift. One server has a slightly different SSH config from before you added keepalive settings. Another has fail2ban with different timeout values. A third has Tailscale installed but joined without the right tags. You don’t notice until something breaks and you’re wondering why this host behaves differently.

The agent follows the same documented steps on the hundredth server that it followed on the first. In my setup the canonical runbook lives in a private repo and the agent reads it through RAG. The sanitized public copy linked above is the same process without fleet-specific details.

Production bootstrap and lockdown scripts for my hosts live in a private infra-ops repo alongside nftables configs and break-glass tooling. The sanitized example runbook on this site is safe to copy and adapt. Same structure for every Debian 12 host in the fleet: Proxmox VMs, bare metal, and VPS.