Sadly, it wasn't until I became a Security Engineer that I truly started to understand the importance of logs. I wrote about that a while back — how a homelab without visibility is just a pile of computers you hope are fine.

What I didn't anticipate is what happens after you fix that. You end up with Graylog holding a few hundred thousand messages a day, a Prometheus instance, a UniFi controller with a zone-based firewall, a Proxmox node with seventeen guests, two Synology NASes, and a reverse proxy fronting thirty-odd vhosts. You have all the visibility in the world. And the bottleneck moves: it's no longer "do I have the data?", it's "do I have ninety minutes tonight to go find it?"

Most nights, I don't.

So over the last several months I've been wiring Claude Code into the lab — not as a chatbot I paste errors into, but as something that can actually query Graylog, read the UniFi firewall config, list LXCs on Proxmox, and SSH into the Ansible controller. This post is about how that's built, and more usefully, what it caught.

The shape of the thing

The core of it is MCP — the Model Context Protocol. The short version: it's a standard way to hand a model a set of tools it can call. Instead of me describing my firewall to Claude, Claude asks the controller directly.

I run four MCP servers, and they've settled into two distinct patterns.

Spec-driven servers — my UniFi and Proxmox ones — expose exactly two tools: search and execute. They ingest the upstream OpenAPI spec, let the model search for the endpoint it needs, then call it. The whole UniFi API surface in two tools. When Ubiquiti ships a new endpoint, I get it for free.

Curated servers — Synology and Graylog — expose typed, hand-written tools like search_photos, backup_status, aggregate_messages. More work per tool, but the model doesn't have to reason about a sprawling vendor API to answer "when did the last Hyper Backup run?"

Both live on the Docker VM, fronted by nginx under a wildcard cert:

ui.mcp.mlester.network  ->  unifimcp    (docker:8082)
pm.mcp.mlester.network  ->  proxmoxmcp  (docker:8085)
ds.mcp.mlester.network  ->  synomcp     (docker:8089)

Every one of those vhosts is LAN-scoped — allow 172.16.0.0/16; deny all — and the two write-enabled ones sit behind Keycloak, with JWT enforcement in the proxy middleware. I want to be very clear about this part: these are not on the internet. An MCP server is, by construction, a remote code execution endpoint with good manners. It belongs on the inside.

What it actually caught

Architecture diagrams are easy to write and prove nothing. Here's the real ledger.

The five-second bug in this blog

Every request to this site's homepage took exactly 5.013 seconds. Not occasionally — every single one, for months.

It returned HTTP 200 the whole time. Uptime Kuma was green. Nothing alerted, because from the outside nothing was wrong.

The tell was in a Graylog aggregation over nginx access logs: a request_time percentile that was pinned at almost exactly five seconds for one vhost and nowhere else. That "exactly" is the whole clue. Nothing organic is that precise. Five seconds is a timeout.

The cause, once we went looking in Ghost's own error log, was this: Ghost fetches its own homepage og:image — by its public URL — so it can emit og:image:width and og:image:height. My firewall had a rule allowing nginx to reach Ghost. It had no rule for the reverse. So Ghost's request to itself, out through the public hostname and back in, was silently dropped.

Not rejected. Dropped. Which is why it took five seconds instead of failing instantly.

One firewall policy later, homepage renders went from 5.013s to about 11ms.

The diagnostic signature is worth stealing: connect=0.000000s paired with the full client timeout elapsing means a silent DROP. A REJECT would have failed in milliseconds. When you see a round-number latency, stop looking at your application.

The alerting that had been dead for months

This one still bothers me.

I have ntfy wired up for alerts — Proxmox backup failures, Ansible run results, the usual. I'd built it, tested it, watched a test notification arrive, and moved on.

At some point after a firewall restructure, the Management network lost egress to the reverse proxy on 443. Every alert from Proxmox and the Ansible controller had been silently failing ever since. The scripts fired, curl hung, the script exited, nobody looked.

The thing that found it was a boring question I asked mostly out of curiosity — "has anything published to the ntfy alerts topic recently?" — and the honest answer was no, not for months.

Two changes came out of it. The publish scripts now verify the publish and report failures via syslog into Graylog. And there's now an alert on the alerting path itself. If you have a notification channel you have never seen fire on a real incident, you do not have a notification channel. You have a hope.

The Ansible run that killed its own SSH connection

I'd rolled out needrestart in automatic mode fleet-wide, so services would restart themselves after library upgrades. Sensible.

Then the weekly patch run started producing a rotating UNREACHABLE — a different host each week, never the same one twice, always recovering by the next run.

needrestart was doing exactly what I told it to. After upgrading libraries it identified stale services and restarted them — including the ssh@ unit that was carrying Ansible's own connection to that host. Ansible killed itself, mid-play, on whichever host it happened to reach at the wrong moment.

The fix is one line of needrestart config excluding the per-connection SSH units. The part I want to remember is the diagnostic: an empty SSH error means the connection was severed underneath Ansible. A timeout or a connection reset means something else entirely — in my case, an intrusion-prevention rate limiter. Same red text in the output, completely different root causes.

A quarter of a million DNS queries a day

Graylog enriches log messages with reverse DNS. The lookup adapter honours each record's TTL — and cloud providers set PTR TTLs around 300 seconds.

The arithmetic was ugly: roughly 268,000 DNS queries a day to resolve a handful of repeat offenders over and over. Setting an explicit cache TTL override took the hit rate to 99.8%.

Worth noting for anyone tuning Graylog lookup caches: the entries gauge in the UI lies. Use the hits/misses counters. And editing an adapter resets them, so measure before you touch anything.

Where it's genuinely bad

I'd be writing an advertisement, not a blog post, if I stopped there.

It confuses "I couldn't reach it" with "it's down." My controller talks to the Observability VLAN through a rate limiter that trips when you probe one host hard. The first time this happened, the conclusion offered to me was that three hosts were offline. They weren't — every one of them answered fine a minute later. A timeout is a statement about the path, never about the host. That one took a while to train out, and it's now written down permanently so it doesn't recur.

Big API responses truncate, and truncation looks exactly like "no data." This is the single most dangerous failure mode I've hit. A query comes back empty, the model reports "no results," and the actual problem is that the response blew the context window and got cut. Every one of my MCP servers now supports a fields projection, and using it is effectively mandatory.

Confidence is not calibrated to correctness. The most useful habit I've built is asking for the query, running it myself, and only then acting on the conclusion. When I proposed an alert definition and asked for it to be replayed against historical data before saving, roughly a third of them didn't match what I expected.

It will happily do the destructive thing. Write access to Proxmox and the UniFi controller is real write access. My rules: snapshot before any LXC change, read-only tokens where writes aren't needed, and nothing that touches the firewall gets applied without me reading the diff. The one time I got sloppy about that, I learned that in a zone-based firewall the zone a network belongs to is not derivable from its VLAN ID — you have to fetch the network object and look. Guessing produced a policy that would have been wrong in an interesting way.

The part that surprised me

The single highest-value piece isn't any of the MCP servers. It's persistent memory — a directory of small files, one fact each, that survive between sessions.

Not "the Ghost LXC is 106." That's in the inventory; anything can read it. The things worth writing down are the ones that cost me hours and are invisible in the config:

  • Proxmox reports 70–85% IO pressure while the disks sit at 1% utilised. The metric is wrong on this hardware. Measure with iostat -x, never alert on it.
  • libtorrent 2.0 defaults to mmap for disk I/O, which OOMs against a network share. One config line fixes it. Nothing in the error message points at it.
  • Absence alerts in Graylog can't use group_by, and count() < 1 only catches total silence. Per-source coverage needs one definition per source, each with a volume floor.

That last category is really what a homelab is, isn't it? Not the hardware. The accumulated, hard-won, extremely specific knowledge of why your particular pile of computers behaves the way it does. I'd been carrying all of it in my head, which meant re-deriving it every time I came back to a subsystem after three months away.

Writing it down for the model turned out to mostly be writing it down for me.

Would I recommend it

With caveats, yes — and the caveats are the interesting part.

Don't start with the MCP servers. Start with the logging. Claude Code was only useful in my lab because there was already a Graylog with parsed, normalised, enriched fields to query. Point a model at unstructured syslog and you get confident nonsense. All four streams in my lab normalise client IP to source_ip and actor to username, so a single query spans nginx, UniFi, Synology and Linux hosts. That normalisation work is what makes the whole thing function, and it isn't glamorous.

Keep it inside. Scope it to the LAN, put the write-enabled paths behind real auth, and think hard before every expansion of blast radius.

And read the queries. Every time.

The five seconds this blog was wasting on every page load were, in the end, found by a human noticing that a number was suspiciously round. The machine pulled the data in about four seconds. I'd been not-getting-around-to-it for months.

That's the actual trade, and I'll take it.