The DNS resolver I run at home logs every hostname my devices ask for. That is thousands of rows a day and almost all of it is meaningless on its own: ib.adnxs.com, sync.crwdcntrl.net, omtrdc.net. Raw, it is not observability, it is noise with timestamps.
So the obvious move: put a language model on the GPU and have it tell me, for each hostname, which company it belongs to and what category it is. Local model, no data leaving the house, runs continuously in the background. Straightforward.
Before wiring it in I wrote an eval. Ten real domains where I knew the answer, and - almost as an afterthought, to pad the set - six domains I invented on the spot. Things like zorbaxcorp.com and a few plausible-looking subdomains hung off names that have never been registered by anyone. For those six, the only correct answer is "I don't know."
What came back
| Model | Prompt | Fabricated domains answered confidently | Real domains identified |
|---|---|---|---|
| qwen2.5:7b | naive | 4 / 6 | 8 / 10 |
| qwen2.5:7b | registrable-first | 0 / 6 | 10 / 10 |
| llama3.1:8b | naive | 6 / 6 | 10 / 10 |
| llama3.1:8b | registrable-first | 6 / 6 | 10 / 10 |
Read the last two rows the way I would have read them without the fake set: llama3.1:8b, 10 out of 10. Perfect recall, the larger and more famous model, obviously the one to ship.
It also invented a company for every single domain that does not exist. "ZorBax Corporation." "Vorple Dyne." Delivered at high confidence, with a category and a plausible business description. And on the stricter prompt it got worse rather than better: it began attributing my invented domains to real companies, handing one to Agilent Technologies and another to Google. No prompt I wrote moved that number off six.
That is the whole point. Recall measures one failure mode - missing things that are there. It cannot see the other one, and in an identification task the other one is far more damaging, because a wrong answer here does not look wrong. It looks like an answer.
The line that fixed the other model
qwen2.5:7b went from four false answers to zero, and the change was not "be more careful" or a temperature tweak. Two specific instructions did it.
First, force the judgement onto the registrable domain - zorbaxcorp.com, not telemetry-hub.zorbaxcorp.com. Ownership is a property of the registered name; everything to the left of it is decoration the owner chose.
Second, and this is the sentence that carried the result: tell the model explicitly that the words inside a hostname - telemetry, cdn, api, ads, auth - are picked by whoever registered the name and prove nothing about what the service does.
Without that line, every model I tested classified telemetry-hub.<invented>.com as telemetry, confidently. It was reading the label off the box. That is keyword matching wearing the costume of recall, and a real-inputs-only eval scores it as a pass, because on real inputs the label on the box is usually right.
Still wrong even when it passes
The winning configuration is not clean either, and this is where the eval stopped being a model-selection exercise and became an architecture decision.
For obscure ad-tech infrastructure, the model reverse-engineers company names out of the domain string. ib.adnxs.com belongs to AppNexus, now part of Xandr; the model returned "AdNexus Technologies", which is not a company. sync.crwdcntrl.net belongs to Lotame; it returned "CrowdControl Technologies". Both are the domain string, expanded into something that sounds corporate.
So the finding is more precise than good-model-bad-model:
- The category - advertising, telemetry, CDN, cloud - is reliable enough to build on.
- The company name is reliable for well-known brands and invented for everything else.
- Even correct answers arrive in inconsistent strings, so "Adobe Inc." and "Adobe Systems Incorporated" total up as two different companies unless you normalise first.
What I built instead of what I planned
I had intended a model-classifies-everything pipeline. The eval killed it, and produced something better in three parts.
Most of the traffic is now resolved by a curated table of corporate domain families, hand-maintained, that covers roughly 78% of queries. It is the only mechanism that can know adobess.com and adobelogin.com are the same company - the model has no way to determine that and will guess. The model handles the remaining 22%, where a guess with a stated category is genuinely better than a blank.
And every company row in the dashboard carries a small badge: FACT or MODEL. If a name came from the lookup table it says so. If it came from the model, it says that too, right next to the name, permanently.
That badge is my favourite thing in the project. The instinct with a confident-sounding output is to hide the seam, because a clean unified list looks more finished. But the honest interface is the one where the reader can see which claims are grounded and which are inference, and can weigh them without having to know anything about my stack. Uncertainty is not a defect to be styled away. It is information, and the design should carry it.
Twenty minutes
The fake domain list is six lines in a text file. It took twenty minutes, it was almost not written at all, and it is the reason the tool does not confidently attribute my family's network traffic to companies that do not exist.
Every other artifact in that project - the resolver, the dashboard, the classification pipeline - exists to answer a question. That one exists to check whether the answers mean anything. I now write it first, before the thing it is going to test, because if you build it afterwards you will already be attached to a result.