How I was created

The two halves

This site is split deliberately into two pieces that live in completely different places:

• A public website — the page you're reading now. It's just static files (HTML, CSS, an image, a little JavaScript) served for free by GitHub Pages. No code runs on GitHub's side; the files are handed to your browser as-is.
• A private chatbot server — a Python program running on Carlo's desktop. It wraps a local language model and exposes exactly one useful HTTP endpoint, POST /chat.

The website is the front door; the chatbot is the brain. They live in separate repositories on purpose — if the model code and the prompt-engineering files were in the website repo, GitHub would happily serve them as plain-text downloads to anyone curious enough. Keeping them apart means the only thing the public can reach is one narrow API.

Nothing about me runs in the cloud. There's no OpenAI key, no hosted inference. The model weights and every generated token come off a single consumer GPU in a desktop.

The model

The brain is Qwen3-4B — a 4-billion-parameter open-source model from Alibaba's Qwen team — loaded in 4-bit quantisation (the pre-quantised unsloth/Qwen3-4B-bnb-4bit build). Storing the weights pre-quantised to 4-bit nf4 means loading skips the full-precision init buffer that would otherwise blow past the memory of a small (8 GB) card; the compute itself runs in bfloat16. The whole thing fits comfortably in a couple of gigabytes of GPU memory.

Qwen3 is a hybrid reasoning model: it can either answer immediately or first write out a private chain of thought inside a <think>…</think> block and then answer. The server decides which mode to use per message:

• Fast mode — for short greetings, "who are you?", and contact questions. Greedy decoding, capped at ~240 new tokens, no reasoning step. Snappy.
• Thinking mode — for anything substantive. Sampled decoding (temperature 0.3), up to ~1500 new tokens, with a hidden reasoning block. The reasoning is split off and not shown to you — you only get the final answer.

The retrieval pipeline

On its own, a 4B model knows nothing specific about Carlo. That's what the RAG (Retrieval-Augmented Generation) layer is for. Carlo keeps his CV and project notes as a set of markdown files in an Obsidian vault, and the system prompt is assembled from them in two tiers:

• The identity card — always included. A persona file (a decision tree for how to behave as "Carlo": how to greet, when to refuse, how technical to get), plus the bio, an experience summary, projects, skills, education, and publications. This is prepended to every turn.
• Deep-dive notes — retrieved per question. Detailed per-topic notes (WHIR, the Valida zkVM, the shielded airdrop, …) are embedded with a small all-MiniLM-L6-v2 model running on the CPU. For each question the four most relevant notes are pulled in by cosine similarity — but only if they clear a relevance floor, so an unrelated question adds nothing. Retrieved notes arrive with an explicit instruction to paraphrase a fact or two, never to recite.

That two-tier split keeps the prompt small on simple turns and only spends context on deep notes when a question actually calls for them.

The path your message travels

Everything is glued together by a Python server (server.py) built on FastAPI and Uvicorn, with Pydantic validating incoming JSON and slowapi enforcing rate limits. When you hit "Send", here's the round trip:

1. The page's JavaScript packages your message plus the recent conversation history into a JSON POST request.
2. That request crosses the public internet to a Tailscale Funnel tunnel, which terminates TLS and forwards it to the desktop. The server itself only listens on 127.0.0.1:8000 — the loopback interface — so the tunnel is the only way in.
3. The server validates the input, checks the rate limit and the daily cap, retrieves relevant notes, and builds the system prompt.
4. It decides fast-vs-thinking, then Qwen3 generates a reply on the GPU. A lock makes sure only one generation runs at a time, so two visitors can't collide on the single GPU.
5. The hidden reasoning is stripped off, the turn is written to a local log (see below), and FastAPI returns JSON — {"reply": "…"} — back through the tunnel, where the JavaScript drops the text into the chat window.

A second endpoint, GET /health, just returns {"ok": true} so the tunnel and any monitoring can check whether I'm awake.

What I remember — and what gets logged

Two different things, worth keeping straight:

• No live session memory. The server holds no per-user state between requests. Your browser sends the recent conversation along with each new message, and the server trims it to the last few turns. The "memory" of an ongoing conversation lives in your browser tab, not on the server.
• But conversations are saved. Every turn is appended to a transcript file on the desktop. Each record holds the timestamp, the requester's IP address, your message, the history sent, which notes were retrieved, and the reply — including the model's hidden reasoning. Carlo reads these back to spot where I answered badly and to improve the persona and retrieval.

That log stays on the desktop, inside the chatbot's sandboxed workspace — it isn't sent to any third party. But it's a real log of real conversations, so: don't tell me anything you wouldn't want written down.

The security model

The model itself is harmless — Qwen can't read a filesystem. The risk is the surrounding software: FastAPI, Uvicorn, PyTorch and a couple hundred transitive dependencies all become internet-reachable the moment the tunnel is on. One remote-code-execution bug in any of them, and an attacker would be running code on a daily-driver desktop. So defence happens at two layers.

Layer one — guards inside the server

These don't stop a machine compromise; they stop cost runaway and obvious abuse.

Layer two — an OS-level sandbox

The server runs as a dedicated, locked-down system user (chatbot) with no login shell, no home directory, and ownership of nothing except one scratch folder. It's launched and supervised by systemd, which applies a long list of kernel-enforced restrictions before any Python runs:

• The entire filesystem is mounted read-only except one writable workspace (/var/lib/chatbot, which also holds the conversation log) and a private /tmp.
• Read access to the project code is granted narrowly via ACLs — the chatbot user can traverse to exactly the project directory and nothing else under /home.
• ~/.ssh and ~/.gnupg are explicitly blanked out, on top of being unreadable already.
• No new privileges, zero Linux capabilities, no setuid escalation, no realtime scheduling, no new namespaces.
• Can't load kernel modules, change sysctls, read kernel logs, touch the clock or hostname, or see other processes.
• Only AF_INET, AF_INET6, and AF_UNIX sockets — no raw or packet sockets.

The upshot: even a complete RCE-class compromise of the Python process lands the attacker in a read-only box, as a user that owns nothing but a scratch directory, with no credentials and no other processes in sight.

What's left as residual risk

Two things this design doesn't fully eliminate. Prompt injection — clever messages trying to make me say something off-script — is partly handled by the persona decision tree and partly an irreducible fact of any public LLM. And a kernel exploit could in principle escape the sandbox; the airtight fix (a VM with GPU passthrough) wasn't viable because the desktop's single GPU shares an IOMMU group with the boot SSD, so it can't be isolated to a VM without faking guarantees the hardware doesn't make. Running on a dedicated GPU VPS is the available "zero-risk" upgrade if it's ever worth it.