Four Reasons Your Config Edit Didn't Take

The file was overwritten by the process we restarted, a duplicate daemon crash-looping since May, and session state — after we'd edited the wrong file entirely.

Summary

Over about a week we made the same configuration change four times and it failed to take effect four times, for four unrelated reasons: we edited a file that was overridden by a profile, then a file that the running process rewrote from memory when we restarted it, then fought a duplicate daemon that had been crash-looping since May and re-writing the config from its own stale copy, and finally landed a correct change that couldn't affect anything already running. Each failure looks identical from the outside — you edit, you restart, you check, and the old value is there.

1. The file you found is not the file in force

The framework supports profiles. The base config lives at the obvious path; each profile has its own file underneath. The profile's file wins, and the base file is not merged in for the keys the profile sets.

We edited the base file. Nothing changed. Nothing warned. There are seven config files in that tree in total and no command that prints which one is actually in force.

Compounding it: the restart command takes the same profile flag. A bare gateway restart leaves the profile's gateway process completely untouched — its PID doesn't change, which is the tell if you think to look.

And when you do go looking for the PID, pgrep -f "gateway run" matches a stderr-timestamping wrapper process as well as the real one. Two matches, one of them irrelevant, and killing the wrong one accomplishes nothing.

Before editing any config in a system with profiles or layers, prove which file is in force — ideally by changing something harmless and unique and watching it appear in a log.

2. The running process holds the config in memory and writes it back

This is the one that cost the most time, because it makes editing look like it worked and then silently undoes it.

The gateway loads its config at startup, holds it, and writes it back to disk on shutdown. So the natural workflow —

edit config.yaml
gateway restart

— is: the process shuts down, writes its in-memory copy over your edit, and starts again from the file it just clobbered. Measured: a value set at 09:47 was gone by 10:15, and the file was byte-identical to the pre-edit backup. diff output empty.

The correct order is stop, edit, start, and never restart:

gateway stop                    # and wait — drain took up to ~180s
<edit the file>
gateway run --replace &
grep -n "context_length:" <the config in force>   # AFTER it's back up

That last line is not optional. Re-grep the file after the process is running again — that's the only proof the value survived, and it takes two seconds.

There's a corollary for anything machine-managed. One of our config files carries a _config_version counter and is alphabetically sorted, because it's rewritten programmatically. Hand-added comments in a file like that are lost on the next write. Values only; keep the rationale somewhere the machine doesn't own.

And the inverse case, from a different service on the same stack: its config is comment-archaeology, hundreds of lines of hard-won context. Editing it with a YAML load-and-dump round-trip deletes every comment in the file. Line-based edits only, and back it up first.

3. A duplicate daemon, crash-looping since May

The write-back explains one clobbered edit. It doesn't explain edits reverting while we watched, with the gateway stopped.

There were two daemons registered against the same config directory:

The old one starts, fails to acquire the runtime lock because the real gateway holds it, exits non-zero, and KeepAlive restarts it. Forever. The logs said so plainly, if you read the right log:

Gateway (re)started 7 times in 120s — backing off 20s to break a respawn storm
Gateway runtime lock is already held by another instance

Every one of those restarts is a process that loaded the config and will write its stale copy back on exit. Two processes each holding config in memory means last writer wins, and one of them was writing every twenty seconds.

We had previously blamed the respawn storm on the framework's own install command. That diagnosis was incomplete — the install command may have created the second job, but the collision between the two is what actually caused the damage, and removing one job fixed it.

When a config file changes without you, count the processes that can write it. A service that's been failing quietly for months is still a running program with an opinion about what's in that file.

4. A correct change that can't affect anything already running

The fourth failure is the one that isn't a bug. Having finally landed the value in the right file with the right procedure, we watched live traffic get worse — average prompt size more than doubled.

Context settings are pinned at session creation. Every running session keeps what it was born with. All three active sessions predated the restart. There is no command to force a live session to re-read config; you compress it by hand, start a new one, or wait for it to end.

Worse, the log line we were using as evidence was stale — it showed the old value with a timestamp from before the gateway process had even started. An absent new value and a lingering old value are indistinguishable in a tail.

Compare the timestamp of your evidence against the start time of the process that would produce new evidence. ps -o lstart= on the pid, against the log line's timestamp. If the line predates the process, you have learned nothing.

The checklist we use now

Before editing:

1. Which file is in force? Profiles, includes, and per-machine overrides all beat the obvious path.
2. Does the process write its config back? If yes: stop, edit, start. Never restart.
3. How many processes can write this file? Check for stale duplicate service definitions, not just the one you know about.

After editing:

4. Re-read the file once the service is back up. Not before.
5. Confirm from a log line written after the restart, verified by timestamp against process start.
6. Remember that in-flight work may have pinned the old value and will not pick up the new one.

Config edits that don't take are usually one of these four — and the on-site setup hands the machine back with notes so you're not rediscovering them.

Key takeaways

1. A layered config system rarely tells you which layer won. Ours had seven files and no command to print the effective one.
2. Edit-then-restart is destructive when the process holds config in memory and writes it back on shutdown. Stop, edit, start.
3. A crash-looping duplicate service rewrites your file every few seconds. "Failing quietly for three months" does not mean "not running."
4. Machine-managed config files eat your comments. Comment-archaeology config files die to a YAML round-trip. Know which kind you have.
5. Config changes are not retroactive to live sessions, and there is usually no command to force them.
6. Verify from a log line newer than the process that would write it. Otherwise a stale line reads exactly like a failed fix.