Anatomy of a 141GB Log File

In 2024, a production Linux server that hosted several Docker services stopped doing its job. API services on the box began failing, and the failures had nothing to do with the API code. The disk was full. Tracing the usage back to its source landed on a single file: the Airbyte server container, part of a data integration platform running on the host, had written 141GB of log output into one log file.

This is a quietly common failure mode. Containers feel self-contained, but everything they write lands on the host disk, and a service that logs heavily with no rotation configured is a slow leak. Nothing looks wrong on day one. Nothing looks wrong on day thirty. Then the filesystem hits 100 percent and every service on the machine starts failing at once, including the ones that had nothing to do with the logging.

That blast radius is what made this more than a cleanup task. The same host served API traffic, so a logging oversight inside one third-party container had turned into an availability problem for the whole machine.

• Production host, live fix. The data platform and its neighbors needed to keep running while the problem was contained. Tearing the environment down for a rebuild was not on the table.
• Third-party image. The service producing the logs is not in-house code. Patching its logging internals, or waiting for an upstream release to change its behavior, was not a same-day option.
• Preserve the log data. The logs had operational value for debugging sync issues. Any fix that amounted to deleting them on a timer traded one problem for another.
• Durable, not one-off. Freeing 141GB fixes today. Without rotation in place, the same file starts growing again the moment the service resumes, and the same outage comes back on a schedule.

Option A: Docker log driver limits. Docker can cap container logs natively with size and file-count options on its logging driver, and for most runaway-log incidents that is the right first move. It did not fit here, because the driver only governs what a container writes to stdout and stderr through the Docker daemon. The runaway growth was in a file the application wrote inside the container filesystem, which the logging driver never sees. Driver options also apply when a container is created, so enforcing them means recreating the container rather than adjusting the running one.

Option B: a host-side cleanup job. A cron job on the host could reach into the container's storage and truncate or delete the file on a schedule. This reaches the right file, but it does so by depending on Docker's internal storage paths, which are an implementation detail rather than an interface, and blind truncation throws the log data away entirely. It also races against a process that is actively writing. It is the kind of fix that works until it quietly does not.

Option C: reconfigure the application's own logging. The cleanest fix in principle is for the service to stop logging unboundedly in the first place. In practice this was a third-party image whose logging behavior was not something the team controlled, and an upstream change does nothing for a disk that is filling up now.

What remained was the option that meets the file where it lives: rotate it inside the container, with standard Linux tooling, on a schedule aggressive enough to match how fast the file grows.

The fix installs the two oldest tools for this job, logrotate and cron, inside the container itself. Cron fires hourly and runs logrotate against the application's log directory. logrotate closes out the current file, starts a fresh one for the service to continue writing to, and keeps the rotated segments as recent history, pruning the oldest so total usage stays inside a fixed budget.

The hourly cadence is the deliberate part. Rotation schedules default to daily, and daily is fine for a service that logs politely. This one had already demonstrated it could produce log volume measured in gigabytes, so a day between rotations leaves a window where the disk can be badly hurt between two runs. Rotating hourly caps how much damage any single window can do, and rotation of an already-bounded file is cheap, so the aggressive schedule costs almost nothing.

Because rotation happens where the file lives, it needs no knowledge of Docker's storage internals, and because the writer keeps running throughout, the data pipeline never notices. The logs stay available for debugging as rotated history instead of being deleted.

The disk stabilized and stayed stable, the API services on the host came back and stayed up, and the log data survived as rotated history rather than being sacrificed for space. The incident itself was over quickly; the rotation strategy is what kept it from becoming a recurring one.

The honest tradeoff is that this is a patch at the point of impact, not a platform fix. Rotation configured inside a container lives and dies with that container: recreate it from the stock image and the fix must be applied again. That is acceptable for containing an incident and wrong as a permanent posture, which is why the durable version of this lesson is to treat log budgets as part of how containers get deployed, not as something bolted on after an outage.

The other thing worth being honest about: the disk should never have reached 100 percent before anyone knew. Done again, disk usage alerting with real headroom comes first, so the conversation happens at 80 percent with weeks of margin instead of during an outage. Log rotation stops a container from eating the disk. Monitoring is what stops a full disk from being a surprise.