DriftAgent: Using a Language Model to Turn Terraform Drift into Reviewed Merge Requests

Abstract

Purpose: This paper describes the design and operation of DriftAgent, a system that detects configuration drift in Terraform-managed cloud infrastructure and prepares fixes for human review. It also reports what we learned from running the system in production at a streaming television company.

Methodology: The study used a single-site case study design to evaluate a production deployment. DriftAgent runs every weekday morning: it asks an Atlantis server to plan 187 Terraform projects in parallel, parses the plan output, and sends each drifted project, together with its source files, to a large language model in a single request. The model returns HCL changes, including Terraform import blocks for resources that exist in the cloud but not in state; the agent opens a GitLab merge request for human review and posts a daily summary to Slack. Over twelve weeks (56 runs), we measured four outcomes: the fate of each merge request (merged unchanged, merged after edits, or closed), plan sweep time, engineering time per drift event, and inference cost. Merge request outcomes were validated by two engineers who independently rated a random sample of 120 merge requests as acceptable, acceptable with edits, or unacceptable against defined criteria, and inter-rater reliability was measured with Cohen’s kappa. Engineering effort was compared with a pre-deployment timing study of 71 manually handled drift events. Data were analyzed with descriptive statistics (counts, percentages, means, medians, and 95th percentiles), edited or closed proposals were classified by their main failure reason, and the findings were synthesized by triangulating production logs, merge request histories, the independent ratings, and the timing data.

Findings: Across 56 runs the system recorded 372 drift events and opened 351 merge requests. Reviewers merged 73.5% of them without semantic changes and 18.2% after edits, and closed 8.3%. Moving plan execution from a sequential loop to Atlantis reduced the median sweep time from 52.0 to 11.4 minutes. Weekly engineering time spent on drift fell from 9.4 to 2.4 hours, at an inference cost of about US$0.08 per drifted project. The most common reasons for edited or rejected proposals were choosing the wrong kind of remediation and formatting an import identifier incorrectly.

Unique contribution to theory, practice and policy: The study gives evidence that a fixed workflow with one model call, a deterministic plan check, and mandatory human approval is a workable way to use language models in change-controlled infrastructure. For practitioners, it documents specific design choices and failure modes. For organizations writing policy on AI in operations, it supports keeping approval and apply rights with people while assigning analysis and drafting to the model.