TriageGPT for DataOps: Large Language Models as Intelligent Incident Classification and Resolution Agents in Data Pipeline Operations
DOI:
https://doi.org/10.47941/ijce.4117Keywords:
Large Language Models, LLM agents, AIOps, DataOps, Incident Classification, Autonomous RemediationAbstract
Purpose: This paper presents TriageGPT for DataOps, an architecture in which large language model agents classify data pipeline incidents, retrieve relevant context, and perform constrained autonomous remediation, in order to reduce the manual triage effort that drives incident duration and on-call fatigue.
Methodology: TriageGPT combines a retrieval-augmented classification agent that assigns alerts to a data pipeline specific incident taxonomy, a tool-use resolution agent that invokes actions from a vetted, blast-radius-tiered action library, and an audit-logged guardrail layer that gates and can automatically roll back actions, paired with confidence-based human-in-the-loop escalation. We describe the architecture, algorithms and calibration policy, map the design to safety and governance guidance, and report a prototype evaluation on a benchmark pipeline with a fourteen-category taxonomy, 2,800 labeled synthetic incidents and a twelve-week comparative deployment trial.
Findings: TriageGPT achieved 94.3 percent top-one classification accuracy and 95.1 percent factual attribution accuracy with retrieval, against 68.4 percent without it. It resolved 71.5 percent of test incidents without human intervention at an incorrect-action rate of 0.6 percent, every one of which the guardrail caught and reversed. In the deployment trial it reduced median time to resolution from 41 minutes under manual on-call practice to 9.8 minutes.
Unique Contribution to Theory, Policy and Practice: For theory, the paper treats autonomous remediation as something earned through calibrated confidence and bounded blast radius rather than assumed. For practice, it provides an architecture, algorithms and evaluation method for resolving routine pipeline incidents while deferring the rest to a human with an evidence-grounded case. For policy, it maps autonomous remediation to existing guidance, informing how organizations set guardrails, audit requirements and risk tolerance for AIOps agents.
Downloads
References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention Is All You Need," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998-6008.
[2] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, et al., "Language Models are Few-Shot Learners," in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 1877-1901.
[3] OpenAI, "GPT-4 Technical Report," arXiv:2303.08774, 2023.
[4] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training Language Models to Follow Instructions with Human Feedback," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[5] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[6] P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, "A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications," arXiv:2402.07927, 2024.
[7] Y. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models," in Proc. Int. Conf. Learning Representations (ICLR), 2023.
[8] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language Models Can Teach Themselves to Use Tools," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[9] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language Agents with Verbal Reinforcement Learning," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[10] OpenAI, "Function calling," OpenAI API Documentation. Available: https://developers.openai.com/api/docs/guides/function-calling
[11] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework," arXiv:2308.08155, 2023.
[12] LangChain, "LangChain: Building Applications with LLMs Through Composability," Official Documentation. Available: https://python.langchain.com
[13] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, S. Riedel, and D. Kiela, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459-9474.
[14] A. Singh, A. Ehtesham, S. Kumar, et al., "Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG," arXiv:2501.09136, 2025.
[15] "Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Enterprise Knowledge Management and Document Automation: A Systematic Literature Review," Applied Sciences, vol. 16, no. 1, article 368, 2026.
[16] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, "Drain: An Online Log Parsing Approach with Fixed Depth Tree," in Proc. IEEE Int. Conf. Web Services (ICWS), 2017, pp. 33-40.
[17] M. Du, F. Li, G. Zheng, and V. Srikumar, "DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning," in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2017, pp. 1285-1298.
[18] X. Zhang, Y. Xu, Q. Lin, B. Qiao, H. Zhang, Y. Dang, C. Xie, X. Yang, Q. Cheng, Z. Li, J. Chen, X. He, R. Yao, J.-G. Lou, M. Chintalapati, F. Shen, and D. Zhang, "Robust Log-Based Anomaly Detection on Unstable Log Data," in Proc. ACM Joint Meeting European Software Engineering Conf. and Symp. Foundations of Software Engineering (ESEC/FSE), 2019, pp. 807-817.
[19] V.-H. Le and H. Zhang, "Log-Based Anomaly Detection with Deep Learning: How Far Are We?," in Proc. IEEE/ACM 44th Int. Conf. Software Engineering (ICSE), 2022.
[20] N. Sun, T. Zhao, and D. Pei, "HotSpot: Anomaly Localization for Additive KPIs with Multi-Dimensional Attributes," IEEE Access, vol. 6, pp. 10909-10923, 2018.
[21] L. Wu, J. Tordsson, E. Elmroth, and O. Kao, "MicroRCA: Root Cause Localization of Performance Issues in Microservices," in Proc. IEEE/IFIP Network Operations and Management Symposium (NOMS), 2020.
[22] J. Lin, P. Chen, and Z. Zheng, "Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-Service Environments," in Proc. Int. Conf. Service-Oriented Computing (ICSOC), Lecture Notes in Computer Science, vol. 11236, Springer, 2018, pp. 3-20.
[23] J. Soldani and A. Brogi, "Failure Diagnosis in Microservice Systems: A Comprehensive Survey and Analysis," ACM Transactions on Software Engineering and Methodology, 2025.
[24] Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zeng, S. Ghosh, X. Zhang, C. Zhang, Q. Lin, S. Rajmohan, D. Zhang, and T. Xie, "Automatic Root Cause Analysis via Large Language Models for Cloud Incidents," in Proc. 19th European Conf. Computer Systems (EuroSys), 2024, pp. 674-688.
[25] D. Roy, X. Zhang, R. Bhave, C. Bansal, P. Las-Casas, R. Fonseca, and S. Rajmohan, "Exploring LLM-based Agents for Root Cause Analysis," arXiv:2403.04123, 2024.
[26] J. Jiang, W. Lu, J. Deng, X. Wang, N. Zhang, X. Yu, Y. Wu, Q. Zhu, W. Chen, et al., "Xpert: Empowering Incident Management with Query Recommendations via Large Language Models," in Proc. IEEE/ACM 46th Int. Conf. Software Engineering (ICSE), 2024.
[27] P. Jin, S. Zhang, M. Ma, H. Li, Y. Kang, L. Li, Y. Liu, B. Qiao, C. Zhang, P. Zhao, S. Rajmohan, Q. Lin, and D. Zhang, "Assess and Summarize: Improve Outage Understanding with Large Language Models," arXiv:2305.18084, 2023.
[28] Y. Liu, X. Wang, et al., "OpsEval: A Comprehensive Benchmark Suite for Evaluating Large Language Models' Capability in IT Operations Domain," arXiv:2310.07637, 2023.
[29] "A Survey of AIOps in the Era of Large Language Models," ACM Computing Surveys, 2025.
[30] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang, "AgentBench: Evaluating LLMs as Agents," in Proc. Int. Conf. Learning Representations (ICLR), 2024.
[31] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs," in Proc. Int. Conf. Learning Representations (ICLR), 2024.
[32] Z. Guo, S. Cheng, Y. Wang, P. Li, and Y. Liu, "StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models," in Findings of the Association for Computational Linguistics (ACL), 2024.
[33] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?," in Proc. Int. Conf. Learning Representations (ICLR), 2024.
[34] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[35] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions," ACM Transactions on Information Systems, 2025.
[36] OpenLineage, "OpenLineage: An Open Standard for Lineage Metadata Collection," Official Documentation. Available: https://openlineage.io/
[37] Great Expectations, "GX Core: Open Source Data Quality Platform," Official Documentation. Available: https://greatexpectations.io/
[38] S. Shankar, R. Garcia, J. M. Hellerstein, and A. G. Parameswaran, "Towards Observability for Production Machine Learning Pipelines [Vision Paper]," Proceedings of the VLDB Endowment, vol. 15, no. 13, pp. 4015-4023, 2022.
[39] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing," in Proc. 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2012, pp. 15-28.
[40] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA: O'Reilly Media, 2016.
[41] PagerDuty, "The Blameless Postmortem," Incident Response Documentation. Available: https://postmortems.pagerduty.com/culture/blameless/
[42] OWASP GenAI Security Project, "OWASP Top 10 for LLM Applications 2025," OWASP Foundation, 2025. Available: https://genai.owasp.org/llm-top-10/
[43] International Organization for Standardization, ISO/IEC 42001:2023, Information Technology, Artificial Intelligence, Management System, 2023.
[44] National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile," NIST AI 600-1, Jul. 2024.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Prabhu Ravi, Rajesh Thondamanati

This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution (CC-BY) 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.