Smart Technology and Artificial Intelligence
LLM-Orchestrated Cloud Data Architecture: Leveraging Generative AI for Automated Data Engineering and Knowledge-Driven Databases
Authors: Balaji Salem Balasundram, Riaz Ahmed Mohammed Sait and Chaitanya Kulkarni
Publishing Date: 04-06-2026
ISBN: 978-81-975670-2-5
Abstract
As enterprise data volumes grow and cloud environments become more complex, there is increasing demand for architectures that combine automation, semantic understanding, and adaptive intelligence. This paper explores how large language models (LLMs) can serve as intelligent orchestration engines within cloud-native data platforms, automating data engineering workflows and enabling knowledgedriven database paradigms. We examine LLM roles across schema inference, data transformation, metadata generation, natural language querying, and anomaly detection, and present a layered reference architecture integrating generative AI with data lakes, warehouses, and semantic layers. A multi-agent coordination model is proposed for managing concurrent tasks across distributed systems. Drawing on documented enterprise deployments and published benchmarks, our findings indicate meaningful improvements in pipeline speed, metadata completeness, and query accuracy when LLMs are embedded in the data engineering lifecycle. We also address key challenges hallucination risk, security, regulatory compliance, and cost with targeted mitigation strategies, advancing the case for responsible, production-grade generative AI in enterprise cloud platforms.
Keywords
Large Language Models; Cloud Data Architecture; Data Engineering Automation; Generative AI; Knowledge-Driven Databases; Multi-Agent Systems; Natural Language Querying; Semantic Metadata Management.
Cite as
Balaji Salem Balasundram, Riaz Ahmed Mohammed Sait and Chaitanya Kulkarni, "LLM-Orchestrated Cloud Data Architecture: Leveraging Generative AI for Automated Data Engineering and Knowledge-Driven Databases", In: Mukesh Saraswat, Sandeep Kumar, Manjunatha Sughaturu Krishnappa and Rakesh Keshava (eds), Smart Technology and Artificial Intelligence, SCRS, India, 2026, pp. 21-32. https://doi.org/10.56155/978-81-975670-2-5-3