Cost and speed
Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs
Our short summary of Expedia’s engineering write-up. The full story, with all the detail, is on their blog.
Read the full write-up on medium.com (opens in a new tab)Problem
Engineers spent hours debugging slow Spark jobs without clear clues, leading to over‑provisioned clusters and higher costs.
Solution
They built an LLM‑driven tool that reads structured Spark SQL plans, matches them against defined anti‑patterns, and returns JSON‑formatted fixes tied to specific plan nodes.
Want the architecture, code and numbers? They’re in the original post.
Read the full write-up on medium.com (opens in a new tab)
Elastic