New Benchmark Tests LLMs on XML Generation: Room for Improvement

Large language models (LLMs) have rapidly become indispensable for generating structured outputs, whether it's JSON, SQL, or code. But here's the thing: generating XML that meets strict industry standards, there’s still a lot of room for improvement. Enter Ishigaki-IDS-Bench, a newly released benchmark that puts these models to the test.

what's Ishigaki-IDS-Bench?

Think of Ishigaki-IDS-Bench as a yardstick for evaluating how well LLMs can generate Information Delivery Specification (IDS) XML from Building Information Modeling (BIM) data. The benchmark includes 166 expert-authored examples, expanded from 83 real-world scenarios into both Japanese and English. It doesn’t stop there. it even includes gold-standard IDS files and metadata covering input format, language, and construction domain specifics.

If you’ve ever trained a model, you know the thrill of hitting high accuracy. But here, even the top-performing model only manages 65.6% macro F1 for content agreement. To put it bluntly, just 27.7% of outputs pass the Content audit. In ML-speak, that's not exactly something to brag about.

Why Should We Care?

Let me translate from ML-speak. Generating XML that satisfies both the IDS standard and IFC vocabulary constraints is important for industries relying on precise data transfer and processing. The analogy I keep coming back to is trying to solve a puzzle where the pieces keep changing shape. And until these models can consistently generate compliant XML, their utility in real-world applications is limited.

Here’s why this matters for everyone, not just researchers. We're at a point where the capabilities of LLMs are expanding faster than our ability to evaluate them effectively. Ishigaki-IDS-Bench offers a way to critically assess their limitations and strengths. But, is it too much to ask for a model that can finally nail down XML generation?

What's Next?

The release of Ishigaki-IDS-Bench is a clear call to action. We need to improve constrained structured generation methods to better align with domain standards. And with the evaluation scripts and benchmark data up for grabs on GitHub and Hugging Face under a CC BY 4.0 license, collaboration across the ML community isn't just encouraged, it’s vital.

while we’re seeing some progress, the results are a stark reminder of the limitations LLMs currently face. It's high time we focus our compute budgets and brainpower on solving these intricate puzzles. After all, the wider applications of such technologies hinge on these important improvements.