Researchers at ByteDance have released DAComp, a benchmark created to test how well AI can manage the full analytics lifecycle โ from data processing to insight and recommendation. And they've looked at how well state-of-the art models perform on these benchmarks today.โฃ
โฃ
๐๐๐๐: AI is now an OK analyst (for less complex tasks) and a terrible strategist.โฃ
โฃ
Their paper looks at AI performance across two dimensions:โฃ
โฃ
(๐) ๐๐ก๐ โ๐ก๐๐ซ๐โ ๐๐ข๐ฆ๐๐ง๐ฌ๐ข๐จ๐ง (๐ญ๐๐๐ก๐ง๐ข๐๐๐ฅ ๐๐ฑ๐๐๐ฎ๐ญ๐ข๐จ๐ง)โฃ
โฃ๐ป This includes breaking down a business question, planning analyses, writing queries, running calculations, and producing charts.โฃ AI can handle many of these steps, but performance is uneven โ it does well on straightforward descriptive tasks, yet reliability drops when logic becomes layered or when multiple analytical steps must be sequenced correctly.โฃ
โฃ
(๐) ๐๐ก๐ โ๐ฌ๐จ๐๐ญโ ๐๐ข๐ฆ๐๐ง๐ฌ๐ข๐จ๐ง (๐๐ง๐๐ฅ๐ฒ๐ญ๐ข๐๐๐ฅ ๐ซ๐๐๐ฌ๐จ๐ง๐ข๐ง๐ & ๐ข๐ง๐ญ๐๐ซ๐ฉ๐ซ๐๐ญ๐๐ญ๐ข๐จ๐ง)โฃ
โฃ๐ง This involves interpreting results, identifying drivers, weighing explanations, and turning findings into clear recommendations.โฃ Here, AI still struggles: it can summarise outputs, but often cannot deliver grounded diagnosis or meaningful strategic direction.โฃ
โฃ
My take-away is that AI accelerates parts of the analytical workflow, but its limitations grow as tasks become more complex and interpretive, and human expertise remains essential for context, judgment, and synthesis.โฃ
โฃ
Good initiative to create a dedicated benchmark for this.โฃ
Link to paper:https://arxiv.org/pdf/2512.04324
