What is DA-bench?
DA-bench is the “Data Analyst Benchmark” - it is a series of questions that you would expect Data Analysts to be able to solve.
This benchmark was created to help us test Unsupervised and other data tools against real-world problems, so we can understand the strengths and weaknesses of various approaches to automating analytics.
DA-bench is a visual benchmark, you can see both the score and videos of how every tool performs on every test. See some examples below:
Leaderboard
| Tool | Date Tested | Hallucinations |
Setup
Score
|
Test
Score
|
Overall
Score
|
|
|---|---|---|---|---|---|---|
|
Unsupervised
Full Agent Use your Data Warehouse |
Jan 6, 2025 | 3 | 100.0% | 46.8% | 68.1% | VIEW DETAILS |
|
Databricks Genie
Full Agent Use your Data Warehouse |
Jan 9, 2025 | 9 | 100.0% | 38.3% | 63.0% | VIEW DETAILS |
|
Databricks IQ
Co-pilot Use your Data Warehouse |
Jan 1, 2025 | 6 | 100.0% | 37.0% | 62.2% | VIEW DETAILS |
|
Julius
Full Agent Use Data In Memory |
Jan 8, 2025 | 6 | 57.1% | 63.0% | 60.6% | VIEW DETAILS |
|
MicroStrategy Auto Answers
Full Agent Use your Data Warehouse |
Dec 10, 2024 | 11 | 100.0% | 30.2% | 58.1% | VIEW DETAILS |
|
Amazon Q in QuickSight
Co-pilot Use your Data Warehouse |
Dec 16, 2024 | 4 | 100.0% | 22.3% | 53.4% | VIEW DETAILS |
|
ChatGPT Data Analyst
Full Agent Use Data In Memory |
Dec 31, 2024 | 12 | 57.1% | 37.0% | 45.1% | VIEW DETAILS |
|
Einstein for Tableau
Co-pilot Use your Data Warehouse |
Jan 2, 2025 | 7 | 100.0% | 8.1% | 44.9% | VIEW DETAILS |
|
ThoughtSpot Sage
Natural Language Search Use your Data Warehouse |
Dec 30, 2024 | 0 | 100.0% | 0.0% | 40.0% | VIEW DETAILS |
|
Qlik Sense Insight Advisor
Natural Language Search Use your Data Warehouse |
Dec 27, 2024 | 5 | 100.0% | -8.9% | 34.6% | VIEW DETAILS |
|
Snowflake Copilot
Co-pilot Use your Data Warehouse |
Jan 3, 2025 | 8 | 28.6% | 32.8% | 31.1% | VIEW DETAILS |
|
BigQuery
Co-pilot Use your Data Warehouse |
Dec 31, 2024 | 9 | 28.6% | 21.3% | 24.2% | VIEW DETAILS |
|
Snowflake Cortex Analyst
Full Agent Use Data In Memory |
Jan 7, 2025 | 5 | 0.0% | 39.6% | 23.7% | VIEW DETAILS |
|
IBM Cognos Assistant AI
Natural Language Search Use Data In Memory |
Dec 17, 2024 | 2 | 57.1% | -0.9% | 22.3% | VIEW DETAILS |
|
SAP Just Ask
Natural Language Search Use your Data Warehouse |
Jan 7, 2024 | 0 | 28.6% | 0.0% | 11.4% | VIEW DETAILS |
About
The Data Analyst Benchmark is a collection of datasets and prompts that can be used to test how automated analytics tools handle common data analyst tasks.
We use this information to help us prioritize work to improve our AI. We are making it available publicly to help other companies improve their tools and to help users evaluate which tools are relevant to their problems.
DA-bench currently tests dozens of prompts across 9 categories. Evaluation is performed by manual testing by a third-party, scores and videos of test results are displayed on dabench.com.
Test Scoring
Each test question is worth a maximum of 5 points. Deductions are made as follows:
| User must hunt for the answer in the UI | -1 point |
| Tool requests the table name from the user | -1 point |
| Tool requests the column name from the user | -1 point |
| Tool requests the column value from the user | -1 point |
| User must fix errors | -1 point per error fixed |
| Tool hallucinates an incorrect answer | -5 points |
| Tool says it cannot answer the question because the necessary column or data is not available | -5 points (this is also a hallucination) |
More Info
DA-bench is maintained by Unsupervised.
Suggestions and contributions are welcome on the DA-bench Github Repository.