DA-bench

Visual Benchmark for Data Analytics AI Agents

chatgptRun #172 (2024-12-31)

using 4o model

Overall Score: 45.1%
(40% Setup Score + 60% Test Score)

DA-bench Setup for Run #172 — Setup Score: 57.1% (4 / 7)

See how this tool was set up for the test.

  • Verified Setup Less Than 20 Minutes
  • Unchecked Connects to Data Warehouse
  • Unchecked Handles 1TB Table
  • Verified No Individual Upload of Files
  • Verified No Python for Setup
  • Verified No SQL for Setup

DA-bench Results for Run #172 — Test Score: 37.0% (87 / 235)

Data Querying (37 / 125)
15 Correct Answers, 7 Hallucinations
Question Date Tested Overall Score Video Recording

dq01
Perform an aggregation on an explicit column

2024-12-31 5
Video Unavailable

dq02
Perform an aggregation with an explicit table but not an inferred column

2024-12-31 5
Video Unavailable

dq03
Perform an aggregation with an implicit table and implicit column

2024-12-31 -5
Video Unavailable

dq04
Find and compare information across tables without joins

2024-12-31 5
Video Unavailable

dq05
Work with non-literal values

2024-12-31 5
Video Unavailable

dq06
Work with non-literal values and non-SQL data manipulation

2024-12-31 5
Video Unavailable

dq07
Deal with common acronymns and more advanced aggregations

2024-12-31 3
Video Unavailable

dq08
Multi-step queries

2024-12-31 -5
Video Unavailable

dq09
Aggregations with numeric predicates to filter

2024-12-31 5
Video Unavailable

dq10
Aggregations with categorical predicates to filter

2024-12-31 -5
Video Unavailable

dq11
Schema review

2024-12-31 0
Video Unavailable

dq12
Aggregate records that are filtered with a predicate requiring a join

2024-12-31 -5
Video Unavailable

dq13
Recognizes truly ambiguous queries.

2024-12-31 0
Video Unavailable

dq14
Can handle boolean features

2024-12-31 5
Video Unavailable

dq15
Handles ambiguous column names

2024-12-31 5
Video Unavailable

dq16
Understands set operates require consideration of overlap

2024-12-31 -5
Video Unavailable

dq17
Finds relevant values inside a Column to answer questions

2024-12-31 -5
Video Unavailable

dq18
Lookup a single record by ID

2024-12-31 5
Video Unavailable

dq19
Perform an aggregation by a different name and a second query from that

2024-12-31 4
Video Unavailable

dq20
Perform an aggregation based on a very different question name

2024-12-31 5
Video Unavailable

dq21
Perform a filter and an unusually-phrased aggregation in the correct order

2024-12-31 5
Video Unavailable

dq22

2024-01-08 -5
Video Unavailable

dq23
Can handle incorrect column names well

2024-12-31 5
Video Unavailable

dq24
Schema review

2024-12-31 0
Video Unavailable

dq25
Work with non-literal values

2024-12-31 5
Video Unavailable
Domain Knowledge (5 / 5)
1 Correct Answer, 0 Hallucinations
Question Date Tested Overall Score Video Recording

dk01
Column Relevance Determination

2024-12-31 5
Video Unavailable
Feature Engineering (25 / 40)
6 Correct Answers, 1 Hallucination
Question Date Tested Overall Score Video Recording

fe1
Make a boolean indicator feature for a criteria set

2024-12-31 5
Video Unavailable

fe2
Make a categorical feature from a criteria set

2024-12-31 5
Video Unavailable

fe3
Minmax normalization

2024-12-31 5
Video Unavailable

fe4
Combining two input columns

2024-12-31 5
Video Unavailable

fe5
Sentiment

2024-12-31 -5
Video Unavailable

fe6
Phrase Identification in Text

2024-12-31 5
Video Unavailable

fe7
Advanced NLQ

2024-12-31 5
Video Unavailable

fe8
Advanced NLQ

2024-12-31 0
Video Unavailable
Insight Identification (20 / 40)
6 Correct Answers, 2 Hallucinations
Question Date Tested Overall Score Video Recording

ii2
Compare an aggregation for two distinct subsets of data

2024-12-31 5
Video Unavailable

ii5
Identifying basic trends on short timelines

2024-12-31 5
Video Unavailable

ii6
Understands statistical significance

2024-12-31 5
Video Unavailable

ii7
Understands derivitives

2024-12-31 5
Video Unavailable

ii8
Can use NLQ feature engineering as part of an insight request

2024-12-31 -5
Video Unavailable

ii10

2024-12-31 5
Video Unavailable

ii12

2024-12-31 5
Video Unavailable

ii15

2024-12-31 -5
Video Unavailable
Learning (-10 / 10)
0 Correct Answers, 2 Hallucinations
Question Date Tested Overall Score Video Recording

l1
Can remember the meanings of oddly-named columns

2024-12-31 -5
Video Unavailable

l2
Can remember criteria sets under a single name

2024-12-31 -5
Video Unavailable
Visualization (10 / 10)
2 Correct Answers, 0 Hallucinations
Question Date Tested Overall Score Video Recording

v1
Basic Charting

2024-12-31 5
Video Unavailable

v2
Charting with two series

0
Video Unavailable

v3
Categorical Charts

2024-12-31 5
Video Unavailable