DA-bench

Visual Benchmark for Data Analytics AI Agents

Run #109

  • Tool: sap
  • Date Tested: October 30, 2024
  • Setup Score: 33.3%
  • Unchecked Connects to Data Warehouse
  • Unchecked No Individual Upload of Files
  • Verified No SQL for Setup
  • Verified Setup Less Than Ten Minutes

See how this tool was set up for the test.


DA-bench Results for Run #109 — Percentage Test Score: 0.0% (0 / 210)

Data Querying (0 / 120)
Question Date Tested Score Video Recording

dq01
Perform an aggregation on an explicit column

October 30, 2024 0

dq02
Perform an aggregation with an explicit table but not an inferred column

October 30, 2024 0

dq03
Perform an aggregation with an implicit table and implicit column

October 30, 2024 0

dq04
Find and compare information across tables without joins

October 30, 2024 0

dq05
Work with non-literal values

October 30, 2024 0

dq06
Work with non-literal values and non-SQL data manipulation

October 30, 2024 0

dq07
Deal with common acronymns and more advanced aggregations

October 30, 2024 0

dq08
Multi-step queries

October 30, 2024 0

dq09
Aggregations with numeric predicates to filter

October 30, 2024 0

dq10
Aggregations with categorical predicates to filter

October 30, 2024 0

dq11
Schema review

October 30, 2024 0

dq12
Aggregate records that are filtered with a predicate requiring a join

October 30, 2024 0

dq13
Recognizes truly ambiguous queries.

October 30, 2024 0

dq14
Can handle boolean features

October 30, 2024 0

dq15
Handles ambiguous column names

October 30, 2024 0

dq16
Understands set operates require consideration of overlap

October 30, 2024 0

dq17
Finds relevant values inside a Column to answer questions

October 30, 2024 0

dq18
Lookup a single record by ID

October 30, 2024 0

dq19
Perform an aggregation by a different name and a second query from that

October 30, 2024 0

dq20
Perform an aggregation based on a very different question name

October 30, 2024 0

dq21
Perform a filter and an unusually-phrased aggregation in the correct order

October 30, 2024 0

dq23
Can handle typos well

October 30, 2024 0

dq24
Schema review

October 30, 2024 0

dq25
Work with non-literal values

October 30, 2024 0
Feature Engineering (0 / 40)
Question Date Tested Score Video Recording

fe1
Make a boolean indicator feature for a criteria set

October 30, 2024 0

fe2
Make a categorical feature from a criteria set

October 30, 2024 0

fe3
Minmax normalization

October 30, 2024 0

fe4
Combining two input columns

October 30, 2024 0

fe5
Sentiment

October 30, 2024 0

fe6
Phrase Identification in Text

October 30, 2024 0

fe7
Advanced NLQ

October 30, 2024 0

fe8
Advanced NLQ

October 30, 2024 0
Insight Identification (0 / 25)
Question Date Tested Score Video Recording

ii2
Compare an aggregation for two distinct subsets of data

October 30, 2024 0

ii5
Identifying basic trends on short timelines

October 30, 2024 0

ii6
Understands statistical significance

October 30, 2024 0

ii7
Understands derivitives

October 30, 2024 0

ii8
Can use NLQ feature engineering as part of an insight request

October 30, 2024 0
Learning (0 / 10)
Question Date Tested Score Video Recording

l1
Can remember the meanings of oddly-named columns

October 30, 2024 0

l2
Can remember criteria sets under a single name

October 30, 2024 0
Visualization (0 / 15)
Question Date Tested Score Video Recording

v1
Basic Charting

October 30, 2024 0

v2
Charting with two series

October 30, 2024 0

v3
Categorical Charts

October 30, 2024 0