chatgpt — Run #157 (2024-12-12)
using 4o model

Overall Score: 48.7%
(40% Scalability Score + 60% Test Score)

DA-bench Setup for Run #157 — Scalability Score: 42.9% (3 / 7)

See how this tool was set up for the test.

Setup Less Than 20 Minutes
Connects to Data Warehouse
Handles 1TB Table
Handles 10+ Tables
No Table Structure Changes
No SQL Expertise for Setup

DA-bench Results for Run #157 — Test Score: 52.6% (113 / 215)

Data Querying (78 / 120)

19 Correct Answers, 3 Hallucinations

Question	Date Tested	Overall Score	Video Recording
dq01 Perform an aggregation on an explicit column	2024-12-12	5	X
dq02 Perform an aggregation with an explicit table but not an inferred column	2024-12-12	5	X
dq03 Perform an aggregation with an implicit table and implicit column	2024-12-12	5	X
dq04 Find and compare information across tables without joins	2024-12-12	5	X
dq05 Work with non-literal values	2024-12-12	5	X
dq06 Work with non-literal values and non-SQL data manipulation	2024-12-12	5	X
dq07 Deal with common acronymns and more advanced aggregations	2024-12-12	3	X
dq08 Multi-step queries	2024-12-12	-5	X
dq09 Aggregations with numeric predicates to filter	2024-12-12	5	X
dq10 Aggregations with categorical predicates to filter	2024-12-12	5	X
dq11 Schema review	2024-12-12	0	X
dq12 Aggregate records that are filtered with a predicate requiring a join	2024-12-12	5	X
dq13 Recognizes truly ambiguous queries.	2024-12-16	-5	X
dq14 Can handle boolean features	2024-12-12	5	X
dq15 Handles ambiguous column names	2024-12-12	-5	X
dq16 Understands set operates require consideration of overlap	2024-12-12	5	X
dq17 Finds relevant values inside a Column to answer questions	2024-12-12	5	X
dq18 Lookup a single record by ID	2024-12-12	5	X
dq19 Perform an aggregation by a different name and a second query from that	2024-12-12	5	X
dq20 Perform an aggregation based on a very different question name	2024-12-12	5	X
dq21 Perform a filter and an unusually-phrased aggregation in the correct order	2024-12-16	5	X
dq23 Can handle incorrect column names well	2024-12-12	5	X
dq24 Schema review	2024-12-12	0	X
dq25 Work with non-literal values	2024-12-12	5	X

Domain Knowledge (5 / 5)

1 Correct Answer, 0 Hallucinations

Question	Date Tested	Overall Score	Video Recording
dk01 Column Relevance Determination	2024-12-12	5	X

Feature Engineering (30 / 40)

7 Correct Answers, 1 Hallucination

Question	Date Tested	Overall Score	Video Recording
fe1 Make a boolean indicator feature for a criteria set	2024-12-13	5	X
fe2 Make a categorical feature from a criteria set	2024-12-13	5	X
fe3 Minmax normalization	2024-12-13	5	X
fe4 Combining two input columns	2024-12-13	5	X
fe5 Sentiment	2024-12-13	-5	X
fe6 Phrase Identification in Text	2024-12-13	5	X
fe7 Advanced NLP	2024-12-13	5	X
fe8 Advanced NLP	2024-12-13	5	X

Insight Identification (15 / 25)

4 Correct Answers, 1 Hallucination

Question	Date Tested	Overall Score	Video Recording
ii2 Compare an aggregation for two distinct subsets of data	2024-12-16	5	X
ii5 Identifying basic trends on short timelines	2024-12-13	5	X
ii6 Understands statistical significance	2024-12-16	5	X
ii7 Understands derivitives	2024-12-13	-5	X
ii8 Can use NLP feature engineering as part of an insight request	2024-12-16	5	X