chatgpt — Run #172 (2024-12-31)
using 4o model

Overall Score: 37.6%
(40% Scalability Score + 60% Test Score)

DA-bench Setup for Run #172 — Scalability Score: 42.9% (3 / 7)

See how this tool was set up for the test.

Setup Less Than 20 Minutes
Connects to Data Warehouse
Handles 1TB Table
Handles 10+ Tables
No Table Structure Changes
No SQL Expertise for Setup

DA-bench Results for Run #172 — Test Score: 34.0% (80 / 235)

Data Querying (37 / 125)

15 Correct Answers, 7 Hallucinations

Question	Date Tested	Overall Score	Video Recording
dq01 Perform an aggregation on an explicit column	2024-12-31	5	X
dq02 Perform an aggregation with an explicit table but not an inferred column	2024-12-31	5	X
dq03 Perform an aggregation with an implicit table and implicit column	2024-12-31	-5	X
dq04 Find and compare information across tables without joins	2024-12-31	5	X
dq05 Work with non-literal values	2024-12-31	5	X
dq06 Work with non-literal values and non-SQL data manipulation	2024-12-31	5	X
dq07 Deal with common acronymns and more advanced aggregations	2024-12-31	3	X
dq08 Multi-step queries	2024-12-31	-5	X
dq09 Aggregations with numeric predicates to filter	2024-12-31	5	X
dq10 Aggregations with categorical predicates to filter	2024-12-31	-5	X
dq11 Schema review	2024-12-31	0	X
dq12 Aggregate records that are filtered with a predicate requiring a join	2024-12-31	-5	X
dq13 Recognizes truly ambiguous queries.	2024-12-31	0	X
dq14 Can handle boolean features	2024-12-31	5	X
dq15 Handles ambiguous column names	2024-12-31	5	X
dq16 Understands set operates require consideration of overlap	2024-12-31	-5	X
dq17 Finds relevant values inside a Column to answer questions	2024-12-31	-5	X
dq18 Lookup a single record by ID	2024-12-31	5	X
dq19 Perform an aggregation by a different name and a second query from that	2024-12-31	4	X
dq20 Perform an aggregation based on a very different question name	2024-12-31	5	X
dq21 Perform a filter and an unusually-phrased aggregation in the correct order	2024-12-31	5	X
dq22 Complex query using the same table multiple times in different ways	2024-01-08	-5	X
dq23 Can handle incorrect column names well	2024-12-31	5	X
dq24 Schema review	2024-12-31	0	X
dq25 Work with non-literal values	2024-12-31	5	X

Domain Knowledge (5 / 5)

1 Correct Answer, 0 Hallucinations

Question	Date Tested	Overall Score	Video Recording
dk01 Column Relevance Determination	2024-12-31	5	X

Feature Engineering (25 / 40)

6 Correct Answers, 1 Hallucination

Question	Date Tested	Overall Score	Video Recording
fe1 Make a boolean indicator feature for a criteria set	2024-12-31	5	X
fe2 Make a categorical feature from a criteria set	2024-12-31	5	X
fe3 Minmax normalization	2024-12-31	5	X
fe4 Combining two input columns	2024-12-31	5	X
fe5 Sentiment	2024-12-31	-5	X
fe6 Phrase Identification in Text	2024-12-31	5	X
fe7 Advanced NLP	2024-12-31	5	X
fe8 Advanced NLP	2024-12-31	0	X

Insight Identification (18 / 40)

6 Correct Answers, 2 Hallucinations

Question	Date Tested	Overall Score	Video Recording
ii2 Compare an aggregation for two distinct subsets of data	2024-12-31	5	X
ii5 Identifying basic trends on short timelines	2024-12-31	5	X
ii6 Understands statistical significance	2024-12-31	5	X
ii7 Understands derivitives	2024-12-31	5	X
ii8 Can use NLP feature engineering as part of an insight request	2024-12-31	-5	X
ii10 Identifying data discrepancies given guidance	2024-12-31	4	X
ii12 Can manage planning aggregations in several tables	2024-12-31	4	X
ii15 Can align timestamps and filter a heavily joined table for sequenced data	2024-12-31	-5	X