LLM SQL Generation Benchmark Results

We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts. The results are shown below and can be compared to a human baseline.

Show metrics relative to human baseline

--

human

human

--

--

--

--

--

332.6 ms

31,006,852

759.83 MB

#1

anthropic

claude-opus-4.7

82.55

99.61

65.48

4.738

1.00

2,800.88 ms

34,437,823

811.31 MB

#2

openai

gpt-chat-latest

82.47

99.90

65.05

1.793

1.00

818.72 ms

39,159,059

941.02 MB

#3

anthropic

82.46

99.86

65.05

7.349

1.14

633.94 ms

38,050,598

949.38 MB

#4

~x-ai

81.91

99.80

64.02

13.747

1.24

2,811 ms

45,211,063

418.09 MB

#5

anthropic

claude-fable-5.1

80.69

99.86

61.53

9.401

1.06

2,656.48 ms

34,977,404

367.42 MB

#6

google

gemini-3-pro-preview

80.04

99.78

60.30

49.440

1.02

682.04 ms

46,150,625

1,055.25 MB

#7

qwen

qwen3.5-plus-02-15

79.90

99.63

60.17

67.609

1.00

1,201 ms

46,773,375

984.67 MB

#8

google

gemini-3.5-flash

79.87

99.69

60.05

9.736

1.00

1,484.88 ms

50,048,646

1,060.75 MB

#9

anthropic

claude-opus-5-fast

79.54

99.54

59.54

3.100

1.02

2,951.48 ms

36,182,829

938.35 MB

#10

moonshotai

kimi-k2-0905:exacto

79.40

99.75

59.06

4.547

1.02

1,397.327 ms

46,877,411

848.80 MB

#11

~anthropic

claude-opus-latest

79.39

99.91

58.87

7.406

1.02

2,658.08 ms

35,558,485

301.20 MB

#12

x-ai

79.37

99.02

59.71

31.499

1.04

3,613.62 ms

37,203,325

921.56 MB

#13

google

gemini-2.5-pro-preview-05-06

79.33

99.73

58.94

21.273

1.00

825.714 ms

41,502,252

947.30 MB

#14

google

gemini-3.1-pro-preview-customtools

79.26

99.03

59.48

20.360

1.02

3,833.3 ms

49,317,675

1,056.97 MB

#15

google

gemini-3.6-flash

79.16

98.84

59.48

7.875

1.00

5,507.52 ms

41,838,459

1,045.51 MB

#16

google

78.95

99.91

57.98

19.100

1.06

443.22 ms

42,878,115

826.88 MB

#17

anthropic

claude-opus-4.8

78.85

99.93

57.77

2.780

1.00

585.18 ms

34,526,743

811.92 MB

#18

google

gemini-3-flash-preview

78.76

99.18

58.35

2.406

1.00

6,401.3 ms

42,874,619

927.15 MB

#19

z-ai

78.72

99.82

57.62

50.516

1.06

2,970.88 ms

39,150,641

318.93 MB

#20

openai

gpt-4o-search-preview

78.59

99.97

57.21

3.757

1.40

525.58 ms

41,771,932

394.36 MB

#21

meta

78.57

99.92

57.22

6.670

1.08

2,934.5 ms

34,692,912

254.72 MB

#22

~google

gemini-pro-latest

78.56

99.31

57.81

11.087

1.04

3,468.7 ms

40,340,295

914.03 MB

#23

bytedance-seed

78.37

99.39

57.35

11.170

1.06

2,954.24 ms

46,000,305

964.12 MB

#24

openrouter

78.32

99.85

56.78

1.401

1.02

1,202.46 ms

72,041,510

1,141.62 MB

#25

~google

gemini-flash-latest

78.28

99.46

57.10

9.730

1.04

3,234.08 ms

32,730,132

781.68 MB

#26

anthropic

claude-3.7-sonnet:thinking

78.27

99.60

56.93

40.858

1.00

1,425.82 ms

40,046,627

947.37 MB

#27

z-ai

78.22

99.48

56.95

16.908

1.00

2,150.16 ms

45,225,460

1,036.61 MB

#28

anthropic

claude-3.7-sonnet

78.01

99.93

56.08

3.149

1.10

374.224 ms

40,099,998

824.57 MB

#29

z-ai

77.96

99.70

56.21

32.466

1.02

968.06 ms

45,783,003

1,078.88 MB

#30

deepseek

deepseek-v4-pro

77.91

99.91

55.91

20.368

1.02

2,224.64 ms

42,412,877

299.77 MB

#31

qwen

qwen3.6-35b-a3b

77.85

99.73

55.97

20.423

1.12

1,065.78 ms

50,827,717

997.11 MB

#32

moonshotai

77.84

99.92

55.77

4.265

1.06

589.22 ms

49,539,148

903.11 MB

#33

x-ai

77.78

99.75

55.81

61.602

1.00

677.06 ms

49,360,869

1,145.95 MB

#34

openai

gpt-5.6-terra-pro

77.73

99.53

55.92

9.575

1.02

2,624.06 ms

45,682,344

923.23 MB

#35

x-ai

77.53

99.92

55.14

0.896

1.02

794.26 ms

35,500,350

846.01 MB

#36

google

gemini-3.7-flash

77.48

99.38

55.57

5.771

1.14

4,475.24 ms

37,623,293

736.10 MB

#37

qwen

qwen3.5-397b-a17b

77.47

99.71

55.23

78.487

1.02

809.92 ms

45,777,471

1,046.79 MB

#38

meta

77.46

99.51

55.42

5.558

1.00

2,944.1 ms

44,070,185

969.39 MB

#39

x-ai

grok-4.20-multi-agent

77.45

99.79

55.12

16.323

1.06

998.54 ms

42,709,671

910.85 MB

#40

x-ai

77.43

99.49

55.37

17.513

1.00

2,168.32 ms

44,026,726

1,009.68 MB

#41

z-ai

77.38

99.84

54.92

32.316

1.02

2,991.12 ms

38,051,000

315.34 MB

#42

openai

77.38

99.92

54.83

9.886

1.14

448.84 ms

49,432,133

844.29 MB

#43

qwen

77.34

99.80

54.89

72.053

1.04

2,453.62 ms

38,011,304

365.27 MB

#44

google

gemini-2.5-pro-preview

77.34

98.43

56.25

20.550

1.00

2,995.771 ms

51,942,301

1,111.37 MB

#45

openai

77.32

99.69

54.95

3.188

1.00

1,898.78 ms

57,024,811

1,137.19 MB

#46

deepseek

deepseek-v3.2-exp

77.29

99.82

54.77

4.786

1.00

1,371.82 ms

42,265,330

838.05 MB

#47

xiaomi

77.20

99.67

54.74

23.199

1.06

1,395.92 ms

43,289,653

927.15 MB

#48

google

gemini-3.8-flash

77.19

99.19

55.20

23.069

1.06

3,776.58 ms

32,718,154

785.15 MB

#49

moonshotai

77.16

99.77

54.54

8.252

1.00

1,468.84 ms

46,652,241

877.99 MB

#50

openai

77.11

99.85

54.36

19.254

1.04

625.82 ms

43,602,960

960.85 MB

#51

mistralai

mistral-medium-3-5

77.10

99.71

54.50

5.209

1.02

1,331.286 ms

52,340,224

974.02 MB

#52

anthropic

claude-opus-4.5

77.07

99.88

54.26

2.907

1.02

4,677.22 ms

42,411,762

294.93 MB

#53

thinkingmachines

77.07

99.82

54.31

6.680

1.06

4,048.72 ms

46,538,689

375.01 MB

#54

moonshotai

77.06

99.46

54.67

68.072

1.00

1,175.633 ms

44,153,829

997.26 MB

#55

qwen

qwen3.6-max-preview

76.97

99.69

54.26

50.610

1.00

1,024.12 ms

40,394,423

964.23 MB

#56

qwen

76.97

99.85

54.09

14.378

1.10

3,435.22 ms

39,311,235

310.13 MB

#57

sakana

76.96

99.63

54.30

58.429

1.06

1,059.8 ms

38,309,339

990.27 MB

#58

qwen

76.95

99.50

54.40

35.404

1.00

1,217.265 ms

42,908,205

1,021.26 MB

#59

deepseek

deepseek-r1-0528

76.93

99.71

54.16

61.820

1.02

682.673 ms

42,768,724

937.48 MB

#60

nex-agi