Skip to content

Ranks Table¤

Overall results

overall

endpoint F1-score recall precison
SPARQL-LLM 0.790387 0.798214 0.799652
INFAI-ETI-AND-FRIENDS-A 0.592826 0.622568 0.592645
ADFR 0.418437 0.651944 0.347553
LIBER-AI-CLAUDE 0.342242 0.379101 0.338292
IRIS 0.254164 0.278771 0.252436
LIBER-AI-QWEN 0.240099 0.255515 0.238520
INFAI-ETI-AND-FRIENDS-C 0.227666 0.295864 0.243855
INFAI-ETI-AND-FRIENDS-B 0.195584 0.235309 0.218478

english

endpoint F1-score recall precison
SPARQL-LLM 0.785100 0.791336 0.797086
INFAI-ETI-AND-FRIENDS-A 0.628963 0.655200 0.631956
ADFR 0.427897 0.647217 0.364585
LIBER-AI-CLAUDE 0.322491 0.351062 0.325556
IRIS 0.289631 0.311784 0.291999
INFAI-ETI-AND-FRIENDS-C 0.263964 0.341816 0.276218
LIBER-AI-QWEN 0.242657 0.263858 0.238547
INFAI-ETI-AND-FRIENDS-B 0.223787 0.257738 0.241415

CK26 results

overall

endpoint F1-score/ndcg recall precison F1-score ndcg
SPARQL-LLM 0.832360 0.831655 0.838328 0.830684 1.000000
INFAI-ETI-AND-FRIENDS-A 0.632524 0.643958 0.634836 0.633378 0.634307
ADFR 0.512887 0.778874 0.417613 0.507700 0.788472
INFAI-ETI-AND-FRIENDS-C 0.452128 0.591728 0.487711 0.455333 0.253565
LIBER-AI-CLAUDE 0.335131 0.393077 0.325941 0.336232 0.348168
IRIS 0.216755 0.223242 0.223268 0.217797 0.174084
LIBER-AI-QWEN 0.205895 0.231558 0.202254 0.205667 0.339904
INFAI-ETI-AND-FRIENDS-B 0.156962 0.183213 0.156840 0.154233 0.424254

german

endpoint F1-score/ndcg recall precison F1-score ndcg
SPARQL-LLM 0.814103 0.816584 0.811731 0.810385 1.000000
INFAI-ETI-AND-FRIENDS-A 0.624160 0.627270 0.626833 0.623015 0.768615
ADFR 0.499284 0.759763 0.397879 0.489171 0.846091
INFAI-ETI-AND-FRIENDS-C 0.379224 0.499823 0.422985 0.382738 0.231106
LIBER-AI-CLAUDE 0.349699 0.419990 0.336803 0.350960 0.348168
LIBER-AI-QWEN 0.239887 0.255296 0.238459 0.238952 0.348168
IRIS 0.143166 0.154145 0.142058 0.146030 0.000000
INFAI-ETI-AND-FRIENDS-B 0.112500 0.153564 0.107584 0.114750 0.000000

english

endpoint F1-score/ndcg recall precison F1-score ndcg
SPARQL-LLM 0.853905 0.846726 0.864924 0.850983 1.000000
INFAI-ETI-AND-FRIENDS-A 0.640923 0.660646 0.642839 0.643741 0.500000
ADFR 0.531892 0.797985 0.437347 0.526229 0.730852
INFAI-ETI-AND-FRIENDS-C 0.521137 0.683633 0.552437 0.527928 0.276024
LIBER-AI-CLAUDE 0.320820 0.366165 0.315080 0.321504 0.348168
IRIS 0.289507 0.292338 0.304479 0.289565 0.348168
INFAI-ETI-AND-FRIENDS-B 0.206665 0.212863 0.206096 0.193715 0.848508
LIBER-AI-QWEN 0.174532 0.207819 0.166049 0.172381 0.331639


DB26 results

overall

endpoint F1-score recall precison
SPARQL-LLM 0.750090 0.764774 0.760977
INFAI-ETI-AND-FRIENDS-A 0.552273 0.601177 0.550454
LIBER-AI-CLAUDE 0.348252 0.365125 0.350643
ADFR 0.329174 0.525014 0.277492
IRIS 0.290531 0.334301 0.281603
LIBER-AI-QWEN 0.274531 0.279473 0.274787
INFAI-ETI-AND-FRIENDS-B 0.236936 0.287405 0.280117
INFAI-ETI-AND-FRIENDS-C 0.000000 0.000000 0.000000

english

endpoint F1-score recall precison
SPARQL-LLM 0.719217 0.735947 0.729248
INFAI-ETI-AND-FRIENDS-A 0.614184 0.649753 0.621074
ADFR 0.329565 0.496449 0.291823
LIBER-AI-CLAUDE 0.323479 0.335960 0.336032
LIBER-AI-QWEN 0.312932 0.319897 0.311046
IRIS 0.289697 0.331231 0.279518
INFAI-ETI-AND-FRIENDS-B 0.253859 0.302613 0.276733
INFAI-ETI-AND-FRIENDS-C 0.000000 0.000000 0.000000

spanish

endpoint F1-score recall precison
SPARQL-LLM 0.780964 0.793601 0.792705
INFAI-ETI-AND-FRIENDS-A 0.490363 0.552601 0.479834
LIBER-AI-CLAUDE 0.373026 0.394291 0.365254
ADFR 0.328783 0.553579 0.263162
IRIS 0.291365 0.337371 0.283688
LIBER-AI-QWEN 0.236129 0.239048 0.238528
INFAI-ETI-AND-FRIENDS-B 0.220013 0.272196 0.283501
INFAI-ETI-AND-FRIENDS-C 0.000000 0.000000 0.000000


Results News¤

2026-05-13 - All results are out!

The final results directory contains two folders for the datasets CK26 dataset and DB26 dataset. Each folder contains the results for the respective dataset according to the endpoint IDs in the CHALLENGERS.yaml.

In each subfolder you will find the following files:

  • *_answers.json - the result of the requested queries per respective dataset and endpoint in the correct folders.
  • *_responses.db - the database file of the responses per respective dataset and endpoint in the correct folders.
  • *_retries.log - the retries log when the endpoint did not respond within the timeout or returned a connection error per respective dataset and endpoint in the correct folders.
  • *_results.json - the final calculated metrics for each question, and the overall and per language averages.
  • the other files are automatically generated and used in the website pages.

2026-04-20 - Endpoint Responses

The results directory contains two folders for the datasets CK26 dataset and DB26 dataset. Each folder contains the results for the respective dataset according to the endpoint IDs in the CHALLENGERS.yaml.

In each subfolder you will find the following files:

  • *_answers.json - the result of the requested queries per respective dataset and endpoint in the correct folders.
  • *_responses.db - the database file of the responses per respective dataset and endpoint in the correct folders.
  • *_retries.log - the retries log when the endpoint did not respond within the timeout or returned a connection error per respective dataset and endpoint in the correct folders.