Content uploaded by Johannes Frey
Author content
All content in this area was uploaded by Johannes Frey on Sep 24, 2023
Content may be subject to copyright.
Benchmarking the Abilities of Large Language
Models for RDF Knowledge Graph Creation and
Comprehension: How Well Do LLMs Speak Turtle?
1,2,3∗† 1,2,3∗† 2,4 1
1,2
1Institute for Applied Informatics, Goerdelerring 9, 04109 Leipzig, Germany, https: // infai.org
2Agile Knowledge Engineering and Semantic Web (AKSW), https: // aksw.org
3Leipzig University, Institute for Informatics, Germany, https:// www.uni- leipzig.de
4eccenca GmbH, Leipzig, Germany, https:// eccenca.com
Abstract
Keywords
1. Introduction
dl4kg2023 @ ISWC: Workshop Deep Learning for Knowledge Graphs, November 6th-7th, 2023, Athen, Greece
∗
†
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
LLM-KG-Bench
LLM-KG-Bench
2. Related Work
BigBench
LLM-KG-Bench
??
BigBench
3. Benchmark Tasks
T2 TurtleErrorsStatic T3 TurtleSampleGeneration
T5 FactExtractStatic
T1 TurtleConnectionExplainStatic T4 TurtleFriendCount
static
scalable
3.1. Task T1: Find Connection in Small Turtle File
Turtle-
ConnectionExplainStatic
Prompt 1:
:Anne :Bob
PREFIX :<https://abc.def/ghi/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX ow l: <http://www.w3.org/2002/07/owl#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX vcard: <http://www.w3.org/2006/vcard/ns#>
PREFIX or g: <http://www.w3.org/ns/org#>
:anne afo af :Person ;foaf:firstName "Anne" ;foaf:surname "Miller" ;
vcard:hasAddress [avcard:Home ;vcard:country-name "UK" ] .
:bob afo af :Person ;foaf:firstName "Bob" ;foaf:surname "Tanner" ;
vcard:hasAddress [avcard:Home ;vcard:country-name "US" ] .
:wonderOrg aorg:Organization .
:researchDep aorg:OrganizationalUnit ;org:unitOf :wonderOrg ;
rdfs:label "Research Department" .
:marketingDep aorg:OrganizationalUnit ;org:unitOf :wonderOrg ;
rdfs:label "Marketing Department" .
:chiefResearchOfficer aorg:Role . :marketingManager aorg:Role .
[aorg:Membership ;org:member :anne ;org:organization :researchDep ;
org:ro le :chiefResearchOfficer ] .
[aorg:Membership ;org:member :bob ;org:organization :marketingDep ;
org:ro le :marketingManager ] .
𝑎𝑛𝑛𝑒 𝑜𝑟𝑔∶𝑚𝑒𝑚𝑏𝑒𝑟
←−−−−−−−−− 𝑏𝑛𝑜𝑑𝑒1 𝑜𝑟 𝑔∶𝑜𝑟𝑔𝑎𝑛𝑖𝑧𝑎𝑡 𝑖𝑜𝑛
−−−−−−−−−−−−−→ 𝑟𝑒𝑠𝑒𝑎𝑟 𝑐ℎ𝐷𝑒𝑝 𝑜𝑟 𝑔∶𝑢𝑛𝑖𝑡𝑂𝑓
−−−−−−−−→ 𝑤𝑜𝑛𝑑𝑒𝑟𝑂𝑟 𝑔 𝑜𝑟𝑔 ∶𝑢𝑛𝑖𝑡 𝑂𝑓
←−−−−−−−−
𝑚𝑎𝑟 𝑘𝑒𝑡 𝑖𝑛𝑔𝐷𝑒𝑝 𝑜𝑟 𝑔∶𝑜𝑟𝑔𝑎𝑛𝑖𝑧 𝑎𝑡𝑖𝑜𝑛
←−−−−−−−−−−−−− 𝑏𝑛𝑜𝑑𝑒2 𝑜𝑟 𝑔∶𝑚𝑒𝑚𝑏𝑒𝑟
−−−−−−−−−→ 𝑏𝑜𝑏
𝑎𝑛𝑛𝑒, 𝑟 𝑒𝑠𝑒𝑎𝑟𝑐ℎ𝐷 𝑒𝑝, 𝑤 𝑜𝑛𝑑𝑒𝑟 𝑂𝑟 𝑔, 𝑚𝑎𝑟 𝑘𝑒𝑡𝑖𝑛𝑔𝐷𝑒𝑝, 𝑏𝑜𝑏
3.2. Task T2: Find Errors in Small Turtle File
Prompt 2:
TurtleErrorsStatic
rdflib
3.3. Task T3: Create Sample Graphs
TurtleSampleGeneration
FOAF
Prompt 3:
𝑛
foaf:Person
foaf:knows
𝑛
𝑛
rdf:type foaf:Person
persons_relative_error
= 0
> 0
< 0
−1
3.4. Task T4: Count Links in Person Graph
Prompt 4:
TurtleFriendCount
foaf:Person
foaf:knows
foaf:Person
foaf:Person
3.5. Task T5: Create Knowledge Graph from Factsheet
Prompt 5:
FactExtractStatic
Figure 1:
4. Benchmark Study Results and Discussion
LLM-KG-Bench
ggml-vicuna-13b-1.1-q4_2.bin
ggml-model-gpt4all-falcon-q4_0.bin
T1:
Figure 2:
T2:
T3:
T4:
T5:
5. Conclusion and Future Work
Acknowledgments
References
arXiv:2303.12712
10.4230/
DAGREP.12.9.60
arXiv:2306.08302
arXiv:2307.06917
arXiv:2308.16622
arXiv:2305.04676
arXiv:2307.01128
arXiv:2305.13168
arXiv:2305.15066
arXiv:2206.04615
arXiv:2306.05685
10.5281/zenodo.
5371628
A. Online Resources
