Evaluating the Adaptability of Large Language Models for Knowledge-aware Question and Answering

Thakkar, Jay; Kolekar, Suresh; Gite, Shilpa; Pradhan, Biswajeet; Alamri, Abdullah

doi:10.2478/ijssis-2024-0021

Figures & Tables

Workflow of a user knowledge level-adaptive language model system.

Comparison of the Flesch–Kincaid grade level scores across various language models categorized by user familiarity level.

Evaluation of text complexity using the SMOG index for different language models based on user familiarity. SMOG, Simple Measure of Gobbledygook.

Gunning Fog Index Scores demonstrating text readability across models and user familiarity levels.

Flesh–Kincaid grade level

Question	Level	Chat Bison	Chat Bison 32k	Test Bison	Text Bison 32k	Gemini Pro	Gemini Pro Chat
What does this webpage contain?	None	7.67	13.23	6.62	6.62	11.44	11.68
	Basic	10.36	14.94	12.03	6.42	13.07	10.28
	Completely familiar	11.76	16.4	12.79	8.18	15.54	13.83

What exactly is this documentation covering?	None	7.4	9.53	11.85	11.73	12.41	12.2
	Basic	10.34	14.68	15.95	14.77	13.01	14.5
	Completely familiar	10.77	14.94	16.76	15.16	15.88	15.88

What are the most significant takeaways within this webpage?	None	10.46	7.44	7.11	9.06	11.95	12.17
	Basic	10.91	9.56	11.53	14.66	11.11	13.11
	Completely familiar	10.94	11.83	15.56	15.16	14.37	14.25

What is the core purpose or focus of these webpages?	None	10.4	9.8	10.8	9.42	13.81	10.86
	Basic	11.2	11.13	11.7	11.76	15.38	11.63
	Completely familiar	11.23	12.01	13.4	15.59	14.63	11.83

Who comprises the target audience for this Google Cloud Storage content?	None	13.56	10.14	9.26	8.87	12.56	12.56
	Basic	13.76	11.72	9.47	17.57	12.89	11.89
	Completely familiar	16.84	12.88	11.58	20.13	13.44	12.44

What is the primary intention this documentation is aiming to achieve?	None	10.13	7.37	12.91	14.95	12.04	14.61
	Basic	10.52	9.53	13.53	15.74	14.4	14.4
	Completely familiar	13.14	16.08	13.79	18.08	16.06	16.06

What tangible benefits can the information within these pages provide?	None	9.83	11.13	7.83	8.22	9.55	9.55
	Basic	10.04	11.32	8.13	10.84	10.25	10.25
	Completely familiar	11.13	11.96	9.55	11.19	10.65	10.65

Are practical tips, guidelines, or advice offered through this documentation?	None	7.74	13.61	12.03	13.48	12.14	12.14
	Basic	10.27	14.07	12.18	14.08	14.32	14.32
	Completely familiar	13.45	15.06	12.36	15.25	15.73	15.73

What real-world technology skills and knowledge can be attained from the diligent study of the pages?	None	5.84	14.63	9.96	9.69	11.92	11.92
	Basic	10.06	14.63	10.29	13.16	10.98	10.98
	Completely fmiliar	14.12	15.42	11.92	17.35	14.09	13.5

Who represents the primary intended readership in terms of backgrounds and use cases?	None	7.22	9.68	12.76	12.76	12.93	12.95
	Basic	15.27	11.44	14.64	13.96	13.01	13.54
	Completely familiar	16.71	14.74	15.57	16.19	14.93	14.99

SMOG

Question	Level	Chat Bison	Chat Bison 32k	Test Bison	Text Bison 32k	Gemini Pro	Gemini Pro Chat
What does this webpage contain?	None	23.5	22.5	13.02	14.55	23.9	22.5
	Basic	20.12	24.8	20.74	16.4	24.11	22.86
	Completely familiar	26.82	26.5	24.18	17.12	25.15	23.29

What exactly is this documentation covering?	None	24.56	23.33	18.31	22.5	26.51	25.1
	Basic	25.98	24.98	21.86	23.73	26.91	26.96
	Completely familiar	26.19	25.25	24.6	27.03	27.25	27.25

What are the most significant takeaways within this webpage?	None	20.89	17.12	19.54	21.06	25.25	25.25
	Basic	24.31	22.29	25.07	27.37	26.92	26.92
	Completely familiar	25.64	23.12	29.45	28.86	27.59	27.52

What is the core purpose or focus of these webpages?	None	23.08	23.12	21.27	18.67	24.69	22.77
	Basic	24.17	25.64	21.45	20.79	27.65	24.39
	Completely familiar	25.68	26.76	23.33	26.33	28.33	25.1

Who comprises the target audience for this Google Cloud Storage content?	None	21.49	20.27	21.19	21.55	20.19	20.19
	Basic	22.64	21.27	22.19	22.98	20.27	20.27
	Completely familiar	25.98	22.08	24.31	24.35	21.61	21.61

What is the primary intention this documentation is aiming to achieve?	None	20.27	16.53	25.8	25.44	25.07	24.29
	Basic	23.73	20.58	25.8	27.63	26.33	26.33
	Completely familiar	27.03	25.4	25.42	28.4	28.84	28.84

What tangible benefits can the information within these pages provide?	None	17.12	17.69	22.92	19.76	19.78	19.78
	Basic	19.03	24.81	23.01	20.27	24.25	24.25
	Completely familiar	28.52	25.74	24.69	20.89	25.95	25.95

Are practical tips, guidelines, or advice offered through this documentation?	None	20.27	18.6	20.03	20.52	24.68	24.83
	Basic	18.24	21.19	21.86	23.19	24.83	24.88
	Completely familiar	23.73	22.64	22.92	23.63	24.88	26.68

What real-world technology skills and knowledge can be attained from diligent study of the pages?	None	22.59	22.92	22.36	21.06	21.49	21.49
	Basic	23.53	24.83	23	21.19	22.67	22.67
	Completely familiar	22.59	25.74	23.5	31.12	26.33	24.88

Who represents the primary intended readership in terms of backgrounds and use cases?	None	20.27	20.08	25.07	25.46	25.8	23.8
	Basic	21.79	21.79	26.25	26.45	24.5	24.5
	Completely familiar	24.76	27.03	28.36	28.25	26.8	25.8

Gunning Fog Index

Question	Level	Chat Bison	Chat Bison 32k	Test Bison	Text Bison 32k	Gemini Pro	Gemini Pro Chat
What does this webpage contain?	None	36.32	37.31	31.6	31.6	39.52	36.13
	Basic	39.7	40.54	36.47	36.4	40.98	34.3
	Completely familiar	40.89	42.98	40.07	39.2	42.17	38.73

What exactly is this documentation covering?	None	40.27	39.33	36.55	37.98	40.59	40.59
	Basic	41.28	40.41	36.77	38.97	40.67	41.31
	Completely familiar	43.84	41.77	37.16	42.67	41.38	42.38

What are the most significant takeaways within this webpage?	None	39.55	33.2	36.3	35.53	39.43	37.83
	Basic	39.57	38.73	38.2	41.53	38	38
	Completely familiar	36.57	39.23	44.06	42.17	41.53	41.45

What is the core purpose or focus of these webpages?	None	37.26	39.59	35.74	35.85	38.75	37.06
	Basic	38.95	40.75	38.4	36.4	41.54	37.45
	Completely familiar	40.25	41.93	38.95	39.47	42.43	37.83

Who comprises the target audience for this Google Cloud Storage content?	None	34.45	38	34.17	36.8	38.95	34.95
	Basic	35.7	39.14	36.17	40.51	39.52	35.52
	Completely familiar	40	40.2	41.68	41.26	41.92	38.92

What is the primary intention this documentation is aiming to achieve?	None	36.3	34.23	38.46	39.71	39.29	40.52
	Basic	40.05	36.67	40.43	41	41.65	41.65
	Completely familiar	41.44	40.98	42.23	42.79	42.98	42.98

What tangible benefits can the information within these pages provide?	None	32.09	35.7	32.49	39.52	41.38	41.38
	Basic	34.34	37.68	37.01	35.34	39.21	39.21
	Completely familiar	43.41	40.15	38.59	37.81	37.67	37.67

Are practical tips, guidelines, or advice offered through this documentation?	None	33.73	37.87	36.22	38	39.08	39.08
	Basic	36.3	38.13	37.25	38.89	39.46	39.46
	Completely familiar	41.66	38.93	37.9	40.15	40.23	40.23

What real-world technology skills and knowledge can be attained from the diligent study of the pages?	None	38.68	39.29	37.92	37.83	37.37	37.37
	Basic	40.43	39.67	38.07	38.95	37.7	37.7
	Completely familiar	40.62	40.25	38.24	45.13	40.4	39.28

Who represents the primary intended readership in terms of backgrounds and use cases?	None	37.13	37.03	39.63	38.57	39.59	40.59
	Basic	40.36	37.33	40.01	43.18	40.54	41.54
	Completely familiar	41.64	42.67	44.88	43.56	41.11	41.91

Evaluating the Adaptability of Large Language Models for Knowledge-aware Question and Answering

Figures & Tables

Figure 1:

Figure 2:

Figure 3:

Figure 4:

Flesh–Kincaid grade level

SMOG

Gunning Fog Index

Paradigm

My account