An evaluation of orthodontic information quality regarding artificial intelligence (AI) chatbot technologies: A comparison of ChatGPT and google BARD

Arslan, Can; Kahya, Kaan; Cesur, Emre; Cakan, Derya Germec

doi:10.2478/aoj-2024-0012

Figures & Tables

An analysis of the relationship between the variables (word and reference counts) with Investigator 1’s rating scale for ChatGPT

		Investigator 1’s ChatGPT scores (Mean)	Number of words (ChatGPT) (Mean)
Number	r	0.332
of words	p	0.113
(ChatGPT)
Number of	r	0.685	0.237
references	p	<0.001*	0.266
(ChatGPT)

Comparison of the numerical rating scale, word and reference count values according to the tool used

Tool
	ChatGPT		BARD		Mann Whitney U test	p
	Mean ± SD	Median (Min–Max)	Mean ± SD	Median (Min–Max)	Mann Whitney U test	p
Investigator 1	5.17 ± 0.76	5 (4–6)	4.63 ±0.65	5 (4–6)	U = 176.500	0.014*
Investigator 2	5.17 ± 0.87	5 (3 - 6)	4.67 ± 0.76	5 (3 - 6)	U = 189.000	0.030*
Number of Words	337.04 ± 82.81	349.5 (125 - 464)	328.46 ± 89.89	321.5 (189 - 551)	t = 0.344	0.732
Number of References	2.13 ± 1.51	2.5 (0 - 6)	1.96 ± 1.76	2.5 (0 - 5)	U = 276.000	0.796

An analysis of the relationship between the variables (word and reference counts) with Investigator 2’s rating scale for ChatGPT

		Investigator 2’s ChatGPT scores (Mean)	Number of words (ChatGPT) (Mean)
Number	r	0.133
of words	^p	0.536
(ChatGPT)
Number of	r	0.708	0.237
references	^p	<0.001*	0.266
(ChatGPT)

An analysis of the relationship between the variables (word and listed reference counts) with Investigator 2’s rating scale for BARD

		Investigator 2’s BARD Scores (Mean)	Number of Words (BARD) (Mean)
Number of	r	-0.187
words (BARD)	p	0.382
Number of	r	0.691	-0.030
references	P	<0.001*	0.889
(BARD)

Descriptive statistics of the questions asked in the study

	Questions	Investigator 1 (XX) (Mean ± SD)	Investigator 2 (XX) (Mean ± SD)	Number of words (Mean ± SD)	Number of references (Mean ± SD)
1	What iss the best age to visit the orthodontist?	5 ± 0	5 ± 0	240 ± 41.01	2 ± 1.41
2	Will my braces interfere with my sport activities?	4.5 ± 0.71	4.5 ± 0.71	294 ± 48.08	0.5 ± 0.71
3	What can I eat with braces?	5 ± 0	5 ± 0	270 ± 97.58	2 ± 1.41
4	How do I care for my braces?	5.5 ± 0.71	5 ± 0	363.5 ± 94.05	3 ± 2.83
5	Can an adult do orthodontic treatment?	5 ± 1.41	5 ± 1.41	188 ± 89.1	2 ± 2.83
6	Do the braces interfere with playing an instrument?	4.5 ± 0.71	5.5 ± 0.71	381.5 ± 47.38	2.5 ± 0.71
7	How long does orthodontic treatment take?	5 ± 1.41	5 ± 1.41	285 ± 48.08	1.5 ± 2.12
8	Does orthodontic treatment hurt?	4.5 ± 0.71	4.5 ± 0.71	313.5 ± 53.03	1 ±1.41
9	How to brush your teeth with braces?	5 ± 0	5 ± 0	378.5 ± 78.49	3.5 ± 0.71
10	Do I have to wear orthodontic retainers and how long will I have to wear my retainer?	4 ± 0	3.5 ± 0.71	316.5 ± 111.02	0 ± 0
11	How often will I have to go to the orthodontist?	4 ± 0	4 ± 0	279.5 ± 38.89	0 ± 0
12	How to get orthodontic treatment without metal braces?	5 ± 1.41	4 ± 1.41	387.5 ± 53.03	4.5 ± 2.12
13	What is orthognathic surgery?	5 ± 0	5.5 ± 0.71	381 ± 86.27	3 ± 0
14	How do braces move teeth in orthodontics?	5.5 ± 0.71	5.5 ± 0.71	361.5 ± 2.12	3 ± 0
15	What are the types of braces?	5 ± 1.41	5 ± 1.41	407.5 ± 30.41	15 ± 2.12
16	What is clear aligner treatment?	5.5 ± 0.71	5.5 ± 0.71	436.5 ± 113.84	4 ± 1.41
17	What are the risks and side effects of the orthodontic treatment?	4.5 ± 0.71	4 ± 0	507.5 ± 61.52	0 ± 0
18	What should I do if a bracket comes off or an appliance breaks?	4.5 ± 0.71	5 ± 0	373.5 ± 50.2	3 ± 0
19	What should I do if I have pain or other problems with my braces?	4.5 ± 0.71	4.5 ± 0.71	364 ± 1.41	1 ± 1.41
20	Will wisdom teeth cause teeth to shift?	5.5 ± 0.71	6 ± 0	305.5 ± 82.73	2.5 ± 0.71
21	Which treatment is faster ? Clear aligners or braces?	5.5 ± 0.71	5.5 ± 0.71	289.5 ± 142.13	2 ± 0
22	Can I undergo orthognathic surgery with clear aligners?	5 ± 1.41	5 ± 1.41	252.5 ± 26.16	2 ± 2.83
23	Why do some people need extractions before orthodontic treatment?	5.5 ± 0.71	6 ± 0	327.5 ± 12.02	3 ± 0
24	Can I have orthognathic surgery without orthodontic treatment?	4.5 ± 0.71	4.5 ± 0.71	282 ± 38.18	1.5 ± 2.12

Question and answer format for ChatGPT and BARD

Answers

Question 1. What’s the best age to visit the orthodontist?

ChatGPT

The best age to visit the orthodontist can vary depending on individual circumstances, but there are generally two main stages when people seek orthodontic treatment:

Early Treatment (Phase I): This usually occurs between the ages of 7 and 10. At this stage, a child’s adult teeth are beginning to come in, and an orthodontist can identify potential issues with tooth and jaw development. Early treatment may be recommended for problems like crowding, crossbites, or severe overbites.
Adolescent/Teen Years (Phase II): This is the most common time for orthodontic treatment. Typically, it begins between the ages of 11 and 15. By this age, most of the adult teeth have erupted, and any existing dental issues are more easily identified. This is also a period when the jaw is still growing, which makes it an ideal time for more complex treatments.
Adult Treatment: Orthodontic treatment can be done at any age. Many adults seek orthodontic care to address issues that were not treated during childhood or to correct new problems that have developed over time.

It’s important to note that while there are optimal ages for certain types of treatment, there’s no strict rule. Each person’s case is unique, and the best time to visit the orthodontist ultimately depends on their specific dental situation and the recommendation of their dentist or orthodontist.Regular dental check-ups are essential, and if a child or adult shows signs of orthodontic issues, it’s a good idea to consult with an orthodontist as early as possible. They can provide personalized advice on when to start treatment and what type of treatment is needed.

Google Bard

The American Association of Orthodontists (AAO) recommends that all children have their first orthodontic evaluation by the age of 7. This is because most permanent teeth have erupted by this age, and an orthodontist can identify any potential problems with the development of the jaw and teeth. Early intervention can help to prevent more serious problems down the road, and it can also make orthodontic treatment shorter and easier. For example, if a child has a crossbite (where the upper jaw overlaps the lower jaw), early treatment can help to correct the problem before it becomes more severe. Of course, not all children will need orthodontic treatment. But having an early evaluation can give parents and orthodontists the opportunity to identify any potential problems and develop a plan to address them.It is also important to note that it is never too late to see an orthodontist. Adults can also benefit from orthodontic treatment, and there are many different types of braces and other appliances available that can be tailored to meet the needs of individual patients.If you are concerned about the alignment of your child’s teeth or jaws, or if you are an adult who is considering orthodontic treatment, schedule a consultation with an orthodontist to discuss your options.

Inter-observer reliability scores

	ICC (95% CI)	p
ChatGPT	0.739 (0.485–0.878)	<0.001*
BARD	0.784 (0.563–0.9)	<0.001*

An analysis of the relationship between the variables (word and reference counts) with Investigator 1’s rating scale for BARD

		Investigator 1’s BARD scores (Mean)	Number of words (BARD) (Mean)
Number of	r	-0.326
words (BARD)	p	0.120
Number of	r	0.621	-0.030
References	P	0.001*	0.889
(BARD)