AI May Improve Genetic Counseling Recommendations for Gynecologic Cancers

“ChatGPT provides accurate answers regarding HBOC syndromes but is less accurate about Lynch Syndrome,” according to Jharna M. Patel, MD and her coauthors.

Using ChatGPT to pose genetic counseling questions was found to provide accurate answers for those with gynecologic cancers, according to a presentation from the Society of Gynecologic Oncology (SGO) 2024 Annual Meeting on Women’s Cancer.

A score of 1 was assigned to all (100%) genetic counseling questions asked, indicating that these questions were comprehensive, correct, and did not contain any inaccurate information. Overall, 40 questions were asked, with 20 including genetic counseling questions and 20 including those on Lynch syndrome and BRCA associated Hereditary Breast and Ovarian Cancer (HBOC) syndrome.

“ChatGPT provides accurate answers regarding HBOC syndromes but is less accurate about Lynch Syndrome,” Jharna M. Patel, MD, a gynecologic fellow at NYU Langone Health, said during the presentation. “[There was] a low rate of inter-reviewer variability: only 7.5% of questions required additional reviewers to assign a final score.”

The authors wanted to quantify the accuracy of ChatGPT to answer commonly asked questions that may pertain to genetic testing and counseling for patients with gynecologic cancers. They wrote that ChatGPT is a growing informational source for patients, and that use of the tool represents an area of interest. Additionally, they noted genetic counseling is important for personalized management of treatment based on each patient’s genetic predisposition.

ChatGPT version 3.5 was used to ask the questions. Investigators calculated scores that had assigned responses overall and within each category. Additionally, there were some questions that required additional reviewers to resolve scoring discrepancies.

Two gynecologic oncologists scored each answer. They utilized 4 main criteria:

• A score of 1 was comprehensive and correct: the information was accurate, and an independent oncologist would have nothing to add.
• A score of 2 was correct but not comprehensive: all the information is correct, but another oncologist may have more information to add.
• A score of 3 was somewhat correct/somewhat incorrect: there is some misinformation.
• A score of 4 was completely inaccurate

Different scores were assigned based on ChatGPT queries overall. Overall, 82% of answers had a score of 1, 15% had a score of 2, and 3% had a score of 3. For scores assigned to Lynch syndrome questions, 67% had a score of 1, and 33% had a score of 2. Scores assigned to HBOC questions included 65% with a score of 1, 29% with a score of 2, and 6% with a score of 3. For this section, 3 questions were assigned to Lynch syndrome. and 17 pertained to HBOC syndrome.

The reviewer concordance rate for the genetic counseling questions was 100%. For HBOC syndrome, it was 88.2%, and for Lynch syndrome, it was 66.6%.

The study authors noted that there were some limitations to this study, which included inherent scoring bias. Additionally, fewer questions associated with Lynch syndrome, and authors noted that ChatGPT does not provide a source or citation for user reference.

Moving forward, the authors hope to evaluate the safety and accuracy of different ChatGPT models. They also want to focus on the impact of physician input to create more accurate responses. Additionally, the hope is to evaluate genetic testing uptake if ChatGPT is used as a patient resource.

Reference

Patel JM, Hermann CE, Growdon WB, Aviki E, Stasenko M. ChatGPT accurately performs genetic counseling for gynecologic cancers. Presented at the Society of Gynecologic Oncology 2024 Annual Meeting on Women’s Cancer; San Diego, CA; March 16-18, 2024.