Understanding the value of a physician’s intuition when assessing risk factors for surgery

Key takeaways

• Researchers investigated whether a physician’s intuition, including their training and past experiences, could be used in risk prediction similar to a standardized surgical risk calculator developed by the American College of Surgeons.
• While preoperative surgeon intuition is an independent predictor of 30-day postoperative complications, the study found that its predictive power isn’t as strong when compared with the standardized risk calculator.
• Combining surgeon intuition with the risk calculator was not any more accurate than the risk calculator alone.

CHICAGO (February 23, 2023): Preoperative surgeon intuition is an independent predictor of 30-day postoperative complications; however, when compared to the standard risk calculator derived from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP^®), its predictive power isn’t as strong, according to a study published in the Journal of the American College of Surgeons (JACS).

Surgeons weigh many different factors, such as medical history and current health status, when deciding what type of surgery to perform. A physician’s intuition — training, past experiences, and their “gut” feeling about a patient –– also plays a role in the assessment.

However, even when the diagnosis is the same, there is still significant variability in physician decision-making. According to one study, patients seeking a second opinion only receive the same diagnosis from both physicians about 12% of the time.^*

“The integration of the explicit, the intangible, and experience together form what we call surgeon intuition. Surgeons with a certain level of training and experience will have relatively similar intuition in certain cases,” said senior study author Gabriel A. Brat, MD, MPH, FACS, a trauma surgeon and assistant professor of surgery at Beth Israel Deaconess Medical Center and Harvard Medical School. “However, intuition is dynamic. It depends on the characteristics of the provider. One surgeon can see one patient and believe one thing about that patient’s outcome, and another surgeon can see the same patient and predict a different outcome.” 

For the study, researchers sought to quantify the value of intuition in predicting outcomes among surgical patients. They investigated whether preoperative intuition could be used in risk prediction in a similar way that the ACS NSQIP Surgical Risk Calculator is currently used. The NSQIP Risk Calculator is a tool used to estimate patient-specific postoperative complication risks for almost all operations and includes a surgeon intuition adjustment.

“We wanted to know if it’s possible to adjust for intuition in a more precise way,” said Jayson S. Marwaha, MD, MBI, lead study author and general surgery resident at Georgetown University.

The researchers developed a new algorithm that predicts postoperative outcomes using surgeon preoperative intuition alone. To do this, researchers surveyed general surgeons between October 2021 and September 2022 at Beth Israel Deaconess Medical Center right before starting their surgery. A one-question text message asked them to predict the patient’s likelihood of having a negative outcome, specifically if the patient was lower than average risk, average risk, or higher than average risk for postoperative complications or death. In total, 216 patients were included in this analysis.

In a separate model, the research team collected NSQIP data on 9,182 patients who underwent general surgery between January 2017 and September 2022 at the medical center. They predicted patient outcomes by analyzing clinical data captured by the NSQIP Risk Calculator.

After comparing the two models, a third model was built, combining both preoperative intuition and the NSQIP Risk Calculator, to determine if this third model might outperform the other two models. 

Key findings

• Nearly half of surgeons who responded to the survey (45.4%) indicated that their patient’s risk of any complication was average, with 40.3% responding higher than average risk and 14.4% responding lower than average risk.
• Preoperative surgeon intuition was an independent predictor of postoperative complications. A model of preoperative surgeon intuition predicting complications had an area under the curve (AUC) of 0.70, where an AUC of 1.0 is perfect prediction and 0.5 is a random result.
• Surgeon intuition in predicting any complication was less accurate than the ACS NSQIP risk calculator, which had an AUC of 0.83.
• A combined model using both surgeon intuition and the NSQIP risk calculator did not do better, with an AUC of 0.83, than the NSQIP risk calculator alone.
• A subset analysis showed that the intuition of more experienced attending surgeons in predicting the outcomes of patients was more accurate than less experienced residents.

“The value of surgical intuition for preoperative prediction was not improved by including human intuition in the model and this suggests that, at least for most presurgical prediction, the information that is gathered by the NSQIP Risk Calculator is better at predicting those outcomes than the gut feeling that surgeons have when looking at patients,” Dr. Brat said.

“Human intuition takes into account a lot of information that is not available to the calculator, but it doesn’t weight it in any explicit way. We don’t have a weighting system in our head that says, ‘We know that this piece of information is more important than another by a certain amount,’ whereas the NSQIP Risk Calculator does have that explicit weighting system. So, in certain situations, it’s the case that an explicit algorithm is going to be better at prediction. The value of the clinician is to integrate information that’s not available to the risk calculator.”

“Data from the NSQIP Risk Calculator are amongst the best for predicting outcomes – demonstrably superior to claims, billing, and administrative data. However, what is done with the data is the next important step,” said Clifford Y. Ko, MD, MS, MSHS, FACS, FASCRS, Director, ACS Division of Research and Optimal Patient Care, who wasn’t involved with the study. “It will be the surgeon and patient’s responsibility together, based on the data, to decide whether to proceed with surgery, and how best to prepare for the phases of care before, during, and after surgery.”

The major limitations of the study are the small size of the dataset and the fact that most of the surgeons who participated in the research were trauma and emergency surgeons. These results might not apply to other types of surgeons or scenarios.

Coauthors are Jayson S. Marwaha, MD, MBI; Brendin R. Beaulieu-Jones, MD, MBA; Margaret Berrigan, MD; William Yuan, PhD; Stephen R. Odom II, MD, FACS; Charles H. Cook, MD, FACS; Benjamin B. Scott, MD; Alok Gupta, MD, FACS; Charles S. Parsons, MD, FACS; and Anupamaa J. Seshadri, MD.

The study was supported by the National Library of Medicine.

This article is published as an article in press on the JACS website.

Citation: Marwaha JS, Beaulieu-Jones BR, Berrigan M, et al. Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparing Surgeon Intuition and Clinical Risk Prediction, as derived from the ACS NSQIP Risk Calculator. Journal of the American College of Surgeons. DOI: 10.1097/XCS.0000000000000658.

_______________________

^*Gao J, Xiao C, Glass LM, Sun J. Dr. Agent: Clinical predictive model via mimicked second opinions. Journal of the American Medical Informatics Association: JAMIA. 2020;27(7):1084-1091.

# # #

About the American College of Surgeons
The American College of Surgeons is a scientific and educational organization of surgeons that was founded in 1913 to raise the standards of surgical practice and improve the quality of care for all surgical patients. The College is dedicated to the ethical and competent practice of surgery. Its achievements have significantly influenced the course of scientific surgery in America and have established it as an important advocate for all surgical patients. The College has more than 84,000 members and is the largest organization of surgeons in the world. “FACS” designates a surgeon is a Fellow of the American College of Surgeons.

The Journal of the American College of Surgeons (JACS) is the official scientific journal of ACS. Each month, JACS publishes peer-reviewed original contributions on all aspects of surgery, with the goal of providing its readership the highest quality rapid retrieval of information relevant to surgeons.

---

---