Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation Among Residents

Hayashi, F.K. et al. (2022) "Simulation-based assessment to measure proficiency in mechanical ventilation among residents," ATS Scholar, 3(2), pp. 204–219. Available at: https://doi.org/10.34197/ats-scholar.2021-0130oc.

Summary

Mechanical ventilation (MV) management is a complex topic to master.  However, due to the shortage of trained intensivists, more and more providers are expected to manage ventilated patients without dedicated critical care training.  Numerous studies over the past two decades have consistently shown that providers feel undertrained in MV management, with a desire for structured training programs with dedicated learning objectives.  As a result, there has been an increased focus on the ideal method of MV instruction.

Dr. Ferreira along with several other providers from numerous institutions worked together to develop a simulation-based assessment to determine baseline resident competency in MV management.  The assessment involved a simulation-based assessment consisting of thirty-two questions administered to eighty second-year internal medicine residents in Brazil.  In a novel fashion, the assessment had an Objective Structured Clinical Exam (OSCE) structure, utilizing a lung and mechanical ventilator simulator.  Some residents were additionally evaluated with a multiple-choice exam both at the beginning and the end of the rotation, separately from the study.  At the beginning of the rotation, performance was lower on the simulation-based assessment than the multiple-choice exam, with the authors suggesting the need for greater practical mechanical ventilation training in residency.  We can look forward to additional studies from this group on this important topic.

Interview

CH: What was the impetus to this study?

JF: Management of mechanical ventilation is a core competency in critical care training, and yet studies show that trainees report low confidence in managing patients under mechanical ventilation and few training programs use a structured curriculum to teach mechanical ventilation.
Traditional assessments, based on multiple-choice exams are useful for verifying theoretical knowledge, but do not predict whether trainees will be able to implement knowledge and skills at the bedside. We were interested in creating an assessment of competency in mechanical ventilation that addressed not only theoretical knowledge, but also performance in simulated scenarios.

CH: What was the discussion behind including only second year Internal Medicine residents in the study, as opposed to first year residents?

JF: In our program, a 2-year internal medicine residency is a prerequisite for several subspecialties, including pulmonary and critical care fellowship.  In Year 1, there is one medical ICU rotation, and in Year 2 there are two ICU rotations: the medical ICU and the Respiratory ICU. We wanted to use the assessment to identify gaps in mechanical ventilation training acquired in the first year that could inform the curriculum for the Respiratory ICU rotation, which focuses primarily on teaching MV skills. But the assessment tool can be used in other contexts, such as at the end of a training program.

CH: First off, I think the number skills assessed in the competency list is incredibly complete. How was the content decided upon?

JF: A few years ago, when we were creating learning objectives for the ICU rotation, we used an adaptation of the competency list proposed by Goligher et al in 2012 (1). This list informed the development of a multiple-choice exam in a prior medical education project and has been used for a few years in our ICU.

For this study, we tried to include the most important items of the competency list in the simulation scenarios, and we invited a group of six experts (three pulmonary and critical care physicians and three respiratory therapists) to give feedback on content, face validity, and clarity of the scenarios using a modified Delphi process.

CH: Could you discuss the decision to avoid a post-test, considering only 42 residents were included in the MCE analysis?

JF: That is a great point. When we were designing the study, we considered pros and cons of applying the assessment at baseline, at the end of rotation, or both. The decision to do it at baseline was based on two points: first, we wanted to identify what competencies and skills were lacking for most residents at their second year of training, in order to focus on those competencies/skills in our rotation. Second, we were worried that residents would remember the scenarios if we repeated the SBA after 4 weeks, and that repeating the examination could decrease resident participation. After considering these points, we decided not to repeat the evaluation at the end of the rotation, because, for our primary aim - to develop and validate the simulated-based assessment – it was not an essential step.

CH: Finally, any future areas of research or curriculum development related to this topic?

JF: One drawback of the assessment is that it depends on the availability of the simulator and a ventilator and requires time availability from both the learner and examiner. As a second step, we are working on a project to recreate these same simulated scenarios using an online simulator, for a self-directed examination. We also developed an online, interactive course on mechanical ventilation for residents at our institution and will use this online assessment to measure competency at the end of the course. Our team has successfully completed the course and final evaluations, and we are working on data analysis to report on the results soon.

• Goligher EC, Ferguson ND, Kenny LP. Core competency in mechanical ventilation: development of educational objectives using the Delphi technique. Crit Care Med. 2012;40(10):2828-2832. https:// doi.org/10.1097/CCM.0b013e31825bc695

Blog Post Author

Christiana Hayward is a third pulmonary critical care medicine fellow at the University of California Los Angeles, where she currently serves as chief fellow.  Clinical interests include drug induced pulmonary diseases and a burgeoning interest in asthma.  Her medical education interests focus on curriculum development, specifically the optimization of trainee learning within the intensive care unit setting.

Twitter: @HeyDrHay_Ward

Article Author

Juliana Ferreira, MD, is an Associate Professor in Pulmonary and Critical Care Medicine at the University of Sao Paulo, Brazil. She currently serves as Associate Program Director for the Pulmonary and Critical Care Medicine Fellowship and research interests include mechanical ventilation, global health, and medical education. She is also the Director of the ATS Methods in Epidemiologic, Clinical, and Operations Research (MECOR) program in Latin America, whose mission is to aid the improvement of global lung health through the development of local, country, and regional lung disease research capacity in low- and middle-income countries.

Twitter: @FerreiraJu