Abstract
Objectives: To assess the learning environment of the Saudi psychiatry board program using the Dutch Residency Educational Climate Test (D-RECT) and to explore residents’ perception of different domains of the learning environment.
Methods: This was a descriptive cross-sectional study. The D-RECT instrument was distributed to all residents at all training sites of the Saudi psychiatry training program. It is a reliable and valid instrument to measure educational environment at the postgraduate level. Mean scores are presented, t-tests, analysis of variance, and post hoc analysis were used to compare subgroups and pearson’s correlation was used to assess relationships.
Results: Seventy-eight out of 96 residents responded (81.25%), one third of them were female. Overall D-RECT score was 2.76±0.55. The supervision subscale scored 2.83±0.83, coaching and assessment scored 2.60±0.73, feedback scored 2.00±0.85, team work scored 2.81±0.86, peer collaboration scored 3.54±0.84, professional relations between consultants scored 2.71±0.95, work is adapted to residents’ competence scored 2.71±0.86, consultants’ attitudes scored 2.71±0.86, formal education scored 2.68±0.72, and patient handover subscales scored 3.25±1.06. Female residents scored significantly higher than their male counterparts and there were no statistical significant difference between years of residency. Cronbach’s alpha was 0.936.
Conclusion: Most of the learning climate domains scored poorly, which necessitates a rigorous plan for reevaluation and improvement. Furthermore, D-RECT proved to be a reliable instrument and could help in evaluation and improvement of postgraduate training programs.
The learning climate or environment in residency programs is an essential marker for post graduate medical education (PGME) quality1,2 and may play a major role in successful implementation of an effective curriculum.3-5 Furthermore, it would be good to establish regular feedback in order to improve medical education, especially in new training centers and developing nations.6 Residency’s learning climate points to the context in which residents’ learning happens as far as the setting, shared recognitions on approaches, practices, and policies.7 Healthy educational climate is helpful for effective learning,3-5 resident wellbeing,8-10 physician competency,6 professionality,11 and training satisfaction.12,13 As an acknowledgement to the role of learning environment in improving medical education quality, several tools have been developed for its evaluation at pre- and postgraduate levels. A systematic literature review, evaluated13 instruments for measuring learning environment in order to identify the most reliable and valid instrument. They found that 9 of these instruments are suitable for measuring learning climate in postgraduate medical training.14 A more recent review of educational environment instruments, including the newly invented Dutch Residency Educational Climate Test (D-RECT), concluded that D-RECT has the best items representing sociocultural aspects.15 The D-RECT Dutch Residency Educational Climate Test is a theoretically grounded instrument built through the application of a rigorous scientific method.16 It is a valid, reliable instrument with good generalizability.16,17 Despite its recent invention (2011), D-RECT has become one of the renowned and most widely used postgraduate learning environment scales.15,18 The Saudi Psychiatry Residency Training program is a 4 year training joint program between different psychiatry departments and mental hospitals.19 It was established and supervised by the Saudi Commission for health specialties (SCFHS) in 1997 in order to graduate competent, qualified, well trained psychiatrists.19 Currently, there are 3 local training committees located in Riyadh, Dammam, and Jeddah. We aim to assess the training program by using the D-RECT instrument and to explore residents’ perception of different domains of the learning environment.
Methods
Study Setting, participants, and data collection
This is a descriptive cross-sectional study where all residents at all training sites of the Saudi psychiatry training program were targeted and asked to participate in the study. A D-RECT questionnaire in addition to questions about residency level and main demographic information were distributed electronically to each resident via email. Furthermore, a paper-based questionnaire was distributed to the residents during their weekly half day activities by one of their colleagues, in order to ensure a high response rate. In order to avoid duplication of responses, a clear instruction stated “those who answered electronic questionnaire should not fill in the paper-based one” was prominently written on the questionnaire and emphasized by the resident who distributed the paper-based questionnaire. Furthermore, the electronic questionnaire responses represent less than 20% of the total responses and data were furtherly crossly rechecked for possible duplication. Data collection was carried out by assigned residents during April and May 2016. The research was approved by the IRB of the college of medicine at Al Imam Mohammad Ibn Saud Islamic University, Riyadh, Kingdom of Saudi Arabia. Responding to the questionnaire was voluntary and full anonymity of each responding resident was ensured.
D-RECT instrument
The D-RECT was developed through a sophisticated scientifically grounded method.16 At first, an intake of 40 residents from different programs and different institutes was taken. The 83 item questionnaire was constructed using this qualitative method, it was then reviewed by a group of 14 experts and modified accordingly to 75 items. It was then submitted to a Delphi panel of 38 experts and at the same time filled out by 1251 residents. The final 50 item instrument was constructed based on interpretation of response outcomes and the Delphi panel results. The questions in the final version of the instruments are organized into 11 learning climate domains: supervision, coaching and assessment, feedback, teamwork, peer collaboration, professional relations between consultants, work adaptation to residents’ competence levels, consultant’s attitude towards residents, formal education, role of the specialty tutor, and patient handover.16 Each question has a five-point Likert scale response (1=totally disagree, 2=disagree, 3=neutral, 4=agree, 5=totally agree) in addition to the option of (not applicable).16 According to the curriculum and setting of the Saudi psychiatry training program, items of ‘role of the specialty tutor’ subscale were removed as it is not applicable. Furthermore, we deleted the option of (not applicable) from the answer options since all the rest of items are applicable to the training program. The total items of the modified D-RECT became 44 with 5 answer options for each item. We used the original English-written version of the questionnaire as the training program is conducted in English language and all of the residents were fluent in English. However, a few phrases and terminologies that may have been misunderstood were translated or illustrated in Arabic.
Statistical analysis
Data were analyzed using the Statistical Package for Social Sciences version 21 (SPSS, Armonk, NY, USA). We calculated the mean of the total D-RECT score and also the mean for each subscale. The following descriptive statistical data are presented: mean values, standard deviations, frequency, and percentages. T-tests, analysis of variance (ANOVA), and post hoc analysis (Bonferroni procedure) were used to compare subgroups. We used pearson’s correlation to assess the relationship between different variables. Cronbach’s Alpha was used to measure the internal consistency. Only statistically significant differences at p< 0.05 were considered.
Results
Seventy-eight out of 96 residents responded, a response rate of 81.25%. Approximately one third of the sample (27%) was female. There was an equal distribution of participants across the 3 cities. Second year residents had the highest participation level (35%) while the fourth year residents had the lowest (13%). Table 1 shows the demographic data including age and marital status of the sample. The mean of overall learning climate score was 2.76±0.55.
Demographic characteristics of all the residents responded at all training sites of the Saudi psychiatry training program. n=78
Female residents showed higher mean overall D-RECT scores than male residents with statistical significance. This was also the case for most sub-scales; all of them were statistically significantly higher for female residents except for the supervision, feedback, and peer collaboration subscales as shown in Table 2. The Jeddah training site scored the highest with a mean of 2.92±0.46 followed by Riyadh 2.82±0.62 then Dammam 2.42±0.47 with p-value=0.012. In post hoc analysis (Bonferroni procedure), the significant difference was only between Jeddah and Dammam training programs. There is no clear pattern in subscales regarding the site of training. Some subscales were higher in Riyadh and others were higher in Jeddah (Table 3). Table 3 shows training sites means and standard deviations. All the subscales showed statistically significant differences among training sites except supervision, feedback, peer collaboration, and consultants’ attitudes. There is no statistically significant difference between residency years in the overall mean D-RECT scores. This was also the case for all subscales except coaching and assessment where the p-value was 0.046. The mean score of the coaching and assessment item was highest in year one residents; it then decreased through the years of training until it reached its lowest in fourth year residents (Table 4).
Mean and standard deviation for the D-RECT subscales and gender.
Mean and standard deviation for the D-RECT subscales and city of training.
Mean and standard deviation for the D-RECT subscales and year of training.
Overall D-RECT mean score did not differ significantly among age groups (p-value=0.843). Even when we combined the last 3 groups together to make them 2 groups only (less than and more than 30 years old), there was no statistically significant differences.
Being married or not was not a significant factor that affected overall D-RECT mean score.
Among all items of the D-RECT scale, the item “We residents, as a group, make sure the day’s work gets done” from the peer collaboration subscale scored the highest. While “Observation forms are used periodically to monitor my progress” item from the feedback subscale scored the lowest. Table 5 shows D-RECT items means and standard deviations. The D-RECT scale showed very good internal consistency with cronbach’s alpha=0.936. Furthermore, all subscales showed acceptable reliability with alpha ranging between 0.705 (supervision subscale) and 0.895 (formal education subscale) as shown in Table 6.
Mean and standard deviation of each D-RECT item.
Reliability of the total D-RECT scale and subscales.
Discussion
This study aimed to evaluate the learning climate for residents in the Saudi psychiatry training program. The high cronbach’s alpha values obtained for the D-RECT scale, and also the subscales, shows that it is highly reliable and it can be used for assessing the learning environment in psychiatry training programs in Saudi Arabia. The high response rate in our study could be due to the combined way of using a paper-questionnaire and an electronic version. This also indicates the residents’ motivation to share their opinion about the learning climate of the training program. Being anonymous may also have helped increase the response rate.
A ‘good’ clinical learning climate is considered when the overall mean of D-RECT scores ≥4. Scores below 4 are considered a cause for concern.20 In our sample mean D-RECT score was 2.76, which is less than most of the previous studies using the D-RECT instrument. Unfortunately, there is no previous study on the same training program studied here. A study in the Netherlands on 451 trainees of 45 residency programs and different disciplines in 17 hospitals found a mean score of 3.71±0.33.7 In an Irish study where 407 residents participated, D-RECT scored 3.46±-0.66.18 In a study on orthopedic trainees aiming to assess burnout and quality of life in a modern educational program, the mean of D-RECT scored 3.8±0.4.20 The learning climate in gynecological oncology fellowship programs across Europe was found to have a mean score of 3.67±0.86.21
Female residents assessed the learning climate in a more positive way compared to male residents in this study. However, some subscales showed no difference. They include: supervision, feedback, and peer collaboration. This difference could be due to either female residents having special care and more privilege from the learning climate or that they overestimate their assessment. The previous studies showed no differences between the 2 genders.18,20,21 The Dammam program scored the lowest among the three training programs while Jeddah was the highest. All subscales were low in the Dammam program except peer collaboration, which was the only one that scored >3.5. The residents clearly have a belief that they are trying their best to do whatever necessary to get this training complete. They attribute the defect in the learning climate to the other parties in this process. This is not the case in the Jeddah program where the highest subscale was formal education. The program in Jeddah is the newest site which started in 2011 while Riyadh and Dammam both started in 1997. Recent accreditation of the program in Jeddah may justify its relatively high scores since SCFHS adopted higher standards for accreditation of training centers in recent years, which was not applied in the previous centers. Similar to the gender factor: supervision, feedback, and peer collaboration did not differ across training sites. In addition to these three subscales, the consultants’ attitudes subscale did not differ across centers.
Interestingly, junior and senior residents share the same view about the learning environment even though their duties are not the same. For example, junior residents (first and second year) are “first on-calls” and they stay in the hospital during their calls while senior (third and fourth year) are “second on-calls” and can stay at home. Also, junior residents have mandatory learning lectures weekly while seniors do not. This agreement between juniors and seniors may add to the reliability of their assessment.
There are a few positive aspects of the training experience in the Saudi psychiatry residency program. Residents in general have good peer collaboration. Formal education in the form of availability and continuity of educational activities are better than other aspects of the learning climate. On the other hand, there are specific elements of training which are weak and need improvement. The biggest deficiency was in getting feedback. There is no regular feedback and no implanted observational forms in the training program. The second lowest subscale was coaching and assessment. Only one item in this subscale scored above 3 which is “I am regularly asked to provide a rationale for my management decisions and actions”. The worst item in this subscale was “My consultants occasionally observe me taking a history”. Unfortunately, it seems that the presence of the consultant when the trainee is taking a history or doing a mental state examination is rare. There are regular activities where trainees interview patients in front of the consultant and other trainees (interview technique activities), however this is not the case in the daily clinical work. In general, it is well known that there is a mismatch between trainees and trainers regarding feedback in post graduate training programs. Trainers report giving frequent and adequate feedback whereas trainees often indicate that feedback is given infrequently and ineffective.22 To decrease this controversy, it is advised that the trainers clearly indicate that they will give feedback e.g. “I will give you feedback on your clinical work today”. Feedback should be given timely and it is better to be specific, narrative, and to include positive notes as well as notes on things that need to be improved.23
Study limitation
Although almost all of the learning domains of the training program were studied, however ‘role of the specialty tutor’ domain items were omitted from the D-RECT test and not included in our study as it is not applied in Saudi psychiatry training program which may limits full evaluation of the program environment and affect comparisons with other programs.
In conclusion, according to D-RECT instrument most of the assessed areas in the learning climate of the Saudi psychiatry training program are inadequate. Rigorous re-evaluation, meeting, and discussion with trainees and trainers, and revising the roles and policies in the training program are needed in order to analyze and diagnose the problem. Strict quality control measures and high standards for training site accreditation are needed. The D-RECT could help in evaluation and quality improvement of the Saudi psychiatry residency training program.
Acknowledgment
We would like to thank the residents who participated in this study. Also, we thank Dr. Mohammed A. AlShareef, Dr. Haitham K. Qurashi, Dr. Hatim Y. AlHarbi, and Dr. Deemah A. AlAteeq for helping us with data collection.
Footnotes
Disclosure. Author has no conflict of interests, and the work was not supported or funded by any drug company. This project was financially funded by King Saud University, Vice Deanship of Research Chairs, SABIC Psychological Health Applications and Research Chair, Riyadh, Kingdom of Saudi Arabia.
- Received December 13, 2016.
- Accepted March 9, 2017.
- Copyright: © Saudi Medical Journal
This is an open-access article distributed under the terms of the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
References
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.