Strategies to improve the quality of reference networks for rare diseases
Abstract
BACKGROUND:
In 2017, 24 European Reference Networks (ERNs) have been recognized in order to improve rare diseases’ management. However, a lack of practical information persists regarding the most adequate ways to fulfil ERNs’ quality criteria.
OBJECTIVE:
Identify the most efficient strategies to improve ERNs’ quality.
METHODS:
67 strategies, related to 8 activity domains (Quality Management System, healthcare coordination, clinicians’ training, inter-ERNs’ members collaborations, clinical data management, ERNs’ external collaborations/supports, information provided to patients, and ERNs’ financing) were selected from peer-reviewed scientific publications. Their usefulness, feasibility and adequacy was evaluated by 17 ERNs-related clinicians and 10 Belgian health policy makers.
RESULTS:
Results showed that the most efficient strategies are the participation of quality controls, satisfaction surveys among patients and external physicians, diffusion of information about the hospitals’ research activities, specific trainings addressed to clinicians, exchange of medical samples between hospitals, and development of adequate clinical databases. The poor practical feasibility of 7 highly-useful strategies was also highlighted, such as major clinicians’ concerns regarding legal/ethical frameworks for the reimbursement of cross-country care and exchange of patients’ data.
CONCLUSIONS:
This study enabled identification of efficient strategies to improve and validate ERNs’ quality, expertise and functioning, as well as weaknesses that should be addressed.
Abbreviations
EQA | External Quality Assessment |
ERN | European Reference Network |
NGS | Next Generation Sequencing |
OR | Odds Ratio |
RAND/UCLA | Research And Development Corporation/University of California, Los Angeles |
1.Introduction
Hospitals have to continuously evaluate their strengths and weaknesses in order to validate their expertise, improve healthcare’s quality and fulfill the health policy requirements [1–4]. This continuous improvement is supported by national and international political authorities. For instance, in 2016, the European Commission opened a Call for application for the recognition of “European Reference Networks (ERNs)”. These networks have to be composed of at least 10 Members (healthcare professionals), across 8 European Member States [5]. Their missions are to provide highly specialized healthcare for rare or complex diseases (diagnosis, treatment, follow-up), and to share knowledge [5–7]. Twenty-four ERNs, involving more than 300 hospitals, were officially recognized in March 2017 [8]. Their selection was the continuation of a strategy that started in 2007 with the recognition and funding of a limited number of pilot-ERNs [9]. The European Commission frequently publishes recommendations in order to harmonize ERNs’ duties and quality/recognition criteria (cf. Table 1 [6, 7]).
Table 1
Operational criteria and conditions for the recognition of European Reference Networks
| Governance &Coordination | •Governance and coordination (by representatives of each hospital, a governing board &a network coordinator) |
| Specific tasks for healthcare providers of the network: | |
| ► Organization and management procedures (including procedures for cross-country care &transparency of tariffs) | |
| ► Business continuity plan | |
| ► Access to good facilities (e.g. surgery theatres, intensive care unit, emergency ward, medical laboratories, etc.) | |
| ► Capacity to communicate with relevant post-discharges services, including for cross-border communication | |
| Good practice guidelines, quality control &patient safety | •Fostering of proper diagnosis, treatment, follow-up and management of patients across the network |
| •Development and implementation of clinical guidelines and cross-border patient pathways | |
| •Design and implementation of outcomes and performance indicators | |
| •Development and maintenance of a quality, patient safety and evaluation framework | |
| Specific tasks for healthcare providers of the network: | |
| ► Documentation of the competences, experience and activities of the network | |
| ► Evidence of good clinical care and outcomes according to standards/indicators/evidence-based medicine requirements | |
| ► Quality management system (including plans for the governance and evaluation of this system) | |
| ► Patient safety program (with specific goals/procedures/standards/outcome/indicators) and a reporting/learning system for adverse events, educative activities, safe use of medications/medical procedures, information provided to patients, etc. | |
| ► Use of the best knowledge- and evidence-based health technologies and treatments | |
| ► Development and use of clinical guidelines and pathways related to the area of expertise | |
| Multidisciplinary approach | •Identification of areas and best practices for multi-disciplinary work |
| •Multidisciplinary healthcare teams | |
| •Offer and promotion of multidisciplinary advice for complex cases | |
| Expertise | •Gathering and dissemination of knowledge/best practices/expertise within and outside the Network, in particular on the different alternatives, therapeutic options and best practices. |
| •Bringing healthcare closer to patients (promote expertise &support local/regional/national healthcare providers). | |
| Contribution to research | •Identification and filling of research gaps |
| •Promotion of collaborative research within the network | |
| •Set up of shared registries to reinforce research &epidemiological surveillance | |
| Teaching &training activities | •Identification and filling of training gaps |
| •Encourage and facilitate the development of training/continuous education programs/tools for healthcare providers | |
| Specific tasks for healthcare providers of the network: | |
| ► Academic, university or specialized level training | |
| ► Necessary human, technical and structural resources | |
| ► Demonstrated research experience in the field of expertise of the network, at national and international levels | |
| Collaborations | •Exchange of expertise/information systems/e-health tools at the national and international level |
| •Set up of networking elements (e.g. communication tools) and methodologies to exchange clinical information and develop clinical guidelines/protocols, training alternatives and models/operation/coordination practices, etc. | |
| •Collaboration with other medical centers that don’t belong to the Network | |
| Specific tasks for healthcare providers of the network: | |
| ► Exchange of expertise with other healthcare providers in order to support them | |
| ► Procedures and framework for the management, safeguarding and exchange of medical data (outcomes, process indicators and patient registers for specific area of expertise) | |
| ► Fostering of the use of telemedicine and other e-health tools | |
| ► Use a standardized information and nationally or internationally recognized coding systems | |
| Patient empowerment &patient-centered care | •Empowerment and involvement of patients in order to improve the safety and quality of care |
| Specific tasks for healthcare providers of the network: | |
| ► Strategies to ensure patient-centered care and the respect of patients’ rights (e.g. informed consent; information about health, access to medical records, privacy, complain and compensations, patient’s empowerment and participation) | |
| ► Clear and transparent information about complaint procedures and the remedies/forms of redress | |
| ► Feedback on patient experience and active evaluation of patient experience. | |
| ► Personal data protection and access to medical records and clinical information | |
| ► Informed consent procedure complying with legal requirements | |
| ► Transparency (e.g. information about clinical outcomes, treatment options, the quality and safety standards) |
These criteria and conditions were initially published in the COMMISSION DELEGATED DECISION of 10 March 2014 (2014/286/EU) [22].
However, in-depth analysis reveals that some of these criteria remain vague and lack practical information in order to advise hospitals and clinicians on the most adequate ways to adapt their activities in order to fulfill the recognition criteria defined for ERN’s members and, thereby, to be able to valorize their quality and expertise [10].
In this context, we conducted a literature screening of peer-reviewed scientific publications in order to identify possible interesting strategies for the improvement of various processes within the Quality Management System, healthcare coordination, clinicians’ training, collaborations between ERNs’ members and data management within reference hospitals. Then we conducted two surveys: a first one in order to ask the opinions of experts of the field regarding the usefulness and feasibility of inventoried strategies. The second one aimed to investigate the adequacy of collaborations between ERNs and external actors, of different types of logistic supports for ERNs, of information provided to patients, and several sources of funding for the activities of ERNs’ members.
2.Methods
In a first instance, a screening of the scientific literature was performed in order to identify potentially efficient strategies. This initial step took place between December 2015 and March 2016, and was performed by a single investigator. Strategies that were retained for the second step of the study were only selected from peer-reviewed scientific papers that could demonstrate their relevance and benefits in well-defined practical applications, frameworks or models (for more details, please see the bibliographic references mentioned in Table 2).
Table 2
Sixty-seven strategies or aspects investigated during the study
| Domains of activities | Strategies | Definitions of particular terms or concepts | References supporting strategies’ relevance and providing practical examples |
| Quality Management System | Participation to ring tests | Ring test: one/several reference laboratory send(s) identical clinical samples to other (reference) medical laboratories in order to compare results and participate to the quality assurance. | [23] |
| Participation to international EQA programs | EQA: External Quality Assessments organized by official providers that send identical samples to medical laboratories, one or several times per year, in order to assess the quality of results and laboratory effectiveness. Samples have to be sufficiently stable and homogeneous, according to the ISO norm 17043, which also refers to the ISO norm 13528 as model to assess these aspects. | [5, 7, 24] | |
| Establishment of multidisciplinary quality manuals for the management of specific diseases | [5, 25] | ||
| Financing of quality coordinators | Quality coordinator: person responsible for the different aspects of the Quality Management System of the institution (audits, accreditation, participation to EQAs/ring tests, norms/SOPs/DOCs, etc.). | [4, 26] | |
| Healthcare coordination | Satisfaction surveys among patients | Use of questionnaires and/or interviews between patients and clinicians in order to collect and measure patient’s satisfaction regarding some specific aspects of healthcare/services provided by the hospital, based on appropriated scores of satisfaction and quality of life. | [17, 27, 28] |
| Participation to weekly multidisciplinary sessions of analysis of clinical cases | [7, 25, 29, 30] | ||
| Optimization of patient flows | Patient flow: process of quickly and efficiently attending to patients through the different stages of care, from the time they walk into the medical facility to the time of discharge. | [3, 31, 32] | |
| Structural/spatial organization of the hospital in “poles of care” | Poles of care: Grouping some specific medical departments within the spatial architecture of the medical facility in order to improve the management of some specific diseases/groups of diseases that require these specific services. | [32, 33] | |
| Financing of healthcare coordinators | Healthcare coordinator: person who ensures patients’ follow-up between different medical departments of the hospital or between the hospital and the general practitioner/external hospitals. | [30] | |
| Structural/spatial organization of the hospital in “technical poles” | Technical poles: Grouping some departments with specific technical equipment in order to improve medical procedures and/or reduce technical costs. | [32, 33] | |
| Financing of communication coordinators | Communication coordinator: person who ensures exchanges of information and the partnership between the hospital and patients’ organizations. | [21, 27, 28, 30] | |
| Clinicians’ training | Easier access to medico-scientific databases/guidelines/tools | [1, 3, 30, 34, 35] | |
| More information about the research activities of the hospital | [11, 27] | ||
| Trainings about specific diagnostic guidelines | [5, 7, 29, 30, 35, 36] | ||
| Financing of the participation to international scientific conferences | [3, 21] | ||
| Establishment of healthcare algorithms | [5, 30, 35] | ||
| Training about genetic counseling | [12, 19] | ||
| Financing of long-term stays of healthcare professionals in foreign hospitals | Rotational site visits in order to observe and practice new therapeutic ways, surgery procedures, etc. | [29, 37] | |
| Collaborations between ERNs’ members | Exchange of medical samples | [23] | |
| Exchange of medical/epidemiological data | [3, 7, 10, 34, 35, 38] | ||
| Collaborative research projects | [3, 4, 7, 11, 25, 39] | ||
| Sharing patients | [7] | ||
| Satisfaction surveys among external physicians | Use of questionnaires in order to solicit clinicians’ perception of the ease and quality of the exchange of information in the context of the management and follow-up of patients that they referred to the hospital. | [3, 21] | |
| Telemedicine | [7, 30, 38] | ||
| Pooling of equipment | [7, 21] | ||
| Pooling of financial resources | [7] | ||
| Clinical data management | Patient’s and family medical history | Development or purchase of additional databases, encoding systems or specific registries for these types of medical information | [3, 34, 35] |
| Reports of medical consultations/exams | |||
| Reports of drugs-related adverse events/side effects | |||
| Results of analyses of clinical biology and anatomical pathology | |||
| Results of genetic analyses other than NGS | |||
| Results of NGS analyses | |||
| Financing of data managers | Data Manager: persons who ensure the storage and update of medical databases | [12, 26] | |
| Adequacy of the following ERNs’ collaborations and supports | Collaborations between ERN’s members and the ERN’s coordinator | ERN coordinator: medical specialist related to one specific ERN and who centralizes information coming from the different ERN’s members, interacts with the European Commission and coordinates the evaluation of the ERN’s activities every five years. | [34] |
| Collaborations with patients’ organizations | [5, 7, 10, 21, 25, 30, 34] | ||
| Collaborations with second-line hospitals | [3, 5, 21, 35] | ||
| Collaborations with research laboratories | [11, 21, 34] | ||
| Collaborations with pharmaceutical companies | [10, 21, 40] | ||
| Interactions with the European Commission | [7] | ||
| Interactions with national healthcare authorities | [3, 4, 30, 34] | ||
| Sufficiency of medical staff | [21, 30] | ||
| Time dedicated to research activities | [5, 11, 21] | ||
| Existing legal framework for the exchange of clinical data within the ERN | [7] | ||
| Existing legal framework for the reimbursement of cross-country care within the ERN | [7] | ||
| Existing legal framework for the recognition of expertise in Europe | [8] | ||
| Financial support for ERN’s activities and missions | [21] | ||
| Adequacy of the following types of information provided to patients | Information about the symptoms of the disease | [5, 27, 28, 30] | |
| Drugs-related side effects/adverse events | |||
| Evolution of the disease | |||
| Access to psychological support | |||
| Interpretation of medical data | |||
| Compassionate drug use | |||
| Existing patients’ organizations | |||
| Exchange of patients’ medical data between hospitals/physicians | |||
| Length of retention of patients’ personal/medical data | |||
| Legal framework for the reimbursement of cross-country care | |||
| Legal framework for the reimbursement of the share of patients’ data | |||
| Funding of the following ERNs’ activities | Management of clinical data | [1, 10, 12, 30, 34] | |
| Medical equipment | |||
| Quality Management System | |||
| Fundamental research | |||
| Genetic counseling | |||
| Training of healthcare professionals | |||
| Communication with patients and caregivers | |||
| Communication between hospitals | |||
| Clinical research |
A short description and/or definition of highly specific or complex activities has been mentioned in the third column. The fourth column contains the bibliographic sources that support the relevance of each strategy/aspect investigated and provide practical examples of the use of these strategy. Abbreviations: DOC: Document; EQA: External Quality Assessment; ERN: European Reference Network; SOP: Standard Operating Procedure.
Secondly, the opinions of healthcare professionals about inventoried strategies were collected in May 2016 using two surveys. On one hand, the opinions of 17 physicians related to European academic hospitals, who applied to the European call for the selection of ERN’s (2016) and who thereafter obtained a recognition as members of different ERN’s in 2017, were collected in order to evaluate the strategies’ usefulness and practical feasibility, as well as the current adequacy of different types of ERNs’ collaborations and logistic supports. Categorical variables (usefulness [useful; unuseful; no opinion]; practical feasibility [high or normal; poor; no opinion]) were used for that purpose and the results expressed in rates (percentages) of participants who reported each category. In a secondary analysis, strategies’ usefulness and feasibility were investigated for participants stratified based on a prior membership versus no prior membership to pilot-ERNs. On the other hand, a second panel, composed of 10 physicians related to different Belgian health policy institutions (Federal Public Service Health, Food Chain Safety and Environment; Belgian Observatory of chronic diseases; National Institute for Health and Disability Insurance; Fund for Rare Diseases and Orphan Drugs; RaDiOrg.be), was requested to provide an opinion regarding the adequacy (using different categorical variables [excellent; moderately good; insufficient; woefully insufficient or non-existent; no opinion]) of different types of information provided to patients, and the most-adapted sources of funding for ERN members’ activities.
Each survey was accompanied by a cover letter specifying the context and objectives of the study, as well as ensuring that collected data would be stored and treated in complete confidentiality and anonymity. Participants had over one month to complete the surveys.
Figure SM1 (supplementary material) illustrates the study design.
3.Results
3.1.Strategies’ identification
Sixty-seven potentially efficient strategies/aspects (summarized in Table 2) were identified during the literature screening. Aspects related to the same activities were grouped in order to define eight areas for improvement: (i) Quality Management System, (ii) healthcare coordination, (iii) clinicians’ training, (iv) inter-ERNs’ members collaborations, (v) clinical data management, (vi) ERNs’ external collaborations and supports, (vii) information provided to patients, (viii) financing of ERN members’ activities.
3.2.Strategies’ usefulness and feasibility
Table 3 shows the opinions of the physicians coming from ERNs (for the global population [n = 17], and for each stratum of physicians [prior membership (n = 8) versus no prior membership to pilot-ERNs (n = 9)]) about the usefulness and practical feasibility of strategies investigated in order to improve the Quality Management System, healthcare coordination, the training of clinicians, collaborations between ERN’s members, and management of different types of clinical data (cf. the five first areas investigated in this study and mentioned in Table 2). The proportion (%) of participants who reported each strategy as “useful” is shown in the second column. The third column mentions the rates of participants (%) who reported the strategies as “highly” or “normally” feasible.
Table 3
Reported usefulness and feasibility
| Strategies | Rate of reported USEFULLNESS (%)* | Rate of reported high or normal PRACTICAL FEASIBILITY (%)* | ||||
| Mean | Prior membership to pilot-ERNs | Mean | Prior membership to pilot-ERNs | |||
| No | Yes | No | Yes | |||
| Quality Management System: | ||||||
| 1. Participation to ring tests | 87 (13/15) | 100 (7/7) | 75 (6/8) | 81 (13/16) | 100 (9/9) | 57 (4/7) |
| 2. Participation to international EQA programs | 76 (13/17) | 78 (7/9) | 75 (6/8) | 69 (9/13) | 67 (6/9) | 75 (3/4) |
| 3. Establishment of multidisciplinary quality manuals for the management of specific diseases | 81 (13/16) | 87 (7/8) | 75 (6/8) | 57 (8/14) | 44 (4/9) | 80 (4/5) |
| 4. Financing of quality coordinators | 88 (15/17) | 100 (9/9) | 75 (6/8) | 45 (5/11) | 50 (3/6) | 40 (2/5) |
| Healthcare coordination: | ||||||
| 5. Satisfaction surveys among patients | 71 (12/17) | 78 (7/9) | 62 (5/8) | 100 (13/13) | 100 (8/8) | 100 (5/5) |
| 6. Financing of healthcare coordinators | 100 (17/17) | 100 (9/9) | 100 (8/8) | 67 (6/9) | 60 (3/5) | 75 (3/4) |
| 7. Participation to weekly multidisciplinary sessions of analysis of clinical cases | 94 (16/17) | 89 (8/9) | 100 (8/8) | 54 (7/13) | 29 (2/7) | 83 (5/6) |
| 8. Structural/spatial organization of the hospital in “technical poles” | 64 (9/14) | 50 (4/8) | 83 (5/6) | 71 (10/14) | 50 (4/8) | 100 (6/6) |
| 9. Structural/spatial organization of the hospital in “poles of care” | 86 (12/14) | 87 (7/8) | 83 (5/6) | 40 (4/10) | 40 (2/5) | 40 (2/5) |
| 10. Optimization of patients flows | 69 (11/16) | 67 (6/9) | 71 (5/7) | 57 (8/14) | 44 (4/9) | 80 (4/5) |
| 11. Financing of communication coordinators | 56 (9/16) | 87 (7/8) | 25 (2/8) | 37 (6/16) | 50 (4/8) | 25 (2/8) |
| Clinicians’ training: | ||||||
| 12. More information about the research activities of the hospital | 87 (14/16) | 100 (9/9) | 71 (5/7) | 92 (11/12) | 100 (8/8) | 75 (3/4) |
| 13. Training about specific diagnostic guidelines | 88 (15/17) | 89 (8/9) | 87 (7/8) | 100 (12/12) | 100 (7/7) | 100 (5/5) |
| 14. Easier access to medico-scientific databases/guidelines/tools | 100 (16/16) | 100 (9/9) | 100 (7/7) | 85 (11/13) | 86 (6/7) | 83 (5/6) |
| 15. Trainings about specific therapeutic guidelines | 87 (13/15) | 87 (7/8) | 86 (6/7) | 92 (11/12) | 86 (6/7) | 100 (5/5) |
| 16. Financing of the participation to international scientific conferences | 75 (12/16) | 62 (5/8) | 87 (7/8) | 82 (9/11) | 83 (5/6) | 80 (4/5) |
| 17. Establishment of healthcare algorithms | 79 (11/14) | 100 (6/6) | 62 (5/8) | 61 (8/13) | 37 (3/8) | 100 (5/5) |
| 18. Training about genetic counseling | 69 (9/13) | 50 (3/6) | 86 (6/7) | 70 (7/10) | 80 (4/5) | 60 (3/5) |
| 19. Financing of long-term stays of healthcare professionals in foreign hospitals | 69 (11/16) | 100 (9/9) | 29 (2/7) | 54 (6/11) | 62 (5/8) | 33 (1/3) |
| Collaborations between ERNs’ members: | ||||||
| 20. Exchange of medical samples | 93 (14/15) | 100 (7/7) | 87 (7/8) | 79 (11/14) | 57 (4/7) | 100 (7/7) |
| 21. Satisfaction surveys among external physicians | 71 (10/14) | 100 (6/6) | 50 (4/8) | 83 (10/12) | 75 (6/8) | 100 (4/4) |
| 22. Exchange of medical/ epidemiological data | 94 (15/16) | 100 (8/8) | 87 (7/8) | 53 (8/15) | 50 (4/8) | 57 (4/7) |
| 23. Collaborative research projects | 100 (17/17) | 100 (9/9) | 100 (8/8) | 44 (7/16) | 33 (3/9) | 57 (4/7) |
| 24. Sharing patients | 82 (14/17) | 89 (8/9) | 75 (6/8) | 47 (8/17) | 11 (1/9) | 87 (7/8) |
| 25. Telemedicine | 80 (12/15) | 100 (7/7) | 62 (5/8) | 15 (2/13) | 0 (0/7) | 33 (2/6) |
| 26. Pooling of equipment | 47 (7/15) | 57 (4/7) | 37 (3/8) | 22 (2/9) | 25 (1/4) | 20 (1/5) |
| 27. Pooling of financial resources | 43 (6/14) | 50 (3/6) | 37 (3/8) | 20 (2/10) | 20 (1/5) | 20 (1/5) |
| Management of clinical data: | ||||||
| 28. Patient’s and family medical history | 100 (17/17) | 100 (9/9) | 100 (8/8) | 82 (14/17) | 100 (9/9) | 62 (5/8) |
| 29. Reports of medical consultations/exams | 100 (17/17) | 100 (9/9) | 100 (8/8) | 75 (12/16) | 100 (9/9) | 43 (3/7) |
| 30. Reports of drugs-related adverse events/side effects | 100 (17/17) | 100 (9/9) | 100 (8/8) | 91 (10/11) | 89 (8/9) | 50 (2/2) |
| 31. Results of analyses of clinical biology and anatomical pathology | 100 (17/17) | 100 (9/9) | 100 (8/8) | 82 (14/17) | 100 (9/9) | 62 (5/8) |
| 32. Results of genetic analyses other than NGS | 100 (17/17) | 100 (9/9) | 100 (8/8) | 76 (13/17) | 67 (6/9) | 87 (7/8) |
| 33. Results of NGS analyses | 100 (17/17) | 100 (9/9) | 100 (8/8) | 44 (4/9) | 20 (1/5) | 75 (3/4) |
| 34. Financing of data managers | 71 (12/17) | 100 (9/9) | 37 (3/8) | 65 (11/17) | 100 (9/9) | 25 (2/8) |
The second column of Table 3 mentions the global proportion (%) of participants who reported each strategy as useful. The third column mentions the global proportion (%) of participants who reported each strategy as highly or normally easy to develop. *: Results are presented as global mean scores for all participants (without distinction between those having a prior membership to pilot ERNs and those who don’t) and as means scores after participants’ stratification (based on a prior membership versus no prior membership to pilot ERNs). Each mean score is accompanied by the number of participants who reported the categories “useful” or “high or normal feasibility” on the total number of respondents (between brackets). Abbreviations: ERN: European Reference Network; EQA: External Quality Assessment; NGS: Next Generation Sequencing.
Regarding the improvement of the Quality Management System, results show that 81 to 88% of participants reported the participation to ring tests (please see the definition mentioned in Table 2) and External Quality Assessment programs (EQAs), the establishment of multidisciplinary quality manuals focused on specific groups of diseases, and the financing of quality coordinators as useful. However, participants’ opinions about the feasibility of these strategies were globally more negative, especially for the two latest (<60% of participants reported a high or normal feasibility). Of note, a trend suggests that clinicians with a prior membership to pilot-ERNs reported the participation to ring tests as more difficult to develop versus those without previous membership.
Concerning the improvement of healthcare coordination, the (i) use of surveys in order to evaluate patients’ satisfaction and understanding of services/information provided by university hospitals, (ii) financing of healthcare coordinators to ensure patients’ follow-up between medical departments or between the hospital and the general practitioner/external hospitals, (iii) participation to weekly multidisciplinary discussions of clinical cases, (iv) hospitals’ reorganization in “technical poles” (e.g. laboratories, medical imagery, etc.), and (v) the optimization of patients’ flows seem to be efficient (reported as useful by 64–100% of participants and as highly or normally easy to develop by 54–100% of participants). The other strategies (hospital’s reorganization in “poles of care” [by diseases’ types]; financing of coordinators to ensure the communication between the hospital and patients’ organizations) were considered as useful by the majority (56–86%) of participants. However, their implementation seems to be fastidious (high/normal feasibility reported by ≤40% of participants). Of note, participants with a prior membership to pilot-ERN’s reported a lower usefulness for the financing of communication coordinators than those without prior membership.
Regarding clinicians’ training, the study enabled to assess the efficiency of (i) the dissemination of information about the hospital’s research activities, (ii) an easier access to medico-scientific databases/guidelines/informatics tools, (iii) trainings focused on diagnostic/therapeutic guidelines and genetic counseling, (iv) the financing of the participation to international scientific conferences, (v) the development of healthcare algorithms for some specific diseases (usefulness and high/normal feasibility reported by >60% of participants). The financing of clinicians long-term stays in foreign hospitals seem to be less efficient and only 54% of all participants reported it as highly/normally feasible.
As regards the improvement of inter-ERNs’ members collaborations, the exchange of medical samples is very efficient (reported as useful and feasible by respectively 93% and 79% of participants). To a lesser extent, the development of satisfaction surveys among external clinicians who refer patients to the hospital was also reported as useful and feasible by respectively 71% and 83% of participants. Some strategies (exchange of medical/epidemiological data, collaborative research projects, sharing patients, telemedicine) have been reported as useful by the 80 to 100% of participants but quite fastidious to perform. Lastly, the pooling of medico-scientific equipment and funding were reported as poorly useful and difficult to perform. Of note, a higher feasibility for sharing patients between hospitals was expressed by physicians with a prior membership to pilot-ERNs compared to those without a prior membership.
Finally, the development of a good management and storage system for the different types of patients’ data mentioned in Table 3 was considered as useful by 100% of participants. However, physicians with a prior membership to pilot-ERNs seem to be more pessimistic concerning the ease of management of patients and family medical histories, reports of medical consultations/exams, and of analyses of clinical pathology and anatomical pathology. Regarding the feasibility management of results of Next Generation Sequencing (NGS) analyses, we have to notice that a high rate (47%) of participants expressed no opinion. However, even if no statistically significant correlation could be obtained, it seems that the management of results of NGS analyses is considered as easier by physicians coming from pilot-ERNs vs. those without any prior membership. Curiously, the financing of data managers was only considered as useful and highly/normally feasible by respectively 71 and 65% of all participants, with participants with a prior membership to pilot-ERN’s reporting a lower usefulness and feasibility than those without prior membership to ERNs. This may partly be explained by the fact that the current data management issues aren’t related to a lack of staff dedicated to this activity but maybe more to inadequate tools and databases.
3.3.Global efficiency of the strategies selected from peer-reviewed publications
Figure 1 summarizes the global efficiency of the strategies based on their mean rates of usefulness (x axis) and high/normal practical feasibility (y axis) reported in Table 3. It appears that the strategies selected from the literature in order to improve the Quality Management System and the management of clinical data are really useful but their feasibility is heterogeneous. Two strategies presenting a low feasibility (<50%) but high usefulness (>90%) should receive more attention and means: the financing of quality coordinators and the development of adequate storage platforms, especially for NGS data.
Fig.1
Efficiency of the strategies selected from the literature based on their combined usefulness and feasibility. The global efficiency of strategies selected from peer-reviewed scientific publications in order to improve the Quality Management System, management of clinical data, healthcare coordination, collaborations between ERNs’ members, and healthcare professionals’ training has been determined based on their means scores of feasibility (y axis) and usefulness (x axis) reported in (Table 3). Strategies are plotted by activity domains using open circle symbols. The dashed black lines represent 50% of participants reporting a high/normal feasibility and usefulness. Stars within the circles represent strategies for which the reported feasibility is higher than the reported usefulness (data plotted above the feasibility/usefulness correlation line [solid black line; correlation coefficient = 1]).
![Efficiency of the strategies selected from the literature based on their combined usefulness and feasibility. The global efficiency of strategies selected from peer-reviewed scientific publications in order to improve the Quality Management System, management of clinical data, healthcare coordination, collaborations between ERNs’ members, and healthcare professionals’ training has been determined based on their means scores of feasibility (y axis) and usefulness (x axis) reported in (Table 3). Strategies are plotted by activity domains using open circle symbols. The dashed black lines represent 50% of participants reporting a high/normal feasibility and usefulness. Stars within the circles represent strategies for which the reported feasibility is higher than the reported usefulness (data plotted above the feasibility/usefulness correlation line [solid black line; correlation coefficient = 1]).](https://ip.ios.semcs.net:443/media/trd/2020/5-1-2/trd-5-1-2-trd190032/trd-5-trd190032-g001.jpg)
Regarding the strategies proposed to improve the healthcare coordination and collaboration between ERNs’ members, we observe a broad dispersion of the efficiency profiles. This tends to suggest that literature’s proposals are not yet totally optimal, maybe because of the novelty of the activities and proposals for improvements in these domains.
Finally, even if all strategies selected to improve healthcare professionals’ training have been reported as useful and quite feasible, a gradient of efficiency is observed among them and helps to privilege the most efficient ones.
Of note, for some useful strategies, the rate of reported high/normal feasibility is higher than the rate of reported usefulness (*; data plotted above the feasibility/usefulness correlation line). This suggests that these strategies (development of satisfaction surveys among patients and external collaborators, hospitals’ organization in technical poles, development of diagnostic/therapeutic guidelines, clinicians participation to international scientific conferences and a broader diffusion of information about hospitals’ research activities in order to valorize their expertise) might be priority areas.
3.4.Adequacy of ERNs’ external collaborations and supports
Figure 2 shows that 75 to 100% of participants reported excellent or good collaborations between ERNs’ members and the ERNs’ coordinator and between ERNs and patients’ organizations, second-line hospitals, research laboratories and pharmaceutical companies. It seems that the adequacy of the interactions with the European Commission and national healthcare authorities remain quite heterogeneous (excellent/good [50 to 62.5%]; significantly insufficient/non-existent [37.5%]; no opinion [0 to 12.5%]).
Fig.2
Evaluation of the adequacy of different types of collaborations and supports of European Reference Networks (ERNs). The horizontal stacked bar charts represent the rates (expressed in percentages) of participants who reported each of the categories used to evaluate the adequacy of different types of collaborations and supports. The level of adequacy was evaluated in a categorical way (excellent [white]; moderately good [light grey]; insufficient [grey hatched area]; woefully insufficient or non-existent [black]; no opinion [white hatched area]).
![Evaluation of the adequacy of different types of collaborations and supports of European Reference Networks (ERNs). The horizontal stacked bar charts represent the rates (expressed in percentages) of participants who reported each of the categories used to evaluate the adequacy of different types of collaborations and supports. The level of adequacy was evaluated in a categorical way (excellent [white]; moderately good [light grey]; insufficient [grey hatched area]; woefully insufficient or non-existent [black]; no opinion [white hatched area]).](https://ip.ios.semcs.net:443/media/trd/2020/5-1-2/trd-5-1-2-trd190032/trd-5-trd190032-g002.jpg)
Concerning logistic support, the sufficiency of medical staff was reported as excellent/good by 75% of participants. However, the time that clinicians related to ERNs can dedicate to their research activities, the European legal frameworks for the exchanges of clinical data, reimbursement of cross-country care and recognition of clinicians’ expertise, as well as the funding of clinicians’ additional tasks related to their membership to ERNs were reported as significantly insufficient by 37.5 to 75% of participants.
3.5.Adequacy of information provided to patients
Patients’ satisfaction considerably varies depending on the type of information received at the hospital. As shown in (Fig. 3A), information about diseases’ symptoms/evolution, drugs-related side effects/adverse events, available psychological support and interpretation of medical data were considered as good or excellent by >70% of participants. Information about compassionate drug use and existing patients’ organizations should be improved (reported as good by <60% of participants). Intriguingly enough, results showed a significant insufficiency of information about ethical or financial aspects, namely the exchange/retention of patients’ data, the reimbursement of cross-country care, and associated legal aspects (reported as insufficient by 42 to 86% of participants).
Fig.3
Evaluation of the adequacy of information provided to patients and sources of funding for the activities of the European Reference Networks (ERNs). (A) The horizontal stacked bar charts represent the rates (expressed in percentages) of Belgian health policy makers who reported each category used to evaluate the adequacy of the different types of information considered. The level of adequacy was evaluated in a categorical way (excellent [white]; moderately good [light grey]; insufficient [grey hatched area]; woefully insufficient or non-existent [black]; no opinion [white hatched area]). (B) The horizontal stacked bar charts represent the rates (expressed in percentages) of participants who reported each one of the sources of funding proposed in the literature as adequate for the financing of ERNs’ activities.
![Evaluation of the adequacy of information provided to patients and sources of funding for the activities of the European Reference Networks (ERNs). (A) The horizontal stacked bar charts represent the rates (expressed in percentages) of Belgian health policy makers who reported each category used to evaluate the adequacy of the different types of information considered. The level of adequacy was evaluated in a categorical way (excellent [white]; moderately good [light grey]; insufficient [grey hatched area]; woefully insufficient or non-existent [black]; no opinion [white hatched area]). (B) The horizontal stacked bar charts represent the rates (expressed in percentages) of participants who reported each one of the sources of funding proposed in the literature as adequate for the financing of ERNs’ activities.](https://ip.ios.semcs.net:443/media/trd/2020/5-1-2/trd-5-1-2-trd190032/trd-5-trd190032-g003.jpg)
3.6.Financing of ERN’s activities
As shown in (Fig. 3B), the financing of ERN’s members activities should be supported by different actors.
Federal/regional public funding was the most often reported source for the financing of clinical data management, medical equipment, quality management, fundamental research, genetic counseling and training of healthcare professionals. Other sources of funding proposed to a much lesser extent for these activities were the European Commission (8–43%), hospitals’ own resources or universities (0–33%), specific research grants (0–25%), pharmaceutical companies (proposed by 10% of participants for medical equipment) or patient’s organizations (proposed by 8% of participants to fund data management and genetic counseling sessions).
Information provided to patients should also be funded using different sources, including patient’s organizations. The European Commission was the main funding source proposed for inter-hospitals communication. Finally, clinical research’s costs should be covered by academic hospitals themselves (or related universities), the European Commission and national/regional healthcare authorities.
4.Discussion
Choosing the best ways to improve healthcare’s quality, clinicians’ expertise, hospitals’ collaborations and financing remains a big challenge, especially for highly-qualified institutions, which already face budgetary shortfalls [11, 12].
To the best of our knowledge, this study is the first one that investigated experts’ opinions in order to deal with these combined issues. This work presents five main strengths: (i) focused on strategies described in peer-reviewed papers, (ii) its bottom-up approach (based on experts’ opinions), (iii) its large scope, (iv) the involvement of both clinicians and representatives of health policy institutions, (v) its methodology, validated by the Donabedian model and RAND/UCLA appropriateness method, two well-known approaches used to evaluate healthcare’s quality and appropriateness [13–15]. Its limitations are (i) the limited number of participants and the fact that sometimes participants decided to provide no opinion. This may affect results’ power, especially for ERNs-related clinicians (n = 17), even if this number represents a significant proportion of the total number of ERNs recognized in 2017 (n = 24) because participants are related to different ERNs. This enables to have a good representation of the opinion of most ERNs, (ii) the use of only categorical variables to report participants’ opinions and, (iii) the fact that health policy makers came only from Belgium and not from different European states. Moreover, we have to take into account that results are based on participants’ opinions and remain thereby quite subjective. This study should thus be considered as an initial and exploratory step in the improvement process of the quality of ERNs.
The results strengthen the importance of quality assessment and of involving both clinicians and patients in the improvement of healthcare coordination. Indeed, participative management is a key factor of hospitals’ quality, efficiency, and patients’ satisfaction [16]. To this aim, patients’ opinions should also be discussed during clinicians’ multidisciplinary meetings and adapted communication tools developed to this aim [17, 18].
This study also highlighted some clinicians’ concerns (e.g. lack of information or trainings in human genetics or telemedicine). Indeed, only a few existing training programs cover, for instance, all aspects of genetic counseling [19, 20]. Awareness campaigns should therefore be performed and adequate tools and data storage platforms developed at the national or international levels for genetic data. The study also enabled to highlight the imbalance between physicians’ clinical and research activities that affects their contribution to scientific knowledge [5, 11, 21].
The very positive results obtained during this study regarding the collaborations between the ERNs recognized in 2017 and other actors from the public health sector (patients’ organizations, second-line hospitals, research laboratories, pharmaceutical companies), and between the ERNs and their respective coordinators, demonstrate the robustness and efficiency of the concept of network as improvement process for patients management. However, it seems that the partnership between ERNs and European or national healthcare authorities is still lagging when it comes to support ERNs’ work, especially financially. This can be explained by a lack of budgetary resources at the European and/or national level(s). Healthcare authorities from the different European states may also consider that they are not competent enough to evaluate the quality of the activities performed by the network, especially in other countries or to have an accurate idea of ERNs’ needs. Closer and frequent discussions should therefore be organized between ERNs and European or national political powers in order to improve this aspect and guarantee ERNs’ sustainability.
A priority should be given to the establishment of European legal frameworks for the reimbursement of cross-country care, share of patients’ data and international recognition of clinicians’ expertise.
The dissemination of adapted information to patients requires a clear definition of funding sources and a close collaboration between all healthcare actors, including political instances. This aspect is crucial in terms of quality of patients’ management and well-being. However, it seems that the quality level differs among the different types of delivered information considered. This means that there is still room for improvement for this ERNs’ duty. ERNs’ members and healthcare authorities should therefore encourage the reflection on an harmonized list of quality criteria that ERNs should fulfilled with accurate subcategories (e.g. different types of information provided to the patients) for specific activities with outcomes that can be assessed objectively.
In conclusion, the set-up of panels of multidisciplinary and international clinicians and of health policy makers enables to identify efficient strategies to deal with ERNs’ challenges. It also enables to balance unjustified concerns and to highlight possible threats.
Acknowledgments
The author thanks all participants for their contribution to this work, as well as Professor Frédéric Thys for his advices during the study design and Dr. Wim Coucke for his pertinent suggestions during the redaction of the article. NMV is a scientific collaborator from the Sciensano Institute, previously called Scientific Institute of Public Health, and is supported by the Belgian National Institute for Health and Disability Insurance (RIZIV-INAMI; grant W4043.0100.8).
Supplementary Materials
[1] The supplementary material is available in the electronic version of this article: http://dx.doi.org/10.3233/TRD-190032.
References
[1] | D.L. Coleman , R.M. Wardrop, III , W.S. Levinson , M.L. Zeidel P.E. Parsons , Strategies for Developing and Recognizing Faculty Working in Quality Improvement and Patient Safety, Acad Med, May (2016) . |
[2] | C.D. Stevens , Five Tactics to Quickly Build Quality Improvement and Patient Safety Capacity at Academic Health Centers, Acad Med, May (2016) . |
[3] | P. Dennehy , M.P. White , A. Hamilton , J.M. Pohl , C. Tanner , T.J. Onifade K. Zheng , A partnership model for implementing electronic health records in resource-limited primary care settings: Experiences from two nurse-managed health centers, J Am Med Inform Assoc 18: (6) ((2011) ), 820–826. |
[4] | Z. Terzic-Supic , V. Bjegovic-Mikanovic , D. Vukovic , M. Santric-Milicevic , J. Marinkovic , V. Vasic U. Laaser , Training hospital managers for strategic planning and management: A prospective study, BMC Med Educ 15: ((2015) ), 25. |
[5] | The European Commission, European Reference Networks - Assessment Manual for Applicants : 3.- Operational Criteria for the Assessment of Networks, (2016) . |
[6] | The European Commission, COMMISSION DELEGATED DECISION of 10 March 2014 setting out criteria and conditions that European Reference Networks and healthcare providers wishing to join a European Reference Network must fulfil (Text with EEA relevance) (2014/286/EU) , Official Journal of the European Union 147: ((2014) ), 71–78. |
[7] | The european parliament and of the council of the european union, directive 2011/24/eu of the european parliament and of the council of 9 March 2011 on the application of patients' rights in cross-border healthcare, Official Journal of the European Union 88: ((2011) ), 45–65. |
[8] | V. Heon-Klin , European Reference networks for rare diseases: What is the conceptual framework? Orphanet J Rare Dis 12: (1) ((2017) ), 137. |
[9] | European Union Committee of Experts on Rare Diseases (EUCERD), EUCERD Report - Preliminary analysis of the outcomes and experiences of pilot European Reference Networks for rare diseases, May (2011) . |
[10] | C.E. Hollak , M. Biegstraaten , M.R. Baumgartner , N. Belmatoug , B. Bembi , A. Bosch , M. Brouwers , H. Dekker , D. Dobbelaere , M. Engelen , M.C. Groenendijk , R. Lachmann , J.G. Langendonk , M. Langeveld , G. Linthorst , E. Morava , Poll-The BT , S. Rahman , M.E. Rubio-Gozalbo , U. Spiekerkoetter , E. Treacy , R. Wanders , J. Zschocke R. Hagendijk , Position statement on the role of healthcare professionals, patient organizations and industry in European Reference Networks, Orphanet J Rare Dis 11: ((2016) ), 7. |
[11] | H.M. Brandt , V.M. Young , D.A. Campbell , S.K. Choi , J.S. Seel D.B. Friedman , Federally Qualified Health Centers’ Capacity and Readiness for Research Collaborations: Implications for Clinical-Academic-Community Partnerships, Clin Transl Sci 8: (4) ((2015) ), 391–393. |
[12] | S.A. Lynch I. Borg , Wide disparity of clinical genetics services and EU rare disease research funding across Europe, J Community Genet 7: (2) ((2016) ), 119–126. |
[13] | A. Donabedian , Evaluating the quality of medical care. (1966) , Milbank Q 83: (4) ((2005) ), 691–729. |
[14] | K. Fitch , S.J. Bernstein , M.D. Aguilar , B. Burnand , J.R. LaCalle , P. Lazaro , M. van het Loo , J. McDonnell , J.P. Vader J.P. Kahan , The RAND/UCLA Appropriateness Method User’s Manual,. RAND Corporation, Ed. (2001) . |
[15] | M.R. Patel , J.A. Spertus , R.G. Brindis , R.C. Hendel , P.S. Douglas , E.D. Peterson , M.J. Wolk , J.M. Allen I.E. Raskin , ACCF proposed method for evaluating the appropriateness of cardiovascular imaging, J Am Coll Cardiol 46: (8) ((2005) ), 1606–1613. |
[16] | A.P. van Beek D.L. Gerritsen , The relationship between organizational culture of nursing staff and quality of care for residents with dementia: Questionnaire surveys and systematic observations in nursing homes, Int J Nurs Stud 47: (10) ((2010) ), 1274–1282. |
[17] | R. Al-Abri A. Al-Balushi , Patient satisfaction survey as a tool towards quality improvement, Oman Med J 29: (1) ((2014) ), 3–7. |
[18] | D.W. Hamilton , B. Heaven , R.G. Thomson , J.A. Wilson C. Exley , Multidisciplinary team decision-making in cancer and the absent patient: A qualitative study, BMJ Open 6: (7) (2016) . |
[19] | G.W. Hooker , K.E. Ormond , K. Sweet B.B. Biesecker , Teaching genomic counseling: Preparing the genetic counseling workforce for the genomic era, J Genet Couns 23: (4) ((2014) ), 445–451. |
[20] | K.M. Ivan , S. Hassed , A.G. Darden , C.E. Aston C. Guy , Influence of Genetic Counseling Graduate Program Websites on Student Application Decisions, J Genet Couns 26: (6) ((2017) ), 1213–1220. |
[21] | H. Hannemann-Weber , M. Kessel C. Schultz , Research performance of centers of expertise for rare diseases—the influence of network integration, internal resource access and operational experience, Health Policy 105: (2-3) ((2012) ), 138–145. |
[22] | The European Commission, COMMISSION IMPLEMENTING DECISION of 10 March 2014 setting out criteria for establishing and evaluating European Reference Networks and their Members and for facilitating the exchange of information and expertise on establishing and evaluating such Networks (Text with EEA relevance) (2014/287/EU), Official Journal of the European Union 147: ((2014) ), 79–87. |
[23] | J. Depuydt , A. Baeyens , S. Barnard , C. Beinke , A. Benedek , P. Beukes , I. Buraczewska , F. Darroudi , S.S. De , I. Dominguez , G.O. Monteiro , V. Hadjidekova , E. Kis , U. Kulka , F. Lista , K. Lumniczky , R. M’kacher , J. Moquet , D. Obreja , U. Oestreicher , J. Pajic , N. Pastor , L. Popova , E. Regalbuto , M. Ricoul , L. Sabatier , J. Slabbert , S. Sommer , A. Testa , H. Thierens , A. Wojcik A. Vral , RENEB intercomparison exercises analyzing micronuclei (Cytokinesis-block Micronucleus Assay), Int J Radiat Biol 93: (1) ((2017) ), 36–47. |
[24] | W. Coucke , C. Charlier , W. Lambert , F. Martens , H. Neels , J. Tytgat , P. Van de Walle , A. Vanescote , P. Wallemacq , S. Wille A.G. Verstraete , Application of the characteristic function to evaluate and compare analytical variability in an external quality assessment scheme for serum ethanol, Clin Chem 61: (7) ((2015) ), 948–954. |
[25] | E. Schmiedeke , B. de , I.M. Lacher , S. Grasshoff-Derr , A. Garcia-Vazquez , S. Giuliani , P. Midrio , P. Gamba , B. Iacobelli , P. Bagolan , G. Brisighelli , E. Leva , C. Cretolle , S. Sarnacki , P. Broens , C. Sloots , R. van , I, N. Schwarzer , D. Aminoff , M. Haanen E. Jenetzky , Towards the perfect ARM center: The European Union's criteria for centers of expertise and their implementation in the member states. A report from the ARM-Net, Pediatr Surg Int 31: (8) ((2015) ), 741–745. |
[26] | J.C. Parmalee , T.O. Scholomiti, P. Whitman , M. Sees , D. Smith , E. Gardner C. Bastian , Achieving empowerment through information, Top Health Inf Manage 13: (4) ((1993) ), 15–29. |
[27] | V.S. Lee , T. Miller , C. Daniels , M. Paine , B. Gresh A.L. Betz , Creating the Exceptional Patient Experience in One Academic Health System, Acad Med 91: (3) ((2016) ), 338–344. |
[28] | J. Morisset , B.P. Dube , C. Garvey , J. Bourbeau , H.R. Collard , J.J. Swigris J.S. Lee , The Unmet Educational Needs of Interstitial Lung Disease Patients: Setting the Stage for Tailored Pulmonary Rehabilitation, Ann Am Thorac Soc (2016) . |
[29] | M.J. Wolf , E.K. Lee , S.C. Nicolson , G.D. Pearson , M.K. Witte , J. Huckaby , M. Gaies , L.S. Shekerdemian W.T. Mahle , Rationale and methodology of a collaborative learning project in congenital cardiac care, Am Heart J 174: ((2016) ), 129–137. |
[30] | European Union Committee of Experts on Rare Diseases (EUCERD), Centres of Expertise and Quality of Care for Rare Diseases - Report from the EUCERD Joint Action WP7, Jan. (2015) . |
[31] | S. Khanna , J. Boyle , N. Good , A. Bell J. Lind , Analysing the emergency department patient journey: Discovery of bottlenecks to emergency department patient flow, Emerg Med Australas, Nov. (2016) . |
[32] | T. Meschi , A. Ticinesi , B. Prati , A. Montali , A. Ventura , A. Nouvenne L. Borghi , A novel organizational model to face the challenge of multimorbid elderly patients in an internal medicine setting: A case study from Parma Hospital, Italy, Intern Emerg Med 11: (5) ((2016) ), 667–676. |
[33] | P.H. Brechat , L. Antoine , M.C. Mathieu-Grenouilleau, R. Rymer , F. Matisse , D. Baraille P. Beaufils , [Hospital governance: Between crisis management and implementation of public health policy], Sante Publique 22: (5) ((2010) ), 571–580. |
[34] | S. Baldovino , A.M. Moliner , D. Taruscio , E. Daina D. Roccatello , Rare Diseases in Europe: From a Wide to a Local Perspective, Isr Med Assoc J 18: (6) ((2016) ), 359–363. |
[35] | F. Schurhoff , G. Fond , F. Berna , E. Bulzacka , J. Vilain , D. Capdevielle , D. Misdrahi , M. Leboyer P.M. Llorca , A National network of schizophrenia expert centres: An innovative tool to bridge the research-practice gap, Eur Psychiatry 30: (6) ((2015) ), 728–735. |
[36] | F. Veneziani , F. Panza , V. Solfrizzi , R. Capozzo , M.R. Barulli , A. Leo , M. Lozupone , A. Fontana , S. Arcuti , M. Copetti , V. Cardinali , A. Grasso , M. Tursi , A. Iurillo , B.P. Imbimbo , D. Seripa G. Logroscino , Examination of level of knowledge in Italian general practitioners attending an education session on diagnosis and management of the early stage of Alzheimer’s disease: Pass or fail? Int Psychogeriatr 28: (7) ((2016) ), 1111–1124. |
[37] | G.S. Bloomfield M.D. Huffman , Global chronic disease research training for fellows: Perspectives, challenges and opportunities, Circulation 121: (11) ((2010) ), 1365–1370. |
[38] | A. Geissbuhler , C.O. Bagayoko O. Ly , The RAFT network: 5 years of distance continuing medical education and tele-consultations over the Internet in French-speaking Africa, Int J Med Inform 76: (5-6) ((2007) ), 351–356. |
[39] | M. Aelvoet , Arrêté royal relatif a l'agrément des laboratoires de biologie clinique par le Ministre qui a la Santé publique dans ses attributions - Koninklijk besluit betreffende de erkenning van de laboratoria voor klinische biologie door de Minister tot wiens bevoegdheid de Volksgezondheid behoort, (1999) . |
[40] | M. Toner R.G. Tompkins , Invention, innovation, entrepreneurship in academic medical centers, Surgery 143: (2) ((2008) ), 168–171. |
