Analysis of the organizational knowledge management system of nurse education centers with hybrid Fuzzy DEMATEL-network DEA method
Abstract
On-the-job training is important for the growth of organizational human capital. The present paper evaluates the efficiency of on-the-job training using the DEA model and knowledge management system to illustrate the interactions of organizational knowledge management concepts for measuring the efficiency of on-the-job training courses for employees within the organizational knowledge management systems. In this research, the DEA method was used to evaluate the efficiency of knowledge management system and analyze the interdependencies of the system. Also, the EU Knowledge and Innovation Management Measurement and the Kirkpatrick models were employed to identify the indicators. This study was conducted on 27 education centers in Iran. The results of evaluating the effectiveness of the knowledge management system show that the factor of “educational resources and facilities” is the most affecting variable and the most affected variable is “educational effectiveness”. Then, based on the DEA method, the weights of the sub-factors of the model were determined, and the efficiency of each step and as a result the efficiency of the whole system was obtained. Inefficient units were discarded and finally the efficient units were ranked. This study can help the organizations to identify the factors affecting the effectiveness of knowledge management programs.
1.Introduction
The growth of human resources and the movement of organizations towards learning organizations are prerequisites for success in today’s dynamic, challenging and competitive business environment. Organizations must be able to acquire, create and disseminate the knowledge needed to achieve the organizational goals and missions. Staff training is one of organizations’ actions to acquire and create the knowledge required by the organization and its transfer and sharing among employees [1]. Some researchers such as Wang [2] consider staff training and their educational levels as one of the main affecting factors of the improvement and success of knowledge management in organizations. However, it should be noted that any type of training may not be effective in achieving the desirable results except for wasting the organizational scarce resources [3]. On-the-job training is a tool for the growth and development of human capital. There are factors that can help to achieve the goals of training courses and make them more effective. In organizations, usually training programs are poorly evaluated due to the lack of seriousness of employees and managers about training, appropriate evaluation of courses and their feedback [4]. However, it is of significant importance for governmental and private organizations such as the nurse education centers to ensure the effectiveness of training programs and identify the factors affecting the effectiveness of these programs. Despite the importance of employee training management in organizations as well as the crucial role of improving the quality of human resources in the health sector, the lack of knowledge and skills of the health sector employees is one of the most important obstacles to the implementation of sustainable development projects in the health sector in Iran [5].
Therefore, this study measures and evaluates the effectiveness of the on-the-job training courses with the DEA method due to the importance of knowledge management and on-the-job training in preventing the loss of organizational resources and achieving the organizational goals. The research question is how to measure the quality indicators for analyzing the organizational knowledge management systems, which has always been one of the main problems in engineering and management sciences and has received less attention. In other words, the purpose of this study is to apply the DEA model based on an organizational knowledge management model that can answer the following research question by examining the causal relationships and interdependencies in the decision-making units (DMUs): Is it possible to evaluate the effectiveness of training courses of the nurse education centers by using the DEA method as well as the knowledge management model?
In the present study, a qualitative indicator (the Kirkpartric model) is first quantified, then presented in the form of a quantitative research model and finally measured. After examining the interdependencies, the weights of the main factors are determined. Further to the authors’ knowledge, there has been no study used the DEA method based on a knowledge management model to evaluate the effectiveness of on-the-job training courses that is one of the substantial challenges of learning organizations.
2.Literature review
2.1Educational evaluation in knowledge management system
Performance evaluation, evaluation of on-the-job training courses, and their effectiveness are important for educational evaluation in the knowledge management system. Knowledge management processes are divided into different categories. Wiig [6] classifies knowledge management processes into creation, collection, dissemination and application of knowledge. Darroch [7] categorized knowledge management processes into three components: acquisition, creation, and accountability. Dalkir [8] divided knowledge management processes into three main processes of knowledge acquisition/creation, knowledge sharing/dissemination, knowledge understanding and application as well as three sub-processes of knowledge evaluation, adaptation and updating. Carlsson et al. [9] introduced a model to measure the performance of knowledge management and the national innovation system. They assumed that it would be difficult to measure the performance of the entire system due to the size and complexity of the systems, so as a solution they proposed limiting the level of analysis to a product, industry or group of industries. The following table shows the indicators that have been extracted in each aspect to adapt to the system in question in this research.
Table 1
Performance criteria of technological systems measurement model
| Indicators of knowledge creation | Indicators of knowledge dissemination | Indicators of knowledge application |
|
|
|
|
The purpose of organizations in implementing knowledge management processes is to improve the level of knowledge, skills and experience of employees, and the main competencies of organizations and employees. Therefore, it complies with the staff training system that is responsible for training and promoting the professional knowledge of human resources [1]. After understanding the processes of the knowledge management system, the second problem is holding training courses and on-the-job training for employees. Werner and Desimone [10] categorized human resource training methods into three categories: on-the-job training, classroom training, and self-action training. Nowadays, by recognizing knowledge as the greatest source for gaining competitive advantage, and stabilizing the knowledge management system at the inclusive level, organizations have been encouraged to utilize the knowledge management capacities and process along with the staff training system in order to achieve their goals. On-the-job training is one of the types of training that organizations provide for the development of themselves and their employees. On-the-job training is nothing more than an effort to improve the knowledge, awareness, technical skills and well-being of the employees of an organization and prepare them for the optimal performance. On-the-job training is one of the important and effective factors in the success of organizations and achieving their goals. Planning and implementing on-the-job training in organizations is a vital need, but what is more important is to measure the effectiveness of these training courses [11]. Evaluation of the educational programs often appears as the last step in a systematic education method with the purpose of improving education and its effectiveness [12]. The definition that is more comprehensive and includes most of the elements discussed in evaluation is the definition introduced by Bybee et al. [13]. They believe that evaluation is a systematic process of gathering and interpreting evidence that ultimately leads to value judgment with the expectation of a certain action. One of the best evaluation methods is the Kirkpatrick four-level evaluation model introduced by Kirk Patrick in 1960 which facilitates complex evaluation processes of educational programs [14]. Most popular evaluation models in recent years have been based on the four-level educational evaluation model proposed by Kirkpatrick. This model has been described as a comprehensive, simple and practical model for many educational situations and is recognized by many experts as a standard index in this field. Kirkpatrick defines evaluation as a determinant of effectiveness in a training program and divides the evaluation process into four levels or steps including reaction which refers to the degree of reaction that learners show to all the factors influencing the implementation of a training course, learning, behavior, and results [15]. Participants’ knowledge, skills, or attitudes that have changed simply because of their participation in a particular training course are measured. Behavior refers to how and to what extent changes occur in the behavior of participants as a result of participation in training courses, and it can be clarified by continuing the assessment in the real work environment. Results means the extent to which outcomes are directly related to the organizational goals [16]. One of the strengths of this model is the simplification of complex evaluation processes that facilitate the understanding of these processes for novice evaluators [17]. Another advantage of this model is that it does not depend on the investigation of the impact of individual and environmental variables [18]. This model helps evaluators figure out important questions in evaluation and gradually move towards gathering evidence at higher levels and not limiting their judgment to just one or two levels [19]. In a study of UK industrial companies, Hale showed that the Kirkpatrick educational evaluation model has been widely used only at the reaction level (level 1); learning (level 2): 32%, behavior (level 3): 9%, and results (level 4): 7% [20, 21].
Bartel [22] examined the effectiveness of on-the-job training courses from 1983 to 1986 and showed that labor productivity increases significantly with the increase of training programs. Dilley and Wolf [23] found that the tendency of the learners of agricultural training courses after participating in the courses has increased and their motivation for engaging in agricultural activities has increased. Shakoor, Ghumman, and Mahmood [24] demonstrated that on-the-job training has a significant impact on teacher competence as well as the implementation of the curriculum and improved teachers’ attitudes toward professional teaching. Borate et al. [25] used the Kirkpatrick model to evaluate the effectiveness of staff training and development programs. Gregory et al. [26] suggested that environmental concepts should be included in the educational programs to promote and support more agricultural products. Jeong and Choi [27] investigated agricultural literacy in South Korean schools and concluded that teachers’ agricultural education has a considerable impact (13.6%) on students’ agricultural literacy.
Ardahan-Sevgili and Yardimci [28] evaluated the educational approach using the Kirkpatrick model in the field of medicine and among nurses. Alsalamah and Callinan [19] stated that the effectiveness of educational programs is measured by assessing their ability to achieve their goals and identifying areas that need improvement. They evaluated the training programs for female teachers using the Kirkpatrick model in Saudi Arabia. Findings showed that Kirkpatrick assessment model is very effective in teachers’ educational evaluation.
Data envelopment analysis (DEA) technique is a quantitative performance evaluation model which is particularly applied to the fields of operations and production management and production systems. The DEA mathematical models consider decision-making units (DMUs) as completely independent parts and boxes. The performance indicators are measured based on system inputs and outputs. However, there may be several interdependencies and cause-and-effect relationships within these DMUs that are often neglected. But what our research question is how to measure the quality indicators in these models to perform various analyzes, which has always been one of the main problems in engineering and management sciences and has received less attention. In the present study, a qualitative index and concept (effectiveness with the components of the Kirkpatrick model) is first quantified, then presented in the form of a quantitative research model and finally measured. The other studies have commonly determined the weights of the main factors first using the DEMATEL method, then, identified the weights of the variables or sub-factors of the model using different network analysis techniques. However, in this research, the interdependencies are examined first, and the weights and intensity of the effects of main factors are determined. In the next step, the weights of these variables are calculated as the optimal solutions to the problem. In other words, the DEA method is used to identify the weights of variables (sub-factors) and the DEMATEL technique is applied to determine the weights of main factors that no studies have been conducted in this way so far. The review of the relevant studies shows that there has been no research that evaluated the effectiveness of on-the-job training of employees with a single DEA model based on knowledge management. The contribution of this study is to use the DEMATEL technique and DEA method to measure and evaluate the effectiveness of on-the-job training courses in the organization.
2.2Two-stage DEA model
Data Envelopment Analysis (DEA) method is used to measure the relative efficiency of homogeneous decision-making units (DMUs) that have multiple inputs and outputs. Several studies have considered two stages for DMUs so that in the first stage the initial inputs are converted to the outputs which are used as the inputs to the second stage. The outputs of the first stage are called intermediate products. Figure 1 shows a two-stage network structure.
Figure 1.
Two-stage network structure.
Fare and Grosskopf [29] indicated the importance of the network DEA method. Lewis and Sexton [3] used the network DEA method to formulate a network of related units for analyzing the performance of organizations in which some of the resources generated are consumed by other units. Kao [31] proposed a model for performance analysis with the network DEA model so that each network system is transformed into series systems using virtual processes, each step of which includes a parallel structure. Wolszczak-Derlacz [32] used a two-stage DEA model to quantitatively assess the performance of a number of European and American higher education institutions with different inputs and outputs. The results showed that the amount of government investment had a negative impact on the performance of European educational institutions, which was not the case with American institutions. Afsharian Ahn, and Thanassoulis [33] evaluated the motivational system in centralized multi-unit organizations such as banks and public health centers through the DEA model.
3.Material and method
This study is basic research; as mathematical programming model is proposed to evaluate the effectiveness of on-the-job training courses for the personnel of the Ministry of Agriculture. It is also practical; because this approach is used for all higher education centers that are under the supervision of the Ministry of Agriculture. Planning and implementing training courses and evaluating their effectiveness are extremely important for empowering the human resources of the agricultural sector and ultimately fulfilling the missions of the Ministry of Agriculture. Hence, the duties of the higher education centers of the Ministry of Agriculture have been defined in the form of improving productivity, updating job skills, facilitating the process of transferring knowledge and new technologies, developing the capacities of agricultural education and promotion, and establishing knowledge management and agricultural information network with the aim of achieving knowledge-based agriculture.
The first phase of this research involves the Fuzzy DEMATEL technique. The DEMATEL method is used for two purposes: first, to determine the weights of the main variables of the model and second, to calculate the dependency matrix between the main factors to identify the causal relationships and how these main variables affect each other. The DEMATEL technique is a type of decision-making method based on pairwise comparisons introduced by Fontela and Gabus in 1976 [34] This technique provides a hierarchical structure of existing factors together with the causal relationships among them through the expert judgement. The Fuzzy DEMATEL technique comprises five steps as follows: (1) Formation of the initial direct-relation matrix between system factors, (2) Defuzzification of the matrix, (3) Normalization of the matrix, (4) Calculation of total relation matrix, and (5) Preparation of causal diagram and calculation of threshold values [35].
In the second phase, the data of the previous phase are analyzed and the research model is designed. The steps of this phase are as follows:
• Selection of input and output indicators of the two-stage model.
• Implementation of the knowledge management model by using the variables of the Knowledge Management and Innovation Model of the European Union and also consulting experts. According to the availability of information, input and output indicators of the research are determined and adapted to the current educational system.
• Collection of data and statistical information of provincial higher education centers of the Ministry of Agriculture in 2021.
• Determination of the weights of sub-variables by solving the model in MATLAB software.
• Calculation of the efficiency
To design the research model, almost all two-stage DEA models were reviewed, and consequently a comprehensive and practical model was defined for the organization under study through adapting all components and indicators. This model includes the following two steps:
Stage one: The output of the first stage of the model (the components of knowledge development) actually indicates how to implement the knowledge management process in the organization. The indicators of the three main phases of the knowledge management process are given in the outputs of the first stage of the model. These three main phases are “knowledge creation”, “knowledge dissemination” and “knowledge application” which are specified in the output components of the first stage through the indicators shown in Table 1 and extracted from the valid model of measuring knowledge management and innovation in the European Union and adapted to the current education system.
Table 2
First stage variables
| Input variables | X1 | 1-Number of educational classes |
| X2 | 2-Number of laboratories, workshops and practical training environments | |
| X3 | 3- Capacity of computer sites | |
| X4 | 4-Number of professors (faculty and guests) | |
| X5 | 5-Personnel working in the staff training department | |
| Output variables | Z1 | 1-Manager-teacher (teachers with management background) |
| Z2 | 2-Researcher-teacher (teachers with exclusive innovative, articles and books) | |
| Z3 | 3-Specialized and sectoral audiences (areas that have outputs during the courses) | |
| Z4 | 4-Per capita education |
Stage two: The Kirkpatrick model, which has been used as one of the most famous models to evaluate and measure human resource management for more than 30 years, is exploited in the output of the second stage of the model Quantifying and measuring the qualitative variables and using them in the quantitative research model is one of the contributions of the current study. It should be noted that the fourth level of the Kirkpatrick model is "results", which can be compared with the expected results of the decision-making. It is very difficult to measure the fourth level in which the evidence of results such as cost reduction, rework, replacement, accident and quality increase are examined In the present research, due to the aforementioned reason and lack of sufficient information, the evaluation of the “results” section has been dismissed.
Table 3
Second stage variables
| Input variables | Z1 | 1-Manager-teacher (teachers with management background) |
| Z2 | 2-Researcher-teacher (teachers with exclusive innovative, articles and books) | |
| Z3 | 3-Specialized and sectoral audiences (areas that have outputs during the courses) | |
| Z4 | 4-Per capita education | |
| Output variables | Y1 | 1-Reaction |
| Y2 | 2-Learning | |
| Y3 | 3-Behavior changes (performance) |
3.1Quantification and analysis of the last level of the model
Measuring the level of reaction: Reaction level analyzes are performed separately for each factor (content, teacher and organizing) for all courses in the whole country, a specific course in an educational center, and a comparison is made according to the educational centers, and etc. All required analyzes are performed through drawing quality control charts with the Minitab software. These control analyzes are subject to statistical quality control methods. Based on the above analyzes, the upper and lower limits are determined. If the obtained score is less than the lower limit, the course is considered out of control and tracking is needed to find out the cause(s)
Assessing the level of learning: Pre and post-course tests are used to assess the level of knowledge acquisition. Subsequently, the score of each test is calculated, and using the learning score formula, the amount of change in the knowledge level is measured comparing to before training.
Behavior measurement: Each training course has a separate questionnaire for assessing this level. These questionnaires are based on the five-point Likert scale and include five common questions and several other questions, in which, code 1 was assigned to “very weak”, code 2 to “weak”, code 3 to “average”, code 4 to “good”, and code 5 to “very good”. Subsequently, the score of each questionnaire is calculated using the following formula for measuring the change in behavior:
The functional model considered for this research is depicted in Fig. 2.
Figure 2.
Functional model of research.
The assumptions of the model are as follows:
1. The return to scale is constant.
2. All inputs and outputs are non-negative.
3. There are no zero and negative inputs.
4. The weights of inputs and outputs are different in each stage.
(1)
(2)
If the efficiency of each stage is separately considered, we have:
(3)
In addition, the efficiency of each stage is equal to:
(4)
(5)
4.Results
In order to measure the effectiveness of on-the-job training courses, the desirability of the courses is calculated based on the Kirkpatrick model in the higher provincial education centers of the Ministry of Agriculture. In evaluating the reaction, the importance weights of three components of content, teacher, and organizing to express the level of satisfaction with the training course and course evaluation should not be equal. Due to the fact that the purpose of the training course is to increase the level of knowledge, the effect of the course teacher is more than the content and organizing. The desirability of the training course is obtained based on the importance weights of 20% for content, 60% for teacher, and 20% for organizing. For virtual training courses, the teacher component is removed and a number of questions are added, and the desirability is obtained based on the importance weights of 50% for content and 50% for organizing.
The main factors of the model in evaluating the effectiveness of the training courses held together with the effectiveness of each course are identified using the Fuzzy DEMATEL technique, shown in the following figures and tables. After forming the direct-relation matrix between the main factors of the system and defuzzifying the experts’ opinions about the main factors, the initial direct-relation matrix is normalized. The normalized matrix is obtained using the Eq. (4).
(6)
Table 4
The main factors of the model
| Symbol | The main factors of the model |
|---|---|
| C1 | Educational resources and facilities |
| C2 | Knowledge development (implementation of knowledge management process) |
| C3 | Educational effectiveness |
Table 5
The degree of influence of the factors on each other
| Influence | R-D | R+D | Column sum (D) | Row sum (R) | Factor |
|---|---|---|---|---|---|
| C1 | 2.4803 | 0.8432 | 3.3235 | 1.6370 | 1 |
| C2 | 1.7930 | 2.0069 | 3.7999 | 2 | |
| C3 | 1.1708 | 2.5940 | 3.7648 | 3 |
Figure 3.
Defining relationships between variables based on the DEMATEL technique.
According to the above tables and figures, the most influential main variable of the model is “educational resources and facilities” and the most affected main variable is “educational effectiveness”. Once efficient and inefficient units are identified, inefficient units should pay attention to the most affecting and least affected factors to improve their performance. Subsequently, the weights of the sub-factors of the model (the weights of the sub-variables of the problem in each stage) were determined using the DEA method and solving the research model in the MATLAB software, finally, the efficiency of each stage and the total efficiency were obtained. Inefficient units were discarded and the efficient units were ranked. Table 6 shows the efficiency of provincial higher education centers and their final rankings.
Table 6
Efficiency of provincial higher education centers and their rankings
| Higher education center number (efficient unit) |
| Final ranking | ||
| 20 | 7.16 | 1 | ||
| 3 | 6.34 | 2 | ||
| 6 | 5.65 | 3 | ||
| 12 | 5.06 | 4 | ||
| 19 | 5.05 | 5 | ||
| 28 | 4.69 | 6 | ||
| 26 | 4.26 | 7 | ||
| 23 | 4.07 | 8 | ||
| 21 | 4.06 | 9 | ||
| 30 | 3.52 | 10 | ||
| 8 | 3.19 | 11 | ||
| 9 | 3 | 12 | ||
| 14 | 2.98 | 13 | ||
| 2 | 2.82 | 14 | ||
| 10 | 2.79 | 15 | ||
| 18 | 2.61 | 16 | ||
| 16 | 2.50 | 17 | ||
| 13 | 2.34 | 18 | ||
| 24 | 2.28 | 19 | ||
| 31 | 2.20 | 20 | ||
| 22 | 2.03 | 21 | ||
| 11 | 2 | 22 | ||
| 29 | 1.86 | 23 | ||
| 4 | 1.76 | 24 | ||
| 1 | 1.71 | 25 | ||
| 15 | 1.36 | 26 | ||
| 7 | 1.18 | 27 |
5.Discussion and conclusion
The most important social capital of organizations is their human resources. Today, learning and progressive knowledge-based organizations need people with intellectual knowledge and practical skills. Therefore, organizations are paying more attention to creating the appropriate training systems for continuous updating of knowledge and skills of employees. The on-the-job training system consists of four operational sections: design, selection, implementation and monitoring, and evaluation and modification of training courses. One of the important goals of any training is to ensure that the training process is provided effectively and efficiently. In fact, education is the improvement of the required attitude, knowledge, skills and behavioral patterns of a person to perform well in a given task or job. Effective training is a process that can take responsibility for changes in human resources in accordance with changes in structure, culture, technology and product to adapt the organization to the impacts of driving factors.
The purpose of this study was to evaluate the effectiveness of on-the-job training courses for employees in the Ministry of Agriculture through providing a DEA model that has been adapted to the system under study. The proposed model was based on the concepts of the knowledge management model and its indicators have been derived from the EU model. One of the topics that has been addressed in recent years is the measurement of efficiency of decision-making units (DMUs) with a two-stage network structure, in which the first stage outputs are used as the second stage inputs. Network DEA models measure the overall efficiency of the organization and the efficiency of each of the sub-processes of the organization In this study, the factors of educational resources and facilities, knowledge development and the components of the effectiveness evaluation of training courses were assessed simultaneously In addition, a two-stage series model was proposed to measure the efficiency of higher education centers under the supervision of the Ministry of Agriculture. In the present research, the DEA model which was based on an organizational knowledge management model was applied to answer the aforementioned research question by examining the causal relationships and interdependencies in the decision-making units (DMUs).
In this study, the proposed model was developed in two stages as follows: In the first stage, the three main phases of the knowledge management process (knowledge creation, dissemination and application) have been considered. In the second stage, the Kirk Patrick model as one of the most well-known models for evaluating and measuring human resource management has been exploited “Educational resources and facilities” has been identified as the most influential variable and “educational effectiveness” has been identified as the most influential variable. According to the results, the organization should pay special attention to training and improving the knowledge of employees, design practical contents in relation with the job requirements of learners, and implement training courses based on needs assessment. In addition, in the training courses, attention should be paid to the desired level for staff improvement and the personnel should be assured that their performance is the main criterion for evaluation. On the other hand, the design of training courses should be in accordance with the job skills of employees in order to motivate learners to make the most of training courses. Also, effective and continuous communication should be formed with scientific and educational centers. Moreover, the opinions of teachers and personnel who passed the courses should be exploited in designing the following training courses.
The results of this research can be used as a framework for necessary reviews in the fields of needs assessment, design, presentation and evaluation of on-the-job training courses. Lack of the relevant resources and how to evaluate on-the-job training of employees can be stated as one of the limitations of the current study. As some suggestions for future research, the Nonaka knowledge management model can be used. Also, the structural equation modelling (SEM) method may be exploited to determine the relationships between the variables.
References
[1] | Noe RA, Kodwani AD. Employee training and development, 7e. McGraw-Hill Education, (2018) . |
[2] | Wang S. Knowledge maps for managing webbased business. Industrial Management & Data Systems. (2002) ; 102: (7): 357-364. |
[3] | Swanson RA, Holton E. Foundations of human resource development. |
[4] | Omar M, Gerein N, Tarin E, Butcher, C, Pearson S, Heidari G. Training evaluation: a case study of training Iranian health managers. Human resources for Health, (2009) ; 7: (1): 1-14. |
[5] | Chizari M, Lindner JR, Karjoyan S. Factors affecting involvement of volunteers in extension education activities in Talesh Township, Iran. Journal of Agricultural Education. (1999) ; 40: : 61-62. |
[6] | Wiig KM. Knowledge management: an introduction and perspective. Journal of knowledge Management. (1997) ; 1: (1): 6-14. |
[7] | Darroch J. Knowledge management, innovation and firm performance. Journal of Knowledge Management. (2005) ; 9: (3): 101-115. |
[8] | Dalkir K. Knowledge management in theory and practice. |
[9] | Carlsson BS, Jacobsson M, Holmén M, Rickne A. Innovation systems: analytical and methodological issues. Research Policy. (2002) ; 31: (2): 233-245. |
[10] | Werner JM, DeSimone RL. Human resource development. |
[11] | Hashimoto M. Investment in Employment Relations in Japanese Firms. Journal of the Japanese and International Economies. (1995) ; 9: (1): 75-95. |
[12] | Youens B, Smethem L, Simmons M. Move over Nelly: lessons from 30 years of employment-based initial teacher education in England. Teachers and Teaching. (2018) ; 24: (7): 854-869. |
[13] | Bybee RW, Taylor JA, Gardner A, Van Scotter AP, Powell JC, Westbrook A, Landes N. The BSCS 5E instructional model: origins, effectiveness, and applications. Colorado Springs: BSCS. International Journal of Man-Machine Studies. (2006) ; 29: : 407-427. |
[14] | Curran VR, Fleet L. A review of evaluation outcomes of webbased continuing medical education. Medical Education. (2005) ; 39: (6): 561-567. |
[15] | Lee H, Song Y. Kirkpatrick Model Evaluation of Accelerated Second-Degree Nursing Programs: A Scoping Review. Journal of Nursing Education. (2021) ; 60: (5): 265-271. |
[16] | Kaufman R, Keller JM. Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly. (1994) ; 5: (4): 371-80. |
[17] | Kirkpatrick DL. Great ideas revisited: revisiting Kirkpatrick’s four-level model. Training and Development. (1996) ; 50: (1): 54-57. |
[18] | Praslova L. Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability. (2010) ; 22: (3): 215-225. |
[19] | Alsalamah A, Callinan C. Adaptation of Kirkpatrick’s Four-Level Model of Training Criteria to Evaluate Training Programmes for Head Teachers. Education Sciences. (2021) ; 1: (3): 116. |
[20] | Hale R. How training can add real value to the business: part 1. Industrial and commercial Training. (2003) ; 35: (1): 29-32. |
[21] | Hale R. How training can add real value to the business: part 2. Industrial and Commercial Training. (2003) ; 3: (2): 49-52. |
[22] | Bartel AP. Productivity gains from the implementation of employee training programs. Industrial Relations: A Journal of Economy and Society. (1994) ; 33: (4): 411-425. |
[23] | Dilley J, Wolf KL. Homeowner interactions with residential trees in urban areas. Arboriculture and Urban Forestry. (2013) ; 39: (6): 267-277. |
[24] | Shakoor A, Ghumman MS, Mahmood T. Effect of in-service training on the working capacity and performance of science teachers at secondary level. Journal of Educational and Social Research. (2013) ; 3: (3): 337. |
[25] | Borate N, Gopalkrishna, D, Shiva Prasad HC, Borate S. A case study approach for evaluation of employee training effectiveness and development program. The International Journal of Business & Management. (2014) ; 2: (6): 201-211. |
[26] | Gregory M, Leslie TW, Drinkwater LE. Agroecological and social characteristics of New York city community gardens: contributions to urban food security, ecosystem services, and environmental education. Urban Ecosystems. (2016) ; 19: (2): 763-794. |
[27] | Jeong SJ, Choi SJ. Agricultural literacy in the context of agricultural education in South Korea: using hierarchical linear modeling. The Journal of Agricultural Education and Extension. (2020) ; 26: (4): 401-419. |
[28] | Ardahan-Sevgili S, Yardimci F. Educational Study Based on Kirkpatrick’s Model for Reducing False Positive Blood Cultures. International Journal of Caring Sciences. (2020) ; 13: (3): 1604. |
[29] | Fare R, Grosskopf S. Network DEA. Socio-economic Planning Sciences. (2000) ; 34: (1): 35-49. |
[30] | Lewis HF, Sexton TR. Network DEA: efficiency analysis of organizations with complex internal structure. Computers & Operations Research. (2004) ; 31: (9): 1365-1410. |
[31] | Kao C. Efficiency decomposition in network data envelopment analysis: A relational model. European Journal of Operational Research. (2009) ; 192: (3): 949-962. |
[32] | Wolszczak-Derlacz J. An evaluation and explanation of (in) efficiency in higher education institutions in Europe and the US with the application of two-stage semi-parametric DEA. Research Policy. (2017) ; 46: (9): 1595-1605. |
[33] | Afsharian MH, Thanassoulis E. A DEA-based incentives system for centrally managed multi-unit organisations. European Journal of Operational Research. (2017) ; 259: (2): 587-598. |
[34] | Fontela E, Gabus A. The DEMATEL observer, DEMATEL 1976 report. Battelle Geneva Research Center, Switzerland Geneva, (1976) . |
[35] | Alinezhad A, Khalili J. DEMATEL Method. In New Methods and Applications in Multiple Attribute Decision Making (MADM) (pp. 103-108). Springer, Cham, (2019) . |
