Evaluating the performance of healthcare research institutions is exceptionally significant and of vital importance for the further development of Russia’s research potential. The importance of this analysis was dictated by the need to increase efficiency amidst modernization of the public sector of science [1, 2]. On the other hand, modern theoretical and practical developments and tools used to assess the output of medical research institutions often do not facilitate comprehensive monitoring of their activities.

The peculiarity of the Russian methodological approach to analysis of the performance of healthcare research organizations is that a large number of criteria are used, unlike what is obtainable in a number of Western countries [3, 4]. A typical evaluation method applied in Russia [2] is based predominantly on quantitative rather than qualitative indicators. Additional criteria are used for systemic monitoring, including for institutions with bed space facilities. Since the outcome of an analysis directly determines financing decisions, identification of key criteria is particularly important.

The activities of medical research institutions have characteristic specificity and important differences from the activities of other research institutions [5, 6]. Performance indicators approved for healthcare research institutions include monitoring of the following areas:

• output and relevance of scientific research;
• human resource development;
• integration into the international scientific community, dissemination of scientific knowledge and promotion of the prestige of science;
• resource support for the activities of the research institution.

METHODS

Performance indicators for 55 research institutions subordinated to the Russian Ministry of Health for the period 2011–2015 were investigated.In accordance with the accepted classification, the above research institutions subordinated to the Russian Health Ministry (n = 55) were divided into three groups: those with bed space facilities (n = 41), those without bed space facilities (n = 4), expert and socially important organizations (n = 10).

Statistical analysis was performed using software package Statistica 10.0. The Kolmogorov–Smirnov test by Lilliefors was used to test statistical hypotheses about type of distribution. It was found that in all cases, the data were not subject to the normal law of distribution; so nonparametric tests were used for statistical analysis.

Kruskal–Wallis test and the median test were used to determine the key performance indicators for healthcare research organizations. Additionally, the years 2011 and 2015 were compared in terms of Mann–Whitney U test and it was found that these figures change sufficiently enough.

Correlation analysis was performed using the Spearman’s rank correlation coefficient. Indicators that change over time were included in the correlation analysis with background indicators for 2011, since it is the starting point of reference. P-value < 0.05 was taken as the level of statistical significance.

RESULTS 

Most research institutions (74.5 %) under the Russian Ministry of Health have bed space facilities. However, this does not simplify the assessment procedure — you cannot apply indicators characterizing healthcare delivery to the other research organizations. In this regard, additional criteria are applied in analyzing the performance of research institutions with bed space facilities.

Statistical analysis of the 5-year dynamics (2011–2015) of performance criteria uncovered 16 key performance indicators (p < 0.05). The structure of the criteria earmarked and their proportion in the total number of parameters are shown in fig. 1 and fig. 2.

For the period 2011–2015, the following key indicators were identified.

Key performance indicators and relevance of scientific research:

1. Cumulative impact factor of journals where the articles of the organization are published. It should be noted that this indicator was found to have a moderate correlation with the number of Doctors of Sciences (the highest academic degree in Russia and many other post-Soviet states obtained after obtaining a PhD degree) (r_s= 0.455, p < 0.01), with intramural (r_s = 0.449, p < 0.01) and extramural (r_s = 0.411, p < 0.01) current expenditure on scientific research and development, and with the number of articles of the organization published in journals indexed in Web of Science (r_s = 0.406, p < 0.01).

2. Number of used intellectual property transferred under a license agreement. It was found that this indicator has moderate correlation with intramural current expenditure on research and development, including expenditure on exploratory research (r_s= 0.475, p < 0.01).
3. Number of used intellectual property contributed into the authorized capital.
4. Number of small innovative enterprises created with the participation of that institution.
5. Cumulative average staff number at small innovative enterprises.
6. Cumulative income at small innovative enterprises.
7. Financial impact of the research institution by income sources. This indicator was found to have a weak relationship with intramural current expenditure on research and development (r_s= 0.302, p < 0.05), including a positive correlation with basic research (r_s = 0.351, p < 0.01) and negative correlation with exploratory research (r_s = –0.324, p < 0.05).

Key indicator: integration into the global research community, dissemination of scientific knowledge and enhancement of prestige of science 

The number of traffic (visits) to official sites and/or web pages of the institution on the Internet is defined by a single key indicator. The criterion was found to have a moderate correlation with the number of positive and neutral mentions of the institution in the federal media (r_s = 0.357, p < 0.01), including in online publications (r_s = 0.325, p < 0.05), and with the number of publications of the organization in journals indexed in the Russian Science Citation Index (r_s = 0.339, p < 0.05).

Key indicators: resource support for the research institution

1. Expenditure on fixed assets and intangible assets, including buildings and structures, machinery and equipment. The indicator was found to have a moderate correlation with financial performance of research institution by type of work performed and services rendered, including production goods, works and services (r_s= 0.491, p < 0.01).
2. Intramural current expenditure on basic research. This indicator correlated with the financial performance of research institution by type of work performed and services rendered: by number of research and developments (r_s= 0.548, p < 0.01); with number of employees engaged in research and development (r_s = 0.516, p < 0.01), including researchers (r_s = 0.418, p < 0.01), among whom are PhD holders (r_s = 0.405, p < 0.01) and Doctors of Sciences (r_s = 0.368, p < 0.01) aged not above 39 years (r_s = 0.505, p < 0.01).

All the key indicators of resource support for a research institution were interconnected with the financial performance of the research institution by type of work performed, by services rendered and by income sources.

Key additional performance indicators for a research institution:

1. Number of research critical technologies from the list approved by the Scientific Council of the Russian Ministry of Health. This indicator correlated with the number of publications in journals indexed in Scopus (r_s= 0.367, p < 0.01), and with the number of innovative medical technologies used at the institution and approved by the Scientific Council of the Russian Ministry of Health (r_s = 0.356, p < 0.01). The indicator was found to be weakly correlated with the number of positive and neutral mentions of the organization in the federal media (r_s = 0.295, p < 0.05), including in the federal print media, television and radio (r_s = 0.291, p < 0. 05), and with the number of researchers sent to work in leading Russian and international research and educational organizations (r_s = 0.268, p < 0.05).
2. The proportion of highly skilled medical doctors out of the total number of medical doctors. Curiously, this indicator was found to have a negative correlation with the total number of scientific, design and technological works (r_s= –0.273, p < 0.05), with the number of positive and neutral mentions of the organization in the media (r_s = –0.311, p < 0.05), with financial performance of the research institution by type of work performed and services rendered (by research and development) (r_s = –0.308, p < 0.05), and with the number of prepared draft healthcare delivery standards and procedures, clinical practice guidelines and other regulations (r_s = –0.276, p < 0.05).
3. Number of innovative medical technologies used at the institution and approved by the Scientific Council of the Russian Ministry of Health. This indicator correlated with the number of publications of the organization in journals indexed in Scopus (r_s= 0.367, p < 0.01), with the number of intellectual property created (r_s = 0.406, p < 0.01), including those with state registration and/or legal protection in Russia (r_s = 0.392, p < 0.01), and with the percentage of residents of other regions that received high-tech health care (HHC) at that institution (r_s = 0.360, p < 0.01).
4. Federal nature of the institution: the percentage of people from other regions who have received specialized medical care in that institution. This indicator moderately correlated with the number of people from other regions who received HHC (r_s= 0.474, p < 0.01), and with the number of Federal subjects of Russia, whose residents were treated in the reporting year (r_s = 0.437, p < 0.01).
5. Percentage of medical care expenses under mandatory health insurance (percentage of the total cost). This indicator correlated with the number of intellectual property created (r_s= 0.350, p < 0.01), including those with state registration and/or legal protection in Russia (r_s = 0.459, p < 0.01), with financial performance of the research institution by income sources (r_s = 0.369, p < 0.01), with intramural current expenditure on basic research (r_s = 0.417, p < 0.01).
6. Number of prepared draft healthcare delivery standards, draft healthcare delivery procedures, clinical practice guidelines (treatment protocols) and other regulations. This indicator was found to be correlated with the number of publications of the organization in journals indexed in Scopus (r_s= 0.384, p < 0.01), with the financial performance of the research organization by type of work performed and services rendered, including educational services (r_s = 0.394, p < 0.01), specialist training under continuing professional education programs on unique technologies along the profile of that research institution (r_s = 0.449, p < 0.01), and with the percentage of residents of other regions who received HHC (r_s = 0.358, p < 0.01).

DISCUSSION

Analysis of performance indicators for research institutions subordinated to the Russian Health Ministry identified 16 key criteria, most (43.75 %) of which characterize the output and relevance of scientific research.

Scientometric analysis, which is of particular importance for systemic evaluation [7, 8, 9], revealed that among the criteria analyzed (number of publications in journals indexed in the Russian Science Citation Index, Web of Science and Scopus, total number of citations of publications of the institution, etc.), the cumulative impact factor of journals where the articles of the organization are published is a key indicator. This is probably due to a more careful selection of journals — since publications in top-ranking journals, including foreign journals, increase not only the citation of authors, but also the interest of foreign colleagues in Russian research and future collaboration.

The innovative development model of Russian science involves closer relationship with the economic sector [10]. Against this background, a special role belongs to small innovative enterprises and key indicators reflecting their activities. However, over the studied period — 2011–2015 — medical research institutions showed virtually no interest in commercialization of their intellectual products. It was found that small innovative enterprises created only 14.5 % of organizations with bed space facilities.

The standard human resource evaluation for healthcare research organizations includes four quantitative indicators, none of which is a key indicator. The absence of qualitative factors and the different importance of investigated parameters create obstacles to a full-fledged analysis. According to a number of researchers [5, 11], an effective human resource evaluation requires a systemic approach, which includes monitoring of professional competence, learning ability, conditions and wages, and many other indicators.

Since the output of institutions depends on the research performed and other works related to the main activities of the institutions, as well as on conformity of the state of national science to modern world standards [12, 13, 14, 15], there is need to develop quality criteria for a particular area.

During the period under consideration — 2011–2015 — despite a decline in the number of mentions of research organizations in the media, a sharp increase in visits to the official websites of medical research institutions was revealed. Presumably, this is due to increased online activity by interested persons.

The use of additional performance criteria for evaluation of healthcare research organizations has proved its worth — they accounted for 37.5 % of the total number of key indicators. It is curious to note that the criterion reflecting the number of used scientific critical technologies from the list approved by the Scientific Council of the Russian Health Ministry correlates with the number of publications of the organization in journals indexed in Scopus and with the number of researchers sent to work in leading Russian and international research and educational organizations.

CONCLUSIONS

In assessing the performance of healthcare research organizations in the period 2011–2015, 16 key performance indicators were identified, among which are especially significant criteria, reflecting the output and relevance of scientific research.

During the evaluation, the additional criteria were found to be relevant. Data obtained clearly show that the used research critical and innovative medical technologies from the list approved by the Scientific Council of the Russian Health Ministry, prepared draft healthcare delivery standards, procedures and clinical practice guidelines are crucially important for further development of the scientific potential of institutions.

For institutions with bed space facilities, the proportion of highly qualified medical doctors, the percentage of people from other regions who have received specialized medical care, and the percentage of medical care expenses under mandatory health insurance served as additional key indicators.

It was shown that the existing standard indicators for human resource analysis are insufficient and that there is need to come up not only with quantitative but also with qualitative criteria.