ArticlePDF AvailableLiterature Review

The national Indigenous health performance measurement system

Authors:

Abstract

This article reviews the development of the national Indigenous performance measurement system over the last decade. Data were collected from the published and unpublished literature and review of government websites, facilitated by key informant interviews which provided information about the policy context. A number of innovations have occurred over the last decade, including the development of a conceptual framework to underpin a system-wide approach to performance measurement that is aligned with nationally agreed strategic goals. The development of mechanisms to oversee Indigenous health strategy and health data development create formal mechanisms that potentially link data development and performance measurement priorities. Innovation in the development of processes to support health system performance improvement is evident, but this needs to be prioritised, particularly with respect to those components of the health system that are not Indigenous-specific.
A preview of the PDF is not available
... Despite expanding scholarship and related ethical, legal, and policy recommendations that challenge the historic and ongoing marginalization of Indigenous peoples in the planning, implementation, and ongoing management and governance of Indigenous health information systems [46][47][48][49][50], substantive Indigenous participation continues to be the exception rather than the norm. The first author's systematic review of Indigenous involvement in Indigenous health information systems in Canada and internationally [51,52] identified some advances both domestically and globally , but also important gaps. Serious deficits were identified in the development of local and regional health information systems, feedback of health mation to these systems, and the processes of Indigenous input into these systems, including the application of Indigenous-specific frameworks and indica- tors [51]. ...
... The first author's systematic review of Indigenous involvement in Indigenous health information systems in Canada and internationally [51,52] identified some advances both domestically and globally , but also important gaps. Serious deficits were identified in the development of local and regional health information systems, feedback of health mation to these systems, and the processes of Indigenous input into these systems, including the application of Indigenous-specific frameworks and indica- tors [51]. These findings are not surprising since the large majority of policy makers, practitioners and researchers involved in the development of population health datasets have had limited exposure to Indigenous peoples and their diverse systems of knowledge and practice with respect to health. ...
... Métis regions across the country have also been working on health information systems development, with Métis specific provincial health surveys (convenience samples) completed in BC and Saskatchewan and linkages with provincial health data bases in Manitoba and On- tario [66,67]. Globally, Indigenous scholars and policymakers have recommended an approach that incorporates Indigenous specific measures to complement more universally accepted measures [51,68,69]. There are a number of recent survey initiatives in Canada that have adapted this approach, including the Aboriginal Children's Survey [70], the Inuit Health Survey [71], the Our Health Counts Survey [72] and the First Nations Regional Longitudinal Health Survey [73]. ...
Article
Full-text available
Canada is known internationally for excellence in both the quality and public policy relevance of its health and social statistics. There is a double standard however with respect to the relevance and quality of statistics for Indigenous populations in Canada. Indigenous specific health and social statistics gathering is informed by unique ethical, rights-based, policy and practice imperatives regarding the need for Indigenous participation and leadership in Indigenous data processes throughout the spectrum of indicator development, data collection, management, analysis and use. We demonstrate how current Indigenous data quality challenges including misclassification errors and non-response bias systematically contribute to a significant underestimate of inequities in health determinants, health status, and health care access between Indigenous and non-Indigenous people in Canada. The major quality challenge underlying these errors and biases is the lack of Indigenous specific identifiers that are consistent and relevant in major health and social data sources. The recent removal of an Indigenous identity question from the Canadian census has resulted in further deterioration of an already suboptimal system. A revision of core health data sources to include relevant, consistent, and inclusive Indigenous self-identification is urgently required. These changes need to be carried out in partnership with Indigenous peoples and their representative and governing organizations.
... 38,88 Comprehensive, consistent, and coherent health data are crucial for decision making across the health sector for all populations, including Indigenous peoples. 92,93 Accordingly, we recommend action against four interrelated priorities for Indigenous data development. Action on these recommendations is also needed to strengthen the monitoring of the SDGs, and to extend the coverage of Indigenous populations and health indicators in future international studies such as this. ...
... Ascertainment bias (usually under-reporting of Indigenous status) is a key issue that has been shown through data linkage, although the extent of this problem might vary between databases. [27][28][29][93][94][95][96][97][98] Consistent and comparable data defi nitions of Indigenous status present a challenge that needs to be addressed. Internationally, Indigenous data defi nitions based on social identity have increasingly replaced questions based on ancestry, with an international survey of census questionnaires 99 (from 1994 to 2008) documenting the variability in the recording of Indigenous status. ...
Article
Background: International studies of the health of Indigenous and tribal peoples provide important public health insights. Reliable data are required for the development of policy and health services. Previous studies document poorer outcomes for Indigenous peoples compared with benchmark populations, but have been restricted in their coverage of countries or the range of health indicators. Our objective is to describe the health and social status of Indigenous and tribal peoples relative to benchmark populations from a sample of countries. Methods: Collaborators with expertise in Indigenous health data systems were identified for each country. Data were obtained for population, life expectancy at birth, infant mortality, low and high birthweight, maternal mortality, nutritional status, educational attainment, and economic status. Data sources consisted of governmental data, data from non-governmental organisations such as UNICEF, and other research. Absolute and relative differences were calculated. Findings: Our data (23 countries, 28 populations) provide evidence of poorer health and social outcomes for Indigenous peoples than for non-Indigenous populations. However, this is not uniformly the case, and the size of the rate difference varies. We document poorer outcomes for Indigenous populations for: life expectancy at birth for 16 of 18 populations with a difference greater than 1 year in 15 populations; infant mortality rate for 18 of 19 populations with a rate difference greater than one per 1000 livebirths in 16 populations; maternal mortality in ten populations; low birthweight with the rate difference greater than 2% in three populations; high birthweight with the rate difference greater than 2% in one population; child malnutrition for ten of 16 populations with a difference greater than 10% in five populations; child obesity for eight of 12 populations with a difference greater than 5% in four populations; adult obesity for seven of 13 populations with a difference greater than 10% in four populations; educational attainment for 26 of 27 populations with a difference greater than 1% in 24 populations; and economic status for 15 of 18 populations with a difference greater than 1% in 14 populations. Interpretation: We systematically collated data across a broader sample of countries and indicators than done in previous studies. Taking into account the UN Sustainable Development Goals, we recommend that national governments develop targeted policy responses to Indigenous health, improving access to health services, and Indigenous data within national surveillance systems. Funding: The Lowitja Institute.
... The endorsement of the United Nations' 2007 Declaration on the Rights of Indigenous Peoples [1] by most circumpolar countries (with the exception of Russia), and current international discourses on the importance of health equity, have led to important debates in Indigenous communities. These conversations have centered on what indicators best capture the health of Indigenous peoples, and which investigate existing disparities [2][3][4][5][6][7][8]. This work has critiqued the quasiexclusive use of classic epidemiological indicators (suicide rates, obesity rates, infant mortality rates, etc.), and advocated for the addition of strengths-based indicators that can be used by communities as pathways on which to build successful health and wellness programs [7,[9][10][11]. ...
Article
Aims: Historically, health research in the Arctic has focused on documenting ill-health using a narrow set of deficit-oriented epidemiologic indicators (i.e., prevalence of disease and mortality rates). While useful, this type of research does not adequately capture the breadth and complexities of community health and well-being, and fails to highlight solutions. A community's context, strengths, and continued expressions of well-being need to guide inquiries, inform processes, and contextualize recommendations. In this paper, we present a conceptual framework developed to address the aforementioned concerns and inform community-led health and social research in the Arctic. Methods: The proposed framework is informed by our collective collaborations with circumpolar communities, and syntheses of individual and group research undertaken throughout the Circumpolar North. Our framework encourages investigation into the contextual factors that promote circumpolar communities to thrive. Results: Our framework centers on the visual imagery of an iceberg. There is a need to dive deeper than superficial indicators of health to examine individual, family, social, cultural, historical, linguistic, and environmental contexts that support communities in the Circumpolar North to thrive. A participatory community-based approach in conjunction with ongoing epidemiologic research is necessary in order to effectively support health and wellness. Conclusions: The iceberg framework is a way to conceptualize circumpolar health research and encourage investigators to both monitor epidemiologic indicators and also dive below the surface using participatory methodology to investigate contextual factors that support thriving communities.
... Responses include: 1"We have a lot of savings", 2"We have some savings", 3"We have just enough to get us to the next payday", 4"We run out of money before payday", 5"We are spending more than we get", 6"Unsure". These were grouped as "Savings" [1,2], "Just enough" [3], "No savings" [4,5], Unsure [6]. First language was defined as: "Aboriginal or Torres Strait Islander language", "English" or "other". ...
Article
Full-text available
Background: Evidence on the effectiveness of postal recruitment methods for Indigenous peoples is lacking. Mayi Kuwayu, the National Study of Aboriginal and Torres Strait Islander Wellbeing, uses multi-staged sampling. We aimed to test postal surveys as a primary recruitment method, analysing preliminary response rate data to inform the Study's ongoing sampling approach. Methods: Twenty thousand adults aged ≥16 years were sampled from Aboriginal and Torres Strait Islander people enrolled in the Medicare Australia Enrolment Database. We calculated response rates at 4 and 15 weeks, overall and by age group, gender, state/territory and remoteness. Results: The overall response rate was 2.3% (n = 456/20000). Highest response rates were observed among males and females ≥50 years from major cities (6.0, 95%CI 4.4-7.9 and 5.5%, 4.1-7.2, respectively) and regional areas (6.0%, 4.6-7.6 and 6.2%, 4.9-7.7, respectively). Younger age groups and remote areas had lower response rates; all remote age groups < 50 years had a response rate ≤ 0.6%. While most participants responded on the paper surveys, online responses were more common among younger age groups and, respondents with higher education levels and whose first language was not English. Conclusion: Using a postal survey, we observed response rates of ≥5.5% among older Aboriginal and Torres Strait Islander adults in major cities and regional areas; response rates were lower in other groups. A two-stage postal distribution approach provided an opportunity to adapt sampling approaches to different demographic groups. Based on initial response rates, the sampling strategy was revised to send postal surveys to groups with higher response rates groups and focus field recruitment strategies on low response groups.
... It is internationally recognised that improvements in data systems are needed to support action and progress in Indigenous peoples' health outcomes. (1,2) This includes the need for longitudinal data on wellbeing measures that are defined by Indigenous people. These calls also recognise the importance of Indigenous peoples' role in determining data processes, including the sampling approaches applied. ...
Preprint
Full-text available
Background: Evidence on the effectiveness of postal recruitment methods for Indigenous peoples is lacking. Mayi Kuwayu, the National Study of Aboriginal and Torres Strait Islander Wellbeing, uses multi-staged sampling. We aimed to test postal surveys as a primary recruitment method, analysing preliminary response rate data to inform the Study’s ongoing sampling approach. Methods: 20,000 adults aged ≥16 years were sampled from Aboriginal and Torres Strait Islander people enrolled in the Medicare Australia Enrolment Database. We calculated response rates at 4 and 15 weeks, overall and by age group, gender, state/territory and remoteness. Results: The overall response rate was 2.3% (n=456/20000). Highest response rates were observed among males and females ≥50 years from major cities (6.0%, 95%CI 4.4–7.9 and 5.5%, 4.1–7.2, respectively) and regional areas (6.0%, 4.6–7.6 and 6.2%, 4.9–7.7, respectively). Younger age groups and remote areas had lower response rates; all remote age groups <50 years had a response rate ≤0.6%. While most participants responded on the paper surveys, online responses were more common among younger age groups and, respondents with higher education levels and whose first language was not English. Conclusion: Using a postal survey, we observed response rates of ≥5.5% among older Aboriginal and Torres Strait Islander adults in major cities and regional areas; response rates were lower in other groups. A two-stage postal distribution approach provided an opportunity to adapt sampling approaches to different demographic groups. Based on initial response rates, the sampling strategy was revised to send postal surveys to groups with higher response rates groups and focus field recruitment strategies on low response groups.
... It is internationally recognised that improvements in data systems are needed to support action and progress in Indigenous peoples' health outcomes. (1,2) This includes the need for longitudinal data on wellbeing measures that are defined by Indigenous people. These calls also recognise the importance of Indigenous peoples' role in determining data processes, including the sampling approaches applied. ...
Preprint
Full-text available
Background Evidence on the effectiveness of postal recruitment methods for Indigenous peoples is lacking. Mayi Kuwayu, the National Study of Aboriginal and Torres Strait Islander Wellbeing, uses multi-staged sampling. We aimed to test postal surveys as a primary recruitment method, analysing preliminary response rate data to inform the Study’s ongoing sampling approach. Methods Our sample included 20,000 Aboriginal and Torres Strait Islander adults >16 years enrolled in the Medicare Australia Enrolment Database. We calculated response rates at 4 and 15 weeks, overall and by age group, gender, state/territory and remoteness. Results The overall response rate was 2.3% (n=456/20000). Highest response rates were observed among males and females ≥50 years from major cities (6.0%, 95%CI 4.4–7.9 and 5.5%, 4.1–7.2, respectively) and regional areas (6.0%, 4.6–7.6 and 6.2%, 4.9–7.7, respectively). Younger age groups and remote areas had lower response rates; all remote age groups <50 years had a response rate ≤0.6%. While most participants responded on the paper surveys, online responses were more common among younger age groups and, respondents with higher education levels and whose first language was not English. Conclusion Using a postal survey, we observed response rates of ≥5.5% among older Aboriginal and Torres Strait Islander adults in major cities and regional areas; response rates were lower in other groups. A two-stage postal distribution approach provided an opportunity to adapt sampling approaches to different demographic groups. Based on initial response rates, the sampling strategy was revised to send postal surveys to groups with higher response rates groups and focus field recruitment strategies on low response groups.
... Ball [7] and Fremantle et al. [8] have argued that high quality data is an essential precursor to understanding the problems, identifying solutions, and improving health equity for Aboriginal children in Canada. This data must provide detailed health information that is meaningful at the local level, if we are to identify solutions910111213. The Aboriginal Children's Health and Well-being Measure (ACHWM) was developed to meet the needs of local communities to assess their children's health. ...
Article
Full-text available
Background: Aboriginal children experience challenges to their health and well-being, yet also have unique strengths. It has been difficult to accurately assess their health outcomes due to the lack of culturally relevant measures. The Aboriginal Children's Health and Well-Being Measure (ACHWM) was developed to address this gap. This paper describes the validity of the new measure. Methods: We recruited First Nations children from one First Nation reserve in Canada. Participants were asked to complete the ACHWM independently using a computer tablet. Participants also completed the PedsQL. The ACHWM total score and 4 Quadrant scores were expected to have a moderate correlation of between 0.4 and 0.6 with the parallel PedsQL total score, domains (scale scores), and summary scores. Results: Paired ACHWM and PedsQL scores were available for 48 participants. They had a mean age of 14.6 (range of 7 to 19) years and 60.4 % were girls. The Pearson's correlation between the total ACHWM score and a total PedsQL aggregate score was 0.52 (p = 0.0001). The correlations with the Physical Health Summary Scores and the Psychosocial Health Summary Scores were slightly lower range (r = 0.35 p = 0.016; and r = 0.51 p = 0.0002 respectively) and approached the expected range. The ACHWM Quadrant scores were moderately correlated with the parallel PedsQL domains ranging from r = 0.45 to r = 0.64 (p ≤ 0.001). The Spiritual Quadrant of the ACHWM did not have a parallel domain in the PedsQL. Conclusions: These results establish the validity of the ACHWM. The children gave this measure an Ojibway name, Aaniish Naa Gegii, meaning "how are you?". This measure is now ready for implementation, and will contribute to a better understanding of the health of Aboriginal children.
... However, such data collection remains underdeveloped in Canada (e.g. Smylie, Anderson, Ratima, Crengle, & Anderson, 2006;). As well there are several methodological limitations to using this type of data. ...
Article
Full-text available
The objective of this paper is to document the knowledge and experiences of healthcare professionals and researchers in Australia about the barriers to controlling Strongyloides stercoralis in Australian Indigenous communities. Qualitative research methods were used to conduct in-depth semi-structured interviews, which were digitally recorded, transcribed, and participant-checked. Data were thematically analysed to identify significant themes. Five major themes were identified: 1) Barriers to health/treatment; 2) Access to healthcare; 3) Policy; 4) Learning opportunity; and 5) Ideas for intervention. The findings suggest that Australian Indigenous communities will continue to suffer increased morbidity and mortality due to a lack of control or prevention of Strongyloides stercoralis. Issues such as institutional racism, improvements to health promotion, education, socioeconomic determinants, and health care system policy and procedures need to be addressed. This study identifies several direct implications for Indigenous health: The need for increased knowledge and understanding of the risks to health for Indigenous community members; The need for prevention policy development for neglected tropical diseases in Indigenous communities; The need for increased knowledge and understanding of the treatment, diagnosis, and healthcare access concerning Strongyloides stercoralis for health professionals and policymakers who work within Indigenous health; The need to raise awareness of systematic institutional racism in the control and prevention of neglected tropical diseases in Indigenous communities; and The need for a health promotion framework that can provide the basis for multiple-level interventions to control and prevent Strongyloides in Indigenous communities.
Article
Background International studies of the health of Indigenous and tribal peoples provide important public health insights. Reliable data are required for the development of policy and health services. Previous studies document poorer outcomes for Indigenous peoples compared with benchmark populations, but have been restricted in their coverage of countries or the range of health indicators. Our objective is to describe the health and social status of Indigenous and tribal peoples relative to benchmark populations from a sample of countries.
Article
Full-text available
Since, at the health system level, there is little research into the possible interrelationships among the various indicators of health, healthcare performance, non-medical determinants of health, and community and health system characteristics, we conducted this study to explore such interrelationships using the Canadian Health Indicators Framework. We conducted univariate correlational analyses with health and healthcare performance as outcomes using recent Canadian data and the ten Canadian provinces and three territories as units of the analyses. For health, 6 indicators were included. Sixteen healthcare performance indicators, 12 non-medical determinants of health and 16 indicators of community and health system characteristics were also included as independent variables for the analysis. A set of decision rules was applied to guide the choice of what was considered actual and preferred performance associations. Health (28%) correlates more frequently with non-medical determinants than healthcare does (12%), in the preferred direction. Better health is only correlated with better healthcare performance in 13% of the cases in the preferred direction. Better health (24%) is also more frequently correlated with community and health system characteristics than healthcare is (13%), in the preferred direction. Canadian health performance is a function of multiple factors, the most frequent of which may be the non-medical determinants of health and the community characteristics as against healthcare performance. The contribution of healthcare to health may be limited only to relatively small groups which stand to benefit from effective healthcare, but its overall effect may be diluted in summary measures of population health. Interpreting multidimensional, multi-indicator performance data in their proper context may be more complex than hitherto believed.
Article
Full-text available
Given the proliferation and the growing complexity of performance measurement initiatives in many health systems, the Netherlands and Ontario, Canada expressed interests in cross-national comparisons in an effort to promote knowledge transfer and best practise. To support this cross-national learning, a study was undertaken to compare health system performance approaches in The Netherlands with Ontario, Canada. We explored the performance assessment framework and system of each constituency, the embeddedness of performance data in management and policy processes, and the interrelationships between the frameworks. Methods used included analysing governmental strategic planning and policy documents, literature and internet searches, comparative descriptive tables, and schematics. Data collection and analysis took place in Ontario and The Netherlands. A workshop to validate and discuss the findings was conducted in Toronto, adding important insights to the study. Both Ontario and The Netherlands conceive health system performance within supportive frameworks. However they differ in their assessment approaches. Ontario's Scorecard links performance measurement with strategy, aimed at health system integration. The Dutch Health Care Performance Report (Zorgbalans) does not explicitly link performance with strategy, and focuses on the technical quality of healthcare by measuring dimensions of quality, access, and cost against healthcare needs. A backbone 'five diamond' framework maps both frameworks and articulates the interrelations and overlap between their goals, themes, dimensions and indicators. The workshop yielded more contextual insights and further validated the comparative values of each constituency's performance assessment system. To compare the health system performance approaches between The Netherlands and Ontario, Canada, several important conceptual and contextual issues must be addressed, before even attempting any future content comparisons and benchmarking. Such issues would lend relevant interpretational credibility to international comparative assessments of the two health systems.
Article
Full-text available
The Australian government's Healthy for Life program is supporting capacity development in Indigenous primary care using continuous quality improvement (CQI) techniques. An important influence on the Healthy for Life program has been the ABCD research project. The key features contributing to the success of the project are described. The ABCD research project: uses a CQI approach, with an ongoing cycle of gathering data on how well organisational systems are functioning, and developing and then implementing improvements; is guided by widely accepted principles of community-based research, which emphasise participation; and adheres to the principles and values of Indigenous health research and service delivery. The potential for improving health outcomes in Aboriginal and Torres Strait Islander communities using a CQI approach should be strengthened by clear clinical and managerial leadership, supporting service organisations at the community level, and applying participatory-action principles.
Article
This article describes recent national performance improvement initiatives in the United States, United Kingdom, and Australia. This comparison is of particular interest because each of these three countries faces similar challenges in delivering health care and improving health. Each has elevated a focus on safety and quality improvement to a national level. Marked differences in the organization and financing of health care across these three countries provide a unique opportunity to compare and contrast approaches. Drawing on the experience of the authors in each of the three countries and publicly available data sources about specific national initiatives, we describe the national context for improvement and outline recent performance improvement initiatives and emerging issues and challenges. Similarities and differences in the current evolution of national performance initiatives are described and conclusions are drawn about challenges that all three countries face, particularly in terms of developing meaningful sets of national indicators of health system performance. The challenges for future work include the importance of information infrastructure, the paucity of accurate and accessible clinical data, the need for effective performance measurement processes at a local level to capture useful data, and the tensions of balancing accountability and improvement agendas for measurement.
Article
This article provides an overview of health care per formance measurement, including a chronological history of the major developments in the performance measurement field. It is not intended to be all-encompassing in its descriptions of events and organizations but, rather, its purpose is to provide a broad historical con-text for describing health care performance measurement activities of the past 50 years. The article also highlights the key con­ stituents driving performance measure­ ment (government payers, private sector regulators, business coalitions, health care providers, and health care consumers), how they have influenced what is measured (the content of per formance measure­ ment), and why. The article concludes by establishing the commonalities among con­ stituents and forecasts what the foreseeable future may hold regarding performance measurement.
Article
Performance measurement is considered to be an important part in improving and controlling contemporary organizations. Despite the increased interest in using and improving performance measurement systems, the number of researchers investigating the design and implementation process in more detail is still very small. The purpose of this article is to increase the understanding of how a performance measurement system, called the flow model, is designed and implemented by development teams in Swedish health care. The purpose of the flow model is to follow up the requirements in the National Care Guarantee through 8 measures. From a multiple case study of 6 local development teams, we conclude that the performance measurement system of the flow model is designed and implemented in Swedish County Councils through 3 development tracks, which are reflected in close interaction between the local development team and people in clinical departments. Enabling factors in the design and implementation are (1) the recognition of a need to change the current situation, (2) the teams' interaction with people in the clinical departments, (3) the national network meetings arranged by the financier, (4) the struggle to motivate and inform the top management, and (5) the establishment of contact with other development teams and ongoing projects.
Article
Unlabelled: Interest in comparative quality measurement and evaluation has grown considerably over the past two decades because of factors such as the recognition of widespread variation in clinical practice, the increased availability of evidence about medical effectiveness, and increasing concern about the cost and quality of health care. This article describes and contrasts two current efforts to develop health performance reporting systems: one, an international initiative-the Health Care Quality Indicator (HCQI) Project, sponsored by the Organization for Economic Cooperation and Development (OECD); and the other, a national project-the National Healthcare Quality Report (NHQR), sponsored by the US Agency for Healthcare Quality and Research. There are a number of lessons learned from a comparison of the two efforts that are relevant for the future of each project and for other indicator-based reporting efforts in quality of health care. These lessons are discussed in the article and include: Conceptual frameworks should be established to guide the selection of indicators. Choices should be made early on in the process to focus on a wide range of clinical conditions or to report on a few priority areas. Methods: should be developed to add and subtract indicators while maintaining a stable set of indicators to track over time. Resources should be allocated to communication strategies and how best to present data results to diverse audiences. Mechanisms should be put in place to maintain project momentum.
Article
A confluence of forces has resulted in massive changes to the collection and use of performance data in the UK health system over the past 25 years. Performance measurement offers major potential benefits for clinical professionals, managers, regulators, politicians, researchers, patients and citizens. However, uncritical reliance on performance data can lead to a number of unintended and adverse consequences. The National Health Service has pioneered the assembly and dissemination of performance data from the early 1980s. This article summarizes the history of performance measurement in the NHS, assesses its effectiveness to date, discusses the major challenges in deploying performance measures, and highlights priorities for policy-makers.
Population characteristics Aboriginal and Torres Strait Islander Australians
  • Australian Bureau
  • Statistics
Australian Bureau of Statistics. Population characteristics Aboriginal and Torres Strait Islander Australians 2001. Canberra: ABS, 2003. (Cat. No. 4713.0.)