Julie Kanter, MD, Melissa Frei-Jones, MD, MSCI, Deepa Manwani, MD, Marsha Treadwell, PhD, Mohan Madisetti, MSc, Robin Miller, MD, Seethal A Jacob, MD, Sana Saif-Ur-Rehman, MD, Andrew Ross Wickman O’Brien, MD, Sophie Lanzkron, MD
Abstract
Introduction
SCD remains the most common inherited blood disorder. Due to a lack of strong evidence, the management of sickle cell is often anecdotal, local to individual centers, states, and countries. Even in areas of practice with high-level data to improve practice, implementation has not been consistent. The historical lack of an agreed-upon national registry has compounded this problem and resulted in a lack of comparative effectiveness data to identify areas of deficiency and improve practice. These barriers have resulted in centres developing local policies and procedures, often with limited communication with other centres, resulting in disparate and inconsistent care.
Objectives
To improve the situation, the National Alliance of Sickle Cell Centres (NASCC) was founded in 2020 and adopted a national registry, the Globin Research Network for Data and Discovery. To provide clear, practical, and measurable recommendations, the NASCC initiated a consensus procedure using a modified Delphi process to enhance the development of practice recommendations for SCD that include current guideline-based recommendations, use consensus where data are limited and identify areas where research is needed.
Methods
This manuscript explains the consensus process used, including the facilitation of the hybrid discussions and the categorisation of practice-based recommendations as standard versus recommended.
Results
This paper describes the methods used to develop consensus recommendations to improve practice harmonisation in SCD.
Conclusions
The recommendations resulting from this process will help us to provide consistent care to affected individuals, enhance the evidence base in SCD management and support quality improvement efforts.
Keywords: Sickle cell disease, Sickle Cell Centers, Delphi, recommendations, quality improvement
Lay Summary
SCD is a common inherited blood disorder both in the United States and globally. Managing this condition has often relied on local practices rather than strong evidence-based guidelines, leading to inconsistencies in care across different centres and regions. To address these (and other) issues, the National Alliance of Sickle Cell Centres (NASCC) was established in 2020. One of its key initiatives is the use of a national registry called the Globin Research Network for Data and Discovery (GRNDaD) to collect detailed data that can identify gaps in care and facilitate improvements in SCD management. The National Alliance of Sickle Cell Centres has also implemented a way to develop more recommendations for care using a consensus process, called a modified Delphi method. By harmonising care practices across NASCC SCD centres and promoting the use of the GRNDaD registry, NASCC aims to enhance the quality and consistency of care for individuals with SCD. This collaborative approach is expected to bolster the evidence base for SCD management and support ongoing quality improvement efforts in the field.
BACKGROUND
SCD is the most common inherited blood disorder in the United States, disproportionately affecting people of African ancestry. Although systemic racism and socioeconomic inequities have adversely affected progress, the last 30 years have demonstrated significant improvement in outcomes, allowing far more individuals to grow into adulthood. A deeper understanding of the pathophysiology of SCD, including the role of vascular endothelium in the disease process as well as the nature of haemoglobin S, has broadened the scope of clinical and academic research. Additionally, greater interest in drug development for SCD has led to more disease-focused treatment options. As new therapies become available, a system of care needs to be effectively developed with systems in place to monitor and understand their impact, to ensure these therapies reach the people who need them, and to guide future avenues of research.
The lack of a strong evidence base for much of SCD care has led to significant variation in practice both within and across individual SCD centres, states, and countries. Even in areas of practice where high-level data exist to improve practice, recent SCD surveillance and implementation data have demonstrated that these evidence-based guidelines are not reaching clinical practice. This problem has been historically compounded by the lack of an agreed-upon national registry and has resulted in a lack of comparative effectiveness data to identify areas of deficiency and improve practice.
INTRODUCTION
To advance the practice of medicine for people with SCD, the National Alliance of Sickle Cell Centres (NASCC) was founded in 2020 and adopted the Globin Research Network for Data and Discovery (GRNDaD) as a national registry to ensure harmonised data collection. The National Alliance of Sickle Cell Centres is a nonprofit organisation formed to support SCD centres in delivering high-quality, comprehensive care and employing the collaborative impact to improve health outcomes, quality of life, and survival in all patients with SCD.
Creating and sustaining more high-quality SCD centres will require the pooling of expertise, resources, perspectives, and community input. At its inception, NASCC identified board members, including physicians representing all regions of the United States, researchers, community members, affected individuals, and other stakeholders to establish a shared vision to enhance the quality of care for people with SCD by improving patient outcomes, reducing health disparities, and working in partnership while collecting data in the GRNDaD registry to better understand where care improvements are needed. Once established, NASCC needed to develop the collective impact framework necessary to measure outcomes for affected individuals. One aspect of this framework is an electronic data capture system (GRNDaD). The second requirement is for shared metrics that can be used to measure outcomes, including increased life expectancy, improved patient satisfaction, and decreased acute care utilisation. Once these metrics are established, NASCC will employ data analytics to identify gaps in care and target interventions effectively.
While there have been efforts in the past to develop consensus recommendations for the treatment of SCD, including recommendations established by several SCD experts in collaboration with the National Heart Lung and Blood institute (NHLBI) in 2014 in the United States, these have not been updated to reflect the aging adult SCD population or to include novel treatment options.1 The Sickle Cell Society in the United Kingdom developed the first edition of the “Standards for Clinical Care of Adults with Sickle Cell Disease in the UK” in 2008, which was revised in 2018 and has been instrumental in improving care for patients with SCD in the United Kingdom. Similar topic-specific standards of care have also been published by the British Society of Haematology (2015, 2018) and the National Institute for Health (2014).1–3 In the United States, the American Society of Haematology (ASH) developed evidence-based guidelines in 2019-2021, which are organ-system-based, that is, cardiovascular, pulmonary, renal, and neurologic or therapy-focused, that is, blood transfusion, pain management, and stem cell transplant.4–7 Comparatively, the ASH guidelines do not sufficiently address the holistic approach to care needed by people living with SCD and the UK guidelines are more specific for their patient population and system of care. Furthermore, neither of these guidelines has been implemented for quality assurance purposes and needs restructuring for long-term quality improvement activities. To truly improve care in SCD, a longitudinal clinical registry combined with practice harmonisation and a plan for quality assurance and improvement activities is desperately needed. People with SCD require specialised and multidisciplinary care, including access to knowledgeable providers who are familiar with available treatment options to reduce both pain and organ complications, decreasing morbidity and mortality.
To formulate updated management recommendations that could be used to measure the delivery of quality care, NASCC opted to use a well-defined, accepted, consensus-based recommendation process using data-driven evidence where available, combined with a consensus-based approach amongst experts to enhance practice standardisation and allow for quality assurance assessments. The initial smaller-scale projects using consensus included those undertaken to define the clinical standards needed to be recognised as an SCD centre for children or adults. The definitions of “SCD centres” that resulted from these efforts allowed SCD centres to apply to NASCC to become a member-centred organisation recognised for delivering multi-disciplinary SCD care.
Once NASCC was fully established and had over 20 member centres, a more defined consensus process was implemented to develop clinical standards of care. The goal of this manuscript is to detail the methodology used to develop the NASCC consensus practice recommendations. As per the Institute of Medicine’s published standards for the production of clinical practice guidelines, this methodology is consistent and systematic; all resulting consensus statements are transparent and have specific notations where evidence is lacking.8 These consensus recommendations are meant to be living documents that will be reviewed and rewritten as needed to ensure the recommendations are up to date and relevant to current treatments and findings in SCD. Finally, these documents are also meant as a call for action to the US Government and healthcare community to increase efforts to provide equitable services to all people living with SCD in the United States.
MATERIALS AND METHODS
This paper will present the method used by NASCC to develop clinical practice recommendations. It is important to note that the number of NASCC-recognised centres has grown substantially since its inception (resulting in 114-NASCC NASCC-recognised full and associate member centres as of this article’s submission). The consensus process is completed from January to December of each calendar year. Thus, each topic selected annually for the development of recommendations will have a different number of centres voting as the Alliance grows. However, the process for achieving consensus has remained unaltered despite the growth of the organisation and the increased number of participating specialists. In addition, the Executive Board within NASCC reviews each set of recommendations to evaluate for relevant updates and changes that need to be made to ensure the data (and recommendations) remain current. If a specific change is needed, a vote will be reconvened across the centres to re-determine consensus for that item. In this way, the recommendations can remain as living documents.
Topic selection
The topics chosen each year are identified by the NASCC Executive Board with input from the participating recognised sickle cell centres. Topics must be deemed important and relevant to both patients and providers, as well as measurable for long-term evaluation and quality assessment. Each topic selected is also approved by the NASCC Board of Directors before the initiation of the process. The NASCC process does not focus on topics where a formal evidence-based review has already been conducted by others using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE)-like methodology.9 The NASCC consensus recommendations may encompass components already included in other guidelines, such as age-specific screening recommendations, but are redefined and organised for SCD management to enhance clarity and readability. Further, in some cases, NASCC consensus processes can also be used to provide recommendations on how to optimally implement existing guidelines.
RAND/UCLA Modified Delphi Panel Methodology
The Delphi panel methodology systematically and quantitatively combines expert opinion and published literature (Figure 1). This method is consistent with established processes used for expert practice harmonisation in health care decision making.10,11 Delphi panels do not involve human subjects as defined by 45 CFR part 46, and therefore, these procedures do not require institutional review board approval. There are 4 key characteristics required when using the original Delphi method: anonymity, iterative data collection, participant feedback, and statistical determination of group response. A classic Delphi process starts with a set of questions that are answered (anonymously) by experts in a series of rounds. The initial rounds are followed by a discussion in which participants can give feedback on the results of the previous round to explain their rationale. Panellists can revise their responses and offer a new vote in subsequent rounds. After each round, the items which have reached consensus are removed from the round, and the remaining items continue to be discussed/voted/revisited. This process can continue until either a predetermined percentage consensus is reached or until the group completes a pre-arranged number of rounds. The RAND/UCLA (University of California, Los Angelos) Modified Delphi method is more specific and includes a comprehensive literature review prior to the initial voting, followed by only 2 rounds of ratings or voting on specific statements and a discussion in between rounds. At the end of the second round of voting, the group will have achieved consensus on each item or identified items as non-conclusive. This process removes the anonymity used in the original Delphi.
The NASCC process uses the same modified-Delphi process but includes both virtual and in-person engagement with a larger group of experts as done using the ExpertLens. The actual ExpertLens tool is not used in this process, but the concepts developed by the RAND Corporation (virtual participation and expansion of the participant pool) are included. The National Alliance of Sickle Cell Centres has 2 virtual voting rounds to allow for the inclusion of geographically distributed participants. In between these rounds, the participants gather for in-person/ hybrid discussion to review the initial voting results, review/ discuss and debate changes, and formulate the final items to be considered in the final voting round. The National Alliance of Sickle Cell Centres asks each centre to have 1 representative who will vote in the process to express their opinions on behalf of their SCD centre. Most often, the representative is the SCD centre director. However, a centre director may select someone else to serve in this capacity. The NASCC process remains iterative, asks participants to provide their judgments independently and encourages discussion before the solicitation of the final anonymous, independent votes.
Pre-consensus ratings survey
An initial survey is completed independently by each centre to assess current practice patterns across all the recognised NASCC member centres (at that time) to identify areas of practice agreement and identify where there is significant variability. The length of the initial ratings survey depends on the topic undertaken.
Following this survey, NASCC member centres are asked to review the current literature available around the specific consensus topic. For the first consensus topics (age-based screening and prevention for children and adults with SCD), a thorough literature review was undertaken by Partnership for Health Analytic Research, LLC (PHAR). Partnership for Health Analytic Research, LLC was directed to conduct targeted literature searches to identify published information on routine screening and assessments for children and adults with SCD, established SCD guidelines, and previously published consensus recommendations or practice harmonisation documents from any state in the United States and countries in Europe. Partnership for Health Analytic Research, LLC was used to assemble and condense the material for the screening and prevention topics due to the breadth of available material that included all age-related (ie, healthy person) screening in addition to SCD-specific screening and guidelines. Subsequent relevant literature reviews are assembled by the topic leaders or recommended by the local experts in the designated practice areas. In all cases, the designated literature is given to all NASCC members to review prior to the meeting.
Consensus ratings
For all consensus recommendations, NASCC adopted 2 grades of practical interventions:
Standards
Being that which providers must do to ensure safe and adequate care, or where omission could lead to poor clinical outcomes. These include key requirements of any service. Where possible, we have tried to ensure these are measurable.
Recommended
Being those that would be beneficial and that providers should try to follow, but for which there is less evidence or that are less certain to have a direct impact on clinical outcomes.
Hybrid meeting
After the literature review is completed, all NASCC member centres as of May 1 (in that year) are invited to attend a consensus meeting to discuss the designated consensus topics for that year. During this meeting, the results of the initial pre-meeting consensus are reviewed as a group and clarifying questions can be asked and discussed. The results of the pre-meeting surveys are presented to specifically call out areas of clear convergence and variation. After review, the centre members are asked to self-select into workgroups. Each consensus topic has designated leaders who serve as moderators for this group discussion. Here, all members can discuss the logic behind their ratings, focusing on areas of difference. The goal is to specifically review the initial findings, discuss and debate areas of variation and divergence, and identify key statements to be included in the final recommendations. National Alliance of Sickle Cell Centres members can attend both in-person and via teleconference. During the live discussion, NASCC members can also add new items to the recommendations that were not included in the initial survey, which may develop during the conversation.
Hybrid meeting voting
During the hybrid meeting, a vote is taken on all items, including those that did reach initial consensus during the pre-meeting survey, as well as those in which there was no agreement and the new items that were added during the discussion. Votes are taken on each item. Items that reached at least 70% agreement were considered agreed upon in this vote. If an item did not reach 70% agreement, it was identified as an area needing further study/ discussion.
Post-meeting assessments and community input
Following the live meeting, the working groups complete a summary of findings and develop the final list of statements based on the votes, including evidence and the level of data available. These statements are presented to a regionally diverse group of community-based organisations of SCD stakeholders for comments, additions, and recommendations. Community-stakeholder feedback is then included in the document prepared for the final vote.
Final vote
As noted, the initial vote is performed virtually and followed by a hybrid discussion on each point, which results in a final list of items to be voted on after undergoing community review. The final vote is also performed virtually. At the time of the final vote, each of the recognised NASCC centres can identify a single voting member for each consensus topic (1 vote/centre). The lifespan or “whole life” NASCC centres that have separate pediatric and adult directors get 2 votes (one for each director). Importantly, any full NASCC centre member can vote during the final vote even if they were not part of the initial pre-meeting survey or were not present at the annual hybrid meeting. The rationale for including all centres in the final vote is to be as inclusive as possible, given the fluctuating number of NASCC centres. It is optimal that the final vote includes the representation from all current centres (which may have changed since the start of this process). Thus, the total number of voting centres may differ within a Delphi topic (from first to third vote) and between different topics as more centres joined the Alliance over time.
For each statement, the voting members rate the action (test, treatment, management change, etc) on a scale of 1-9, where 1 is highly inappropriate, risks outweigh the benefits, to 9 signifying highly appropriate, benefits outweigh the risks. Ratings of 1-3 are used when an action is considered by the voter to be inappropriate, and ratings of 7-9 are used for actions considered appropriate. The definition of consensus used by RAND/UCLA modified Delphi panels is variable. The consensus definition should be the percentage of agreement based on a predetermined cutoff. In previous studies by different groups, the percentage agreement varies from 50% to 97% and is usually arbitrary.20 Importantly, the most common range required for consensus is 70%-80% because it reflects a substantial majority while remaining achievable in large, diverse expert panels. Higher percentages are more difficult when there is a large number of participants, as in this setting. Thus, the NASCC consensus here was pre-defined as being present when at least 70% of the responses were in the same three-point range (1-3, 4-6, 7-9). Thus, for all recommendations and standards, at least 70% consensus is required for inclusion. For items that do not meet consensus, additional information is included in the document (areas of research needed, areas of controversy due to existing guidelines, or other points of inquiry). The modified Delphi process used for the development of the NASCC practice recommendations is shown in Figure 2.
As of January 2025, NASCC has convened to discuss over 12 different topics intended for NASCC consensus voting resulting in 5 sets of finalized recommendations including annual screening needed for infants <2 years of age with SCD, children 2-18 years of age and adults living with SCD as well as the requirements for documentation when patients transition from pediatric to adult care and the requirements needed to be a transformative therapy center (an SCD center optimized to perform allogeneic transplant and autologous gene therapy). In addition, there are several recommendations undergoing their final votes this year, including those for the treatment of uncomplicated pain crisis, optimal management and assessment for iron overload, the definition and training needed to be an SCD specialist, and the standards and recommendations for surveillance of neurocognitive functioning in SCD. The last recommendations will be finalised in the first quarter of 2025.
DISCUSSION
Multiple areas of SCD management remain understudied, and clinical practice has relied on anecdotal experience. While practice guidelines exist in some spaces, they are often specific to a certain organ system or are limited in their ability to draw actionable conclusions due to the paucity of randomised controlled trials, lack of a clinical, longitudinal registry data, and lack of practice harmonisation among experts on which to make evidence-based conclusions. The recommendations developed by the modified Delphi process will be living documents that are re-evaluated every 3 years (or sooner if new data arrives) and updated when change is needed. With improved harmonisation of practice, ongoing clinical assessment and re-appraisal, NASCC can test the validity of these recommendations to make higher-grade evidence-based decisions in the future. To be included as a voting member of NASCC, you must be part of a recognised SCD centre in the United States to ensure the recommendations can be undertaken as a means for practice harmonisation. Notably, NASCC centre directors (all of whom are recognised as SCD specialists) have different durations and types of experiences in the management of people with SCD, allowing for a highly heterogeneous but diverse group of members, adding to the richness of the discussion and deliberations.
The purpose of these recommendations is to help SCD centres foster practice harmonisation in order to ensure people living with SCD receive consistent, high-level care, based on the most recent scientific discovery to guide practice decisions. Furthermore, harmonising practice allows for an improved ability to assess the outcomes of these practices for further decision making in the future. The target audience includes the SCD centres themselves as well as the primary care providers and other clinicians, nurses, and staff who provide emergency or continuity care to individuals with SCD. These recommendations address the care of individuals living with SCD throughout the lifespan with a goal of facilitating high-quality practice, encouraging the use of a common data collection tool (GRNDaD, the national registry used by NASCC), and improving cooperative thought and research within the SCD provider community.
CONCLUSION
The history of SCD both in the United States and worldwide has demonstrated a lack of commitment to clinical care and quality improvement by multiple governing bodies, resulting in a lack of available data, resources, and few high-quality clinical trials. These recommendations reflect the areas of agreement among NASCC member centres. There are noted limitations in data availability and several areas in which topics chosen for consensus will not have data but will require practice harmonisation to ensure the safe and methodical delivery of novel therapies and treatments. Furthermore, as we now have a population of geriatric patients living with SCD, there are uncharted territories for which we need to studiously examine our practices and identify optimal treatments. The clinician-supported consensus statements developed through this validated method allow for faster dissemination of management recommendations compared to waiting on the results of a clinical study that has the potential to be underfunded (or never funded). We hope that these recommendations can aid in promoting research in SCD and further care improvement in a cooperative, data-driven environment with a focus on improving outcomes for all individuals living with SCD.
ACKNOWLEDGEMENTS
There is no funding for this work. National Alliance of Sickle Cell Centres (NASCC) is grateful to the pharmaceutical sponsorship obtained annually used to support the meeting space and NASCC administration used for the consensus conference. All consensus sessions are limited to SCD centre members only and are closed to industry members. Special thanks to all the members of NASCC who took part in this initiative and for their commitment to providing excellent care to people living with SCD.
AUTHOR CONTRIBUTIONS
Julie Kanter (Conceptualization, Data curation, Methodology, Project administration, Writing—original draft, Writing—review & editing), Melissa Frei-Jones (Conceptualization, Data curation, Writing—review & editing), Deepa Manwani (Investigation, Methodology, Project administration, Writing—review & editing), Marsha Treadwell (Con-ceptualization, Writing —review & editing), Mohan Madisetti (Data curation, Formal analysis, Methodology, Project administration, Writing—review & editing), Robin Miller (Data curation, Writing—review & editing), Seethal A Jacob (Data curation, Writing—review & editing), Sana Saif-Ur-Rehman (Data curation, Writing—review & editing), Andrew Ross Wickman O’Brien (Data curation, Writing—review & editing), and Sophie Lanzkron (Conceptualization, Data curation, Methodology, Writing— original draft, Writing— review & editing)
FUNDING
None declared.
CONFLICTS OF INTEREST
J.K.: Consultant Beam, Novo Nordisk, Merck, Sanofi, Optum, Research: HRSA, CDC, NIH. S.J.: Research NIH. D.M.: consultant, Editas, Pfizer, Novartis. S.L.: Consultant: Beam, Novo Nordisk, Bluebird bio, Merck, Pfizer, Research: PCORI, HRSA. M.F., M.M., R.M., S.S.U.R., and A.R.W.O. have no conflict of interest. D.M., S.L., M.T., and J.K. serve on the Executive Board of the National Alliance of Sickle Cell Centres (uncompensated). S.L. is a member of the Editorial Board for the Journal of Sickle Cell Disease. Full peer review for this manuscript was handled by Journal of Sickle Cell Disease Associate Editor Betty Pace.
DATA AVAILABILITY
This paper reports on the methods used by the National Alliance of Sickle Cell Centres to achieve consensus using a blend of the Modified Delphi and Expert Lens techniques, including describing how the data are obtained to achieve practice harmonisation. This manuscript does not include data.
REFERENCES
1 Yawn BP, Buchanan GR, Afenyi-Annan AN, et al. Management of sickle cell disease: summary of the 2014 evidence-based report by expert panel members. JAMA. 2014;312(10): 1033-1048. 10.1001/jama.2014. 10517
2 Howard J, Hart N, Roberts-Harewood M, et al.; BCSH Committee. Guideline on the management of acute chest syndrome in sickle cell disease. Br J Haematol. 2015;169 (4):492-505. 10.1111/bjh.13348
3 Qureshi A, Kaya B, Pancham S, et al.; British Society for Haematology. Guidelines for the use of hydroxy-carbamide in children and adults with sickle cell disease: a British Society for Haematology Guideline. Br J Haematol. 2018;181(4):460-475. 10.1111/bjh.15235
4 Liem RI, Lanzkron S, Coates TD, et al. American Society of Haematology 2019 guidelines for sickle cell disease: cardiopulmonary and kidney disease. Blood Adv. 2019;3 (23):3867-3897. 10.1182/blood- advances.2019000916
5 DeBaun MR, Jordan LC, King AA, et al. American Society of Haematology 2020 guidelines for sickle cell disease: prevention, diagnosis, and treatment of cerebrovascular disease in children and adults. Blood Adv. 2020;4(8):1554-1588.10.1182/ bloodadvances.2019001142
6 Brandow AM, Carroll CP, Creary S, et al. American Society of Hematology 2020 guidelines for sickle cell disease: management of acute and chronic pain. Blood Adv. 2020;4(12):2656-2701. 10.1182/bloodadvances. 2020001851
7 Kanter J, Liem RI, Bernaudin F, et al. American Society of Hematology 2021 guidelines for sickle cell disease: stem cell transplantation. Blood Adv. 2021;5(18):3668-3689. 10.1182/bloodadvances.2021004394C
8 Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice G. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. National Academies Press (US). Copyright 2011 by the National Academy of Sciences. All rights reserved.; 2011.
9 Schunemann HJ, Mustafa R, Brozek J, et al.; GRADE Working Group. GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76:89-98. 10.1016/j.jclinepi.2016.01.032
10 Bojke L, Soares M, Claxton K, et al. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technol Assess. 2021; 25(37):1-124. 10.3310/hta25370
11 Bojke L, Soares MO, Claxton K, et al. Reference case methods for expert elicitation in health care decision making. Med Decis Making. 2022; 42(2):182-193. 10.1177/0272989X21 1028236
12 Broder MS, , Gibbs SN, , Yermilov I. An adaptation of the RAND/UCLA modified Delphi panel method in the time of COVID-19. J Healthc Leadersh. 2022;14:63-70.
13 Dalal S, Khodyakov D, Srinivasan R, Straus S, Adams J. ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge. Technol Forecast Soc Change. 2011;78(8): 1426-1444.10.1016/j.techfore.2011. 03.021.
14 Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manage Sci. 1963;9(3):458-467.
15 Khodyakov D, Grant S, Kroger J, Gadwah-Meaden C, Motala A, Larkin J. Disciplinary trends in the use of the Delphi method: a bibliometric analysis. PLoS One. 2023;18(8): e0289009. 10.1371/journal.pone. 0289009
16 Khodyakov, D. (2023, October 2). Generating evidence using the Delphi method. RAND Corporation. https://www.rand.org/pubs/commentary/2023/10/generating-evidence-using-the-delphi-method.html.
17 Khodyakov D, Chen C. Response changes in Delphi processes: why is it important to provide high-quality feedback to Delphi participants? J Clin Epidemiol. 2020;125:160-161. 10.1016/j.jclinepi.2020.04.029
18 Khodyakov D, Chen C. Nature and predictors of response changes in modified-Delphi panels. Value Health. 2020;23(12):1630-1638. 10.1016/j.jval.2020.08.2093
19 Scott TE, Costich M, Fiorino EK, Paradise Black N. Using a modified Delphi methodology to identify essential telemedicine skills for pediatric residents. Acad Pediatr. 2023;23(3): 511-517.10.1016/j.acap.2022.08.014
20 Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol. 2021;11(4):116-129. 10.5662/wjm.v11.i4.116
Credit: Julie Kanter, Melissa Frei-Jones, Deepa Manwani, Marsha Treadwell, Mohan Madisetti, Robin Miller, Seethal A Jacob, Sana Saif-Ur-Rehman, Andrew Ross Wickman O’Brien, Sophie Lanzkron, The development of consensus recommendation to improve practice harmonisation for Sickle Cell Disease through the National Alliance of Sickle Cell Centres, Journal of Sickle Cell Disease, Volume 2, Issue 1, 2025, yoaf011, https://doi.org/10.1093/ jscdis/yoaf011