Consulting the oracle: ten lessons from using the Delphi technique in nursing research
求教德尔菲：护理研究中运用德尔菲法的十项经验

Accepted for publication 22 March 2005
2005 年 3 月 22 日收稿

Correspondence:  通信：
Felicity Hasson,  费莉西蒂·哈森
Institute of Nursing Research, University of Ulster, Newtownabbey, Shore Road,
阿尔斯特大学护理研究所，纽敦阿比，海岸路
Belfast BT37 0QB, UK.
英国贝尔法斯特 BT37 0QB
E-mail: f.hasson@ulster.ac.uk
电子邮件：f.hasson@ulster.ac.uk

KEENEY S., HASSON F., McKENNA H. (2006) Journal of Advanced Nursing 53(2), 205-212
基尼 S.，哈森 F.，麦肯纳 H.（2006 年）《高级护理杂志》第 53 卷第 2 期，205-212 页
Consulting the oracle: ten lessons from using the Delphi technique in nursing research
咨询神谕：护理研究中运用德尔菲法的十个经验启示

Aim. The aim of this paper was to provide insight into the Delphi technique by outlining our personal experiences during its use over a 10-year period in a variety of applications.
目的。本文旨在通过概述我们十年来在不同应用场景中使用德尔菲法的亲身经历，深入解析这一研究方法。
Background. As a means of achieving consensus on an issue, the Delphi research method has become widely used in healthcare research generally and nursing research in particular. The literature on this technique is expanding, mainly addressing what it is and how it should be used. However, there is still much confusion and uncertainty surrounding it, particularly about issues such as modifications, consensus, anonymity, definition of experts, how ‘experts’ are selected and how non-respondents are pursued.
背景。作为一种在议题上达成共识的方法，德尔菲研究法已在医疗健康研究领域（尤其是护理研究）得到广泛应用。关于该技术的文献数量持续增长，主要探讨其本质特征与操作规范。然而目前仍存在诸多混淆与不确定性，特别是涉及方法改良、共识标准、匿名性、专家定义、"专家"筛选方式以及未回应者跟进策略等问题。
Discussion. This issues that arise when planning and carrying out a Delphi study include the definition of consensus; the issue of anonymity vs. quasi-anonymity for participants; how to estimate the time needed to collect the data, analyse each ‘round’, feed back results to participants, and gain their responses to this feedback; how to define and select the ‘experts’ who will be asked to participate; how to enhance response rates; and how many ‘rounds’ to conduct.
讨论。德尔菲研究在规划与实施过程中面临的关键问题包括：共识的界定标准；参与者完全匿名与准匿名之争；如何预估数据收集周期、每轮结果分析耗时、反馈结果给参与者的时间成本及其响应周期；"专家"群体的定义与筛选机制；提升应答率的有效途径；以及确定合理的研究轮次数量。
Conclusion. Many challenges and questions are raised when using the Delphi technique, but there is no doubt that it is an important method for achieving consensus on issues where none previously existed. Researchers need to adapt the method to suit their particular study.
结论。使用德尔菲法时虽面临诸多挑战与疑问，但毋庸置疑它是针对尚无共识议题达成一致的重要方法。研究者需根据具体研究调整该方法的应用。

Keywords: agreement, consensus, Delphi method, discussion paper, experts, nursing, research methodology
关键词：共识、德尔菲法、讨论文件、专家、护理学、研究方法论

The Delphi technique is a favoured approach for those researchers who are seeking judgement or consensus on a
德尔菲技术是研究者寻求专业判断或共识时青睐的方法
particular issue (Beretta 1996, Green et al. 1999). While the Delphi technique has been used in the fields of business (Kaynak et al. 1994, Addison 2003, Ilbery et al. 2004), defence (Roberts 1969, Gilbride 2002) and education
特定议题（Beretta 1996, Green 等 1999）。虽然德尔菲法已广泛应用于商业领域（Kaynak 等 1994, Addison 2003, Ilbery 等 2004）、国防领域（Roberts 1969, Gilbride 2002）和教育领域
(Dailey & Holmberg 1990, Volk 1993), studies using the technique within health research, and in particular nursing research, are increasing (Alexander & Kroposki 1999, McIlfatrick & Keeney 2003, Lofmark & Thorell-Ekstrand 2004).
（Dailey & Holmberg 1990, Volk 1993），但在健康研究尤其是护理研究中采用该技术的研究正日益增多（Alexander & Kroposki 1999, McIlfatrick & Keeney 2003, Lofmark & Thorell-Ekstrand 2004）。

Nonetheless, much confusion, disagreement and uncertainty exist concerning the principles intrinsic to its application. These include consensus, various modifications, selection and definition of experts, anonymity and the number of ‘rounds’ required. Most researchers who have used the Delphi would acknowledge that it is not until they undertake a study using the technique that its difficulties and challenges become apparent (see Hartman 1981, Green et al. 1999). In this paper, we present a collection of perceptions and insights into our experience of using the Delphi technique with reference to the studies in which we employed the method. Our perceptions were obtained from a series of informal discussion groups within which experiences of using the Delphi technique were recorded. It was apparent from these discussions that there were common experiences from which a number of key lessons could be identified. We felt that these lessons could be of use to any researcher planning to use the classical Delphi or its many modified versions.
然而，关于德尔菲法应用的核心原则仍存在诸多混淆、分歧与不确定性。这些原则包括共识达成、各类变体形式、专家遴选与界定、匿名性要求以及所需"轮次"数量等。多数采用过德尔菲法的研究者都承认，唯有亲历研究过程后，才能真正体会该方法的难点与挑战（参见 Hartman 1981, Green 等 1999）。本文通过梳理我们运用德尔菲技术开展的多项研究，系统呈现实践过程中的认知与洞见。这些认知源自一系列非正式讨论小组的交流记录，其中详细记载了德尔菲法的应用体验。讨论中明显呈现出若干共性经验，由此可提炼出多项关键启示。我们认为这些经验教训对所有计划采用经典德尔菲法或其多种改良版本的研究者都具有参考价值。

The Delphi technique is a struct process that uses a series of questionnaires or ‘rounds’ to gather information which are continued until ‘group’ consensus is reached (Beretta 1996, Green et al. 1999, Hasson et al. 2001, Powell 2003). Its popularity has centred on the fact that it allows the anonymous inclusion of a large number of individuals across diverse locations and expertise and avoids the situation where a specific expert might be anticipated to dominate the consensus process (Jairath & Weinstein 1994).
德尔菲法是一种结构化流程，通过系列问卷或"轮次"收集信息，直至达成"群体"共识（Beretta 1996, Green 等 1999, Hasson 等 2001, Powell 2003）。该方法广受欢迎的核心在于：既能匿名吸纳来自不同地域、不同专业领域的大量参与者，又可避免特定专家主导共识形成过程的情况（Jairath & Weinstein 1994）。
The technique involves the presentation of a questionnaire to a panel of informed individuals in a specific field of application in order to seek their opinion or judgement on a particular issue (McKenna 1994a). After the questionnaires are returned, the data are summarized and a new questionnaire is designed based on the responses from the first round. This second-round questionnaire is then returned to each participant showing the overall group response and the participant’s own response from round one. Participants are asked to reconsider their initial response in the light of the first round’s overall results. Repeat rounds of this process are carried out until consensus has been reached (Beretta 1996, Green et al. 1999).
该技术通过向特定应用领域的专家小组发放问卷，以征询其对特定问题的意见或判断（McKenna 1994a）。问卷回收后对数据进行汇总，并基于首轮反馈设计新问卷。第二轮问卷会向每位参与者展示整体小组回应及其个人首轮回答，要求参与者根据首轮整体结果重新考量初始回答。此过程循环进行直至达成共识（Beretta 1996, Green 等 1999）。

Hasson (2000) used the Delphi technique with 132 qualified midwives and 69 student midwives to gain consensus on what were non-midwifery duties (see McKenna & Hasson 2001). Keeney (2000) used the technique with 38 primary care nurses, 14 general practitioners and eight public representatives. The aim of this study was to try and obtain agreement on the future structure and function of community nursing in Northern Ireland (see McKenna et al. 2000). McKenna (1994b) used the technique with 95 ward managers. He wished to obtain a judgement from these managers on what models of nursing they favoured for use in a long-stay psychiatric setting. Further experience was gained through other studies. McKenna and Hasson (1998) used the technique to gain agreement on what should be included in guidelines for smoking cessation in pregnancy. McIlfatrick & Keeney (2003) also used the Delphi technique to identify cancer nursing research priorities in Northern Ireland.
哈森（2000 年）采用德尔菲法对 132 名合格助产士和 69 名助产专业学生进行调研，就非助产职责范畴达成共识（参见 McKenna & Hasson 2001）。基尼（2000 年）运用该方法对 38 名基层护理护士、14 名全科医生和 8 名公众代表开展研究，旨在就北爱尔兰社区护理的未来结构与功能达成一致意见（参见 McKenna 等人 2000 年）。麦肯纳（1994b）对 95 名病房管理者实施该技术，试图获取这些管理者对长期精神科护理模式的偏好判断。通过其他研究也积累了更多经验：麦肯纳和哈森（1998 年）运用该方法就妊娠期戒烟指南应包含的内容达成共识；麦克尔法特里克与基尼（2003 年）同样采用德尔菲法确定了北爱尔兰肿瘤护理研究优先事项。

The Delphi technique uses a series of data collection ‘rounds’ to capture and structure the knowledge and opinions of a ‘panel’ of participants on a topic with which they are perceived to have expertise. To enable the identification of a wide array of views, the first round is usually qualitative in nature. This is how the ‘Classic Delphi’ commences (Linstone 1978). Data from round one are summarized and a questionnaire is designed based on results of this summary. The round two questionnaire is returned to each participant, indicating their response from round one and how this compares with the overall panel’s response. As a result, participants are given the opportunity to reconsider the issues they identified in round one. Repeat rounds of this process are carried out until consensus has been reached or when no further changes are taking place (Jairath & Weinstein 1994). Agreement among panel members is determined by consensus, voting or through mathematical averaging (McMurray 1994).
德尔菲法通过一系列数据收集"轮次"来获取并结构化"专家组"成员对特定主题的专业知识与观点。为全面识别多样化见解，首轮通常采用定性方式展开——这正是"经典德尔菲法"的起始步骤（Linstone 1978）。研究人员汇总首轮数据后，将基于该总结结果设计问卷。第二轮问卷会反馈给每位参与者，显示其首轮回答与专家组整体意见的对比情况，从而给予参与者重新考量首轮观点的机会。该流程将循环进行直至达成共识或意见趋于稳定（Jairath & Weinstein 1994）。专家组的一致性可通过共识表决、投票或数学平均等方式确定（McMurray 1994）。

In McKenna’s (1994b) study, the views of 95 participants on the appropriateness of nursing models were obtained in round one through one-to-one interviews. In Keeney’s (2000) study, focus groups were used in the first round to generate qualitative comments on the future structure and function of community nursing in Northern Ireland. In both studies these qualitative data formed
在麦肯纳（1994b）的研究中，第一轮通过一对一访谈收集了 95 名参与者对护理模式适用性的看法。基尼（2000）的研究则在首轮采用焦点小组讨论，就北爱尔兰社区护理的未来结构与功能生成质性意见。两项研究均将这些质性数据作为
the basis for closed-ended questionnaires in subsequent rounds.
后续轮次封闭式问卷设计的基础。

In Hasson’s (2000) midwifery skill mix study, participants were encouraged to identify eight duties that could be undertaken by unqualified assistants. It is helpful to place a limit on participants’ listing of the specific items under study, in this case duties. This helps to curtail the possibility of hundreds of issues being named, as well as encouraging participants to prioritize their responses. In round two, duties identified from round one were subsumed into categories and respondents were asked which duties would they allocate confidently to a midwifery assistant or which should remain the responsibility of a registered midwife. However, Hasson (2000) acknowledged that presentation of data under categories may produce a halo effect (Mead & Moseley 2001), where the responses are about the general category, rather than about the individual issues raised by participants. However, researchers have to be cautious as participants could identify many hundreds of issues in round one. Returning these items in their raw non-themed form could overwhelm the participants in round two and affect adversely the response rate.
在哈森（2000 年）的助产士技能组合研究中，研究者鼓励参与者列出八项可由无资质助理承担的职责。对研究对象（此处指职责）的具体条目数量设限是有益的，这既能避免出现数百个被提名事项，也能促使参与者优先考虑最重要的回答。第二轮调查中，首轮确认的职责被归类整合，受访者需判断哪些职责可放心分配给助产士助理，哪些仍应属于注册助产士的责任范围。但哈森（2000 年）也承认，按类别呈现数据可能产生光环效应（Mead & Moseley 2001），导致反馈针对的是整体类别而非参与者提出的具体问题。研究者必须保持警惕，因为首轮调查中参与者可能提出数百个事项。若将这些原始未分类的条目直接反馈给第二轮参与者，可能造成信息过载并显著降低回复率。

As each round unfolds, the participants have to interpret information sent to them by the researcher. Often the qualitative results of round one are fed back to the panel members as medians and ranges or means and standard deviations. This means members must be skilled in written communication and the interpretation of basic statistical averaging. It cannot be assumed that Delphi respondents possess these skills. In the studies undertaken by the authors, there was no additional time to train participants in the interpretation of basic statistics. Therefore, data were returned to participants in simple text and percentage formats. This ensured that the information was basic yet meaningful.
随着每一轮调查的展开，参与者需要解读研究人员发送的信息。通常第一轮的定性结果会以中位数与极差、或平均数与标准差的形式反馈给小组成员。这意味着成员们必须具备书面沟通能力和基础统计平均值的解读能力。不能默认德尔菲调查对象都掌握这些技能。在作者开展的研究中，没有额外时间培训参与者理解基础统计数据。因此，数据以简单文字和百分比形式返还给参与者，确保信息简明而有意义。

One of the basic principles under ing the Delphi technique is to have as many rounds as are required to achieve consensus or until the ‘law of diminishing returns’ occurs. Provision for feedback and opportunity to revise earlier responses obviously requires that the technique has at least two rounds.
德尔菲技术的基本原则之一，是进行足够多轮次直至达成共识或出现"收益递减规律"。由于需要提供反馈和修改前期答复的机会，该方法显然至少需要两轮调查。

In Hasson’s (2000) study, a two-round approach was used, while for Keeney (2000) it took four rounds before consensus occurred. Although there are no strict guidelines on the correct number of rounds, the number can depend upon the time available and whether the project manager ignited the Delphi technique sequence with one broad question or with a
在哈森（2000 年）的研究中采用了双轮德尔菲法，而基尼（2000 年）的研究则需要进行四轮才能达成共识。虽然关于具体轮次数量没有严格规定，但这一数字可能取决于项目可用时间，以及项目经理是以一个宽泛问题还是以一系列问题或事件来启动德尔菲流程。
list of questions or events. In addition, the number of rounds may be decreased to minimize reductions in the amount of new information and reductions in response rates resulting from respondent fatigue (Starkweather et al. 1975). For example, McKenna (1994b) felt that response exhaustion occurred after two rounds, especially with busy ‘experts’ and hard-pressed clinicians. These experiences are reflected in the literature where the number of rounds varies between two and four.
此外，减少轮次可以避免因受访者疲劳导致的新信息量递减和回复率下降（斯塔克韦瑟等，1975 年）。例如麦肯纳（1994b）发现，特别是对于工作繁忙的"专家"和临床医生而言，两轮后就会出现回复疲劳现象。现有文献中反映的实践经验显示，德尔菲法的实施轮次通常在二至四轮之间。

In general, questionnaire research is notorious for its low response rates. Researchers often have to send out two or three reminder letters to non-responders. With anything up to four rounds of questionnaires, the Delphi technique asks much more of respondents than a simple survey and the potential for low responses increases exponentially.
问卷调查研究的低回复率是众所周知的。研究人员通常不得不向未回复者发送两到三封提醒信。而德尔菲法最多需要进行四轮问卷调查，对受访者的要求远高于普通调查，低回复率的可能性呈指数级增长。

To enhance responses in the rounds, it is critical that participants realize and feel that they are partners in the study and are interested (if not fascinated) by the topic. The researcher should take every opportunity to remind participants that each round is constructed entirely on their responses to previous rounds. This encourages interest, ownership and active participation.
为了提高各轮次的回复率，关键在于让参与者意识到并感受到他们是研究的合作伙伴，并对主题产生兴趣（甚至着迷）。研究者应抓住一切机会提醒参与者：每一轮问卷都是完全基于他们对前几轮的回复构建的。这种做法能激发兴趣、增强参与感并促进积极投入。

McKenna (1994a & S gested that the ‘personal touch’ could help enhance return rates. His use of one-to-one interviews in his first round meant that the participants got to know him and he was able to remind them individually that their responses would be fed back to them by questionnaire in further rounds. In effect, he psychologically signed them up to continuing with the study. He achieved a $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% response rate, which is very rare in a Delphi technique study. Starting with initial contact to build mutual rapport and developing and nurturing a relationship is necessary to increase the likelihood of participants’ ongoing commitment. On the one hand, interviewers can show appreciation for valuable information and on the other hand, respondents may be gratified to be an object of interest and by having an audience. In addition, gaining the commitment of the panel and providing information on how the study will be implemented should ensure that respondents feel sufficient ownership of the study. While this initial stage of informing the participants affects the response rates fundamentally, it is often overlooked in the methodological literature.
麦肯纳（1994a）指出"个性化接触"有助于提高问卷回收率。他在首轮采用一对一访谈的方式，使参与者熟悉研究者本人，并能逐一提醒他们：其反馈意见将通过后续轮次的问卷形式返还。实际上，他从心理层面促使参与者持续投入研究。最终他获得了 $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% 的应答率——这在德尔菲法研究中极为罕见。从初次接触建立默契开始，逐步培养并维系这种关系，对于提高参与者的持续投入度至关重要。一方面，访谈者能对宝贵信息表达赞赏；另一方面，受访者因受到关注而获得满足感。此外，获得专家小组的承诺并说明研究实施方式，应确保受访者对研究产生足够的归属感。虽然这个向参与者说明情况的初始阶段从根本上影响应答率，但方法论文献中常忽视这一点。

In her study, Keeney (2000) only had two non-responders after two rounds. This high response rate was due to her tenacity and persistence in following up non-responders and also to participants’ interest in the topic. Hasson (2000) also achieved a high response rate to round two where $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% of
在她的研究中，Keeney（2000 年）经过两轮调查后仅有两位未回应者。如此高的回应率归功于她对未回应者的持续跟进与坚持，同时也得益于参与者对该主题的兴趣。Hasson（2000 年）在第二轮调查中也获得了高达 $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% 的回应率。
qualified staff and $84\mathrm{\%}$$84\mathrm{\%}$84%84 \% of student midwives responded. Accepting the possible introduction of bias, this was helped by the Director of Nursing writing to staff stressing the importance of the study and asking them to return their questionnaires to the researcher. Hasson also acted as a motivator for the participants: for example, before the study began she became familiar with each ward and spoke to individual staff about the research.
合格的工作人员和 $84\mathrm{\%}$$84\mathrm{\%}$84%84 \% 名实习助产士做出了回应。尽管可能引入偏差，护理主任通过致信员工强调研究重要性并要求他们将问卷返还给研究人员，这对此有所帮助。哈森还充当了参与者的激励者：例如，在研究开始前，她熟悉了每个病房并与个别工作人员讨论了这项研究。

In summary, it is important to use every opportunity to encourage project ownership and loyalty among participants. Retaining participants across rounds will only occur if their interest and commitment are retained.
总之，要利用一切机会鼓励参与者对项目产生归属感和忠诚度。只有保持参与者的兴趣和投入，才能确保他们持续参与多轮调研。

At present there are no formal, universally agreed guidelines on the use of the Delphi technique nor does any standardization of methodology exist (Evans 1997). Consequently, flexibility exists in the design and format of the technique and this often depends on the study’s aims and objectives. The most popular formats include: the ‘modified Delphi’ (McKenna 1994a), the ‘policy Delphi’ (Crisp et al. 1997), the ‘decision Delphi’ (Couper 1984) and the ‘real-time Delphi’ (Beretta 1996). Few researchers use a pure Delphi approach and the various hybrids have been criticized by writers such as Sackman (1975).
目前关于德尔菲法的使用尚无正式且普遍认可的指导方针，方法论也缺乏统一标准（Evans 1997）。因此该技术在设计和形式上具有灵活性，通常取决于研究目的。最常用的形式包括："改良德尔菲法"（McKenna 1994a）、"政策德尔菲法"（Crisp 等 1997）、"决策德尔菲法"（Couper 1984）和"实时德尔菲法"（Beretta 1996）。极少研究者采用纯粹的德尔菲法，而各类混合方法曾受到 Sackman（1975）等学者的批评。

As well as the differing Delphi formats, the approaches within these formats differ. For example, in the traditional design (Linstone 1978), the content of the first round is normally obtained from the literature rather than the qualitative views of participants or from other secondary data. Other variations of the Delphi exist; for example, Proctor and Hunt (1994) sent participant nurses three patient profiles with the remit of identifying the care needs of each patient while Jones et al. (1992) involved the use of face-to-face meetings of participants after two initial Delphi rounds.
除了德尔菲法的不同形式外，这些形式内部的操作方式也存在差异。例如，在传统设计（Linstone 1978）中，第一轮的内容通常来自文献资料，而非参与者的定性观点或其他二手数据。德尔菲法还存在其他变体：Proctor 和 Hunt（1994）向参与研究的护士发送了三份患者档案，要求其识别每位患者的护理需求；而 Jones 等人（1992）则在完成两轮德尔菲调查后安排了参与者的面对面会议。

As researchers, McKenna (1994b), Keeney (2000) and Hasson (2000) all used the literature as the basis for their first round interview/focus group questions. Although the researchers gave Round 1 participants ample opportunity to raise fresh issues to avoid early closure on ideas, it should be noted that this action could introduce bias by making participants feel psychologically pressured to alter their views according to the recognized literature.
作为研究者，McKenna（1994b）、Keeney（2000）和 Hasson（2000）均以文献作为首轮访谈/焦点小组问题的基础。尽管研究者们为第一轮参与者提供了充分机会提出新议题以避免过早形成定论，但需注意的是，这种做法可能因使参与者感受到心理压力而根据公认文献改变观点，从而引入偏差。
In summary, the lesson here is to acknowledge that modification which is not systematic and rigorous may be problematic. Without care, this could result in a self-fulfilling prophecy where, like sheep, participants could be steered to agree on a highly visible issue in the literature. The researcher
总之，这里的经验是要认识到非系统性和不严谨的修改可能会带来问题。若不加以注意，这可能导致一种自我实现的预言——就像羊群效应一样，参与者可能会被引导去认同文献中某个高度显眼的议题。研究者
should allow participants freedom to bring their views to the first round.
应当给予参与者在首轮讨论中自由表达观点的权利。

Most Delphi studies recruit individuals who (based on criteria) are perceived to have expertise in the subject under investigation. A researcher must decide on these criteria before the study commences, such as gender, professional experience, education, employment or designation. As the exact composition of the panel can affect the results obtained, the potential for bias is considerable. Moreover, working on the principle that experts must have knowledge of the subject area, participants cannot be selected randomly. For instance, it makes little sense to seek consensus among non-professionals on developing smoking cessation guidelines in pregnancy when they may have little knowledge of the topic or the context (McKenna & Hasson 1998).
大多数德尔菲研究都会根据既定标准招募被认为具备相关领域专长的个体。研究者必须在研究开始前确定这些遴选标准，例如性别、专业经验、教育背景、职业或职称。由于专家小组的具体构成会影响研究结果，潜在的偏见风险相当大。此外，基于"专家必须掌握该领域知识"的原则，参与者不能随机选择。举例而言，若向缺乏相关知识的非专业人士征求关于制定孕期戒烟指南的共识（McKenna & Hasson 1998），这种做法就毫无意义。

However, there is clearly a potential for bias in the selection as those who respond to the initial invitation are those who are more likely to be interested in the subject matter. Therefore, the exact composition of the panel can affect the results obtained. Indeed, Strauss and Zeigler (1975) have criticized the claim that the Delphi technique represents valid expert opinion as scientifically untenable and overstated.
然而，专家遴选过程显然存在潜在偏见，因为最初接受邀请的往往是那些对该主题更感兴趣的人士。因此，专家小组的确切构成会影响最终研究结果。事实上，Strauss 和 Zeigler（1975）曾批评德尔菲法代表有效专家意见的说法，认为这种主张在科学上站不住脚且被过分夸大。

There are no universally agreed criteria for the selection of experts, and no guidance exists on the minimum or maximum number of experts on a panel; rather it appears to be related to common sense and practical logistics. Linstone (1978) reported studies using several hundred participants to a Japanese study that used several thousand participants.
目前专家遴选既没有全球统一标准，也没有关于专家小组最低或最高人数的指导原则——这似乎更多取决于常识判断和实际运作条件。Linstone（1978）记载的研究案例中，参与专家从数百人到日本某项研究的数千人不等。

It could be stated that this problem of panel size is not unique to the Delphi technique. Other types of surveys employ non-probability sampling techniques. This does not ensure accurate representation and results cannot be generalized to other settings or to a wider population.
可以说专家小组规模问题并非德尔菲法独有。其他类型的调查同样采用非概率抽样技术，这既无法保证样本的精确代表性，其结果也不能推广到其他情境或更广泛人群。

In his Northern Ireland study, McKenna (1994b) included as experts the entire population of nurse managers on longstay psychiatric wards across six hospitals ( $n=95$$n=95$n=95n=95 ). He reasoned that only they had the expertise to decide on what model of nursing care would be the most appropriate for long-stay psychiatric patients. The decision was based on the assertion in the literature that ward managers were the experts so far as clinical setting and patient care were concerned. Hasson (2000) pursued the entire population of midwifery staff and students in the largest maternity hospital in the Republic of Ireland. She too assumed that these were the most appropriate experts when it came to what was and
麦肯纳（McKenna，1994b）在北爱尔兰的研究中，将六家医院长期住院精神科病房的所有护士长都纳入了专家群体（ $n=95$$n=95$n=95n=95 ）。他认为只有这些人才具备专业能力，能够决定哪种护理模式最适合长期住院的精神病患者。这一决策基于文献中的主张，即在临床环境和患者护理方面，病房管理者就是专家。哈森（Hasson，2000）则选择了爱尔兰共和国最大妇产医院的全部助产士工作人员和学生作为研究对象。她也假设这些人在界定助产士职责范围时是最合适的专家群体。
what was not a midwifery duty. In contrast, when investigating the future role of community nurses, Keeney (2000) selected her panellists on the principle of specialist inclusion. She wanted to ensure that general medical practitioners, members of the public and the range of specialist community nurses had the opportunity to be involved.
与之形成对比的是，基尼（Keeney，2000）在研究社区护士未来角色时，依据专业包容性原则选择专家组成员。她希望确保全科医生、公众代表以及各类社区专科护士都有机会参与其中。

In summary, the fifth lesson is that there is no magic formula to help researchers decide on who are the experts and how many there should be. Rather, as in other survey techniques, this decision is often based on funding, logistics and rigorous inclusion and exclusion criteria.
简而言之，第五个经验是：并不存在某种神奇公式能帮助研究者确定专家人选及其数量。与其他调查技术类似，这一决策往往取决于资金支持、后勤保障以及严格的纳入与排除标准。

A critical factor in using the Delphi technique is time. This includes time to allow interviews to be undertaken, questionnaires to be designed, distributed, returned, analysed and re-developed for subsequent rounds. This is something that can be underestimated by researchers using the Delphi technique for the first time. Duffield (1993) reported that each round of the technique can take up to 8 weeks to complete; this is a realistic approximation. While the studies outlined here were well planned and executed, each one of the authors underestimated the time it would take to gain consensus. For example, before entering the field, Hasson (2000) assumed it would take 3 months to complete the Delphi process. In reality, it took over 4 months to complete because of the need to follow up non-respondents and the time needed to adequately analyse the data. Keeney’s (2000) study took 12 months to complete in total, with a full 6 months devoted totally to the Delphi stage of the study, whereas McKenna took 16 months to complete his Delphi study.
运用德尔菲法的关键因素在于时间。这包括开展访谈、设计问卷、分发回收、分析数据以及为后续轮次重新调整所需的时间。初次使用德尔菲法的研究者往往会低估这一过程。达菲尔德（1993）指出，该方法的每一轮可能耗时长达 8 周——这是个相当实际的预估。尽管本文所述研究都经过周密规划和执行，但每位作者都低估了达成共识所需的时间。例如哈森（2000）在实地调研前预计德尔菲流程可在 3 个月内完成，实际却因需要追踪未回复者及充分分析数据而耗时逾 4 个月。基尼（2000）的研究总计花费 12 个月，其中整整 6 个月完全用于研究的德尔菲阶段，而麦肯纳的德尔菲研究则历时 16 个月才完成。

In summary, there is no doubt that a Delphi study is timeconsuming. Researchers should consider that, while a single questionnaire will provide respondents’ views or opinions on an issue, a Delphi study will provide a level of consensus or judgement among the respondents on the issue. The price for this added value is time.
总之，毫无疑问德尔菲研究非常耗时。研究者应当意识到，虽然单次问卷调查能获取受访者对某个问题的观点或意见，但德尔菲研究能得出受访者对该问题的共识程度或集体判断。这种增值效果的代价就是时间。

According to Polit and Hunger (1995), true anonymity is ensured when no one (including the researcher) can link a response to a respondent. Because of the iteration process inherent in the Delphi technique. True anonymity cannot be guarantee: an individual panel member’s responses are unknown to other participants but they are known to the researcher. McKenna (1994a) termed this ‘quasi-anonymity’. This means that non-respondents experience the important
根据 Polit 和 Hunger（1995）的观点，当任何人（包括研究者）都无法将回答与特定受访者对应时，才能确保真正的匿名性。由于德尔菲技术固有的迭代过程，无法保证完全匿名：虽然专家组成员的回答对其他参与者不可见，但研究者是知晓的。McKenna（1994a）将这种情况称为"准匿名"。这意味着未回应者仍能感受到前文强调的重要
‘personal touch’ as highlighted above and can be followed up using reminder letters and telephone calls. However, this may lead to researcher and subject bias and, as Beretta (1996) suggests, this could cause respondents to feel forced into returning the questionnaire.
"人情化接触"，研究者可通过提醒信函和电话进行跟进。但这也可能导致研究者与被试产生偏见，正如 Beretta（1996）所指出的，这可能使受访者产生被迫完成问卷的心理压力。

Hasson (2000) carried out her study in one large maternity hospital where she included $90\mathrm{\%}$$90\mathrm{\%}$90%90 \% of the staff. Therefore, all the participants knew that other colleagues were taking part in the Delphi study. It is possible that some may have discussed their preferred responses with colleagues. This is a limitation with any kind of self-report survey and it is acknowledged that it may threaten quasi-anonymity. Keeney (2000) was also aware that, while the participants in her study did not work together, they knew they were taking part in the same study. This was because Keeney brought her entire sample together for a workshop at the start of the process to introduce them to the study. Accepting this, both Hasson and Keeney believe that, because participants were aware that they were part of a larger research team, it helped to increase response rates.
Hasson（2000）在一家大型妇产医院开展研究时纳入了 $90\mathrm{\%}$$90\mathrm{\%}$90%90 \% 的员工。因此所有参与者都知晓其他同事正参与这项德尔菲研究。部分人可能曾与同事讨论过倾向性回答，这是所有自陈式调查共有的局限性，研究者承认这可能威胁准匿名性。Keeney（2000）也注意到，虽然其研究对象彼此不共事，但都知道参与的是同一项研究——因为 Keeney 在流程初期通过研讨会将所有样本集中并介绍研究内容。Hasson 与 Keeney 均认为，正是这种"参与者知晓自己属于更大研究团队"的设定，有效提升了问卷回收率。

In addition, a problem related to the iterative process within Delphi studies is the potential for experts to withdraw after the first round (Evans 1997). This behaviour may lead to a response bias if the attrition rate is substantial. For example, in Hasson’s (2000) study, round one was sent to 194 qualified midwives and 79 student midwives. Despite the use of follow up reminders and personal visits, 138 (68%) qualified midwives and $69(87\mathrm{\%})$$69(87\mathrm{\%})$69(87%)69(87 \%) student midwives completed and returned the questionnaire in round one. In round two, 100 ( $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% ) (-32) qualified midwives and 58 ( $84\mathrm{\%}$$84\mathrm{\%}$84%84 \% ) $(-11)$$(-11)$(-11)(-11) student midwives responded to the final Delphi questionnaire. A number of techniques are offered to help encourage non-respondents. For example, one technique is to invite the experts to take part at the beginning, sending them a postcard to return which lets them indicate if they would like to be included in the study or to decline. If they decline, they are asked to provide the reason (for example, not enough time) and these data then help to provide a fuller picture of the final sample.
此外，德尔菲研究中与迭代过程相关的一个问题是专家可能在首轮后退出（Evans 1997）。若退出率较高，这种行为可能导致回应偏差。例如在 Hasson（2000）的研究中，首轮问卷发送给 194 名合格助产士和 79 名助产学生。尽管采用了跟进提醒和实地走访，仍有 138 名（68%）合格助产士和 $69(87\mathrm{\%})$$69(87\mathrm{\%})$69(87%)69(87 \%) 名助产学生完成并返回了首轮问卷。第二轮中，100 名（ $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% ）（减少 32 名）合格助产士和 58 名（ $84\mathrm{\%}$$84\mathrm{\%}$84%84 \% ） $(-11)$$(-11)$(-11)(-11) 名助产学生回复了最终德尔菲问卷。研究提出了若干技术来鼓励未回应者参与，例如在初始阶段邀请专家时随附明信片供其选择是否参与研究。若选择拒绝，则需说明原因（如时间不足），这些数据有助于更完整地呈现最终样本特征。

In summary, there is a possible weakness in the Delphi technique in that anonymity cannot be guaranteed. While quasi-anonymity is possible, it can be threatened when the study is carried out with work colleagues where they may discuss the study. Nonetheless, knowing who the other respondents are can also help to increase participation.
总之，德尔菲法可能存在一个潜在缺陷，即无法完全保证匿名性。虽然可以实现准匿名状态，但当研究在同事间开展时，他们可能会讨论研究内容，这种匿名性就会受到威胁。不过，了解其他参与者的身份也有助于提高参与度。

As outlined above, the raison d’être for using the Delphi technique is to gain consensus or a judgement among a group
如前所述，采用德尔菲法的根本目的在于获取群体共识或形成集体判断。
of perceived experts on a topic. However, experts can differ and it would be difficult to gain $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% agreement on all issues. Therefore, a key question in any Delphi study is what percentage agreement would a researcher accept as synonymous with consensus. The answer may lie with the importance of the research topic. For instance, if it were a life and death issue such as whether or not to switch off a respirator in an intensive care unit, a $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% consensus level may be desirable. Alternatively, if the topic was related to the selection of a new nurses’ uniform, a consensus of $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% may be acceptable. As with most aspects of the Delphi technique, the literature provides few clear guidelines on what consensus level to set. Loughlin and Moore (1979) suggested that consensus should be equated with $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% agreement amongst respondents. By contrast, Green et al. (1999) employed an $80\mathrm{\%}$$80\mathrm{\%}$80%80 \% consensus level. Establishing the standard is crucial as the level chosen determines what items are discarded or retained as the rounds unfold. It is good practice for the research team to establish a definition of consensus before data collection begins.
关于某一主题的专家认知可能存在差异，很难在所有议题上达成 $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% 完全一致。因此，德尔菲研究中的关键问题是：研究者会将多大比例的意见统一视为共识？答案可能取决于研究主题的重要性。例如，对于是否关闭重症监护室呼吸机这类生死攸关的议题，可能需要达到 $100\mathrm{\%}$$100\mathrm{\%}$100%100 \% 的共识水平；而若是选择新护士制服这类议题， $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% 的共识度或许即可接受。与德尔菲法大多数环节类似，文献中对于共识标准的设定鲜有明确指导。Loughlin 和 Moore（1979）提出共识应等同于受访者间 $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% 的同意度，而 Green 等人（1999）则采用了 $80\mathrm{\%}$$80\mathrm{\%}$80%80 \% 的共识标准。确立这一标准至关重要，因为选定阈值将决定多轮调研中议题的取舍。最佳实践要求研究团队在数据收集前就明确定义共识标准。

The reason why this is so important is based on the fact that many Delphi studies are undertaken to formulate policy for funding new research or service developments. Depending on the importance of the policy, a $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% consensus cut-off point could lead to low morale or unrest among those who favoured those views which only gained $50\mathrm{\%}$$50\mathrm{\%}$50%50 \% agreement. Having only $1\mathrm{\%}$$1\mathrm{\%}$1%1 \% between whether there is consensus on an issue or not seems harsh and may be difficult to justify. However, while a $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% requirement for consensus may appear more robust, an obvious question is how this ‘cut-off point’ is chosen. The success of the Delphi technique is predicated on the principle of consensus and more research needs to be undertaken into how it is determined. One possible approach is to use confidence intervals. This would begin to address an obvious weakness in the method.
这一重要性源于许多德尔菲研究旨在为资助新研究或服务发展制定政策。根据政策的重要性，设定 $51\mathrm{\%}$$51\mathrm{\%}$51%51 \% 的共识临界点可能导致支持仅获得 $50\mathrm{\%}$$50\mathrm{\%}$50%50 \% 认同观点的人士气低落或产生不满。仅凭 $1\mathrm{\%}$$1\mathrm{\%}$1%1 \% 来判断议题是否达成共识显得过于严苛，且可能难以自圆其说。然而，虽然要求 $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% 的共识看似更为可靠，但一个显而易见的问题是这个"临界点"如何选定。德尔菲技术的成功建立在共识原则之上，关于如何确定共识仍需更多研究。一种可行的方法是使用置信区间，这将开始解决该方法中一个明显的缺陷。

In his study, McKenna (1994b) selected Loughlin and Moore’s (1979) level of consensus. While this is a methodological limitation as outlined above, McKenna found that most of the statements achieved a consensus of over $70\mathrm{\%}$$70\mathrm{\%}$70%70 \%. Keeney (2000) used a $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% level of consensus as the standard set by the research steering group for the study. Hasson (2000) used the same level for the same reason.
麦肯纳（McKenna，1994b）在其研究中采用了劳克林与摩尔（Loughlin & Moore，1979）提出的共识度标准。虽然如前述这构成方法论局限，但麦肯纳发现大多数陈述达成了超过 $70\mathrm{\%}$$70\mathrm{\%}$70%70 \% 的共识度。基尼（Keeney，2000）则以研究指导小组设定的 $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% 共识度作为标准。哈森（Hasson，2000）基于相同理由采用了同等标准。

There are no recognized guidelines on an appropriate level of consensus. From the authors’ perspective, $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% appears to be the minimal level but there is no obvious scientific rationale for this. We would suggest that researchers should decide on the consensus level before commencing the study and consider using confidence intervals with high level of importance equalling high level of consensus and high confidence interval.
目前尚无公认的共识度适用标准。从研究者视角看， $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% 似为最低阈值，但缺乏明确的科学依据。我们建议研究者在启动前确定共识水平，并考虑将高重要性等同于高共识度与高置信区间结合使用。

Most published Delphi studies focus primarily upon the findings. However, according to Evans (1997), the terms agreement and consensus are essentially two different ideologies. Is there a difference between the extent to which each participant agrees with the issue under consideration and the extent to which participants agree with each other? When reporting findings, few studies do so in the context of these different principles. Most researchers prefer instead to rely upon participants agreeing with each other. Yet it is important to note that the extent to which participants agree with each other does not mean that consensus exists nor does it mean that the ‘correct’ answer has been found. This is especially the case when the issues have ethical implications. For example, $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% of a nursing panel may agree that very elderly and ill patients should not be put through the rigours of active resuscitation. This reflects consensus in the Delphi sense, but it may not be the correct way to care for such patients.
大多数已发表的德尔菲研究主要聚焦于研究结果。然而，根据埃文斯（1997）的观点，"认同"与"共识"本质上是两种不同的理念。每位参与者对讨论议题的认同程度，与参与者彼此间的认同程度是否存在差异？在报告研究结果时，很少有研究会基于这两种不同原则进行阐述。多数研究者更倾向于依赖参与者之间的相互认同。但必须注意的是，参与者彼此认同的程度既不代表共识已经形成，也不意味着找到了"正确"答案。当议题涉及伦理影响时尤其如此。例如，某护理专家小组可能 $75\mathrm{\%}$$75\mathrm{\%}$75%75 \% 认同不应让高龄重症患者承受积极复苏的折磨。这符合德尔菲法意义上的共识，但未必是对这类患者最恰当的护理方式。

A panel member may be reluctant to share a view contrary to the majority of panel members. However, this is how the Delphi technique works. It is only through seeing the (anonymous) responses from other panel members that individual participants are encouraged to reconsider their views. Delphi purists would argue that panel members change their minds and move towards consensus because they see that someone else has identified a more relevant issue that they had not thought of. Delphi cynics would assert that panel members are inclined to change their minds because of a possible mistaken belief that the views expressed by the majority of the panel must be right. The obvious conclusion of this assertion is that strong-willed panel members hold rigidly to their views across rounds and weak-willed panel members alter theirs. If true, this conclusion challenges seriously the validity and reliability of Delphi findings.
专家组成员可能不愿发表与多数成员相左的意见。然而这正是德尔菲法的运作机制。只有当参与者看到其他成员（匿名）的反馈时，才会被激励重新审视自身观点。德尔菲纯粹主义者认为，成员改变立场达成共识是因为发现他人提出了自己未曾考虑的更关键问题；而怀疑论者则声称，成员倾向于改变观点是由于可能错误地认为多数人的意见必然正确。这一论断的明显推论是：意志坚定的成员会坚持己见贯穿多轮调查，而意志薄弱者则会改变立场。若属实，该结论将严重挑战德尔菲研究结果的有效性与可靠性。

Therefore, there is a danger of placing too much reliance upon the final results without acknowledging the influence of bias and other factors on validity and reliability. To enhance authenticity, a number of strategies can be used. For instance, pilot testing could be undertaken, additional methodological techniques such as focus groups could be integrated or comparison made with secondary validated data.
因此，若过度依赖最终结果而不承认偏见及其他因素对效度和信度的影响，将存在风险。为提高真实性，可采用多种策略，例如进行预测试、整合焦点小组等补充性方法技术，或与经过验证的二手数据进行比对。

In McKenna’s study, a quasi-experimental design was used as a follow-up method to test the results of the Delphi process (McKenna et al. 1995). In Keeney’s (2000) study, a ‘consensus conference’ was organized to discuss the validity of the Delphi results. In Hasson’s (2000) study, a series of interviews was organized to explore further the Delphi results with qualified midwives; secondary documentary data
麦肯纳的研究采用准实验设计作为后续方法，用以检验德尔菲法的结果（McKenna 等，1995）。基尼（2000）的研究则通过组织"共识会议"来讨论德尔菲结果的效度。哈森（2000）的研究安排了一系列访谈，与专业助产士深入探讨德尔菲结果；同时辅以二手文献数据

In summary, the ninth lesson is that researchers must be aware that validity may be compromised due to panel members changing highly relevant views in the face of a different view from the main body of the panel. The extent to which participants agree with each other (consensus) does not mean that the ‘correct’ answer has been found.
总之，第九条经验是研究者必须意识到：当专家组成员面对与主流意见相左的观点时改变高度相关的立场，可能会影响研究效度。参与者之间的同意程度（共识）并不意味着已经找到了"正确"答案。

The success of the Delphi technique relies upon the analytical and administrative skills of the researcher. These include establishing an administration system and analysing and presenting both qualitative and quantitative data. While these skills are not given much weight in the literature, their presence is vital for an effective and efficient Delphi study.
德尔菲法的成功取决于研究者的分析与组织能力。这包括建立管理系统，以及分析和呈现定性与定量数据。虽然文献中对这些技巧着墨不多，但它们对高效开展德尔菲研究至关重要。

As stated above, the ‘classic Delphi’ collects qualitative data in the first round and quantitative data in subsequent rounds. Little guidance is offered in the literature on the type of qualitative analysis to employ and how to present the results in the second round of the Delphi study. In the authors’ studies, opinions collected in round one were content analysed, with similar items combined into themes. These statements constituted the qualitative feedback to the
如前所述，"经典德尔菲法"首轮收集定性数据，后续轮次收集定量数据。文献中对于采用何种定性分析方法，以及如何在德尔菲研究第二轮呈现结果，提供的指导十分有限。在作者的研究中，首轮收集的意见会进行内容分析，将相似条目合并为主题。这些陈述构成了对参与者的定性反馈
sample in that they could view the variety of statements generated by the panel. No items were discarded in round one and the statements for round two were presented in the participants’ own words as much as possible. This results in a lengthy and detailed list of round two statements. However, as Green et al. (1999) reported, active intervention justified by the need to categorize and reduce statements actually resulted in the movement of statements further from the verbatim responses on which they were originally based.
样本成员可以看到专家小组生成的各种陈述。第一轮未删除任何条目，第二轮陈述尽可能保留参与者的原话表述。这导致第二轮陈述清单冗长而详细。但正如 Green 等人(1999 年)指出的，出于分类和精简陈述的需要而进行的主动干预，实际上导致陈述内容逐渐偏离最初基于的原始应答文本。

In Hasson’s (2000) and Keeney’s (2000) studies, ongoing rounds were composed of simple statistical summaries that illustrated collective opinion. Measures of central tendency and variance were favoured. The results were presented in a sample-friendly manner so that participants could easily see their contribution to the consensus view. This allows the sample to see the group response and their own individual response to each statement.
Hasson(2000 年)和 Keeney(2000 年)的研究中，持续进行的回合采用简单统计摘要来呈现集体意见，主要使用集中趋势和离散程度的测量指标。研究结果以便于样本理解的方式呈现，使参与者能清晰看到自身对共识观点的贡献。这种方式让样本成员既能查看群体响应，也能对照每个陈述查看自己的个人响应。

Each of the authors designed their own administrative system to allow for tracking of individual responses for each round; this enabled non-respondents to be tracked and targeted. Employing self-administrative questionnaires requires a mail base to be established and physical and financial resources allocated to cover costs of postage, printing, telephone bills and photocopying. These are important for any research study but vital for the smooth running of a Delphi survey.
每位作者都设计了独立的管理系统，以便追踪每轮调研中每位专家的反馈情况，从而实现对未回应者的追踪与定向跟进。采用自填式问卷需要建立邮寄数据库，并配置人力与财力资源以支付邮费、印刷费、电话费和复印费等成本。这些要素对所有研究都至关重要，对于德尔菲调查的顺利开展更是不可或缺。

In summary, the tenth lesson is that, like most studies, the Delphi survey will fail if proper administrative systems are not in place to ensure the proper processes are followed. Because of the multiple rounds that make up a Delphi study, high quality administrative systems are crucial for success.
总之，第十条经验是：与大多数研究一样，若缺乏完善的管理体系来确保流程规范执行，德尔菲调查必将失败。由于德尔菲研究包含多轮次调研，高质量的管理系统是其成功的关键所在。

In this paper we present the major lessons we have learned in employing the Delphi technique. As it can be applied in a number of modified ways, not all of these lessons will be applicable to all settings. Nevertheless, some issues presented in this paper, such as the selection of experts, quasianonymity and consensus, are universal problems faced by researchers using this technique.
本文阐述了我们在运用德尔菲法过程中总结的核心经验。鉴于该方法存在多种改良应用形式，并非所有经验都适用于各类场景。但本文探讨的专家遴选、准匿名机制和共识达成等问题，确是所有采用该技术的研究者面临的共性问题。

Many challenges and questions are raised when using the Delphi technique, but there is no doubt that it is an important method for achieving consensus on issues where none previously existed. If researchers merely want to gauge the knowledge or opinions of a group of people, there are many research approaches from which to choose. However, if they wish to establish consensus or obtain a judgement on an issue, a Delphi survey may be the appropriate method.
运用德尔菲法时会产生诸多挑战与疑问，但毫无疑问，这是在原本缺乏共识的议题上达成一致的重要方法。若研究者仅想评估某群体的知识或观点，可选用多种研究方式；但若希望就某个议题建立共识或获取专业判断，德尔菲调查或许是恰当之选。

SK, FH and HMcK were responsible for the study conception and design, data collection and analysis, drafting of the manuscript, critical revisions of the paper for important intellectual content, statistical expertise, and administrative, technical and material support.
SK、FH 与 HMcK 共同负责研究构思设计、数据收集分析、文稿起草、对论文重要学术内容进行关键修订，并提供统计学专业支持及行政、技术与物资支持。

Addison T. (2003) E-commerce project development risks: evidence from a Delphi survey. International Journal of Information Management 23(1), 25-40.
艾迪生 T.（2003）电子商务项目开发风险：德尔菲调查证据。《国际信息管理杂志》23(1), 25-40.
Alexander J. & Kroposki M. (1999) Outcomes for community health nursing practice. Journal of Nursing Administration 29(5), 49-56.
亚历山大 J. 与克罗波斯基 M.（1999）社区护理实践成果。《护理管理杂志》29(5), 49-56.
Beretta R. (1996) A critical review of the Delphi technique. Nurse Researcher 3(4), 79-89.
贝雷塔 R.（1996）德尔菲技术批判性评述。《护理研究者》3(4), 79-89.
Couper M.R. (1984) The Delphi technique: characteristics and sequence model. Advances in Nursing Sciences 7, 72-77.
库珀 M.R.（1984）德尔菲技术：特征与序列模型。《护理科学进展》7, 72-77.
Crisp J., Pelletier D., Duffield C., Adams A. & Nagy S. (1997) The Delphi method? Nursing Research 46(2), 116-118.
克里斯普 J.、佩尔蒂埃 D.、达菲尔德 C.、亚当斯 A.与纳吉 S.（1997）德尔菲法？《护理研究》46(2), 116-118.
Dailey A.L. & Holmberg J.C. (1990) Delphi: a catalytic strategy for motivating curriculum revision by faculty. Community/Junior College Quarterly of Research and Practice 14(2), 129-136.
戴利 A.L.与霍尔姆伯格 J.C.（1990）德尔菲：激励教师修订课程的催化策略。《社区/初级学院研究与实践季刊》14(2), 129-136.
Duffield C. (1993) The Delphi technique: a comparison of results obtained using two expert panels. International Journal of Nursing Studies 30(3), 227-237.
达菲尔德 C.（1993）德尔菲技术：两个专家小组所得结果的比较。《国际护理研究杂志》30(3), 227-237.
Evans C. (1997) The use of consensus methods and expert panels in pharmacoeconomic studies. Pharmacoeconomics 12, 121-129.
埃文斯 C.（1997）共识方法与专家小组在药物经济学研究中的应用。《药物经济学》12, 121-129.
Gilbride J.T. (2002) An Empirical Investigation of Critical Factors Affecting the Ability of Public Entities to Compete in Public Private Companies. Thesis, Department of the Air Force Air University, Air Force Institute of Technology, Wright-Patterson Air Fource Base, Ohio. Available at: http://research.maxwell.af.mil/papers/ay2002/ afit/afit-gaq-env-02m-09.pdf, accessed on 20 October 2004.
吉尔布莱德 J.T.（2002）《影响公共实体参与公私合营企业竞争能力关键因素的实证研究》。俄亥俄州赖特-帕特森空军基地空军技术学院空军大学系论文。访问地址：http://research.maxwell.af.mil/papers/ay2002/afit/afit-gaq-env-02m-09.pdf，2004 年 10 月 20 日查阅。
Green B., Jones M., Hughes D. & Williams A. (1999) Applying the Delphi technique in a study of GPs information requirements. Health and Social Care in the Community 7(3), 198-205.
格林 B.、琼斯 M.、休斯 D. 与威廉姆斯 A.（1999）《德尔菲法在全科医生信息需求研究中的应用》。《社区健康与社会关怀》7(3)，198-205 页。
Hartman A. (1981) Reaching consensus using the Delphi technique. Educational Leadership 38(6), 495-97.
哈特曼 A.（1981）《运用德尔菲技术达成共识》。《教育领导力》38(6)，495-497 页。
Hasson F. (2000). See McKenna H.P., Hasson F. & Smith (2000) A Multimethod Study of Skill Mix Issues in Midwifery. Conference Proceedings. Royal College of Surgeons of Ireland Annual Research Conference (Feb, unpublished).
哈森 F.（2000）。参见麦肯纳 H.P.、哈森 F. 与史密斯（2000）《助产士技能组合问题的多方法研究》。爱尔兰皇家外科医师学院年度研究会议论文集（2 月，未出版）。
Hasson F., Keeney S. & McKenna H. (2001) Research guidelines for the Delphi survey technique. Journal of Advanced Nursing 32, 1008-1015.
哈森·F、基尼·S 和麦肯纳·H（2001）德尔菲调查技术的研究指南。《高级护理杂志》32 卷，1008-1015 页。
Ilbery B., Maye D., Kneafsey M., Jenkins T. & Walkley C. (2004) Forecasting food supply chain developments in lagging rural regions: evidence from the UK. Journal of Rural Studies 20(3), 331-344.
伊尔伯里·B、梅·D、尼夫西·M、詹金斯·T 和沃克利·C（2004）预测落后农村地区食品供应链发展：来自英国的证据。《农村研究杂志》20 卷 3 期，331-344 页。
Jairath N. & Weinstein J. (1994) The Delphi Methodology (Part One): a useful administrative approach. Canadian Journal of Nursing Administration 7(3), 29-40.
杰拉特·N 和温斯坦·J（1994）德尔菲方法论（第一部分）：一种实用的管理方法。《加拿大护理管理杂志》7 卷 3 期，29-40 页。
Jones J., Sanderson C. & Black N. (1992) What will happen to the quality of care with fewer junior doctors? A Delphi study of consultant physicians’ views. J Roy Coll Phys 26, 36-40.
琼斯·J、桑德森·C 和布莱克·N（1992）初级医生减少将如何影响护理质量？顾问医师观点的德尔菲研究。《皇家医师学院杂志》26 卷，36-40 页。

Kaynak E., Bloom J. & Leibold M. (1994) Using the Delphi technique to predict future tourism potential. Marketing Intelligence & Planning 12(7), 18-29.
凯纳克·E.、布鲁姆·J. 和莱博尔德·M.（1994）运用德尔菲法预测未来旅游业潜力。《营销情报与规划》12(7)，18-29 页。
Keeney S. (2000) See McKenna H.P., Bradley M. & Keeney S. (2000) Primary Care Nursing: A Study Exploring Key Issues For Future Developments. University of Ulster, Ulster.
基尼·S.（2000）参见麦肯纳·H.P.、布拉德利·M. 和基尼·S.（2000）《初级护理护理学：探索未来发展的关键问题研究》。阿尔斯特大学，北爱尔兰。
Linstone H.A. (1978) The Delphi technique. In Handbook of Futures Research (Fowles R.B., ed.), Greenwood, Westport, CT, USA, pp. 273-300.
林斯通·H.A.（1978）德尔菲技术。载于《未来研究手册》（福尔斯·R.B.编），格林伍德出版社，美国康涅狄格州西港，273-300 页。
Lofmark A. & Thorell-Ekstrand I. (2004) An assessment form for clinical nursing education: a Delphi study. Journal of Advanced Nursing 48(3), 291-298.
洛夫马克·A. 和托雷尔-埃克斯特兰德·I.（2004）临床护理教育评估表：德尔菲研究。《高级护理学杂志》48(3)，291-298 页。
Loughlin K.G. & Moore L.F. (1979) Using Delphi to achieve congruent objectives and activities in a paediatrics department. Journal of Medical Education 54(2), 101-106.
洛林 K.G. 与摩尔 L.F. (1979) 运用德尔菲法实现儿科部门目标与活动的协调一致。《医学教育杂志》54(2), 101-106.
McIlfatrick S.J. & Keeney S. (2003) Identifying cancer nursing research priorities using the Delphi technique. Journal of Advanced Nursing 42(6), 629-636.
麦克尔法特里克 S.J. 与基尼 S. (2003) 采用德尔菲技术确定肿瘤护理研究重点。《高级护理杂志》42(6), 629-636.
McKenna H.P. (1994a) The Delphi technique: a worthwhile approach for nursing? Journal of Advanced Nursing 19 1221-1225.
麦肯纳 H.P. (1994a) 德尔菲技术：护理领域值得采用的方法？《高级护理杂志》19 1221-1225.
McKenna H.P. (1994b) The essential elements of a practitioners’ nursing model: a survey of clinical psychiatric nurse managers. Journal of Advanced Nursing 19, 870-877.
麦肯纳 H.P. (1994b) 从业者护理模式的核心要素：临床精神科护理管理者调查。《高级护理杂志》19, 870-877.
McKenna H.P. & Hasson F. (1998) The Formulation of Guidelines of Smoking in Pregnancy. University of Ulster, Ulster.
麦肯纳 H.P.与哈森 F.（1998）《妊娠期吸烟指南制定》。阿尔斯特大学，北爱尔兰。
McKenna H.P. & Hasson F. (2001) A study of skills mix issues in midwifery: a multi-method approach. Journal of Advanced Nursing 37(1), 52-61.
麦肯纳 H.P.与哈森 F.（2001）《助产士技能组合问题的多方法研究》。《高级护理杂志》37(1), 52-61。
McKenna H.P., Parahoo K.A. & Boore J. (1995) The evaluation of a nursing model for long-stay psychiatric patient care Part 2; presentation and discussion of findings. International Journal of Nursing Studies 31(1) 95-113.
麦肯纳 H.P.、帕拉胡 K.A.与布尔 J.（1995）《长期住院精神病患者护理模式的评估 第二部分：研究结果展示与讨论》。《国际护理研究杂志》31(1) 95-113。
McKenna H.P., Bradley M. & Keeney S. (2000) Primary Care Nursing: A Study Exploring Key Issues For Future Developments. University of Ulster, Ulster.
麦肯纳 H.P.、布拉德利 M.与基尼 S.（2000）《初级护理护理学：未来发展的关键问题研究》。阿尔斯特大学，北爱尔兰。
McMurray A.R. (1994) Three decision making aids - brainstorming, nominal group and the Delphi technique. Journal of Nursing Staff Development 10(2), 62-65.
麦默里 A.R.（1994）三种决策辅助工具——头脑风暴法、名义群体法与德尔菲技术。《护理人员发展杂志》10(2), 62-65.
Mead D. & Moseley L. (2001) The use of the Delphi as a research approach. Nurse Researcher 8(4), 4-23.
米德 D. 与莫斯利 L.（2001）德尔菲法作为研究途径的应用。《护士研究者》8(4), 4-23.
Polit D.F. & Hunger B.P. (1995) Nursing Research Principles & Methods, 5th edn. JB Lippincott Company, Philadelphia.
波利特 D.F. 与亨格 B.P.（1995）《护理研究原理与方法》第五版。费城：JB Lippincott 公司出版。
Powell C. (2003) The Delphi technique: myths and realities. Journal of Advanced Nursing 41(4), 376-382.
鲍威尔 C.（2003）德尔菲技术：误解与现实。《高级护理杂志》41(4), 376-382.
Proctor S. & Hunt M. (1994) Using the Delphi survey technique to develop a professional definition of nursing for analysing nursing workload. Journal of Advanced Nursing 19, 1003-1014.
普罗克特·S.与亨特·M.（1994）运用德尔菲调查技术构建护理专业定义以分析护理工作量。《高级护理杂志》19 卷，1003-1014 页。
Roberts E. (1969) Exploratory and normative technological forecasting: a critical appraisal. Technological Forecasting 1(2), 113-127.
罗伯茨·E.（1969）探索性与规范性技术预测：批判性评价。《技术预测》1(2)，113-127 页。
Sackman H. (1975) Delphi Critique. Lexington Books, MA, USA.
萨克曼·H.（1975）《德尔菲法批判》。莱克星顿图书公司，美国马萨诸塞州。
Starkweather D.B., Gelwicks L. & Newcomer R. (1975) Delphi forecasting of health care organisations. Inquiry 12, 37-46.
斯塔克韦瑟·D.B.、格尔威克斯·L.与纽科默·R.（1975）医疗机构德尔菲预测研究。《质询》12 卷，37-46 页。
Strauss H. & Zeigler H.L. (1975) The Delphi technique and its uses in social science research. Journal of Creative Behaviour 9(4), 253259.
施特劳斯 H. 与齐格勒 H.L.（1975）德尔菲技术及其在社会科学研究中的应用。《创造性行为期刊》9(4)，253-259 页。

Volk K. (1993) Curriculum development using the Delphi technique. Technology Research 52(4), 35-36.
沃尔克 K.（1993）运用德尔菲技术开发课程。《技术研究》52(4)，35-36 页。