Test changes systematically
系统地测试变化
Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an "eval").
如果能够对性能进行衡量，那么改进性能就更容易。在某些情况下，对提示进行修改可能会在一些孤立的示例上获得更好的性能，但在更具代表性的示例集上可能会导致更差的整体性能。因此，为确保改变对性能的净增益，可能需要定义全面的测试套件（也称为“评估”）。
Sometimes it can be hard to tell whether a change — e.g., a new instruction or a new design — makes your system better or worse. Looking at a few examples may hint at which is better, but with small sample sizes it can be hard to distinguish between a true improvement or random luck. Maybe the change helps performance on some inputs, but hurts performance on others.
有时很难判断一个变化（例如新的指令或新的设计）是使系统变得更好还是更糟。通过查看几个示例可能可以暗示哪个更好，但是对于小样本量来说，很难区分是真正的改进还是随机运气。也许这个变化在某些输入上有助于性能，但对其他输入有害。
Evaluation procedures (or "evals") are useful for optimizing system designs. Good evals are:
评估程序（或“evals”）对于优化系统设计非常有用。良好的评估应该具备以下特点：
- Representative of real-world usage (or at least diverse)
- 代表现实世界的使用情况（或至少多样化）
- Contain many test cases for greater statistical power (see table below for guidelines)
- 包含许多测试用例以增加统计功效（请参考下表以获取指导方针）
- Easy to automate or repeat
- 易于自动化或重复进行
DIFFERENCE TO DETECT 差异的检测 | SAMPLE SIZE NEEDED FOR 95% CONFIDENCE 需要的样本大小以达到95%的置信度 |
| 30% | ~10 |
| 10% | ~100 |
| 3% | ~1,000 |
| 1% | ~10,000 |
Evaluation of outputs can be done by computers, humans, or a mix. Computers can automate evals with objective criteria (e.g., questions with single correct answers) as well as some subjective or fuzzy criteria, in which model outputs are evaluated by other model queries. OpenAI Evals is an open-source software framework that provides tools for creating automated evals.
所做的评估可以由计算机、人类或二者混合进行。计算机可以根据客观标准（例如，具有单个正确答案的问题）自动化评估，也可以根据其他模型查询来评估模型输出的一些主观或模糊标准。OpenAI Evals 是一个开源软件框架，提供了创建自动化评估的工具。
Model-based evals can be useful when there exists a range of possible outputs that would be considered equally high in quality (e.g. for questions with long answers). The boundary between what can be realistically evaluated with a model-based eval and what requires a human to evaluate is fuzzy and is constantly shifting as models become more capable. We encourage experimentation to figure out how well model-based evals can work for your use case.
基于模型的评估在存在一系列可能的输出被认为具有相等高质量的情况下（例如，对于答案较长的问题）可能会很有用。基于模型的评估能够进行实际评估和需要人类评估的边界是模糊的，并且随着模型能力的提升而不断变化。我们鼓励进行实验，以确定基于模型的评估在您的用例中能够发挥多大作用。
Tactic: Evaluate model outputs with reference to gold-standard answers
技巧：根据黄金标准答案评估模型输出
Suppose it is known that the correct answer to a question should make reference to a specific set of known facts. Then we can use a model query to count how many of the required facts are included in the answer.
假设我们已知正确答案应该引用一组特定的已知事实。然后，我们可以使用模型查询来计算答案中包含的必需事实的数量。
For example, using the following system message:
例如，使用以下系统消息：
SYSTEM 系统 | You will be provided with text delimited by triple quotes that is supposed to be the answer to a question. Check if the following pieces of information are directly contained in the answer: 您将获得由三个反引号分隔的文本，这应该是一个问题的答案。检查以下信息是否直接包含在答案中： - Neil Armstrong was the first person to walk on the moon. 尼尔·阿姆斯特朗是第一个在月球上行走的人。 - The date Neil Armstrong first walked on the moon was July 21, 1969. 尼尔·阿姆斯特朗首次登上月球的日期是1969年7月21日。 For each of these points perform the following steps: 对于这些要点，执行以下步骤： 1 - Restate the point. 1 - 重新阐述要点。 2 - Provide a citation from the answer which is closest to this point. 2 - 提供离此要点最近的答案中的引用。 3 - Consider if someone reading the citation who doesn't know the topic could directly infer the point. Explain why or why not before making up your mind. 3 - 考虑如果阅读引用的人不了解这个主题，能否直接推断出这个要点。在做出决定之前解释为什么或为什么不。 4 - Write "yes" if the answer to 3 was yes, otherwise write "no". 4 - 如果答案是“是”，则写“yes”，否则写“no”。 Finally, provide a count of how many "yes" answers there are. Provide this count as {"count": <insert count here>}. 最后，提供有多少个“yes”答案的计数。将此计数提供为{"count": <插入计数>}。 |
Here's an example input where both points are satisfied:
以下是满足两个要点的示例输入：
SYSTEM 系统 | <insert system message above> <插入上面的系统消息> |
USER 用户 | """Neil Armstrong is famous for being the first human to set foot on the Moon. This historic event took place on July 21, 1969, during the Apollo 11 mission.""" """尼尔·阿姆斯特朗因成为第一个登上月球的人而闻名。这一历史性事件发生在1969年7月21日，属于阿波罗11号任务。""" |
Here's an example input where only one point is satisfied:
以下是只有一个满意要点的示例输入：
SYSTEM 系统 | <insert system message above> <插入上面的系统消息> |
USER 用户 | """Neil Armstrong made history when he stepped off the lunar module, becoming the first person to walk on the moon.""" """当尼尔·阿姆斯特朗踏出登月舱，成为第一个踏上月球的人时，他创造了历史。""" |
Here's an example input where none are satisfied:
这是一个示例输入，其中没有满足要求的部分：
SYSTEM 系统 | <insert system message above> <插入上述系统消息> |
USER 用户 | """In the summer of '69, a voyage grand, Apollo 11, bold as legend's hand. Armstrong took a step, history unfurled, "One small step," he said, for a new world.""" """在'69年的夏天，一次伟大的航行， 阿波罗11，勇敢如传说之手。 阿姆斯特朗迈出一步，历史展开， 他说：“迈出小小的一步”，为了一个新世界。 """ |
There are many possible variants on this type of model-based eval. Consider the following variation which tracks the kind of overlap between the candidate answer and the gold-standard answer, and also tracks whether the candidate answer contradicts any part of the gold-standard answer.
这种基于模型的评估可以有很多可能的变体。考虑以下变种，跟踪候选答案与黄金标准答案之间的重叠类型，并跟踪候选答案是否与黄金标准答案的任何部分相矛盾。
SYSTEM 系统 | Use the following steps to respond to user inputs. Fully restate each step before proceeding. i.e. "Step 1: Reason...". 使用以下步骤来回应用户输入。在继续之前，完整重新陈述每一步，例如“步骤1：逐步推理...”。 Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset). 步骤1：逐步推理提交的答案与专家答案的信息是否是不相交、相等、子集、超集或重叠（即有一些交集但不是子集/超集）。 Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer. 步骤2：逐步推理提交的答案是否与专家答案的任何方面相矛盾。 Step 3: Output a JSON object structured like: {"type_of_overlap": "disjoint" or "equal" or "subset" or "superset" or "overlapping", "contradiction": true or false} 步骤3：输出一个结构化的JSON对象，形式如下：{"type_of_overlap": "不相交"或"相等"或"子集"或"超集"或"重叠"，"contradiction": true或false}。 |
Here's an example input with a substandard answer which nonetheless does not contradict the expert answer:
以下是一个具有不太理想的答案但并不与专家答案相矛盾的示例输入：
SYSTEM 系统 | <insert system message above> <插入上述系统消息> |
USER 用户 | Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 问题："尼尔·阿姆斯特朗最出名的事件是什么？它发生在哪个日期？假设使用UTC时间。" Submitted Answer: """Didn't he walk on the moon or something?""" 提交的答案："他是不是在月球上行走了什么的？" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 专家答案："尼尔·阿姆斯特朗最出名的是成为第一个在月球上行走的人。这一历史事件发生在1969年7月21日。" |
Here's an example input with answer that directly contradicts the expert answer:
以下是一个与专家答案直接相矛盾的示例输入：
SYSTEM 系统 | <insert system message above> <插入上述系统消息> |
USER 用户 | Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 问题："尼尔·阿姆斯特朗最出名的事件是什么？它发生在哪个日期？假设使用UTC时间。" Submitted Answer: """On the 21st of July 1969, Neil Armstrong became the second person to walk on the moon, following after Buzz Aldrin.""" 提交的答案："在1969年7月21日，尼尔·阿姆斯特朗成为第二个登上月球的人，继巴兹·奥尔德林之后。" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 专家答案："尼尔·阿姆斯特朗最出名的是成为第一个在月球上行走的人。这一历史事件发生在1969年7月21日。" |
Here's an example input with a correct answer that also provides a bit more detail than is necessary:
以下是一个正确的答案，但提供了比必要的细节更多的示例输入：
SYSTEM 系统 | <insert system message above> <插入上述系统消息> |
USER 用户 | Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 问题："尼尔·阿姆斯特朗最出名的事件是什么？它发生在哪个日期？假设使用UTC时间。" Submitted Answer: """At approximately 02:56 UTC on July 21st 1969, Neil Armstrong became the first human to set foot on the lunar surface, marking a monumental achievement in human history.""" 提交的答案："在1969年7月21日UTC时间的02:56左右，尼尔·阿姆斯特朗成为第一个踏上月球表面的人，标志着人类历史上的一个重大成就。" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 专家答案："尼尔·阿姆斯特朗最出名的是成为第一个在月球上行走的人。这一历史事件发生在1969年7月21日。 |
“点赞有美意，赞赏是鼓励”
