12 Safe Agent Strategy Comparison

这一份 notebook 专门比较两种安全机制：

1. Stability-based Safe Gate
2. Meta-Safety Classifier

我们的目标不是再去追一点点普通 accuracy，而是比较：

• 谁更安全
• 谁的 coverage 更高
• 谁的 selective Macro-F1 更好

最后我们会把两种策略和之前 11 号文件里的 rule-based baseline 放到同一张表里。

from pathlib import Path
from datetime import datetime
import ast
import json

import numpy as np
import pandas as pd

from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

PROJECT_ROOT = Path('/Users/applesues01/Documents/Medical_Agent')
SUPPORT_DIR = PROJECT_ROOT / 'supports'

DYNAMIC_CASES_PATH = SUPPORT_DIR / '2026-08-04_104443_dynamic_agent_cases.csv'
IMAGE_ONLY_CASE_PATH = SUPPORT_DIR / 'image_only_case_level_predictions.csv'

print(DYNAMIC_CASES_PATH)
print(IMAGE_ONLY_CASE_PATH)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_104443_dynamic_agent_cases.csv
/Users/applesues01/Documents/Medical_Agent/supports/image_only_case_level_predictions.csv

1. 读取数据并做基础清洗

这里我们仍然用已经跑好的 dynamic agent 病例级结果，不重新训练主模型。

dynamic_df = pd.read_csv(DYNAMIC_CASES_PATH)
image_only_case_df = pd.read_csv(IMAGE_ONLY_CASE_PATH)

def parse_list_cell(value):
    if pd.isna(value):
        return []
    if isinstance(value, list):
        return value
    text = str(value).strip()
    if text == '' or text == '[]':
        return []
    try:
        return ast.literal_eval(text)
    except Exception:
        return [text]

bool_cols = [
    'initial_correct', 'final_correct', 'changed_prediction', 'improved',
    'worsened', 'still_wrong', 'still_correct', 'initial_high_conf_wrong',
    'final_high_conf_wrong', 'unsafe_confidence_increase'
]
for col in bool_cols:
    dynamic_df[col] = dynamic_df[col].astype(str).str.lower().map({'true': True, 'false': False})

dynamic_df['asked_questions_list'] = dynamic_df['asked_questions'].apply(parse_list_cell)
dynamic_df['known_keys_list'] = dynamic_df['known_keys'].apply(parse_list_cell)
dynamic_df['confidence_gain'] = dynamic_df['final_max_prob'] - dynamic_df['initial_max_prob']
dynamic_df['sex_only'] = dynamic_df['known_keys_list'].apply(lambda xs: xs == ['sex'])
dynamic_df['used_age'] = dynamic_df['known_keys_list'].apply(lambda xs: 'age' in xs)
dynamic_df['used_sex'] = dynamic_df['known_keys_list'].apply(lambda xs: 'sex' in xs)
dynamic_df['used_location'] = dynamic_df['known_keys_list'].apply(lambda xs: 'location' in xs)
dynamic_df['num_questions'] = dynamic_df['num_questions'].astype(int)

image_only_case_df = image_only_case_df.rename(columns={
    'pred_label': 'image_only_pred_label',
    'max_prob': 'image_only_max_prob'
})

merge_cols = ['image_id', 'image_only_pred_label', 'image_only_max_prob']
prob_cols = [c for c in image_only_case_df.columns if c.startswith('prob_')]
dynamic_df = dynamic_df.merge(image_only_case_df[merge_cols + prob_cols], on='image_id', how='left')

dynamic_df.head()

	image_id	true_label	initial_pred_label	initial_max_prob	initial_correct	final_pred_label	final_max_prob	final_correct	asked_questions	num_questions	...	used_location	image_only_pred_label	image_only_max_prob	prob_akiec	prob_bcc	prob_bkl	prob_df	prob_mel	prob_nv	prob_vasc
0	ISIC_0025837	bkl	bkl	0.972961	True	bkl	0.972961	True	[]	0	...	False	bkl	0.972961	0.004813	0.001014	0.972961	0.000886	0.018995	0.001147	0.000185
1	ISIC_0025209	bkl	bkl	0.407122	True	bkl	0.874424	True	['age', 'sex']	2	...	False	bkl	0.407122	0.220992	0.070574	0.407122	0.207246	0.073726	0.019543	0.000797
2	ISIC_0029161	bkl	bkl	0.793358	True	bkl	0.987926	True	['sex']	1	...	False	bkl	0.793358	0.000883	0.001113	0.793358	0.000149	0.074286	0.127832	0.002381
3	ISIC_0026273	bkl	bkl	0.802558	True	bkl	0.802558	True	[]	0	...	False	bkl	0.802558	0.000078	0.005748	0.802558	0.000180	0.094588	0.055271	0.041577
4	ISIC_0025819	bkl	bkl	0.978878	True	bkl	0.978878	True	[]	0	...	False	bkl	0.978878	0.000174	0.000190	0.978878	0.001796	0.009443	0.009492	0.000025

5 rows × 35 columns

2. 定义统一评估函数

无论是哪种安全策略，最后都统一输出：

• coverage
• abstain_rate
• selective_accuracy
• selective_balanced_accuracy
• selective_macro_f1
• unsafe_rate_among_diagnosed
• worsened_rate_among_diagnosed

def evaluate_abstention_policy(df, diagnose_mask, policy_name):
    out = df.copy()
    out['diagnose'] = pd.Series(diagnose_mask, index=out.index).astype(bool)
    out['abstain'] = ~out['diagnose']

    diagnosed = out[out['diagnose']].copy()
    coverage = len(diagnosed) / len(out)
    abstain_rate = 1.0 - coverage

    if len(diagnosed) == 0:
        return {
            'policy_name': policy_name,
            'coverage': 0.0,
            'abstain_rate': 1.0,
            'diagnosed_cases': 0,
            'selective_accuracy': np.nan,
            'selective_balanced_accuracy': np.nan,
            'selective_macro_f1': np.nan,
            'unsafe_rate_among_diagnosed': np.nan,
            'worsened_rate_among_diagnosed': np.nan,
        }, diagnosed

    summary = {
        'policy_name': policy_name,
        'coverage': coverage,
        'abstain_rate': abstain_rate,
        'diagnosed_cases': int(len(diagnosed)),
        'selective_accuracy': accuracy_score(diagnosed['true_label'], diagnosed['final_pred_label']),
        'selective_balanced_accuracy': balanced_accuracy_score(diagnosed['true_label'], diagnosed['final_pred_label']),
        'selective_macro_f1': f1_score(diagnosed['true_label'], diagnosed['final_pred_label'], average='macro'),
        'unsafe_rate_among_diagnosed': diagnosed['unsafe_confidence_increase'].mean(),
        'worsened_rate_among_diagnosed': diagnosed['worsened'].mean(),
    }
    return summary, diagnosed

3. 先放一个规则基线，后面方便比较

这里沿用 11 号文件里最强的 rule-based 方案之一：

• prob >= 0.90
• confidence_gain <= 0.10

我把它当作这次对比里的基线版本。

rule_mask = (
    (dynamic_df['final_max_prob'] >= 0.90) &
    (dynamic_df['confidence_gain'] <= 0.10)
)
rule_summary, rule_diagnosed = evaluate_abstention_policy(dynamic_df, rule_mask, 'rule_based_balanced')
rule_summary

{'policy_name': 'rule_based_balanced',
 'coverage': 0.5165428764348413,
 'abstain_rate': 0.4834571235651587,
 'diagnosed_cases': 765,
 'selective_accuracy': 0.9464052287581699,
 'selective_balanced_accuracy': 0.8591156401262591,
 'selective_macro_f1': 0.8375490004215721,
 'unsafe_rate_among_diagnosed': np.float64(0.0),
 'worsened_rate_among_diagnosed': np.float64(0.0)}

4. Strategy A: Stability-based Safe Gate

这一版不只看最后置信度高不高，而是看系统是否“稳定”。

我们先构造一些稳定性特征：

• prediction_changed
• confidence_gain
• question_count
• 是否只问了 sex
• image-only 的 top1 置信度
• image-only 的 top1-top2 margin

然后用规则扫描做一版稳定性门。

prob_cols = [c for c in dynamic_df.columns if c.startswith('prob_')]

def top2_margin_from_row(row):
    vals = sorted([row[c] for c in prob_cols], reverse=True)
    return vals[0] - vals[1]

dynamic_df['image_only_margin'] = dynamic_df.apply(top2_margin_from_row, axis=1)
dynamic_df['prediction_changed'] = dynamic_df['changed_prediction'].astype(int)
dynamic_df['sex_only_int'] = dynamic_df['sex_only'].astype(int)

dynamic_df[[
    'image_id', 'image_only_max_prob', 'image_only_margin', 'confidence_gain',
    'prediction_changed', 'num_questions', 'sex_only'
]].head()

	image_id	image_only_max_prob	image_only_margin	confidence_gain	prediction_changed	num_questions	sex_only
0	ISIC_0025837	0.972961	0.953966	0.000000	0	0	False
1	ISIC_0025209	0.407122	0.186130	0.467302	0	2	False
2	ISIC_0029161	0.793358	0.665526	0.194568	0	1	True
3	ISIC_0026273	0.802558	0.707970	0.000000	0	0	False
4	ISIC_0025819	0.978878	0.969386	0.000000	0	0	False

stability_results = []

for final_prob_thr in [0.80, 0.85, 0.90, 0.95]:
    for gain_cap in [0.10, 0.20, 0.30]:
        for margin_thr in [0.10, 0.20, 0.30, 0.40]:
            for allow_changed in [False, True]:
                mask = dynamic_df['final_max_prob'] >= final_prob_thr
                mask &= dynamic_df['confidence_gain'] <= gain_cap
                mask &= dynamic_df['image_only_margin'] >= margin_thr
                if not allow_changed:
                    mask &= (dynamic_df['prediction_changed'] == 0)

                summary, _ = evaluate_abstention_policy(
                    dynamic_df,
                    mask,
                    policy_name=f'stability_prob>={final_prob_thr}_gain<={gain_cap}_margin>={margin_thr}_allowChanged={allow_changed}'
                )
                summary['strategy'] = 'stability_gate'
                summary['final_prob_thr'] = final_prob_thr
                summary['gain_cap'] = gain_cap
                summary['margin_thr'] = margin_thr
                summary['allow_changed'] = allow_changed
                stability_results.append(summary)

stability_df = pd.DataFrame(stability_results)
stability_df = stability_df[stability_df['coverage'] >= 0.30].copy()
stability_df.sort_values(
    ['unsafe_rate_among_diagnosed', 'selective_macro_f1', 'coverage'],
    ascending=[True, False, False]
).head(20)

	policy_name	coverage	abstain_rate	diagnosed_cases	selective_accuracy	selective_balanced_accuracy	selective_macro_f1	unsafe_rate_among_diagnosed	worsened_rate_among_diagnosed	strategy	final_prob_thr	gain_cap	margin_thr	allow_changed
72	stability_prob>=0.95_gain<=0.1_margin>=0.1_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.1	False
73	stability_prob>=0.95_gain<=0.1_margin>=0.1_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.1	True
74	stability_prob>=0.95_gain<=0.1_margin>=0.2_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.2	False
75	stability_prob>=0.95_gain<=0.1_margin>=0.2_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.2	True
76	stability_prob>=0.95_gain<=0.1_margin>=0.3_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.3	False
77	stability_prob>=0.95_gain<=0.1_margin>=0.3_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.3	True
78	stability_prob>=0.95_gain<=0.1_margin>=0.4_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.4	False
79	stability_prob>=0.95_gain<=0.1_margin>=0.4_all...	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0	stability_gate	0.95	0.1	0.4	True
48	stability_prob>=0.9_gain<=0.1_margin>=0.1_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.1	False
49	stability_prob>=0.9_gain<=0.1_margin>=0.1_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.1	True
50	stability_prob>=0.9_gain<=0.1_margin>=0.2_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.2	False
51	stability_prob>=0.9_gain<=0.1_margin>=0.2_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.2	True
52	stability_prob>=0.9_gain<=0.1_margin>=0.3_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.3	False
53	stability_prob>=0.9_gain<=0.1_margin>=0.3_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.3	True
54	stability_prob>=0.9_gain<=0.1_margin>=0.4_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.4	False
55	stability_prob>=0.9_gain<=0.1_margin>=0.4_allo...	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0	stability_gate	0.90	0.1	0.4	True
24	stability_prob>=0.85_gain<=0.1_margin>=0.1_all...	0.588116	0.411884	871	0.928817	0.827409	0.796048	0.0	0.0	stability_gate	0.85	0.1	0.1	False
25	stability_prob>=0.85_gain<=0.1_margin>=0.1_all...	0.588116	0.411884	871	0.928817	0.827409	0.796048	0.0	0.0	stability_gate	0.85	0.1	0.1	True
26	stability_prob>=0.85_gain<=0.1_margin>=0.2_all...	0.588116	0.411884	871	0.928817	0.827409	0.796048	0.0	0.0	stability_gate	0.85	0.1	0.2	False
27	stability_prob>=0.85_gain<=0.1_margin>=0.2_all...	0.588116	0.411884	871	0.928817	0.827409	0.796048	0.0	0.0	stability_gate	0.85	0.1	0.2	True

5. Strategy B: Meta-Safety Classifier

这一版我们训练一个很小的二分类器，不预测病种，只预测：

当前这个病例，应该 diagnose 还是 abstain？

为了让目标更贴近安全，我们把“安全可诊断”定义成：

• final_correct == True
• unsafe_confidence_increase == False
• worsened == False

也就是说，它必须既答对，又没有出现危险信号。

dynamic_df['safe_target'] = (
    dynamic_df['final_correct'] &
    (~dynamic_df['unsafe_confidence_increase']) &
    (~dynamic_df['worsened'])
).astype(int)

feature_cols = [
    'initial_max_prob',
    'final_max_prob',
    'confidence_gain',
    'num_questions',
    'prediction_changed',
    'sex_only_int',
    'used_age',
    'used_sex',
    'used_location',
    'image_only_max_prob',
    'image_only_margin'
]

X = dynamic_df[feature_cols].copy()
X['used_age'] = X['used_age'].astype(int)
X['used_sex'] = X['used_sex'].astype(int)
X['used_location'] = X['used_location'].astype(int)
y = dynamic_df['safe_target'].astype(int)

X_train, X_valid, y_train, y_valid, idx_train, idx_valid = train_test_split(
    X, y, dynamic_df.index,
    test_size=0.30,
    random_state=42,
    stratify=y
)

meta_safety_clf = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=2000, class_weight='balanced'))
])

meta_safety_clf.fit(X_train, y_train)
valid_proba = meta_safety_clf.predict_proba(X_valid)[:, 1]
print('Validation ROC-AUC:', roc_auc_score(y_valid, valid_proba))

Validation ROC-AUC: 0.8392582059485721

meta_results = []

all_safe_proba = meta_safety_clf.predict_proba(X)[:, 1]
dynamic_df['safe_proba'] = all_safe_proba

for safe_thr in [0.50, 0.60, 0.70, 0.80, 0.90]:
    mask = dynamic_df['safe_proba'] >= safe_thr
    summary, _ = evaluate_abstention_policy(dynamic_df, mask, f'meta_safety_thr>={safe_thr}')
    summary['strategy'] = 'meta_safety_classifier'
    summary['safe_prob_threshold'] = safe_thr
    meta_results.append(summary)

meta_df = pd.DataFrame(meta_results)
meta_df = meta_df[meta_df['coverage'] >= 0.30].copy()
meta_df.sort_values(
    ['unsafe_rate_among_diagnosed', 'selective_macro_f1', 'coverage'],
    ascending=[True, False, False]
).head(20)

	policy_name	coverage	abstain_rate	diagnosed_cases	selective_accuracy	selective_balanced_accuracy	selective_macro_f1	unsafe_rate_among_diagnosed	worsened_rate_among_diagnosed	strategy	safe_prob_threshold
3	meta_safety_thr>=0.8	0.375422	0.624578	556	0.983813	0.953370	0.938426	0.000000	0.0	meta_safety_classifier	0.8
2	meta_safety_thr>=0.7	0.484808	0.515192	718	0.958217	0.873238	0.851442	0.000000	0.0	meta_safety_classifier	0.7
1	meta_safety_thr>=0.6	0.553005	0.446995	819	0.931624	0.836900	0.796945	0.001221	0.0	meta_safety_classifier	0.6
0	meta_safety_thr>=0.5	0.621877	0.378123	921	0.918567	0.793345	0.771019	0.010858	0.0	meta_safety_classifier	0.5

6. 三种方法放到一起比较

这里我们统一比较：

• 规则基线
• 最佳稳定性安全门
• 最佳 meta-safety classifier

best_stability_row = stability_df.sort_values(
    ['unsafe_rate_among_diagnosed', 'selective_macro_f1', 'coverage'],
    ascending=[True, False, False]
).iloc[0]

best_meta_row = meta_df.sort_values(
    ['unsafe_rate_among_diagnosed', 'selective_macro_f1', 'coverage'],
    ascending=[True, False, False]
).iloc[0]

best_stability_mask = dynamic_df['final_max_prob'] >= best_stability_row['final_prob_thr']
best_stability_mask &= dynamic_df['confidence_gain'] <= best_stability_row['gain_cap']
best_stability_mask &= dynamic_df['image_only_margin'] >= best_stability_row['margin_thr']
if not bool(best_stability_row['allow_changed']):
    best_stability_mask &= (dynamic_df['prediction_changed'] == 0)

best_stability_summary, best_stability_cases = evaluate_abstention_policy(
    dynamic_df,
    best_stability_mask,
    'best_stability_gate'
)

best_meta_mask = dynamic_df['safe_proba'] >= best_meta_row['safe_prob_threshold']
best_meta_summary, best_meta_cases = evaluate_abstention_policy(
    dynamic_df,
    best_meta_mask,
    'best_meta_safety_classifier'
)

comparison_df = pd.DataFrame([
    {'strategy_family': 'rule_based', **rule_summary},
    {'strategy_family': 'stability_gate', **best_stability_summary},
    {'strategy_family': 'meta_safety_classifier', **best_meta_summary},
])

comparison_df

	strategy_family	policy_name	coverage	abstain_rate	diagnosed_cases	selective_accuracy	selective_balanced_accuracy	selective_macro_f1	unsafe_rate_among_diagnosed	worsened_rate_among_diagnosed
0	rule_based	rule_based_balanced	0.516543	0.483457	765	0.946405	0.859116	0.837549	0.0	0.0
1	stability_gate	best_stability_gate	0.411209	0.588791	609	0.978654	0.917516	0.906395	0.0	0.0
2	meta_safety_classifier	best_meta_safety_classifier	0.375422	0.624578	556	0.983813	0.953370	0.938426	0.0	0.0

7. 保存结果

这一格会把：

• stability gate 搜索表
• meta-safety 搜索表
• 三策略最终对比表

全部按时间戳保存下来。

timestamp = datetime.now().strftime('%Y-%m-%d_%H%M%S')

stability_path = SUPPORT_DIR / f'{timestamp}_stability_safe_gate_results.csv'
meta_path = SUPPORT_DIR / f'{timestamp}_meta_safety_classifier_results.csv'
comparison_path = SUPPORT_DIR / f'{timestamp}_safe_strategy_comparison.csv'

stability_df.to_csv(stability_path, index=False)
meta_df.to_csv(meta_path, index=False)
comparison_df.to_csv(comparison_path, index=False)

print(stability_path)
print(meta_path)
print(comparison_path)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_150030_stability_safe_gate_results.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_150030_meta_safety_classifier_results.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_150030_safe_strategy_comparison.csv

8. 你最后怎么判断谁更好

你最后重点看这几件事：

1. unsafe_rate_among_diagnosed 能不能继续保持很低
2. 谁的 coverage 更高
3. 谁的 selective_macro_f1 更高
4. 有没有哪一类策略能明显优于 rule-based baseline

如果有，我们下一步就可以把它升级成论文里的正式安全模块。

保守的男人没有魅力

当前 3 类安全策略的优缺点

1. rule_based

当前结果：

• coverage = 0.5165
• selective_macro_f1 = 0.8375
• unsafe_rate = 0.0
• worsened_rate = 0.0

优点：

• 三者里最平衡
• 覆盖率最高
• 已经能把危险高置信错误压到 0
• 规则直观，论文里最好解释

缺点：

• 还是比较保守，拒答接近一半病例
• 本质上还是手工阈值
• 没有真正理解“为什么这个病例安全”

适合论文中的角色：

• 安全机制基线
• 当前最实用版本

2. stability_gate

当前结果：

• coverage = 0.4112
• selective_macro_f1 = 0.9064
• unsafe_rate = 0.0
• worsened_rate = 0.0

优点：

• 比 rule-based 更关注“预测是否稳定”
• selective 性能明显更高
• 比单纯看最终置信度更有研究意味

缺点：

• coverage 掉得比较明显
• 还是偏保守
• 说明“稳定”能筛出好病例，但还不够会保留更多病例

适合论文中的角色：

• 更强的安全门对照方法
• 证明“稳定性信号”是有价值的

3. meta_safety_classifier

当前结果：

• coverage = 0.3754
• selective_macro_f1 = 0.9384
• unsafe_rate = 0.0
• worsened_rate = 0.0

优点：

• 三者里 selective 表现最好
• 最会挑“最安全的病例”
• 说明数据驱动的安全判别是可行的

缺点：

• coverage 最低
• 太保守
• 现在更像“高精度筛子”，不像临床上真正可用的 agent

适合论文中的角色：

• 上限型安全方法
• 证明 learned safety scoring 有潜力，但当前实用性不足

---

总体结论

这 3 条线共同说明了一件事：

只靠最后一层“拒答/安全门”做修补，确实能提升安全性，但很容易以牺牲 coverage 为代价。

换句话说：

• 安全问题我们已经抓住了
• 但当前解决方式都偏“事后过滤”
• 它们能让系统更稳，却不太能让系统“又稳又多答对一些”

---

为什么会越来越保守

因为这 3 种方法本质上都在做同一件事：

从已经产生的 dynamic agent 结果里，筛掉风险大的病例

所以它们天然更擅长：

• 找出“不该答”的病例

但不擅长：

• 找出“问哪个问题会安全地变好”

这就是为什么：

• unsafe_rate 可以压到 0
• 但 coverage 很难一起上去

---

这对下一步的启发

所以真正该前移研究的位置，不是“最后要不要拒答”，而是：

在提问阶段，怎样选择既有收益、又更安全的问题。

也就是下一步应该做：

Risk-Aware Question Selection

它和现在的区别是：

• 现在：问完了，再决定答不答
• 下一步：问之前，就评估这个问题值不值得问、安不安全

---

现在这一步可以写进论文的表述

可以这样概括：

We found that post-hoc abstention substantially improved the reliability of diagnosed cases, but consistently reduced coverage. This suggests that safety control should not rely solely on final-stage rejection, and instead should be incorporated into the question selection policy itself.

中文就是：

我们发现，事后拒答机制虽然能够显著提高被诊断病例的可靠性，但会稳定地降低覆盖率。这说明安全控制不应仅依赖最终阶段的拒答，而应进一步前移到问题选择策略中。