16 Case-Specific Risk-Aware Agent¶
这一份 notebook 开始做真正的 病例级问题选择。
前面 15 号的问题在于：
- 虽然名字叫 adaptive
- 但打分仍然只依赖全局统计和状态级分数
- 所以在同一个状态下，不同病例仍然会走同一条路
这一版要解决的就是这个问题：
让不同病例根据自己的当前预测分布，选择不同的问题。
核心思路：
- 先读取 image-only 的病例级概率分布
- 对每个病例，提取当前最可能的候选类别结构
- 用“候选类别不确定性 + 问题风险惩罚”来给
age / sex / location打分 - 选对当前病例最合适的问题，而不是全局统一排序
In [3]:
from pathlib import Path
from datetime import datetime
import json
import numpy as np
import pandas as pd
PROJECT_ROOT = Path('/Users/applesues01/Documents/Medical_Agent')
DATA_DIR = PROJECT_ROOT / 'data' / 'HAM10000'
SPLIT_DIR = DATA_DIR / 'splits'
SUPPORT_DIR = PROJECT_ROOT / 'supports'
BASELINE_RESULTS_PATH = SUPPORT_DIR / 'baseline_results.csv'
MODEL_REGISTRY_PATH = SUPPORT_DIR / '2026-08-03_194704_all_saved_metadata_models.csv'
QUESTION_SCORE_PATH = SUPPORT_DIR / '2026-08-04_163052_risk_aware_score_table.csv'
IMAGE_ONLY_CASE_PATH = SUPPORT_DIR / 'image_only_case_level_predictions.csv'
VALIDATED_AGENT_PATH = SUPPORT_DIR / '2026-08-03_185733_validated_agent_comparison.csv'
print(IMAGE_ONLY_CASE_PATH)
print(QUESTION_SCORE_PATH)
print(MODEL_REGISTRY_PATH)
/Users/applesues01/Documents/Medical_Agent/supports/image_only_case_level_predictions.csv /Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_163052_risk_aware_score_table.csv /Users/applesues01/Documents/Medical_Agent/supports/2026-08-03_194704_all_saved_metadata_models.csv
1. 读取数据和模型分数¶
In [4]:
train_df = pd.read_csv(SPLIT_DIR / 'train.csv')
val_df = pd.read_csv(SPLIT_DIR / 'val.csv')
test_df = pd.read_csv(SPLIT_DIR / 'test.csv')
baseline_df = pd.read_csv(BASELINE_RESULTS_PATH)
model_registry_df = pd.read_csv(MODEL_REGISTRY_PATH)
question_score_df = pd.read_csv(QUESTION_SCORE_PATH)
image_only_case_df = pd.read_csv(IMAGE_ONLY_CASE_PATH)
validated_agent_df = pd.read_csv(VALIDATED_AGENT_PATH)
image_only_row = baseline_df[baseline_df['Method'] == 'Image Only'].iloc[0]
policy_model_scores = {
'image_only': {'macro_f1': float(image_only_row['Macro-F1'])},
}
report_model_scores = {
'image_only': {
'accuracy': float(image_only_row['Accuracy']),
'balanced_accuracy': float(image_only_row['Balanced Accuracy']),
'macro_f1': float(image_only_row['Macro-F1']),
}
}
for _, row in model_registry_df.iterrows():
key = str(row['Method'])
policy_model_scores[key] = {'macro_f1': float(row['Best Val Macro-F1'])}
report_model_scores[key] = {
'accuracy': float(row['Test Accuracy']),
'balanced_accuracy': float(row['Test Balanced Accuracy']),
'macro_f1': float(row['Test Macro-F1']),
}
len(test_df), len(image_only_case_df)
Out[4]:
(1481, 1481)
In [5]:
global_question_score = {
row['question']: float(row['risk_aware_score'])
for _, row in question_score_df.iterrows()
}
global_question_score
Out[5]:
{'age': -0.4508196721311475,
'sex': -0.6773255813953488,
'location': -0.8148148148148148}
2. 合并病例级 image-only 预测分布¶
这一步是关键。我们后面的提问决策，第一次真正依赖具体病例自己的概率分布。
In [6]:
case_df = test_df.merge(image_only_case_df, on=['image_id'], how='left')
prob_cols = [c for c in case_df.columns if c.startswith('prob_')]
class_names = [c.replace('prob_', '') for c in prob_cols]
case_df[['image_id', 'dx', 'pred_label', 'max_prob'] + prob_cols[:3]].head()
Out[6]:
| image_id | dx | pred_label | max_prob | prob_akiec | prob_bcc | prob_bkl | |
|---|---|---|---|---|---|---|---|
| 0 | ISIC_0025837 | bkl | bkl | 0.972961 | 0.004813 | 0.001014 | 0.972961 |
| 1 | ISIC_0025209 | bkl | bkl | 0.407122 | 0.220992 | 0.070574 | 0.407122 |
| 2 | ISIC_0029161 | bkl | bkl | 0.793358 | 0.000883 | 0.001113 | 0.793358 |
| 3 | ISIC_0026273 | bkl | bkl | 0.802558 | 0.000078 | 0.005748 | 0.802558 |
| 4 | ISIC_0025819 | bkl | bkl | 0.978878 | 0.000174 | 0.000190 | 0.978878 |
3. 提取病例级不确定性结构¶
我们先从 image-only 概率分布里提取几个最直观的结构：
- top1 类别
- top2 类别
- top1-top2 margin
- 熵（entropy）
这些量会帮助我们判断：当前病例最像哪些病，以及它到底有多纠结。
In [7]:
def extract_case_structure(row):
probs = np.array([row[c] for c in prob_cols], dtype=float)
order = np.argsort(-probs)
top1_idx = int(order[0])
top2_idx = int(order[1])
top1_cls = class_names[top1_idx]
top2_cls = class_names[top2_idx]
top1_prob = float(probs[top1_idx])
top2_prob = float(probs[top2_idx])
margin = top1_prob - top2_prob
entropy = float(-(probs * np.log(probs + 1e-12)).sum())
return pd.Series({
'top1_cls': top1_cls,
'top2_cls': top2_cls,
'top1_prob': top1_prob,
'top2_prob': top2_prob,
'margin': margin,
'entropy': entropy,
})
case_structure_df = case_df.apply(extract_case_structure, axis=1)
case_df = pd.concat([case_df, case_structure_df], axis=1)
case_df[['image_id', 'dx', 'top1_cls', 'top2_cls', 'top1_prob', 'top2_prob', 'margin', 'entropy']].head()
Out[7]:
| image_id | dx | top1_cls | top2_cls | top1_prob | top2_prob | margin | entropy | |
|---|---|---|---|---|---|---|---|---|
| 0 | ISIC_0025837 | bkl | bkl | mel | 0.972961 | 0.018995 | 0.953966 | 0.150215 |
| 1 | ISIC_0025209 | bkl | bkl | akiec | 0.407122 | 0.220992 | 0.186130 | 1.487569 |
| 2 | ISIC_0029161 | bkl | bkl | nv | 0.793358 | 0.127832 | 0.665526 | 0.669199 |
| 3 | ISIC_0026273 | bkl | bkl | mel | 0.802558 | 0.094588 | 0.707970 | 0.723783 |
| 4 | ISIC_0025819 | bkl | bkl | nv | 0.978878 | 0.009492 | 0.969386 | 0.123900 |
In [8]:
def get_model_key_from_known_set(known_set):
if known_set == set():
return 'image_only'
if known_set == {'age'}:
return 'image_age'
if known_set == {'sex'}:
return 'image_sex'
if known_set == {'location'}:
return 'image_location'
if known_set == {'age', 'sex'}:
return 'image_age_sex'
if known_set == {'age', 'location'}:
return 'image_age_location'
if known_set == {'sex', 'location'}:
return 'image_sex_location'
if known_set == {'age', 'sex', 'location'}:
return 'image_all_metadata'
raise ValueError(f'Unknown known_set: {known_set}')
def build_initial_state(row):
return {
'image_id': row['image_id'],
'true_label': row['dx'],
'known_metadata': {},
'asked_questions': [],
'done': False,
'top1_cls': row['top1_cls'],
'top2_cls': row['top2_cls'],
'top1_prob': float(row['top1_prob']),
'top2_prob': float(row['top2_prob']),
'margin': float(row['margin']),
'entropy': float(row['entropy']),
}
def ask_question(state, row, question):
assert question in ['age', 'sex', 'location']
new_state = {
'image_id': state['image_id'],
'true_label': state['true_label'],
'known_metadata': dict(state['known_metadata']),
'asked_questions': list(state['asked_questions']),
'done': state['done'],
'top1_cls': state['top1_cls'],
'top2_cls': state['top2_cls'],
'top1_prob': state['top1_prob'],
'top2_prob': state['top2_prob'],
'margin': state['margin'],
'entropy': state['entropy'],
}
if question == 'age':
new_state['known_metadata']['age'] = row['age']
elif question == 'sex':
new_state['known_metadata']['sex'] = row['sex']
elif question == 'location':
new_state['known_metadata']['location'] = row['localization']
new_state['asked_questions'].append(question)
return new_state
def get_model_key_from_state(state):
return get_model_key_from_known_set(set(state['known_metadata'].keys()))
def score_state(state):
model_key = get_model_key_from_state(state)
return policy_model_scores[model_key]['macro_f1']
In [9]:
def case_specific_question_score(state, question, alpha=1.0, lambda_uncertainty=0.2):
current_known = set(state['known_metadata'].keys())
if question in current_known:
return -1e9
current_key = get_model_key_from_known_set(current_known)
future_key = get_model_key_from_known_set(current_known | {question})
local_gain = policy_model_scores[future_key]['macro_f1'] - policy_model_scores[current_key]['macro_f1']
risk_prior = global_question_score[question]
uncertainty_bonus = (1.0 - state['margin']) + 0.1 * state['entropy']
return local_gain + lambda_uncertainty * uncertainty_bonus + alpha * risk_prior
def case_specific_risk_aware_policy(state, max_questions=2, alpha=1.0, lambda_uncertainty=0.2):
if len(state['asked_questions']) >= max_questions:
return 'diagnose'
candidates = [q for q in ['age', 'sex', 'location'] if q not in state['asked_questions']]
if not candidates:
return 'diagnose'
best_question = None
best_score = -1e18
for q in candidates:
s = case_specific_question_score(state, q, alpha=alpha, lambda_uncertainty=lambda_uncertainty)
if s > best_score:
best_score = s
best_question = q
if best_question is None:
return 'diagnose'
return best_question
5. 先看几个单病例轨迹¶
In [10]:
def run_case_specific_episode(row, max_questions=2, alpha=1.0, lambda_uncertainty=0.2):
state = build_initial_state(row)
trajectory = []
while not state['done']:
action = case_specific_risk_aware_policy(
state,
max_questions=max_questions,
alpha=alpha,
lambda_uncertainty=lambda_uncertainty,
)
trajectory.append({
'step': len(trajectory),
'action': action,
'model_key_before_action': get_model_key_from_state(state),
'score_before_action': score_state(state),
'known_metadata_before_action': dict(state['known_metadata']),
'top1_cls': state['top1_cls'],
'top2_cls': state['top2_cls'],
'margin': state['margin'],
})
if action == 'diagnose':
state['done'] = True
state['final_action'] = 'diagnose'
state['final_model_key'] = get_model_key_from_state(state)
state['final_score'] = score_state(state)
break
state = ask_question(state, row, action)
return trajectory, state
In [11]:
sample_row = case_df.iloc[0]
trajectory, final_state = run_case_specific_episode(sample_row, max_questions=2, alpha=1.0, lambda_uncertainty=0.2)
trajectory, final_state
Out[11]:
([{'step': 0,
'action': 'age',
'model_key_before_action': 'image_only',
'score_before_action': 0.6083234281767498,
'known_metadata_before_action': {},
'top1_cls': 'bkl',
'top2_cls': 'mel',
'margin': 0.9539660867303611},
{'step': 1,
'action': 'sex',
'model_key_before_action': 'image_age',
'score_before_action': 0.5926280556025649,
'known_metadata_before_action': {'age': np.float64(70.0)},
'top1_cls': 'bkl',
'top2_cls': 'mel',
'margin': 0.9539660867303611},
{'step': 2,
'action': 'diagnose',
'model_key_before_action': 'image_age_sex',
'score_before_action': 0.5964635518225668,
'known_metadata_before_action': {'age': np.float64(70.0), 'sex': 'female'},
'top1_cls': 'bkl',
'top2_cls': 'mel',
'margin': 0.9539660867303611}],
{'image_id': 'ISIC_0025837',
'true_label': 'bkl',
'known_metadata': {'age': np.float64(70.0), 'sex': 'female'},
'asked_questions': ['age', 'sex'],
'done': True,
'top1_cls': 'bkl',
'top2_cls': 'mel',
'top1_prob': 0.972960650920868,
'top2_prob': 0.0189945641905069,
'margin': 0.9539660867303611,
'entropy': 0.15021531890606907,
'final_action': 'diagnose',
'final_model_key': 'image_age_sex',
'final_score': 0.5964635518225668})
6. 全测试集跑第一版病例级 risk-aware adaptive agent¶
In [12]:
def evaluate_case_specific_agent(alpha=1.0, lambda_uncertainty=0.2, max_questions=2):
records = []
for idx in range(len(case_df)):
row = case_df.iloc[idx]
trajectory, final_state = run_case_specific_episode(
row,
max_questions=max_questions,
alpha=alpha,
lambda_uncertainty=lambda_uncertainty,
)
records.append({
'image_id': row['image_id'],
'true_label': row['dx'],
'top1_cls': row['top1_cls'],
'top2_cls': row['top2_cls'],
'margin': row['margin'],
'asked_questions': list(final_state['asked_questions']),
'num_questions': len(final_state['asked_questions']),
'final_model_key': final_state['final_model_key'],
'final_policy_score': final_state['final_score'],
'known_metadata': dict(final_state['known_metadata']),
})
cases_df = pd.DataFrame(records)
summary_df = cases_df.groupby('final_model_key').size().reset_index(name='count')
summary_df['ratio'] = summary_df['count'] / len(cases_df)
summary_df['test_accuracy'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['accuracy'])
summary_df['test_balanced_accuracy'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['balanced_accuracy'])
summary_df['test_macro_f1'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['macro_f1'])
result = {
'agent_name': 'case_specific_risk_aware',
'alpha': alpha,
'lambda_uncertainty': lambda_uncertainty,
'max_questions': max_questions,
'avg_questions': float(cases_df['num_questions'].mean()),
'expected_accuracy': float((summary_df['ratio'] * summary_df['test_accuracy']).sum()),
'expected_balanced_accuracy': float((summary_df['ratio'] * summary_df['test_balanced_accuracy']).sum()),
'expected_macro_f1': float((summary_df['ratio'] * summary_df['test_macro_f1']).sum()),
}
return cases_df, summary_df, result
In [13]:
case_specific_cases_df, case_specific_summary_df, case_specific_result = evaluate_case_specific_agent(
alpha=1.0,
lambda_uncertainty=0.2,
max_questions=2,
)
case_specific_result
Out[13]:
{'agent_name': 'case_specific_risk_aware',
'alpha': 1.0,
'lambda_uncertainty': 0.2,
'max_questions': 2,
'avg_questions': 2.0,
'expected_accuracy': 0.7886563133018231,
'expected_balanced_accuracy': 0.5856809903171467,
'expected_macro_f1': 0.5837050128198616}
7. 和之前 baseline agent 对比¶
In [14]:
comparison_df = pd.concat([
validated_agent_df,
pd.DataFrame([case_specific_result])
], ignore_index=True)
comparison_df
Out[14]:
| agent_name | max_questions | avg_questions | expected_accuracy | expected_balanced_accuracy | expected_macro_f1 | threshold | alpha | lambda_uncertainty | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | fixed_order | 2 | 2.0 | 0.793383 | 0.582281 | 0.581666 | NaN | NaN | NaN |
| 1 | uncertainty | 2 | 2.0 | 0.793383 | 0.582281 | 0.581666 | 0.605 | NaN | NaN |
| 2 | lookahead | 2 | 2.0 | 0.792708 | 0.598458 | 0.596497 | NaN | NaN | NaN |
| 3 | case_specific_risk_aware | 2 | 2.0 | 0.788656 | 0.585681 | 0.583705 | NaN | 1.0 | 0.2 |
8. 保存结果¶
In [15]:
timestamp = datetime.now().strftime('%Y-%m-%d_%H%M%S')
cases_path = SUPPORT_DIR / f'{timestamp}_case_specific_risk_aware_cases.csv'
summary_path = SUPPORT_DIR / f'{timestamp}_case_specific_risk_aware_summary.csv'
result_path = SUPPORT_DIR / f'{timestamp}_case_specific_risk_aware_result.json'
comparison_path = SUPPORT_DIR / f'{timestamp}_case_specific_risk_aware_comparison.csv'
case_specific_cases_df.to_csv(cases_path, index=False)
case_specific_summary_df.to_csv(summary_path, index=False)
comparison_df.to_csv(comparison_path, index=False)
with open(result_path, 'w', encoding='utf-8') as f:
json.dump(case_specific_result, f, ensure_ascii=False, indent=2)
print(cases_path)
print(summary_path)
print(result_path)
print(comparison_path)
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192036_case_specific_risk_aware_cases.csv /Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192036_case_specific_risk_aware_summary.csv /Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192036_case_specific_risk_aware_result.json /Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192036_case_specific_risk_aware_comparison.csv
9. 如果这一版还是不行，说明什么¶
如果这一版仍然超不过 lookahead，那就说明：
- 仅靠 image-only 概率结构 + 全局风险分数还不够
- 下一步就必须真正做更像信息增益的策略
- 也就是：问题的价值要和当前候选类别分布更直接地绑定起来