14 Risk-Aware Fixed-Order Agent¶

这一份 notebook 专门比较 3 种 fixed-order questioning agent：

  1. 原始 fixed order：age -> sex -> location
  2. risk-aware candidate A：age -> sex -> location
  3. risk-aware candidate B：age -> location -> sex

注意：candidate A 和原始 fixed order 顺序相同，所以这一轮里真正有差异的是：

  • 原始 fixed order / candidate A
  • candidate B

但我们仍然把三者都保留下来，方便论文里写清楚来源。

In [1]:
from pathlib import Path
from datetime import datetime
import ast
import json

import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score

PROJECT_ROOT = Path('/Users/applesues01/Documents/Medical_Agent')
SUPPORT_DIR = PROJECT_ROOT / 'supports'

MODEL_REGISTRY_PATH = SUPPORT_DIR / '2026-08-03_194704_all_saved_metadata_models.csv'
VALIDATED_AGENT_PATH = SUPPORT_DIR / '2026-08-03_185733_validated_agent_comparison.csv'

print(MODEL_REGISTRY_PATH)
print(VALIDATED_AGENT_PATH)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-03_194704_all_saved_metadata_models.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-03_185733_validated_agent_comparison.csv

1. 先整理我们要比较的 3 个顺序¶

In [2]:
comparison_orders = [
    {
        'agent_name': 'original_fixed_order',
        'question_order': ['age', 'sex', 'location'],
        'source': 'advisor_baseline'
    },
    {
        'agent_name': 'risk_aware_candidate_a',
        'question_order': ['age', 'sex', 'location'],
        'source': 'risk_aware_score_sorted'
    },
    {
        'agent_name': 'risk_aware_candidate_b',
        'question_order': ['age', 'location', 'sex'],
        'source': 'risk_aware_conservative'
    }
]

comparison_order_df = pd.DataFrame(comparison_orders)
comparison_order_df
Out[2]:
agent_name question_order source
0 original_fixed_order [age, sex, location] advisor_baseline
1 risk_aware_candidate_a [age, sex, location] risk_aware_score_sorted
2 risk_aware_candidate_b [age, location, sex] risk_aware_conservative

2. 读取之前的 baseline agent 结果¶

这里我们把以前已经跑出来的 validated baselines 一起读进来，方便后面对照。

In [3]:
validated_agent_df = pd.read_csv(VALIDATED_AGENT_PATH)
validated_agent_df
Out[3]:
agent_name max_questions avg_questions expected_accuracy expected_balanced_accuracy expected_macro_f1 threshold
0 fixed_order 2 2.0 0.793383 0.582281 0.581666 NaN
1 uncertainty 2 2.0 0.793383 0.582281 0.581666 0.605
2 lookahead 2 2.0 0.792708 0.598458 0.596497 NaN

3. 这一轮我们先做“实验设计确认”¶

这一份 notebook 的目标不是重写整套 agent 逻辑，而是明确：

  • 哪些顺序需要比较
  • 输出哪些指标
  • 保存成什么格式

你已经把核心顺序跑出来了，所以这里先把对比框架立好。

In [4]:
planned_metrics = [
    'avg_questions',
    'expected_accuracy',
    'expected_balanced_accuracy',
    'expected_macro_f1'
]

planned_metrics
Out[4]:
['avg_questions',
 'expected_accuracy',
 'expected_balanced_accuracy',
 'expected_macro_f1']

4. 当前已知事实¶

由于 candidate_a 和原始 fixed order 完全同序，所以这一轮真正新的实验重点是：

  • risk_aware_candidate_b = age -> location -> sex

如果它优于原始 fixed order，说明：

在固定顺序设定下，把高风险问题往后放，可能带来更好的风险-性能平衡。

In [5]:
print('Original fixed order   :', comparison_orders[0]['question_order'])
print('Risk-aware candidate A :', comparison_orders[1]['question_order'])
print('Risk-aware candidate B :', comparison_orders[2]['question_order'])

Original fixed order   : ['age', 'sex', 'location']
Risk-aware candidate A : ['age', 'sex', 'location']
Risk-aware candidate B : ['age', 'location', 'sex']

5. 保存这一步的顺序配置¶

这一格先把本轮要比较的 fixed-order 配置保存下来，方便后面继续接实验。

In [6]:
timestamp = datetime.now().strftime('%Y-%m-%d_%H%M%S')

order_config_path = SUPPORT_DIR / f'{timestamp}_risk_aware_fixed_order_configs.csv'
comparison_order_df.to_csv(order_config_path, index=False)

print(order_config_path)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_164518_risk_aware_fixed_order_configs.csv

6. 下一步真正要做什么¶

下一步我们就要把 candidate_b 接入你之前的 validated fixed-order agent 代码里，重新跑：

  • original fixed order
  • risk-aware candidate B

然后把结果汇总成一张表。

因为 candidate A 和 original 顺序相同，所以它本轮不用重复跑，只保留为配置来源说明即可。

7. 读取数据划分与最新模型结果¶

这里不再使用旧 notebook 里手写的分数字典，而是直接读取当前已经保存好的最新结果文件。这样本轮比较会和你现在的实验资产保持一致。

In [7]:
DATA_DIR = PROJECT_ROOT / 'data' / 'HAM10000'
SPLIT_DIR = DATA_DIR / 'splits'
BASELINE_RESULTS_PATH = SUPPORT_DIR / 'baseline_results.csv'

train_df = pd.read_csv(SPLIT_DIR / 'train.csv')
val_df = pd.read_csv(SPLIT_DIR / 'val.csv')
test_df = pd.read_csv(SPLIT_DIR / 'test.csv')
model_registry_df = pd.read_csv(MODEL_REGISTRY_PATH)
baseline_df = pd.read_csv(BASELINE_RESULTS_PATH)

len(train_df), len(val_df), len(test_df)
Out[7]:
(7002, 1532, 1481)
In [8]:
image_only_row = baseline_df[baseline_df['Method'] == 'Image Only'].iloc[0]

policy_model_scores = {
    'image_only': {'macro_f1': float(image_only_row['Macro-F1'])},
}
report_model_scores = {
    'image_only': {
        'accuracy': float(image_only_row['Accuracy']),
        'balanced_accuracy': float(image_only_row['Balanced Accuracy']),
        'macro_f1': float(image_only_row['Macro-F1']),
    }
}

for _, row in model_registry_df.iterrows():
    key = str(row['Method'])
    policy_model_scores[key] = {'macro_f1': float(row['Best Val Macro-F1'])}
    report_model_scores[key] = {
        'accuracy': float(row['Test Accuracy']),
        'balanced_accuracy': float(row['Test Balanced Accuracy']),
        'macro_f1': float(row['Test Macro-F1']),
    }

policy_model_scores, report_model_scores['image_only']
Out[8]:
({'image_only': {'macro_f1': 0.6083234281767498},
  'image_age': {'macro_f1': 0.5926280556025649},
  'image_sex': {'macro_f1': 0.5959471751891948},
  'image_location': {'macro_f1': 0.6041929011696215},
  'image_age_sex': {'macro_f1': 0.5964635518225668},
  'image_age_location': {'macro_f1': 0.609501496398334},
  'image_sex_location': {'macro_f1': 0.6059856792062019},
  'image_all_metadata': {'macro_f1': 0.6112034383440658}},
 {'accuracy': 0.7771775827143822,
  'balanced_accuracy': 0.6372819453123182,
  'macro_f1': 0.6083234281767498})

8. 复用 validated fixed-order agent 的状态逻辑¶

下面这部分直接沿用你之前 validated agent 的状态定义，只把固定顺序从写死改成可配置。

In [9]:
def build_initial_state(row):
    return {
        'image_id': row['image_id'],
        'true_label': row['dx'],
        'known_metadata': {},
        'asked_questions': [],
        'done': False,
    }


def ask_question(state, row, question):
    assert question in ['age', 'sex', 'location']
    new_state = {
        'image_id': state['image_id'],
        'true_label': state['true_label'],
        'known_metadata': dict(state['known_metadata']),
        'asked_questions': list(state['asked_questions']),
        'done': state['done'],
    }

    if question == 'age':
        new_state['known_metadata']['age'] = row['age']
    elif question == 'sex':
        new_state['known_metadata']['sex'] = row['sex']
    elif question == 'location':
        new_state['known_metadata']['location'] = row['localization']

    new_state['asked_questions'].append(question)
    return new_state


def get_model_key_from_known_set(known_set):
    if known_set == set():
        return 'image_only'
    if known_set == {'age'}:
        return 'image_age'
    if known_set == {'sex'}:
        return 'image_sex'
    if known_set == {'location'}:
        return 'image_location'
    if known_set == {'age', 'sex'}:
        return 'image_age_sex'
    if known_set == {'age', 'location'}:
        return 'image_age_location'
    if known_set == {'sex', 'location'}:
        return 'image_sex_location'
    if known_set == {'age', 'sex', 'location'}:
        return 'image_all_metadata'
    raise ValueError(f'Unknown known_set: {known_set}')


def get_model_key_from_state(state):
    return get_model_key_from_known_set(set(state['known_metadata'].keys()))


def score_state(state):
    model_key = get_model_key_from_state(state)
    return policy_model_scores[model_key]['macro_f1']

9. 可配置的 fixed-order policy¶

这里的关键改动只有一个：question_order 不再写死，而是作为参数传入。

In [10]:
def configurable_fixed_order_policy(state, question_order, max_questions=2):
    if len(state['asked_questions']) >= max_questions:
        return 'diagnose'
    for question in question_order:
        if question not in state['asked_questions']:
            return question
    return 'diagnose'


def run_fixed_order_episode(row, question_order, max_questions=2):
    state = build_initial_state(row)
    trajectory = []

    while not state['done']:
        action = configurable_fixed_order_policy(state, question_order=question_order, max_questions=max_questions)

        trajectory.append({
            'step': len(trajectory),
            'action': action,
            'model_key_before_action': get_model_key_from_state(state),
            'score_before_action': score_state(state),
            'known_metadata_before_action': dict(state['known_metadata']),
        })

        if action == 'diagnose':
            state['done'] = True
            state['final_action'] = 'diagnose'
            state['final_model_key'] = get_model_key_from_state(state)
            state['final_score'] = score_state(state)
            break

        state = ask_question(state, row, action)

    return trajectory, state


def evaluate_fixed_order_agent(agent_name, question_order, max_questions=2):
    records = []
    for idx in range(len(test_df)):
        row = test_df.iloc[idx]
        trajectory, final_state = run_fixed_order_episode(
            row,
            question_order=question_order,
            max_questions=max_questions,
        )
        records.append({
            'image_id': row['image_id'],
            'true_label': row['dx'],
            'asked_questions': list(final_state['asked_questions']),
            'num_questions': len(final_state['asked_questions']),
            'final_model_key': final_state['final_model_key'],
            'final_policy_score': final_state['final_score'],
            'known_metadata': dict(final_state['known_metadata']),
            'question_order': question_order,
            'agent_name': agent_name,
        })

    cases_df = pd.DataFrame(records)
    summary_df = cases_df.groupby('final_model_key').size().reset_index(name='count')
    summary_df['ratio'] = summary_df['count'] / len(cases_df)
    summary_df['test_accuracy'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['accuracy'])
    summary_df['test_balanced_accuracy'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['balanced_accuracy'])
    summary_df['test_macro_f1'] = summary_df['final_model_key'].map(lambda k: report_model_scores[k]['macro_f1'])

    result = {
        'agent_name': agent_name,
        'question_order': question_order,
        'max_questions': max_questions,
        'avg_questions': float(cases_df['num_questions'].mean()),
        'expected_accuracy': float((summary_df['ratio'] * summary_df['test_accuracy']).sum()),
        'expected_balanced_accuracy': float((summary_df['ratio'] * summary_df['test_balanced_accuracy']).sum()),
        'expected_macro_f1': float((summary_df['ratio'] * summary_df['test_macro_f1']).sum()),
    }
    return cases_df, summary_df, result

10. 正式跑 3 个顺序¶

这里虽然 original 和 candidate_a 顺序相同，但我还是一起跑一遍，方便你后面直接把表拿去用。

In [11]:
all_agent_cases = []
all_agent_summaries = []
all_agent_results = []

for item in comparison_orders:
    cases_df, summary_df, result = evaluate_fixed_order_agent(
        agent_name=item['agent_name'],
        question_order=item['question_order'],
        max_questions=2,
    )

    summary_df = summary_df.copy()
    summary_df['agent_name'] = item['agent_name']
    summary_df['question_order'] = str(item['question_order'])

    all_agent_cases.append(cases_df)
    all_agent_summaries.append(summary_df)
    all_agent_results.append(result)

results_df = pd.DataFrame(all_agent_results)
results_df
Out[11]:
agent_name question_order max_questions avg_questions expected_accuracy expected_balanced_accuracy expected_macro_f1
0 original_fixed_order [age, sex, location] 2 2.0 0.788656 0.585681 0.583705
1 risk_aware_candidate_a [age, sex, location] 2 2.0 0.788656 0.585681 0.583705
2 risk_aware_candidate_b [age, location, sex] 2 2.0 0.777178 0.573000 0.580037

11. 和旧 validated baselines 放到一起看¶

这一步把新跑的 risk-aware fixed-order 结果，和之前的 fixed_order / uncertainty / lookahead 放到同一张表里。

In [12]:
validated_subset_df = validated_agent_df.copy()
validated_subset_df['question_order'] = validated_subset_df.get('question_order', np.nan)

comparison_df = pd.concat([
    validated_subset_df[['agent_name', 'max_questions', 'avg_questions', 'expected_accuracy', 'expected_balanced_accuracy', 'expected_macro_f1', 'threshold']],
    results_df.assign(threshold=np.nan)[['agent_name', 'max_questions', 'avg_questions', 'expected_accuracy', 'expected_balanced_accuracy', 'expected_macro_f1', 'threshold']]
], ignore_index=True)

comparison_df
Out[12]:
agent_name max_questions avg_questions expected_accuracy expected_balanced_accuracy expected_macro_f1 threshold
0 fixed_order 2 2.0 0.793383 0.582281 0.581666 NaN
1 uncertainty 2 2.0 0.793383 0.582281 0.581666 0.605
2 lookahead 2 2.0 0.792708 0.598458 0.596497 NaN
3 original_fixed_order 2 2.0 0.788656 0.585681 0.583705 NaN
4 risk_aware_candidate_a 2 2.0 0.788656 0.585681 0.583705 NaN
5 risk_aware_candidate_b 2 2.0 0.777178 0.573000 0.580037 NaN

12. 保存结果¶

所有结果都按时间戳保存，方便后面回顾和写论文。

In [13]:
run_timestamp = datetime.now().strftime('%Y-%m-%d_%H%M%S')

cases_path = SUPPORT_DIR / f'{run_timestamp}_risk_aware_fixed_order_cases.csv'
summary_path = SUPPORT_DIR / f'{run_timestamp}_risk_aware_fixed_order_summary.csv'
results_path = SUPPORT_DIR / f'{run_timestamp}_risk_aware_fixed_order_results.csv'
comparison_path = SUPPORT_DIR / f'{run_timestamp}_risk_aware_fixed_order_vs_baselines.csv'

pd.concat(all_agent_cases, ignore_index=True).to_csv(cases_path, index=False)
pd.concat(all_agent_summaries, ignore_index=True).to_csv(summary_path, index=False)
results_df.to_csv(results_path, index=False)
comparison_df.to_csv(comparison_path, index=False)

print(cases_path)
print(summary_path)
print(results_path)
print(comparison_path)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_164518_risk_aware_fixed_order_cases.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_164518_risk_aware_fixed_order_summary.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_164518_risk_aware_fixed_order_results.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_164518_risk_aware_fixed_order_vs_baselines.csv

我不想再纠结顺序了¶