分个类别看看¶

17 Class-Wise Agent Analysis¶

这一份 notebook 的目标是回答一个很关键的问题：

agent 效果不明显，到底是因为方法没用，还是因为不同类别上的表现被整体平均掩盖了？

也就是说，我们不再只看整个测试集的总平均，而是按类别拆开看：

  • nv
  • bkl
  • mel
  • bcc
  • akiec
  • df
  • vasc

看看：

  1. 哪些类别本来 image-only 就很强
  2. 哪些类别最容易被提问帮助
  3. 哪些类别反而会被提问带偏
  4. 这些现象是否和样本量有关
In [1]:
from pathlib import Path
from datetime import datetime
import json

import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score

PROJECT_ROOT = Path('/Users/applesues01/Documents/Medical_Agent')
DATA_DIR = PROJECT_ROOT / 'data' / 'HAM10000'
SPLIT_DIR = DATA_DIR / 'splits'
SUPPORT_DIR = PROJECT_ROOT / 'supports'

TEST_PATH = SPLIT_DIR / 'test.csv'
IMAGE_ONLY_CASE_PATH = SUPPORT_DIR / 'image_only_case_level_predictions.csv'
DYNAMIC_CASES_PATH = SUPPORT_DIR / '2026-08-04_104443_dynamic_agent_cases.csv'
SAFE_CASES_PATH = SUPPORT_DIR / '2026-08-04_133250_safe_agent_diagnosed_cases.csv'
DANGER_CASES_PATH = SUPPORT_DIR / '2026-08-04_104443_dynamic_agent_danger_cases.csv'

print(TEST_PATH)
print(IMAGE_ONLY_CASE_PATH)
print(DYNAMIC_CASES_PATH)
print(SAFE_CASES_PATH)

/Users/applesues01/Documents/Medical_Agent/data/HAM10000/splits/test.csv
/Users/applesues01/Documents/Medical_Agent/supports/image_only_case_level_predictions.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_104443_dynamic_agent_cases.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_133250_safe_agent_diagnosed_cases.csv

1. 读取测试集和几个关键结果表¶

In [2]:
test_df = pd.read_csv(TEST_PATH)
image_only_case_df = pd.read_csv(IMAGE_ONLY_CASE_PATH)
dynamic_df = pd.read_csv(DYNAMIC_CASES_PATH)
safe_df = pd.read_csv(SAFE_CASES_PATH)
danger_df = pd.read_csv(DANGER_CASES_PATH)

len(test_df), len(image_only_case_df), len(dynamic_df), len(safe_df)
Out[2]:
(1481, 1481, 1481, 609)

2. 先看测试集类别分布¶

这是后面解释结果最重要的背景。因为如果某个类别样本极少，它的波动会天然更大。

In [3]:
test_class_counts = test_df['dx'].value_counts().sort_values(ascending=False).reset_index()
test_class_counts.columns = ['class', 'test_count']
test_class_counts
Out[3]:
class test_count
0 nv 992
1 bkl 168
2 mel 165
3 bcc 71
4 akiec 46
5 df 20
6 vasc 19

3. 计算 image-only 的类别级正确率¶

这里先看：

  • 哪些类本来就容易
  • 哪些类本来就很难

因为后面 agent 是否能提升，很大程度上和起点难度有关。

In [4]:
image_only_case_df['image_only_correct'] = image_only_case_df['true_label'] == image_only_case_df['pred_label']

image_only_class_perf = (
    image_only_case_df.groupby('true_label')
    .agg(
        image_only_count=('image_id', 'count'),
        image_only_accuracy=('image_only_correct', 'mean'),
        image_only_avg_conf=('max_prob', 'mean')
    )
    .reset_index()
    .rename(columns={'true_label': 'class'})
)

image_only_class_perf.sort_values('image_only_accuracy', ascending=False)
Out[4]:
class image_only_count image_only_accuracy image_only_avg_conf
5 nv 992 0.861895 0.870013
1 bcc 71 0.690141 0.759018
6 vasc 19 0.684211 0.888098
2 bkl 168 0.672619 0.733207
0 akiec 46 0.586957 0.786592
4 mel 165 0.515152 0.728979
3 df 20 0.450000 0.644947

4. 看 dynamic agent 在各类别上的变化¶

这里最重要的是：

  • improved_rate
  • worsened_rate
  • unsafe_rate
  • avg_questions

这样我们就能看出：哪些类别是 agent 真正在帮忙，哪些类别是在制造风险。

In [5]:
bool_cols = [
    'initial_correct', 'final_correct', 'changed_prediction', 'improved',
    'worsened', 'still_wrong', 'still_correct', 'initial_high_conf_wrong',
    'final_high_conf_wrong', 'unsafe_confidence_increase'
]
for col in bool_cols:
    dynamic_df[col] = dynamic_df[col].astype(str).str.lower().map({'true': True, 'false': False})

dynamic_class_perf = (
    dynamic_df.groupby('true_label')
    .agg(
        dynamic_count=('image_id', 'count'),
        initial_accuracy=('initial_correct', 'mean'),
        final_accuracy=('final_correct', 'mean'),
        improved_rate=('improved', 'mean'),
        worsened_rate=('worsened', 'mean'),
        unsafe_rate=('unsafe_confidence_increase', 'mean'),
        avg_questions=('num_questions', 'mean'),
        final_avg_conf=('final_max_prob', 'mean')
    )
    .reset_index()
    .rename(columns={'true_label': 'class'})
)

dynamic_class_perf.sort_values('improved_rate', ascending=False)
Out[5]:
class dynamic_count initial_accuracy final_accuracy improved_rate worsened_rate unsafe_rate avg_questions final_avg_conf
4 mel 165 0.515152 0.545455 0.078788 0.048485 0.266667 0.581818 0.893911
1 bcc 71 0.690141 0.690141 0.070423 0.070423 0.169014 0.563380 0.923349
2 bkl 168 0.672619 0.601190 0.065476 0.136905 0.166667 0.613095 0.905558
5 nv 992 0.861895 0.885081 0.037298 0.014113 0.062500 0.266129 0.949371
0 akiec 46 0.586957 0.565217 0.021739 0.043478 0.282609 0.500000 0.916607
3 df 20 0.450000 0.150000 0.000000 0.300000 0.500000 0.850000 0.928951
6 vasc 19 0.684211 0.631579 0.000000 0.052632 0.157895 0.210526 0.931189

5. 合并成一张主表¶

这张表会告诉我们：

  • 测试集里某个类有多少样本
  • image-only 对它好不好
  • dynamic agent 对它有没有帮助
  • dynamic agent 会不会给它带来风险
In [6]:
classwise_df = (
    test_class_counts
    .merge(image_only_class_perf, on='class', how='left')
    .merge(dynamic_class_perf, on='class', how='left')
)

classwise_df['accuracy_gain'] = classwise_df['final_accuracy'] - classwise_df['initial_accuracy']
classwise_df.sort_values('accuracy_gain', ascending=False)
Out[6]:
class test_count image_only_count image_only_accuracy image_only_avg_conf dynamic_count initial_accuracy final_accuracy improved_rate worsened_rate unsafe_rate avg_questions final_avg_conf accuracy_gain
2 mel 165 165 0.515152 0.728979 165 0.515152 0.545455 0.078788 0.048485 0.266667 0.581818 0.893911 0.030303
0 nv 992 992 0.861895 0.870013 992 0.861895 0.885081 0.037298 0.014113 0.062500 0.266129 0.949371 0.023185
3 bcc 71 71 0.690141 0.759018 71 0.690141 0.690141 0.070423 0.070423 0.169014 0.563380 0.923349 0.000000
4 akiec 46 46 0.586957 0.786592 46 0.586957 0.565217 0.021739 0.043478 0.282609 0.500000 0.916607 -0.021739
6 vasc 19 19 0.684211 0.888098 19 0.684211 0.631579 0.000000 0.052632 0.157895 0.210526 0.931189 -0.052632
1 bkl 168 168 0.672619 0.733207 168 0.672619 0.601190 0.065476 0.136905 0.166667 0.613095 0.905558 -0.071429
5 df 20 20 0.450000 0.644947 20 0.450000 0.150000 0.000000 0.300000 0.500000 0.850000 0.928951 -0.300000

6. 看样本量和收益/风险的关系¶

你刚刚提出的核心问题就在这里：

是否是因为样本量不同，导致 agent 在不同类别上的表现不同？

我们先看简单相关关系。

In [7]:
corr_df = classwise_df[['test_count', 'image_only_accuracy', 'accuracy_gain', 'improved_rate', 'worsened_rate', 'unsafe_rate']].corr()
corr_df
Out[7]:
test_count image_only_accuracy accuracy_gain improved_rate worsened_rate unsafe_rate
test_count 1.000000 0.731702 0.380755 0.130420 -0.398560 -0.569622
image_only_accuracy 0.731702 1.000000 0.583493 0.137568 -0.631899 -0.917529
accuracy_gain 0.380755 0.583493 1.000000 0.592580 -0.961986 -0.802100
improved_rate 0.130420 0.137568 0.592580 1.000000 -0.350067 -0.374950
worsened_rate -0.398560 -0.631899 -0.961986 -0.350067 1.000000 0.805634
unsafe_rate -0.569622 -0.917529 -0.802100 -0.374950 0.805634 1.000000

7. 特别看看危险病例主要集中在哪些类¶

In [8]:
danger_class_counts = danger_df['true_label'].value_counts().reset_index()
danger_class_counts.columns = ['class', 'danger_case_count']
danger_class_counts
Out[8]:
class danger_case_count
0 nv 96
1 bkl 61
2 mel 58
3 bcc 17
4 akiec 16
5 df 16
6 vasc 5
In [9]:
classwise_with_danger_df = classwise_df.merge(danger_class_counts, on='class', how='left')
classwise_with_danger_df['danger_case_count'] = classwise_with_danger_df['danger_case_count'].fillna(0).astype(int)
classwise_with_danger_df['danger_case_ratio'] = classwise_with_danger_df['danger_case_count'] / classwise_with_danger_df['test_count']
classwise_with_danger_df.sort_values('danger_case_ratio', ascending=False)
Out[9]:
class test_count image_only_count image_only_accuracy image_only_avg_conf dynamic_count initial_accuracy final_accuracy improved_rate worsened_rate unsafe_rate avg_questions final_avg_conf accuracy_gain danger_case_count danger_case_ratio
5 df 20 20 0.450000 0.644947 20 0.450000 0.150000 0.000000 0.300000 0.500000 0.850000 0.928951 -0.300000 16 0.800000
1 bkl 168 168 0.672619 0.733207 168 0.672619 0.601190 0.065476 0.136905 0.166667 0.613095 0.905558 -0.071429 61 0.363095
2 mel 165 165 0.515152 0.728979 165 0.515152 0.545455 0.078788 0.048485 0.266667 0.581818 0.893911 0.030303 58 0.351515
4 akiec 46 46 0.586957 0.786592 46 0.586957 0.565217 0.021739 0.043478 0.282609 0.500000 0.916607 -0.021739 16 0.347826
6 vasc 19 19 0.684211 0.888098 19 0.684211 0.631579 0.000000 0.052632 0.157895 0.210526 0.931189 -0.052632 5 0.263158
3 bcc 71 71 0.690141 0.759018 71 0.690141 0.690141 0.070423 0.070423 0.169014 0.563380 0.923349 0.000000 17 0.239437
0 nv 992 992 0.861895 0.870013 992 0.861895 0.885081 0.037298 0.014113 0.062500 0.266129 0.949371 0.023185 96 0.096774

8. 先给出第一版解读框架¶

你跑完上面这几张表之后，后面可以重点问这些问题：

  1. nv 这种大类，是不是 image-only 本来就已经很强，所以提问提升空间有限？
  2. bkl / mel / bcc 这些更容易混淆的类，agent 是在帮忙还是在添乱？
  3. 小样本类别（如 df / vasc / akiec）是不是更容易波动？
  4. 危险病例是不是集中在少数几个类别上？

9. 保存结果¶

In [10]:
timestamp = datetime.now().strftime('%Y-%m-%d_%H%M%S')

classwise_path = SUPPORT_DIR / f'{timestamp}_classwise_agent_analysis.csv'
danger_path = SUPPORT_DIR / f'{timestamp}_classwise_danger_analysis.csv'
corr_path = SUPPORT_DIR / f'{timestamp}_classwise_correlation.csv'

classwise_df.to_csv(classwise_path, index=False)
classwise_with_danger_df.to_csv(danger_path, index=False)
corr_df.to_csv(corr_path)

print(classwise_path)
print(danger_path)
print(corr_path)

/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192705_classwise_agent_analysis.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192705_classwise_danger_analysis.csv
/Users/applesues01/Documents/Medical_Agent/supports/2026-08-04_192705_classwise_correlation.csv