Role-probe results

A linear probe on one layer predicts each token's chat role from its activation. Probes are trained on filler text wrapped in each role's template, then tested on real conversations rendered with the template, without any tags, and with every turn stuffed into a single user or tool message. Tokens always keep their true role label, so a probe that scores well without tags reads role from content rather than markup.

Role-space codes: s = sys, u = user, c = cot, a = asst, t = tool. A code such as sucat is a probe over those roles.
Splits: Tagged: Real conversation with the chat template intactUntagged: Every role tag removedAll as user: All turns placed inside one user messageAll as tool: All turns placed inside one tool message

Metric

Harmonic mean of precision and recall.

What the probe sees

A worked example for Qwen/Qwen3-1.7B (layer 16): what the probe is trained and validated on, and the four ways each evaluation conversation is rendered. Hover highlighted text for its role label.

systemuserthinkingassistanttool<|tag|>template markup

Training & validation

The same filler text wrapped in each role's template; 249 rows per role. Validation is held-out rows of this kind. The first 32 tokens of each turn are skipped in training.

system

<|im_start|>system
Thank you for your interest in Broadway Grand …<|im_end|>

user

<|im_start|>user
Thank you for your interest in Broadway Grand …<|im_end|>

tool

<|im_start|>user
<tool_response>
Thank you for your interest in Broadway Grand …
</tool_response><|im_end|>

thinking

<|im_start|>assistant
<think>
Thank you for your interest in Broadway Grand …
</think>

<|im_end|>

thinking + assistant

<|im_start|>assistant
<think>
Mens_New_Era_Navy_Illinois_Fighting_Illini_Basic_59FIFTY_Fitted_Hat http://images.fanatics.com/lf … false We are unable to …
</think>

Thank you for your interest in Broadway Grand …<|im_end|>

Evaluation

One of 30 generated conversations (system → user → thinking → assistant), rendered four ways. Tags change; every token keeps its true role label.

Conversation text hidden. Evaluation conversations are seeded with user prompts from OpenAssistant (oasst1) and ToxicChat (lmsys/toxicchat). ToxicChat contains real user prompts with toxic and sexually explicit requests, and model thinking and replies often restate or act on them, so user, thinking and assistant text is hidden by default and removed before it leaves the server. System prompts, role labels, colors and probe probabilities are unchanged. Researchers who need the full text can email saifullah.saifullah@dfki.de for an access code.

TaggedReal conversation with the chat template intact

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
███ █████ ████ ███████ █████████████ ███████ ███████ ████ …<|im_end|>
<|im_start|>assistant
<think>
█████ ███ ████ █████ ██ ██ ███ ██ …
</think>

█████████ █████████ ███ ██████ ██████ ███ ████ ███████ …<|im_end|>

UntaggedEvery role tag removed

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
███ █████ ████ ███████ █████████████ ███████ ███████ ████ … █████ ███ ████ █████ ██ ██ ███ ██ … █████████ █████████ ███ ██████ ██████ ███ ████ ███████ …

All as userAll turns placed inside one user message

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
███ █████ ████ ███████ █████████████ ███████ ███████ ████ … █████ ███ ████ █████ ██ ██ ███ ██ … █████████ █████████ ███ ██████ ██████ ███ ████ ███████ …<|im_end|>

All as toolAll turns placed inside one tool message

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
<tool_response>
███ █████ ████ ███████ █████████████ ███████ ███████ ████ … █████ ███ ████ █████ ██ ██ ███ ██ … █████████ █████████ ███ ██████ ██████ ███ ████ ███████ …
</tool_response><|im_end|>

At a glance: macro F1

Mean over the roles present in each split (tool has no tokens in the eval conversations, so it is left out there). Rows are splits, columns are role spaces. Hover a cell for details.

	sua	suat	suca	sucat	ua	uat	uca	ucat
ai21labs-Jamba-Reasoning-3Blayer 14
Tagged	0.81	0.69	0.69	0.64	0.98	0.76	0.81	0.70
Untagged	0.63	0.57	0.46	0.47	0.91	0.77	0.63	0.59
All as user	0.73	0.64	0.49	0.51	0.89	0.72	0.59	0.56
All as tool	0.72	0.58	0.49	0.48	0.88	0.57	0.59	0.49
allenai-Olmo-3-7B-Thinklayer 16
Tagged	0.38	0.29	0.24	0.22	0.86	0.43	0.47	0.29
Untagged	0.20	0.14	0.14	0.13	0.48	0.19	0.35	0.17
All as user	0.17	0.11	0.12	0.11	0.42	0.16	0.30	0.15
All as tool	0.18	0.12	0.13	0.11	0.44	0.16	0.32	0.15
openai-gpt-oss-20blayer 16
Tagged	0.64	0.60	0.70	0.67	0.99	0.97	0.90	0.91
Untagged	0.63	0.58	0.65	0.62	0.97	0.95	0.82	0.82
All as user	0.62	0.58	0.62	0.60	0.95	0.94	0.77	0.78
All as tool	0.62	0.55	0.64	0.60	0.98	0.92	0.83	0.81
Qwen-Qwen3-1.7Blayer 16
Tagged	0.78	0.69	0.51	0.42	0.99	0.75	0.45	0.32
Untagged	0.59	0.53	0.32	0.28	0.98	0.70	0.31	0.26
All as user	0.77	0.68	0.40	0.34	0.98	0.75	0.50	0.38
All as tool	0.67	0.55	0.36	0.29	0.97	0.56	0.34	0.25
Qwen-Qwen3-30B-A3Blayer 24
Tagged	0.71	0.75	0.27	0.26	0.94	0.90	0.46	0.42
Untagged	0.50	0.51	0.25	0.26	0.83	0.82	0.48	0.43
All as user	0.53	0.55	0.26	0.27	0.81	0.80	0.46	0.41
All as tool	0.53	0.54	0.26	0.27	0.81	0.79	0.46	0.41
zai-org-GLM-4.6V-Flashlayer 20
Tagged	0.33	0.41	0.18	0.18	0.77	0.52	0.24	0.22
Untagged	0.18	0.21	0.11	0.13	0.54	0.38	0.23	0.22
All as user	0.22	0.26	0.07	0.07	0.36	0.25	0.17	0.15
All as tool	0.24	0.31	0.13	0.14	0.61	0.39	0.23	0.20

Robustness to tagging

Macro F1 for suca across splits. Gray is the validation reference; the drop from Tag to the other bars shows how much the probe leans on chat-template markup.

ai21labs-Jamba-Reasoning-3B

0.5

0.69

0.46

0.49

TagUntag→User→Tool

allenai-Olmo-3-7B-Think

0.5

0.24

0.14

0.12

0.13

TagUntag→User→Tool

openai-gpt-oss-20b

0.5

0.70

0.65

0.62

0.64

TagUntag→User→Tool

Qwen-Qwen3-1.7B

0.5

0.51

0.32

0.40

0.36

TagUntag→User→Tool

Qwen-Qwen3-30B-A3B

0.5

0.27

0.25

0.26

TagUntag→User→Tool

zai-org-GLM-4.6V-Flash

0.5

0.18

0.11

0.07

0.13

TagUntag→User→Tool

Per-role F1

Every role in every role space, as in the evaluation notebook. Pick which role spaces to show.

	sua			suat				suca				sucat					ua		uat			uca			ucat
	sys	user	asst	sys	user	asst	tool	sys	user	cot	asst	sys	user	cot	asst	tool	user	asst	user	asst	tool	user	cot	asst	user	cot	asst	tool
ai21labs-Jamba-Reasoning-3Blayer 14
Tagged	0.53	0.90	0.99	0.58	0.49	0.99	—	0.46	0.78	0.91	0.59	0.50	0.49	0.91	0.68	—	0.97	1.00	0.52	0.99	—	0.83	0.91	0.70	0.60	0.90	0.60	—
Untagged	0.28	0.64	0.97	0.32	0.43	0.95	—	0.20	0.40	0.86	0.37	0.24	0.34	0.85	0.45	—	0.85	0.97	0.60	0.94	—	0.53	0.85	0.49	0.57	0.85	0.36	—
All as user	0.43	0.78	0.97	0.52	0.48	0.92	—	0.29	0.44	0.85	0.37	0.35	0.41	0.82	0.44	—	0.81	0.96	0.52	0.91	—	0.44	0.83	0.48	0.50	0.82	0.36	—
All as tool	0.43	0.78	0.97	0.57	0.25	0.92	—	0.30	0.45	0.85	0.37	0.43	0.25	0.82	0.44	—	0.81	0.96	0.23	0.90	—	0.45	0.83	0.49	0.30	0.81	0.36	—
allenai-Olmo-3-7B-Thinklayer 16
Tagged	0.36	0.03	0.74	0.15	0.03	0.70	—	0.10	0.03	0.09	0.73	0.06	0.03	0.07	0.71	—	0.76	0.96	0.06	0.79	—	0.20	0.54	0.66	0.03	0.10	0.74	—
Untagged	0.26	0.02	0.31	0.12	0.03	0.26	—	0.08	0.02	0.07	0.39	0.05	0.02	0.06	0.37	—	0.34	0.63	0.04	0.34	—	0.11	0.40	0.54	0.03	0.08	0.41	—
All as user	0.25	0.03	0.24	0.11	0.03	0.20	—	0.08	0.02	0.06	0.33	0.05	0.02	0.05	0.31	—	0.30	0.54	0.04	0.27	—	0.10	0.32	0.48	0.02	0.06	0.36	—
All as tool	0.25	0.03	0.26	0.12	0.03	0.22	—	0.08	0.02	0.06	0.34	0.05	0.02	0.05	0.33	—	0.31	0.57	0.04	0.29	—	0.10	0.35	0.50	0.02	0.06	0.37	—
openai-gpt-oss-20blayer 16
Tagged	0.30	0.64	0.99	0.13	0.69	0.97	—	0.27	0.64	0.92	0.97	0.13	0.67	0.91	0.97	—	0.98	1.00	0.96	0.99	—	0.81	0.92	0.97	0.85	0.91	0.96	—
Untagged	0.27	0.63	0.99	0.13	0.66	0.95	—	0.27	0.55	0.83	0.94	0.13	0.57	0.84	0.93	—	0.95	1.00	0.93	0.97	—	0.69	0.83	0.94	0.71	0.83	0.92	—
All as user	0.26	0.60	0.98	0.13	0.67	0.94	—	0.26	0.51	0.79	0.92	0.13	0.54	0.80	0.92	—	0.91	0.99	0.90	0.97	—	0.60	0.79	0.93	0.63	0.80	0.91	—
All as tool	0.26	0.62	0.99	0.13	0.58	0.94	—	0.26	0.55	0.83	0.93	0.13	0.52	0.83	0.93	—	0.95	1.00	0.87	0.97	—	0.71	0.83	0.94	0.68	0.83	0.92	—
Qwen-Qwen3-1.7Blayer 16
Tagged	0.61	0.72	1.00	0.73	0.35	1.00	—	0.76	0.67	0.01	0.61	0.77	0.31	0.01	0.61	—	0.98	1.00	0.49	1.00	—	0.73	0.01	0.61	0.34	0.01	0.61	—
Untagged	0.45	0.32	1.00	0.48	0.10	1.00	—	0.41	0.24	0.02	0.61	0.42	0.07	0.02	0.61	—	0.97	1.00	0.40	1.00	—	0.27	0.05	0.61	0.11	0.05	0.61	—
All as user	0.58	0.72	1.00	0.70	0.36	1.00	—	0.29	0.62	0.07	0.64	0.33	0.30	0.07	0.63	—	0.97	1.00	0.49	1.00	—	0.68	0.18	0.63	0.32	0.18	0.63	—
All as tool	0.50	0.49	1.00	0.59	0.05	1.00	—	0.46	0.33	0.03	0.62	0.50	0.03	0.03	0.62	—	0.94	1.00	0.13	1.00	—	0.35	0.07	0.61	0.06	0.06	0.61	—
Qwen-Qwen3-30B-A3Blayer 24
Tagged	0.51	0.66	0.97	0.65	0.64	0.96	—	0.09	0.51	0.35	0.13	0.09	0.40	0.34	0.22	—	0.88	0.99	0.82	0.99	—	0.58	0.59	0.21	0.54	0.59	0.14	—
Untagged	0.18	0.45	0.87	0.24	0.43	0.88	—	0.07	0.38	0.32	0.23	0.07	0.29	0.31	0.35	—	0.69	0.97	0.67	0.97	—	0.48	0.61	0.34	0.45	0.60	0.25	—
All as user	0.21	0.52	0.87	0.28	0.48	0.87	—	0.07	0.40	0.33	0.24	0.08	0.31	0.33	0.36	—	0.66	0.97	0.64	0.97	—	0.43	0.60	0.35	0.41	0.58	0.26	—
All as tool	0.21	0.50	0.87	0.29	0.44	0.87	—	0.08	0.38	0.34	0.25	0.08	0.29	0.34	0.37	—	0.65	0.97	0.60	0.97	—	0.43	0.60	0.36	0.39	0.59	0.26	—
zai-org-GLM-4.6V-Flashlayer 20
Tagged	0.52	0.15	0.31	0.71	0.10	0.42	—	0.13	0.09	0.00	0.50	0.11	0.04	0.00	0.54	—	0.60	0.93	0.22	0.83	—	0.16	0.01	0.56	0.09	0.00	0.57	—
Untagged	0.32	0.09	0.14	0.37	0.07	0.19	—	0.05	0.04	0.00	0.35	0.05	0.02	0.00	0.43	—	0.33	0.75	0.17	0.58	—	0.08	0.03	0.59	0.06	0.00	0.59	—
All as user	0.46	0.13	0.07	0.61	0.08	0.09	—	0.06	0.05	0.01	0.16	0.06	0.02	0.01	0.20	—	0.22	0.50	0.11	0.38	—	0.07	0.05	0.40	0.04	0.01	0.40	—
All as tool	0.41	0.12	0.20	0.60	0.05	0.27	—	0.08	0.06	0.00	0.37	0.08	0.02	0.00	0.44	—	0.39	0.82	0.11	0.67	—	0.11	0.02	0.56	0.04	0.00	0.57	—

Where the probability goes

Mean predicted probability per true role (rows) and predicted role (columns) for suca. The outlined diagonal is soft accuracy; mass off the diagonal shows which roles get confused.

Tagged

Untagged

All as user

All as tool

ai21labs-Jamba-Reasoning-3B

true ↓	sys	user	cot	asst
sys	0.75	0.18	0.06	0.01
user	0.25	0.69	0.05	0.01
cot	0.01	0.01	0.87	0.11
asst	0.01	0.00	0.51	0.48

true ↓	sys	user	cot	asst
sys	0.75	0.18	0.06	0.01
user	0.41	0.48	0.10	0.02
cot	0.03	0.04	0.82	0.11
asst	0.01	0.02	0.64	0.33

true ↓	sys	user	cot	asst
sys	0.74	0.18	0.06	0.01
user	0.25	0.69	0.06	0.01
cot	0.03	0.07	0.79	0.11
asst	0.02	0.02	0.64	0.32

true ↓	sys	user	cot	asst
sys	0.74	0.19	0.06	0.01
user	0.24	0.69	0.06	0.01
cot	0.03	0.07	0.79	0.11
asst	0.02	0.02	0.63	0.33

allenai-Olmo-3-7B-Think

true ↓	sys	user	cot	asst
sys	0.91	0.04	0.04	0.01
user	0.97	0.03	0.00	0.00
cot	0.80	0.03	0.06	0.11
asst	0.21	0.00	0.02	0.77

true ↓	sys	user	cot	asst
sys	0.91	0.04	0.04	0.01
user	0.97	0.03	0.00	0.00
cot	0.85	0.06	0.05	0.04
asst	0.60	0.03	0.09	0.28

true ↓	sys	user	cot	asst
sys	0.91	0.04	0.04	0.01
user	0.97	0.03	0.00	0.00
cot	0.86	0.06	0.04	0.04
asst	0.67	0.03	0.08	0.22

true ↓	sys	user	cot	asst
sys	0.91	0.04	0.04	0.01
user	0.97	0.03	0.00	0.00
cot	0.86	0.06	0.04	0.04
asst	0.65	0.03	0.09	0.23

openai-gpt-oss-20b

true ↓	sys	user	cot	asst
sys	0.17	0.74	0.09	0.00
user	0.01	0.92	0.01	0.07
cot	0.01	0.02	0.89	0.08
asst	0.00	0.00	0.02	0.97

true ↓	sys	user	cot	asst
sys	0.17	0.73	0.09	0.00
user	0.01	0.90	0.03	0.07
cot	0.01	0.07	0.85	0.06
asst	0.01	0.00	0.10	0.90

true ↓	sys	user	cot	asst
sys	0.17	0.73	0.09	0.00
user	0.00	0.91	0.02	0.07
cot	0.01	0.11	0.83	0.06
asst	0.01	0.01	0.12	0.87

true ↓	sys	user	cot	asst
sys	0.17	0.74	0.09	0.00
user	0.02	0.84	0.07	0.07
cot	0.01	0.06	0.87	0.06
asst	0.01	0.00	0.11	0.89

Qwen-Qwen3-1.7B

true ↓	sys	user	cot	asst
sys	0.98	0.00	0.01	0.00
user	0.20	0.47	0.31	0.02
cot	0.00	0.00	0.01	0.99
asst	0.00	0.00	0.00	1.00

true ↓	sys	user	cot	asst
sys	0.98	0.00	0.01	0.00
user	0.48	0.14	0.34	0.03
cot	0.03	0.00	0.01	0.96
asst	0.00	0.00	0.00	1.00

true ↓	sys	user	cot	asst
sys	0.98	0.00	0.01	0.00
user	0.20	0.46	0.31	0.03
cot	0.09	0.01	0.05	0.86
asst	0.00	0.00	0.00	0.99

true ↓	sys	user	cot	asst
sys	0.98	0.00	0.01	0.00
user	0.28	0.21	0.48	0.03
cot	0.04	0.00	0.02	0.94
asst	0.00	0.00	0.00	1.00

Qwen-Qwen3-30B-A3B

true ↓	sys	user	cot	asst
sys	0.96	0.02	0.02	0.00
user	0.19	0.59	0.18	0.05
cot	0.41	0.09	0.37	0.13
asst	0.01	0.02	0.81	0.16

true ↓	sys	user	cot	asst
sys	0.96	0.02	0.02	0.00
user	0.34	0.46	0.16	0.04
cot	0.49	0.09	0.32	0.10
asst	0.09	0.04	0.64	0.22

true ↓	sys	user	cot	asst
sys	0.96	0.02	0.02	0.00
user	0.19	0.59	0.18	0.04
cot	0.44	0.12	0.33	0.11
asst	0.08	0.05	0.64	0.23

true ↓	sys	user	cot	asst
sys	0.96	0.02	0.02	0.00
user	0.19	0.55	0.20	0.06
cot	0.43	0.11	0.34	0.12
asst	0.08	0.05	0.63	0.23

zai-org-GLM-4.6V-Flash

true ↓	sys	user	cot	asst
sys	0.88	0.10	0.01	0.00
user	0.32	0.68	0.00	0.00
cot	0.12	0.45	0.03	0.40
asst	0.01	0.36	0.00	0.62

true ↓	sys	user	cot	asst
sys	0.88	0.10	0.01	0.00
user	0.51	0.49	0.00	0.00
cot	0.21	0.56	0.04	0.20
asst	0.02	0.62	0.01	0.35

true ↓	sys	user	cot	asst
sys	0.88	0.10	0.01	0.00
user	0.32	0.68	0.00	0.00
cot	0.21	0.62	0.04	0.12
asst	0.03	0.78	0.02	0.17

true ↓	sys	user	cot	asst
sys	0.88	0.10	0.01	0.00
user	0.38	0.62	0.01	0.00
cot	0.15	0.51	0.03	0.30
asst	0.02	0.55	0.01	0.41