Captain 01
fixed δ̂ = .10Source-blind
It treats a matching answer as 90% likely. It cannot revise that assumption.
Best next shotD6 · 25.4%
Source belief is locked.
Source-aware decision-making · AAAI submission
When should an agent
believe one source?
QuestionWith few questions left, should an agent diagnose the source, solve the task, or act now?
MethodMaintain one belief over the hidden task and source behavior, then price each question by decision value.
ResultAgainst a deterministic liar, final-board F1 rises from .510 for a source-blind planner to .783 for SymmetricMixture.
BoundaryThe method depends on an identifiable, task-relevant source model. Adaptive or misspecified sources can defeat it.
A question matters only if it can change the next decision.
The live system accepts a free-form yes-or-no question. A language model translates it into an executable predicate over candidate boards. The research method then reasons over the predicate's possible effects.
- 01Ask in natural language"Is any hidden ship tile in rows A to D?"
- 02Compile and testTranslate the question to q(B), then execute it on each candidate board.
- 03Score both answersFor yes and no, update the joint task-source belief p(B,S).
- 04Ask or actAsk only if action value plus source information repays the question cost.
The browser demo starts after translation. It exposes compiled question effects, not the language model that writes the predicate.
Guided Battleship replay
Three events. Three different kinds of evidence.
Follow the short path first. The full board opens for free exploration when the source check is complete.
- 1Ask a task questionThe answer changes board beliefs, but its direction is ambiguous.
- 2Reveal B5A shot supplies task-grounded evidence that bypasses the Spotter.
- 3Check the sourceAsk about the revealed cell and watch the source-aware captain infer inversion.
Event 1 of 3One answer can point to two boards when its source is unknown.
B01 replay · deterministic inverter
Same board. Same answer. Two beliefs.
The left captain assumes answers are mostly truthful. The right captain allows both trust-leaning and invert-leaning sources.
Board B01Prior 1,000 legal samplesSource δ = 1Question cost .050
Shared evidence
Choose what both captains see.
Selected cellB5Click any hidden cell to change it.
Best next-shot gain+.001
Source information.224
Net VOI+.004
Decision gateAsk this candidateNet value after question cost.
Captain 02
SymmetricMixtureSource-aware
It updates one posterior over the board and nine stationary source hypotheses.
Best next shotD6 · 25.4%
p(source flip rate δ)MAP: δ = 0 or 1
trust-leaning 52.38%invert-leaning 47.62%
Reliable inversion recovers the task. It still costs questions.
Use one policy across all three source conditions. Each result reports raw final-board F1, average questions, then cost-adjusted F1 with a .05 charge per question.
Trust the channel
Planner.77014.7 Q · adjusted .033
SymmetricMixture.73415.0 Q · adjusted −.015
Discount the channel
Planner.66314.9 Q · adjusted −.083
SymmetricMixture.68215.0 Q · adjusted −.068
SymmetricMixture uses one seed here.
Invert the channel
Planner.51014.8 Q · adjusted −.231
SymmetricMixture.78314.8 Q · adjusted .042
Honest and lying cells pool seeds 42 to 44. The SymmetricMixture noisy cell uses seed 42. Raw F1 and question counts come from the paper's static-corner cost table.
One slider. Two claims that should not be confused.
The theory describes how much information the channel can carry and how source diagnosis grows harder near δ=.50. The experiment measures task performance under a finite question budget.
trustuninformative wallinvert
capacity 53.1%95% probe bound 10
Capacity is 1 minus binary entropy. The probe count is a sufficient bound for learning channel orientation, not an observed question count and not a guarantee of board recovery.
nearest measured δ=.90SymmetricMixture F1 .732 · +.107 over asking nothing
Dots are measured means with one SEM. All three policies share the measured δ=.25 static corner. Only SymmetricMixture adds benign points at δ=.05, .10, and .40. Endpoints pool 54 runs, adversarial interior points pool 36, and the SymmetricMixture benign sweep uses 18. Dotted bridges cross unmeasured regions. The ask-nothing baseline is .625.
A source can behave well on the checks it recognizes.
Choose a source rule. Each view separates the failure, tested defense, measured result, and strength of the evidence.
The best source classifier can still make the worse decision.
The second environment uses categorical evidence and asymmetric losses. An agent may approve, block, hold, or escalate a transfer.
Where the evidence stops
Useful when calibrated. Fragile when the source changes.
- Several Battleship sweeps and the audit study use one seed. The audit evidence for task recovery is preliminary.
- The theory covers stationary binary channels. It does not cover strategic sources.
- The transfer environment uses hand-specified likelihoods and is not a deployment study.
- Live results need language-channel calibration. A new caller behavior can make an old calibration stale.
- This browser replay uses one stationary source and a fixed board sample. It demonstrates the update, not the full experimental pipeline.