game-mediated signal transmission

When you get information from a non-agentic source, the feedback loop contains

• your choices
• the consequences
• your pleasure or displeasure upon seeing those preferences.

In other words, getting a clear signal from the world in this class of scenarios is a decision-theory problem. In your capacity as a bounded, motivated reasoner, you have a lot of tools at your disposal to:

• course-correct towards pleasure (direct movements towards goals)
• operationalise your sense of efficacy (getting what you want) into a metric, and course-correct towards improving it (first-order changes to strategy)
• update what you want on the basis of what consequences seem to follow from acting to get it (first-order changes to goals)
• generalise principles that seem to hold in common between goals or causal structures, and systematically better orient to those (second-order updates)

These practices as stated hold a lot in common with the strategic level CFAR module.

What happens when the information you're getting is mediated by other agents? Things look very different inside the feedback loop:

• your choices
• your pleasure and displeasure
• the other agents' choices
• their pleasure and displeasure
• the consequences
• each agent's perception thereof

Lots of eddy effects emerge.

• common knowledge of the mere possibility of misalignment between my pleasure and somebody else's means that we're always playing a red-queen race to signal both alignment and sincerity (whether we are sincere or not).
  • the most persuasive strategies, paradoxically, will be ones that send the costly signal of willingness to be at odds, which is both why strong friendships often involve countersignalling and why high-drama people read as charismatic.
• if your incentives are aligned with somebody else's, you can trust what they tell you about the world. if they aren't, you can't.
  • this means that it's helpful to have canaries - facts that you know that work as trust proofs.
    • if everybody lives in a corner of the universe where X is true, X is likely to be a canary. If someone then wanders off into a corner where they can observe X doesn't hold, they risk being ostracized from the infromation network if they communicate it.
  • also helpful to have certificate authorities - a network of known-trustworthy authorities.
    • if a network of certificate authorities begin to disagree with each other, there will come critical point of low consensus or low service availability which loses them their ability to function in their roles. Therefore, a trusted network is more likely to survive if it partitions quickly than if it solves for consensus slowly.
    • insults to the information ecosystem can and will kill a consensus network. everyday churn can also. therefore it's strategic for trust networks to undergo regular apoptosis, to keep the meme-ome fresh.