Vote Count Analysis: How Do You Do it?
-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
Vote Count Analysis: How Do You Do it?
Goal of this thread is to establish a concrete grasp of what VCA is on mafiascum.net.
I wonder if people who find this thread could, as precisely as they can while acknowledging the ambiguity of it all, walk me through the nuts and bolts of:
1) How they go through the process of performing vote count analysis
2) What kinds of assumptions and information they use to ground this analysis
3) What they think can and cannot be learned from VCA
Also:
People seem to talk about VCA as if it's some standardized process that can solve games under a lot of conditions even if people's reads are otherwise null or wrong.
How true is this?
How consistent is the process behind VCA from person to person?
And to what extent is it distinct from and can supplant the regular approach of just reading players' posts and looking for behavior that seems disingenuous?You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
i have a lot of voting data scraped from this site that i've had on my todo list to analyze for a while if you ever want to push furtherYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
Yeahh I've looked at some things in a similar way. I used to be very attached to the program of just testing people's different ideas about what scumtells are but there are a few issues with this.
The main issue is that most scumtells are infinitely moving goal-posts, always more context-sensitive than how people initially specify them. When you dig into them, they turn out to only *really* be scumtells when players do the described behaviors "in a scummy way", which pushes the question of what makes a behavior scummy one step further away. This makes "testing" a scumread with data really challenging. And people trying to apply such tells in an actual game either still wrestle with the basic challenge of sorting whether what they're seeing is *really* a scumtell, or stop thinking beyond pattern detection. Not super useful either way. This feature of scumtells means they're primarily useful as a way of focusing attention -- either when trying to read someone or when trying to push a wagon. But they don't necessarily provide much sorting information beyond that.
Have come to think that we ultimately want to discover shared principles across (real) tells that explainwhythey are tells and can be flexibly applied across scenarios to sort players and justify reads. These principles could conceivably even imply more specific scumtells that have thus far been overlooked all their own. They might even tell us something about how people do social deduction and deception in general. This focus might at least help decide which among the massive pile of supposed scumtells to spend time testing...You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
VCA seems like a good place to try finding some principles for two reasons.
First, votes are kind of concrete and easier to extract as a researcher than the stuff people say in their posts.
Second, the baseline theory of how having the scum faction's knowledge and win condition might shape how they vote is really clear, at least on some level:
Scum's most straightforward incentive during gameplay is to see that town are eliminated instead of their teammates. Even one teammate's death substantially reduces their chances of winning, all else equal. If scum could vote anonymously, they would always group all their votes on whichever town slot is closest to being eliminated. But their votes aren't anonymous. So they are also strongly incentived to vote in a way that is town-like. These incentives have to be balanced throughout the game, and run into the greatest conflict when the town-like thing to do is to vote out their teammates.
If right, this suggests that scumtells can be roughly grouped into three classes:
- A flawed idea of town-like behavior. They follow their notion of what town-like play looks like, but this notion is wrong enough that their play isn't town-like.
- Poor execution of town-like behavior. They have a great idea of what town-like play looks like, but they struggle to actually implement it for some reason (e.g., skill, difficulty, resource constraints).
- Scum motivated behavior. They have a great idea of what town-like play looks like and are perfectly capable of mimicking it, but diverge because they've judged it makes them more likely to win.
The latter seems easier to study at first because it doesn't necessarily require a lot of psychologizing to get started. Instead, we just have to examine incentives. For example: Even though scum are incentized to mimic town, the motivation to see town eliminated and their teammates survive is also strong. In aggregate, a decent baseline prediction from all this might that scum-on-scum votes are relatively rare, at least at certain junctures in the game where the votes would actually be impactful.
In the above example, the overarching principle is that scum will break 'character' when it suits their win conditions, and the more specific tell to look out for is hesitance to vote their buddies. This might show up in VCAs as absent, late, and/or shaky commitment to wagons on slots that are only now known by town to be scum. Of course town can also be hesitant to vote a player for any number of reasons, and scum can bus to avoid suspicion from or even to be positively townread by people sensitive to these dynamics. But maybe the incentive built into their win conditions to keep teammates alive overcomes these forces enough that there's at least one consistent pattern in how scum vote.
Finding these in voting data would improve our confidence in this overarching approach to finding scum, while gaps in evidence for these patterns could motivate us to add caveats to the approach or seek out alternatives. Having these higher principles makes it easier to reason about what the downstream tells would like like in odd scenarios, or when they wouldn't apply, or when they would be especially telling. And the reliability of these patterns can help weight how seriously we apply these principles in our reads. All this specifies some kind of research program for building an evidence-based approach to scumhunting and an account of how people play social deduction games in general, even if the initial predictions about what we'll find are wrong.You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
This post has a lot of concrete ideas about vca that i overlooked but look pretty standard. will try to extract relevant stuff eventually (prob using chatgpt to some extent)
In post 5, mhsmith0 wrote:IC POST #2
**note: the following post is PRE WRITTEN, BEFORE I GOT MY ROLE PM. It has ZERO bearing on my alignment**
Since this is a newbie game, I figured I'd share some resources that I myself have found VERY useful (shamelessly copied/paraphrased from other sources, with a few of my own thoughts and examples mixed in as well). I'll be hiding them under spoiler tags, so feel free to ignore if you aren't really interested. This is 100% OPTIONAL, but I found it extraordinarily useful in improving my game. You may or may not.
Spoiler: Voting Analysis
Spoiler: Micro vote analysis
Spoiler: Macro vote analysis
Spoiler: D1 scumhunting
Spoiler: Theories about how scum place their votesYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
Spoiler: fear im moving a little offtopic so i'm spoilering
EDIT: have a separate repo that pulls votes out across games, yeah. but i find the whole thing embarassing atmYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
Someeeee thinking around analysis plan...
My analyses before looked at final votecounts, and measured the rate at which scum landed at each position on the hammered wagon as a function of factors like the hammered slot's faction, the Day, and so on.
For comparison, I generated a null hypothesis dataset where scum/town labels were repeatedly randomly shuffled per game, and applied the same analysis.
The idea was to set up a context for non-parametric statistical testing. But honestly I'm not sure I was even doing it properly, or if the tests were actually telling me much.
Effect sizes seemed small, and I'm not confident in anything I found, let alone in their usability.
Some reflections on that:
- The null hypothesis distribution thing is maybe still a good idea if implemented right, idk. But I'll probably try harder to verify that I understand the approach and its trade-offs and that I've implemented it correctly.
- While I always expected to go further, just looking at final wagons is wasteful (ignores most votes), and also seems sort of perspective-warping in a way harmful for the analysis and our ability to interpret outcomes. End-of-day analyses are still useful, but probably not best to treat as this kind of project's focal analyses.
- Like, there are likely many systematic differences between successful wagons and unsuccessful wagons. Without a full idea of what those are, it's not clear what we can learn about voting behavior in general from just the final votecounts of every game. And final votecounts are sort of a bit removed from the actual thing we're interested in: player decision-making. Players usually don't know when they first place their vote whether their wagon will fail or not, so conditioning analyses on success/failure obscures the context behind their decisions. A result like "days rarely end with scum as the first vote on their teammates' wagons" gives us some insight into their decision-making and might evince that they hesitate to stay on wagons that aren't advantageous to their win condition but this outcome is a few steps removed from the sequences of decisions made to actually lead to this result -- scum watching the wagon they're on grow and reason after each vote whether to move wagons or double down.
- Correspondingly, it's probably better to frame analyses as examining how players navigate "forks in the road" throughout a Day instead of the Day's final state. We can measure players' propensities to join, leave, or stay on a wagon as a function of its size (or other wagons' size) instead of just looking at their position in a final snapshot. In this way, analyses use more data and provide more direct insight into players' decision-making and Days' overall trajectories.
- I also think I'd more aggressively bin together similar outcomes than I did before. First and last on a wagon are super salient, sure, but I'm doubtful that we need to measure separate entry rates to the third and the fourth position on a wagon in a Day it takes 7 votes to achieve an elimination, for instance. Grouping positions means we'll have more data per bin and have an easier time detecting meaningful differences. Maybe four bins in the wagon-positional analyses, tops?
- I still have trouble conceptualizing how to pull it all together into a presentation that's easy to use and understand given the vast range of analyses that could be done on this data, and ways those analyses could be implemented. Focusing on a specific research question helps with the first half of this, but not the second. But I think this will get clearer as I get a better idea of what VCA already is like and how a tool based on this research could most directly support that.You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
are you stats savvy? this is maybe the trickiest part of all of this for me. the stats courses ive taken have almost been theoretical to a fault (and also i didnt do well)You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
but what's the overarching idea or ideas behind all these patterns? that they're attention-avoiding? lack strong opinions?You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
cool idea!In post 23, joqiza wrote:
So it occurs to me that the way to test this kind of NAI result under your shuffling approach might be to, rather than assigning the alignments completely at random, instead assign the alignmentsIn post 17, joqiza wrote:It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.as if it weretowny or scummy, and then rerun the analysis, and then do that 5000 times or whatever and then see how likely it is to observe the NAI result.
it looks like the term for this is equivalence testing
maybe there's also a source directly describing how to do this in a nonparametric way like i think im doing to try to detect differences. i should probably already be tracking this stuff somewhere
but my guess is that it'll say something like what you described? -- that i can use the CI from an analysis applied this "equivalence" distribution and interpret complete overlap with the CI from applying the analysis to my actual data as unambiguous evidence that a behavior is NAIYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
1) Cool. "When wagons are all on villagers" is concrete enough. But I wonder how we make concrete "[not] prominently driving any single wagon in particular " or "[finding] a way not to do anything"? In posts I guess this would look like weak (?) or infrequent advocacy for the wagon. But what about in voting patterns? Maybe more shifting? Less?In post 24, mhsmith0 wrote:In post 21, Psyche wrote: but what's the overarching idea or ideas behind all these patterns? that they're attention-avoiding? lack strong opinions?
1) If the thread environment is such that wolves don't have to do anything to move the game forward in a favorable direction, then typically they find a way not to do anything - why take heat for an incorrect wagon when villagers will do it for you? (this then comes out later in game once you have more data to base things on, though sometimes role claims will give you info earlier)In post 19, mhsmith0 wrote: I'd probably throw on a couple of other things too
1) When wagons are all on villagers, wolves are usually (though not always) less likely to be prominently driving any single wagon in particular (aggressive / powerwolfing wolves exist though)
2) When there's like two clear wagons and they're both town, and you have some number of players off of both wagons, they usually should get a lot of scrutiny - maybe it's something NAI (like a quickhammer or someone who 0 posted the day phase), maybe they're just really obvious towns in that spot (openly anti both wagons in a very prominent way), but otherwise that kind of positioning tends to be > rand wolf
3) Signs of bussing tend to be harder to pick up but are worth considered where they can show up as.
2) similar idea - though here a lot of it is "ok everyone else wanted one of these folks dead, why didn't you, and why should we believe you're not full of it" - keep in mind that when a wolf is under pressure this is much less reliable (for instance, if the third wagon was a wolf but it faded away to make the top two wagons town, then wolves are likelier to have been materially involved)
also, frankly, in a spot where someone is off wagon in a v/v environment it can be REALLY easy to spot obvtowns in that spot, so "you were off wagon and you were NOT obvtown" is the more composite thing to be looking at
3) bussing signs are a different issue entirely, though i do think it can be instructive to look at overall game flow, who gained credibility from that wagon, and then how they acted subsequently - who's trusting who, who's engaging with who, is that what is "supposed" to happen if they were town, etc - this is more cross-referencing possible busses with behaviors that jar with how you'd expect a townie to act given that info, which is DIFFICULT for the most part, but from time to time you can get situations where it's relatively obvious
2) your description seems to cast it as a special case of 1 but w/ some useful caveats.
3) this resonates but still leaves it challenging to detect bussing yeah. there's a level of WIFOM to it that makes the whole deal challenging -- the more scum go out of their way to lim other scum, the townier they might look, making it hard to predict or read them based on an incentive analysis alone.You can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
yeah probably more precise to say that my approach is "monte carlo" or "bootstrapping" than just to say it's nonparametric
i took this computation heavy approach because i figured it was a decent way to avoid thinking about the right way to frame analyses that, say, care about the identities of more than one slot at a time
i'd definitely be happy to take a less heavy approach if i could feel more confident going about itYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
all this is probably enough detail for me. thanks very muchYou can't step in the same river twice.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10889
- Joined: April 28, 2011
- Pronoun: he/they
thanks!
a couple questions
First, can you elaborate some on how you came to learn these practices for performing and reasoning about VCA?
Also,
> For instance, if a townie is mislimmed on a wagon day 1, with like 70% of the players on it, then the chances of a mafia being on the wagon is high.
Wouldn't it always be 50% + 1? Or are you counting everyone who appears on the wagon at all, even if they leave?
Overall, I'm looking for more context on how you infer whether mafia were on a town mislim or not.You can't step in the same river twice.
Copyright © MafiaScum. All rights reserved.