Vote Count Analysis: How Do You Do it?
-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
Vote Count Analysis: How Do You Do it?
Goal of this thread is to establish a concrete grasp of what VCA is on mafiascum.net.
I wonder if people who find this thread could, as precisely as they can while acknowledging the ambiguity of it all, walk me through the nuts and bolts of:
1) How they go through the process of performing vote count analysis
2) What kinds of assumptions and information they use to ground this analysis
3) What they think can and cannot be learned from VCA
Also:
People seem to talk about VCA as if it's some standardized process that can solve games under a lot of conditions even if people's reads are otherwise null or wrong.
How true is this?
How consistent is the process behind VCA from person to person?
And to what extent is it distinct from and can supplant the regular approach of just reading players' posts and looking for behavior that seems disingenuous?-
-
Klick Flash Forward
- Flash Forward
- Flash Forward
- Posts: 12904
- Joined: September 1, 2012
Love this thread
I have nothing to contribute because I think most VCA removed from its context is super sketch
But I'll drop an ego out of interest-
-
Ythan SheWelcome to the HaystackShe
- Welcome to the Haystack
- Welcome to the Haystack
- Posts: 15154
- Joined: August 11, 2009
- Pronoun: She
-
-
DragonEater70 He/HimJack of All TradesHe/Him
- Jack of All Trades
- Jack of All Trades
- Posts: 7785
- Joined: February 4, 2023
- Pronoun: He/Him
- Location: UTC+2
So 2 things:
1. I personally have no clue how to do VCA but at least I'm honest about it. When I do "VCA" I am basically just trying out a fun idea and definitely have no idea what the actual methodology behind it is. Which is why my "VCA" fails. And also why this thread could actually be useful.
2. I do think that when done properly, VCA can strongly support arguments for why someone is town and someone else is scum. For example in Mini Normal 2322, Flavor Leaf (who's known as being pretty good with VCA I think) was able to identify that Pavowski couldn't be scum and Hu Tao had to be scum based on the fact that the wagons were tied, there were 3 confirmed town (out of the required 5 votes for elimination) on Hu Tao, and she still wasn't eliminated. So while I am not an expert on VCA I can tell you it's pretty useful.
As a sidenote:
While looking at VCs and who was town and on which wagon is not something I usually manage to get alignment indicative information out of reliably, I want to address your last question of "to what extent is it distinct from and can supplant the regular approach of just reading players' posts and looking for behavior that seems disingenuous?", and say that while I don't think VCA (which I define as looking at the actual VCs in relation to the knowledge of alignments of flipped players) can supplant the regular approach, I DO think that looking at a person's voting pattern as well as the pattern of votes in relation to them (who votes them, who unvotes them, etc.), CAN supplant the regular approach of looking for disingenuous posts and often works BETTER than that approach. Like often you can catch scum just by noting the timing of someone's vote, or the lack of a vote where a vote is to be expected, where focusing on the actual content of the post wouldn't allow you to do that. To bring up again the example of Mini Normal 2322, in that game I noted that Hu Tao unvoted HPE with a bad reason, and with HPE having flipped red that indicated Hu Tao could be scum. Had I listened only to this and didn't also consider the tone, progression, etc., of Hu Tao's posts, I would've probably been able to catch her (which I ended up not doing because I was very pocketed).-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
Perhaps not quite VCA, but about a year ago I ran an analysis on a particular voting pattern (unvoting without revoting). I didn't end up drawing any definitive conclusions, but I published the dataset and a discussion around it can be found here: https://www.mafiauniverse.com/forums/th ... tistics%29-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
i have a lot of voting data scraped from this site that i've had on my todo list to analyze for a while if you ever want to push further-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
Sure, I'd be happy to help advise if you're looking to run any kind of analysis. Honestly, I think the data collection and sanitization is the tougher part. For the analysis I linked, I picked only mountainous games to keep it simple. Once you start introducing different types of mechanical information, I think a lot more thought than I was willing to do needs to go into how you are going to standardize data across different types of games.
My general approach in the analysis I ran was pretty straightforward. For the "sonny tell" (unvoting without revoting >1 hr) I defined the experiment as "exhibits the tell at least once on a given day." So for example in Game 1 on Day 1, 3 town players exhibited the tell of 12 overall, so I record this as 3 town successes and 9 town failures. For the wolves on Game 1 Day 1, 1 wolf player exhibited the tell of 3 overall, so I recorded 1 wolf success and 2 wolf failures. Etc. for all game days for all games in the dataset.
I found that the simple tell without any criteria was not obviously indicative in either way. A player named soah proposed a list of criteria which, when applied, led to greater separation between the estimated wolf and town parameters. At that point we were potentially data snooping, though.
It was an interesting exercise and I'd be happy to talk to anyone more about it if they have data they want to test. That said, I suspect this is all probably more of interest to people who are into math/stats than really useful to anyone seeking some kind of competitive advantage in games.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
Yeahh I've looked at some things in a similar way. I used to be very attached to the program of just testing people's different ideas about what scumtells are but there are a few issues with this.
The main issue is that most scumtells are infinitely moving goal-posts, always more context-sensitive than how people initially specify them. When you dig into them, they turn out to only *really* be scumtells when players do the described behaviors "in a scummy way", which pushes the question of what makes a behavior scummy one step further away. This makes "testing" a scumread with data really challenging. And people trying to apply such tells in an actual game either still wrestle with the basic challenge of sorting whether what they're seeing is *really* a scumtell, or stop thinking beyond pattern detection. Not super useful either way. This feature of scumtells means they're primarily useful as a way of focusing attention -- either when trying to read someone or when trying to push a wagon. But they don't necessarily provide much sorting information beyond that.
Have come to think that we ultimately want to discover shared principles across (real) tells that explainwhythey are tells and can be flexibly applied across scenarios to sort players and justify reads. These principles could conceivably even imply more specific scumtells that have thus far been overlooked all their own. They might even tell us something about how people do social deduction and deception in general. This focus might at least help decide which among the massive pile of supposed scumtells to spend time testing...-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
VCA seems like a good place to try finding some principles for two reasons.
First, votes are kind of concrete and easier to extract as a researcher than the stuff people say in their posts.
Second, the baseline theory of how having the scum faction's knowledge and win condition might shape how they vote is really clear, at least on some level:
Scum's most straightforward incentive during gameplay is to see that town are eliminated instead of their teammates. Even one teammate's death substantially reduces their chances of winning, all else equal. If scum could vote anonymously, they would always group all their votes on whichever town slot is closest to being eliminated. But their votes aren't anonymous. So they are also strongly incentived to vote in a way that is town-like. These incentives have to be balanced throughout the game, and run into the greatest conflict when the town-like thing to do is to vote out their teammates.
If right, this suggests that scumtells can be roughly grouped into three classes:
- A flawed idea of town-like behavior. They follow their notion of what town-like play looks like, but this notion is wrong enough that their play isn't town-like.
- Poor execution of town-like behavior. They have a great idea of what town-like play looks like, but they struggle to actually implement it for some reason (e.g., skill, difficulty, resource constraints).
- Scum motivated behavior. They have a great idea of what town-like play looks like and are perfectly capable of mimicking it, but diverge because they've judged it makes them more likely to win.
The latter seems easier to study at first because it doesn't necessarily require a lot of psychologizing to get started. Instead, we just have to examine incentives. For example: Even though scum are incentized to mimic town, the motivation to see town eliminated and their teammates survive is also strong. In aggregate, a decent baseline prediction from all this might that scum-on-scum votes are relatively rare, at least at certain junctures in the game where the votes would actually be impactful.
In the above example, the overarching principle is that scum will break 'character' when it suits their win conditions, and the more specific tell to look out for is hesitance to vote their buddies. This might show up in VCAs as absent, late, and/or shaky commitment to wagons on slots that are only now known by town to be scum. Of course town can also be hesitant to vote a player for any number of reasons, and scum can bus to avoid suspicion from or even to be positively townread by people sensitive to these dynamics. But maybe the incentive built into their win conditions to keep teammates alive overcomes these forces enough that there's at least one consistent pattern in how scum vote.
Finding these in voting data would improve our confidence in this overarching approach to finding scum, while gaps in evidence for these patterns could motivate us to add caveats to the approach or seek out alternatives. Having these higher principles makes it easier to reason about what the downstream tells would like like in odd scenarios, or when they wouldn't apply, or when they would be especially telling. And the reliability of these patterns can help weight how seriously we apply these principles in our reads. All this specifies some kind of research program for building an evidence-based approach to scumhunting and an account of how people play social deduction games in general, even if the initial predictions about what we'll find are wrong.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
This post has a lot of concrete ideas about vca that i overlooked but look pretty standard. will try to extract relevant stuff eventually (prob using chatgpt to some extent)
In post 5, mhsmith0 wrote:IC POST #2
**note: the following post is PRE WRITTEN, BEFORE I GOT MY ROLE PM. It has ZERO bearing on my alignment**
Since this is a newbie game, I figured I'd share some resources that I myself have found VERY useful (shamelessly copied/paraphrased from other sources, with a few of my own thoughts and examples mixed in as well). I'll be hiding them under spoiler tags, so feel free to ignore if you aren't really interested. This is 100% OPTIONAL, but I found it extraordinarily useful in improving my game. You may or may not.
Spoiler: Voting Analysis
Spoiler: Micro vote analysis
Spoiler: Macro vote analysis
Spoiler: D1 scumhunting
Spoiler: Theories about how scum place their votes-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
My own thoughts have gone down similar avenues in the past and I wish you luck. I'll check out your scraper when I get home. I'd built something to grab MU vote histories in the past but that's more structured I think. An MS dataset would be really interesting.
A word of caution--there's a rule on this site prohibiting players from creating machine learning networks to classify posts. I don't think that's what you're proposing, but the mention of ChatGPT may make some folks a bit nervous.-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
Oh, this scraper is pretty cool. Nice work. Do you already have a script which constructs a votecount dataset from the json? I could give a shot at that if you'd like.
I imagine one approach might be to create a dataset of votes with variables for the following:
- Day Phase
- Postnum
- Time
- Voter
- Voter Alignment
- Voted
- Voted Alignment
- Vote Type (vote or unvote)
- Vote Count
- Votes Needed for Elimination
- # Town Alive in Day Phase
- # Wolves Alive in Day Phase
- # Town on Wagon
- # Wolves on Wagon
- Alignment of Prior Vote
- Count of Prior Vote
- Alignment of 2nd Highest Wagon
- Vote Count of 2nd Highest Wagon
- # Town on 2nd Highest Wagon
- # Wolves on 2nd Highest Wagon
... (etc for 3rd, 4th ...)
- # People Not VotingLast edited by joqiza on Sat Feb 17, 2024 12:50 pm, edited 1 time in total.-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
If you really want to develop a theory, you'll probably need to find some way to capture things like "oh, this guy just got red-checked so everyone moved on to him." Not sure how to do that in a programmatic way. I just stuck with mountainous games in the past which made things easy but limited my data.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
Spoiler: fear im moving a little offtopic so i'm spoilering
EDIT: have a separate repo that pulls votes out across games, yeah. but i find the whole thing embarassing atm-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
Someeeee thinking around analysis plan...
My analyses before looked at final votecounts, and measured the rate at which scum landed at each position on the hammered wagon as a function of factors like the hammered slot's faction, the Day, and so on.
For comparison, I generated a null hypothesis dataset where scum/town labels were repeatedly randomly shuffled per game, and applied the same analysis.
The idea was to set up a context for non-parametric statistical testing. But honestly I'm not sure I was even doing it properly, or if the tests were actually telling me much.
Effect sizes seemed small, and I'm not confident in anything I found, let alone in their usability.
Some reflections on that:
- The null hypothesis distribution thing is maybe still a good idea if implemented right, idk. But I'll probably try harder to verify that I understand the approach and its trade-offs and that I've implemented it correctly.
- While I always expected to go further, just looking at final wagons is wasteful (ignores most votes), and also seems sort of perspective-warping in a way harmful for the analysis and our ability to interpret outcomes. End-of-day analyses are still useful, but probably not best to treat as this kind of project's focal analyses.
- Like, there are likely many systematic differences between successful wagons and unsuccessful wagons. Without a full idea of what those are, it's not clear what we can learn about voting behavior in general from just the final votecounts of every game. And final votecounts are sort of a bit removed from the actual thing we're interested in: player decision-making. Players usually don't know when they first place their vote whether their wagon will fail or not, so conditioning analyses on success/failure obscures the context behind their decisions. A result like "days rarely end with scum as the first vote on their teammates' wagons" gives us some insight into their decision-making and might evince that they hesitate to stay on wagons that aren't advantageous to their win condition but this outcome is a few steps removed from the sequences of decisions made to actually lead to this result -- scum watching the wagon they're on grow and reason after each vote whether to move wagons or double down.
- Correspondingly, it's probably better to frame analyses as examining how players navigate "forks in the road" throughout a Day instead of the Day's final state. We can measure players' propensities to join, leave, or stay on a wagon as a function of its size (or other wagons' size) instead of just looking at their position in a final snapshot. In this way, analyses use more data and provide more direct insight into players' decision-making and Days' overall trajectories.
- I also think I'd more aggressively bin together similar outcomes than I did before. First and last on a wagon are super salient, sure, but I'm doubtful that we need to measure separate entry rates to the third and the fourth position on a wagon in a Day it takes 7 votes to achieve an elimination, for instance. Grouping positions means we'll have more data per bin and have an easier time detecting meaningful differences. Maybe four bins in the wagon-positional analyses, tops?
- I still have trouble conceptualizing how to pull it all together into a presentation that's easy to use and understand given the vast range of analyses that could be done on this data, and ways those analyses could be implemented. Focusing on a specific research question helps with the first half of this, but not the second. But I think this will get clearer as I get a better idea of what VCA already is like and how a tool based on this research could most directly support that.-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
Don't worry about messy code lol, I'm sure anything I've written is worse. I'm an econ guy not CS, all my programs are an abomination unto god. Anyway I might give a shot tonight or tomorrow at writing something, if I don't get it done by then I might not get it done for awhile tbh but I'll let you know. FWIW I am not aware of any publically available large dataset of votes in mafia games, and I think just creating something like that would be a huge accomplishment.-
-
mhsmith0 Balancing Act
- Balancing Act
- Balancing Act
- Posts: 10830
- Joined: March 7, 2016
- Location: Phoenix, AZ
All yall might enjoy this thread on the topic
viewtopic.php?t=73863Showhttp://wiki.mafiascum.net/index.php?title=Mhsmith0
Conq: you, sir, are great at being town.
BATMAN: Only jugg was the only one we didn’t scum read at least not me
Quick: There is little to no chance this slot is Power-Wolfing.
SR: I want to give him a day
Life is simply unfair, don't you think?-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
Reading through your analysis plan, I'm not sure I quite understand your approach, but I do like the idea of shuffling alignments around and re-running as a test. Yeah I am not immediately sure what that is telling you though. I would need to think about it.In post 14, Psyche wrote: Someeeee thinking around analysis plan...
I suppose if you had found that say, scum were much more likely to land on, idk, position #3 or something, and then you find that it is very unlikely to observe such a result when alignments are assigned at random, maybe that suggests it is a significant result? Hmm. Yeah got to think about it.
My guess is you found everything with scum position on EoD votecounts closer to NAI though. Yeah if that's the case when you start shuffling you'll probably find that it's pretty easy to generate something similarly NAI.
It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
are you stats savvy? this is maybe the trickiest part of all of this for me. the stats courses ive taken have almost been theoretical to a fault (and also i didnt do well)-
-
mhsmith0 Balancing Act
- Balancing Act
- Balancing Act
- Posts: 10830
- Joined: March 7, 2016
- Location: Phoenix, AZ
I'd probably throw on a couple of other things too
1) When wagons are all on villagers, wolves are usually (though not always) less likely to be prominently driving any single wagon in particular (aggressive / powerwolfing wolves exist though)
2) When there's like two clear wagons and they're both town, and you have some number of players off of both wagons, they usually should get a lot of scrutiny - maybe it's something NAI (like a quickhammer or someone who 0 posted the day phase), maybe they're just really obvious towns in that spot (openly anti both wagons in a very prominent way), but otherwise that kind of positioning tends to be > rand wolf
3) Signs of bussing tend to be harder to pick up but are worth considered where they can show up as.Showhttp://wiki.mafiascum.net/index.php?title=Mhsmith0
Conq: you, sir, are great at being town.
BATMAN: Only jugg was the only one we didn’t scum read at least not me
Quick: There is little to no chance this slot is Power-Wolfing.
SR: I want to give him a day
Life is simply unfair, don't you think?-
-
mhsmith0 Balancing Act
- Balancing Act
- Balancing Act
- Posts: 10830
- Joined: March 7, 2016
- Location: Phoenix, AZ
I think there are some sites / metas where wolves are meaningfully > rand to split votes, but there are plenty where that's basically just NAI entirely.Showhttp://wiki.mafiascum.net/index.php?title=Mhsmith0
Conq: you, sir, are great at being town.
BATMAN: Only jugg was the only one we didn’t scum read at least not me
Quick: There is little to no chance this slot is Power-Wolfing.
SR: I want to give him a day
Life is simply unfair, don't you think?-
-
Psyche he/theySurvivorhe/they
- Survivor
- Survivor
- Posts: 10874
- Joined: April 28, 2011
- Pronoun: he/they
but what's the overarching idea or ideas behind all these patterns? that they're attention-avoiding? lack strong opinions?-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
relatively ig. not a phd or anything but i studied math/econ in undergrad and used to work in market risk. I can give my thoughts but don't take me as an authorityIn post 18, Psyche wrote: are you stats savvy? this is maybe the trickiest part of all of this for me. the stats courses ive taken have almost been theoretical to a fault (and also i didnt do well)-
-
joqiza Goon
- Goon
- Goon
- Posts: 939
- Joined: May 3, 2020
So it occurs to me that the way to test this kind of NAI result under your shuffling approach might be to, rather than assigning the alignments completely at random, instead assign the alignmentsIn post 17, joqiza wrote:It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.as if it weretowny or scummy, and then rerun the analysis, and then do that 5000 times or whatever and then see how likely it is to observe the NAI result.-
-
mhsmith0 Balancing Act
- Balancing Act
- Balancing Act
- Posts: 10830
- Joined: March 7, 2016
- Location: Phoenix, AZ
In post 21, Psyche wrote: but what's the overarching idea or ideas behind all these patterns? that they're attention-avoiding? lack strong opinions?
1) If the thread environment is such that wolves don't have to do anything to move the game forward in a favorable direction, then typically they find a way not to do anything - why take heat for an incorrect wagon when villagers will do it for you? (this then comes out later in game once you have more data to base things on, though sometimes role claims will give you info earlier)In post 19, mhsmith0 wrote: I'd probably throw on a couple of other things too
1) When wagons are all on villagers, wolves are usually (though not always) less likely to be prominently driving any single wagon in particular (aggressive / powerwolfing wolves exist though)
2) When there's like two clear wagons and they're both town, and you have some number of players off of both wagons, they usually should get a lot of scrutiny - maybe it's something NAI (like a quickhammer or someone who 0 posted the day phase), maybe they're just really obvious towns in that spot (openly anti both wagons in a very prominent way), but otherwise that kind of positioning tends to be > rand wolf
3) Signs of bussing tend to be harder to pick up but are worth considered where they can show up as.
2) similar idea - though here a lot of it is "ok everyone else wanted one of these folks dead, why didn't you, and why should we believe you're not full of it" - keep in mind that when a wolf is under pressure this is much less reliable (for instance, if the third wagon was a wolf but it faded away to make the top two wagons town, then wolves are likelier to have been materially involved)
also, frankly, in a spot where someone is off wagon in a v/v environment it can be REALLY easy to spot obvtowns in that spot, so "you were off wagon and you were NOT obvtown" is the more composite thing to be looking at
3) bussing signs are a different issue entirely, though i do think it can be instructive to look at overall game flow, who gained credibility from that wagon, and then how they acted subsequently - who's trusting who, who's engaging with who, is that what is "supposed" to happen if they were town, etc - this is more cross-referencing possible busses with behaviors that jar with how you'd expect a townie to act given that info, which is DIFFICULT for the most part, but from time to time you can get situations where it's relatively obviousShowhttp://wiki.mafiascum.net/index.php?title=Mhsmith0
Conq: you, sir, are great at being town.
BATMAN: Only jugg was the only one we didn’t scum read at least not me
Quick: There is little to no chance this slot is Power-Wolfing.
SR: I want to give him a day
Life is simply unfair, don't you think?
Copyright © MafiaScum. All rights reserved.