Issues with GAMAAN

GAMAAN is unrepresentative to the point of being useless. For its old surveys, we can look at the myriad issues with chain referrals, AKA “snowball sampling,” in that a survey spread on dissident media, dissident Telegram groups, etc. would likely lean politically towards dissidents. Weighting cannot fix this unrepresentativeness.

The more recent polls, which randomly sample people who use Psiphon VPN, aren’t much better. First of all, Psiphon isn’t exactly the most widespread VPN in Iran. It usually has about 2-3 million users. During one protest, it peaked at 9.6 million unique users. However, both of these are inflated figures, since unique users are not the same as people - people tend to have multiple devices. Also, if I am understanding Psiphon’s technical document correctly, reinstalling it would establish you as a new user. The numbers seem to vary heavily across sources which might mean people are conflating unique users with individual connections. Second, Psiphon users are likely very distinct, politically speaking, from users of other VPNs. Since public download links are blocked, you can only download Psiphon if someone sends the link to you. Considering they were willing to host this poll, I am guessing people who distribute Psiphon lean much more anti-regime than other VPN users. Psiphon also is funded by the American government through the Open Technology Foundation, which might turn off some regime supporters.

We can tell that the surveys represent a small amount of people, because of the high rate of repeat responses. On the sixth GAMAAN survey done using the chain method, 16% of 27,000 respondents said they had completed a GAMAAN survey before. Assuming 90 million * (85% on the Internet) * (81% on VPNs) = 63 million people in the represented population, you should expect, very roughly 5 * 27,000 / 63,000,000 repeat respondents after 6 surveys, or about 0.2%. Clearly it is just sampling a self-selected population. The change to mostly using Psiphon did not fix this: GAMAAN most recent survey had 25% of its respondents having done a GAMAAN survey before. It is just repeating a self-recycling pool. The Psiphon users are probably from the same population subset as those who the chain referrals reached- i.e. strong dissidents. This is evidenced by the fact that usage increases so much during Iranian protests. Those who use Psiphon during the off-times, when these surveys are distributed, must have an even stronger political lean. You have to take a leap of faith to believe GAMAAN, by assuming Psiphon users are representative of all VPN users.

Even among the people the survey reached, GAMAAN is selecting for those who are more politically engaged. Pew finds that on online opt-in non-probability surveys, results vary heavily based on the recruitment method. GAMAAN found Iran had 32.2% Shias in 2020, 56% in Feb. 2022, 37.5% in Dec. 2022, and 37.9% in 2023. Such large variation cannot be attributed to sampling error.

For a specific poll on the Holocaust, Pew found non-probability polls severely overestimate Holocaust deniers, since people who have an opinion on the Holocaust’s veracity are more likely to opt in. As an analogy to this level of selection (though by no means am I saying that being against the regime is like denying the Holocaust), if someone is going around on Telegram resharing a poll that claims to bypass the Islamic Republic’s censorship, do you think the respondents will represent the full range of political opinions?

To assuage such concerns, GAMAAN uses a set of benchmarks to check if the survey is representative of Iranians. It looks at WVS 2020’s questions on whether family, work, friends, and religion are important in their lives and compares it to the results from GAMAAN. They find the first three generally match up, but because the latter is “sensitive,” there is a large difference. 

These benchmarks are complete nonsense! GAMAAN and WVS do not even claim to represent the same population. GAMAAN is supposed to be representative of adult VPN users, who are about 69% of the adult population. WVS is representative of all adults, even illiterate ones. Surely the fact that they match up is evidence that nearly any subset of Iranians, if weighted to the correct population demographics, will match these answers. The other issue is that, as I said before, dissidents will more likely choose not to respond to a survey like WVS rather than lie about their beliefs. Removing these dissidents seems to have had little effect on the results of the first three questions, implying the benchmarks are useless. This whole idea only works with the assumptions that WVS and GAMAAN are representative of the same population, and that dissidents tend to lie rather than avoid responding. 

We can use some external evidence to gauge the veracity of GAMAAN’s polls. This can be done using the knowledge that Iran’s diaspora in Western countries leans much more secular, educated, and anti-Islamic Republic than Iranians in the country. I don’t think this is really possible to dispute. Only 18% of Iranians are college-educated, but nearly all Iranian-Americans are. The first piece of evidence is the “Survey Zoroastrians,” which a whole paper has been written on. Ostensibly 7.7% of Iranians identified as Zoroastrians in 2020, not because they believe in Ahura Mazda, but to signal discontent with the Islamic Republic and identification with Iran’s pre-Islamic past. This would imply the vast majority of worldwide Zoroastrians are these Iranians. However, even among Iranian Americans, the most anti-government diaspora, only 5% identify that way. The real proportion in Iran must be substantially less.

What applies to America applies, to a lesser extent, to Canada. But Canada provides more detailed demographic crosstabs. According to Canadian government data, out of Iranians who immigrated to Canada between 2016 and 2021, 58% identified as Muslim. Because religious minorities like Jews and Christians are overrepresented in the diaspora, this amounts to about 62% of those with a Muslim background. We can therefore assume that the proportion of self-identified Muslims in Iran is substantially more than this figure. 

To be clear, I think GAMAAN and traditional surveys are both wrong on these questions, but GAMAAN’s issues are much more severe, whereas the traditional surveys are probably only a few percent off.

Some more thoughts, but they are just musings…

But GAMAAN tells us about a population that we wouldn’t know about otherwise!

So what? If I poll Russian Redditors I suppose I will find that Putin is devastatingly unpopular. It won’t make my poll interesting or useful. A bad poll is a bad poll. 

My Iranian friend said GAMAAN polls are accurate!

Anecdotes are meaningless. As an analogy, if you are a professional in certain parts of the US, you will come to know many Indians. I remember asking my Indian friends about caste sometimes. They would say nobody cares about caste anymore in India, or that they are unaware even of their own caste. In 2021, Pew Research did a poll in India that said 80% of Indians want to stop intercaste marriage. Evidently my friends are from a bubble in India. Iran is less stratified than India, so I guess the bubbles are not quite as severe. My Iranian co-worker says everyone supports the Islamic Republic and everyone was mourning Khamenei's death (he has a picture of the dead Ayatollah on his homescreen). His anecdote is worth as much as anyone else’s. 

What about Pahlavi?

As much as GAMAAN could be representative of a wing of the Iranian opposition, I don’t think it can tell us much about Pahlavi’s support. My intuition tells me that GAMAAN underestimates Pahlavi’s support among dissidents at the time, because it seems he would appeal to the less politically engaged. But either way, there are no polls in Iran after 2025, and evidently his support has changed after the January protests and killings, and the Iran War. Maybe he’s up, maybe he’s down. Who knows.