Data Architect, Ph.D, Information Technologist, Gamer
6034 stories

What Patreon did wrong, and how to understand it

1 Share

A tale of content creation, crowdfunding and corporate communication

Credit: Kevin Dooley

The current Patreon PR gaffe fascinates me. Not only because I’m on there, but also because Patreon could have chosen to communicate these changes differently. Whether you agree with these changes or not, the fact remains that they were communicated in a way that makes very little sense to ordinary users, and which undermines both the company’s credibility and the message they wanted to send.

In short, the gaffe is afoot, and it is interesting to watch it unfold in real time.

Moreover, it is interesting to see which messaging strategy they went with. The strategy they actually went with was not set in stone until it went live, and they could have acted differently, framing the issue in a different light. Communication is not a stimulus-response thing, where actors unthinkingly do the thing the circumstances dictate for them. It is always possible to choose one strategy instead of another. The fact that Patreon went with the messaging they went with instead of literally anything else — this tells us something. It is important to understand the role it has in the current situation.

Let’s take a closer look at the messaging. The opening line of the official announcement sets the tone they want to convey: “In order to continue our mission of funding the creative class, we’re always looking for ways we can help creators continue to grow their creative careers.” Here, we can clearly see who is centered as the main beneficiary of these changes: the creators. The use of the adjective “creative” only further underscores who is considered to be at the heart of things. The upcoming changes are unequivocally framed as being made for the benefit of creators. (This is also reiterated in the FAQ.) Later on, the changes are further framed as being done at the behest of creators, and that the changes have in fact been explicitly requested by a large number of creators.

Needless to say, this is a case of clear communication. There is no mistaking what is being done and in whose name.

There is a subtle second component of this messaging, and that is the framing of the upcoming changes as a proactive step by Patreon to make things better in the longer term. Which is to say that there is no immediate circumstance that forced this change to happen with the quickness, and that the initiative for the changes comes entirely from within the house. There is a lot of effort being made to communicate that Patreon has gone to great lengths in considering these changes, and that what we see now is the result of a long process of thinking, pondering, measuring and deliberating.

In short: these changes are 100% portrayed as the preferred strategy of Patreon. They have gathered all the facts, and after careful consideration voted yes.

Understanding these two components — the claim that the upcoming changes are done in the name of creators, and the steadfastness of being in control of the process — is key to understanding the negative reactions that have so resounded over the interwebs. The first component alienates creators who do not see these changes as good (or even in their interest), while the second component underscores that there is no external reason for these changes. Or, rephrased: these changes are perceived as bad, the reasons given are seen as disingenuous, and there are no convincing external factors contributing to a sense of these changes being unfortunate but necessary.

In the shortest terms possible: the changes are seen as bad and unnecessary.

All this stems from the messaging in and of itself. Even if the only thing you read about the upcoming changes was this blog post, you would have sufficient raw material to fuel a non-significant anger. Especially considering the choice of visuals in the update, where the old system had three red dots, and the new system has eight. Those who already think the upcoming changes are bad, are not likely to change their mind by this clear demonstration of the extent of the changes.

To make matters worse, this is 2017. The production of fan fiction vastly outpaces that of non-fan fiction. Which is to say: people will not just read the corporate blog post. They will read — and write — more about the proposed changes, and they will bring their anger with them. This being 2017, things will get worse.

There is, for instance, this blog post about how the changes could be motivated by US financial regulations regarding the handling and storage of money. By making the changes outlined in the announcement, Patreon would avoid red tape that could potentially make life more difficult — or at least more bureaucratic — for everyone involved. It would be counterintuitive, but it would be understandable — much like the current situation with VAT. It wouldn’t be pretty, but there would be a tangible reason to fall back on.

Unfortunately for Patreon, this is not what they communicated in their messaging. They made it absolutely crystal clear that they were motivated by what they perceived to be the interests of the creators, without reference to external constraints that forced them to act in this particular way. The messaging was Patreon doing Patreon, out of its own free will. If it turns out these changes are made as a legalese maneuver, then the messaging was a lie. Which, in PR speak, is bad news bears.

There is also this twitter thread about how Patreon need to increase revenue in order to satisfy their venture capital investors. Given that the announced changes fit the bill in terms of more money going to parties that are not the creators (remember those eight red dots), this line of thinking makes sense. It has also made the rounds, with thousands of retweets and an untold number of quote-tweets. Whether it is true or not, it is part of the discussion, and there is precious little effort made to counter these lines of thinking.

Public relations is not about the truth. It is about perceptions. Having customers — in this case patrons and creators — thinking that what you are doing is a naked money grab is not a good thing. It is in fact a bad thing. Even if it is not true. Especially if it is not true.

Another trending topic is this June essay about various things Patreon has done to make things better. Some of the points discussed are interesting in and of themselves, and could use further elaboration elsewhere. The thing that caught everyone’s attention, though, was the notion of “life-changing” earnings, and the context in which it was introduced. It was explicitly discussed that Patreon made more money and got more attention from big users with big audiences than small users with small audiences, and thus that it would make more sense to focus on big rather than small. Or, to quote:

This posed a conundrum for the growth team. For example, an 80% drop-off at a given onboarding step would typically be alarming to any other growth team working on user activation. But at Patreon, a big drop-off could actually mean something good is happening — unqualified leads are getting weeded out.

If you are in a mind to read this as an explicit statement that small time creators on Patreon mainly receiving small (non-life-changing) donations are generally not worth the company’s time, and that it would make more monetary sense to boot them out in favor of the bigger creators — then this is a smoking gun. And in this particular case, at this particular time, the audience is very much of a mind to read it as just precisely that.

If we return to Patreon’s messaging, this is particularly damaging given the framing of the changes as something done with the best of intentions towards creators. Here, the very definition of “creator” comes into question, and it becomes reasonable to think that when Patreon says “creator”, they actually mean the big creators who bring in the big bucks, and not us small-timers who funnel 100% of our donations into paying our comparatively small bills. The stated intention to support creators is undermined by this blurring of the definition of who is and is not a creator.

At this point, it might be prudent with a reminder that the announced changes include a $0.35 fee on every single pledge made, including one-dollar pledges. Given that one-dollar donations are the bread, butter, meat and potatoes for small creators, making these 35% more expensive is a way of saying that these donations (and by extension creators) do not matter.

Whether Patreon intended to send this message or not is beside the point. It is the message that has been received, and it is the message that the audience is acting on.

There are countless more examples out there, each painting an ever bleaker picture of Patreon and its credibility. I chose these three examples in particular because they have a wide circulation; they are by no means the only ones making the rounds. I suspect new ones might have surfaced while I was busy writing these words.

I began this post by saying that it is fascinating and interesting to see this PR gaffe unfold in real time. It still is, in a morbid kind of way. It is also terrifying — I am not alone in having taken an immense comfort in knowing that there will be a steady stream of donation dollars to cover bills (present and future). While the donations do not cover all bills by any means, they do cover a couple of the big ones, and that is life-changing enough for me.

I can only hope these words will contribute to a change for the better, and not just serve as a contemporary document bearing witness to an unfortunate fait accompli.

[If you liked these words, consider making a donation on Ko-fi.]

Read the whole story
1 day ago
Sydney, Australia
1 day ago
I wrote a bit on it.
Share this story

US Navy not ready for major war as they forgot to train sailors to sail and operate ships

1 Share

The US surface fleet may not be adequately trained for high-intensity combat, four experienced former skippers and the former deputy secretary of defense warned a US Naval Institute conference here on Monday.

The US Navy has huge budgets but has mismanaged so that many sailors are trained for basic tasks

“Navigation and seamanship, these are the fundamental capabilities which every surface warfare officer should have, but I suspect if called to war, we’ll be required to do a lot more than safely navigate the Singapore strait,” where the destroyer USS McCain collided with an oil tanker, said retired Capt. Kevin Eyer, former skipper of the cruisers Shiloh, Chancellorsville, and Thomas Gates. “If our surface forces are unable to successfully execute these fundamental blocking and tackling tasks, how can it be possibly be expected that they are also able to do the much more complex warfighting tasks?”

So $140 billion per year naval budgets and they thought they could skip over training sailors to actually sail and operate their ships.

Sailing around showing the flag and not training to actually fight

This summer, the Navy lost 17 sailors and crippled two destroyers in peacetime accidents, a clear sign the fleet has been run ragged by day-to-day demands to show the flag around world. The fleet, argues former Deputy Defense Secretary Bob Work is demonstrating presence at the expense of training for high-intensity warfighting.

* Every Aegis (antiaircraft and antimissiles ship can go out there and engage airborne targets.

* But intercepting ballistic missiles (ICBMs) like North Korea’s is an even more demanding task requiring special training that many crews may not have.

* Special rocket scientists have to join ships for every SM-3 ballistic missile shot

* The Navy used to have a special Aegis training command to help sailors learn how to get the most out of the complex system but they cut back on that key training

Read the whole story
2 days ago
Sydney, Australia
Share this story

The Amazing Descartes

1 Comment and 3 Shares

And for my next trick, I will move my body around using only my soul! But my soul is immaterial and my body material, how I am doing it? Magic. That's how.
Read the whole story
6 days ago
Sydney, Australia
Share this story
1 public comment
6 days ago
I love these comics.
Los Angeles, CA

End of privacy rights in the UK public sector?


There has already been serious controversy about the “Henry VIII” powers in the Brexit Bill, which will enable ministers to rewrite laws at their discretion as we leave the EU. Now Theresa May’s government has sneaked a new “Framework for data processing in government” into the Lords committee stage of the new Data Protection Bill (see pages 99-101, which are pp 111–3 of the pdf). It will enable ministers to promulgate a Henry VIII privacy regulation with quite extraordinary properties.

It will cover all data held by any public body including the NHS (175(1)), be outside of the ICO’s jurisdiction (178(5)) and that of any tribunal (178(2)) including Judicial Review (175(4), 176(7)), wider human-rights law (178(2,3,4)), and international jurisdictions – although ministers are supposed to change it if they notice that it breaks any international treaty obligation (177(4)).

In fact it will be changeable on a whim by Ministers (175(4)), have no effective Parliamentary oversight (175(6)), and apply retroactively (178(3)). It will also provide an automatic statutory defence for any data processing in any Government decision taken to any tribunal/court 178(4)).

Ministers have had frequent fights in the past over personal data in the public sector, most frequently over medical records which they have sold, again and again, to drug companies and others in defiance not just of UK law, EU law and human-rights law, but of the express wishes of patents, articulated by opting out of data “sharing”. In fact, we have to thank MedConfidential for being the first to notice the latest data grab. Their briefing gives more details are sets out the amendments we need to press for in Parliament. This is not the only awful thing about the bill by any means; its section 164 will be terrible news for journalists. This is one of those times when you need to write to your MP. Please do it now!

Read the whole story
7 days ago
Sydney, Australia
Share this story

Sleep Walking into a War


The Great Cyber War: Part 1

Russian rioters handcuffed by police, Tallinn, Estonia 2007. Photo: STRINGER/AFP/Getty Images

We are 10 years into a war. A war most people don’t realise is happening, and which our governments are only just beginning to see. This is the Great Cyber War.

If this was a conventional war, by now your street would have been bombed, friends would have been killed at the Front, and food would be running short. But this is a new war, a hybrid war, an information war. So instead, you are confused. Whenever you think you are sure of something, someone else will either counter it with an alternative truth, or will disagree with you so strongly that you wonder if your take on reality is correct. Our houses are still standing but our perception lies in ruins.

The Great Cyber War started in 2007, and developed into two fronts. The Eastern Front began when Russia cyber-attacked Estonia, and the Western Front when Robert Mercer and Cambridge Analytica used social media to manipulate the Brexit Referendum, and then, alongside Russia, the Trump election. Along the way a smaller South Eastern Front opened up when kids in Macedonia realized they could cash in on the chaos. Other countries and dictators are also carving out their bits of the action, but the main war is Russia in the East, and Mercer in the West. Through highly targeted uses of Facebook and Twitter to disseminate fake stories, these forces have been working to undermine the democratic and open world order that rose from the ashes of the Second World War. In its place they seek to create a state of chaos, uninhibited capitalism, authoritarianism, and nationalism.

This is the story of the Great Cyber War. In writing about it I link to articles that go into much more detail about each part of the story. If you want to understand what’s happening right now, you have to read a lot and connect the dots. What should we make of Brexit and Trump when it’s now clear they were the prefered outcomes of the Russian intelligence services, and a coterie of secretive billionaires?

Traditionally, war is a noisy thing. Bombs and guns are loud, and tanks and soldiers are pretty obvious wherever they go. When Hitler started to invade Germany’s neighbours, whether they surrendered or fought, in the end either columns of soldiers and tanks rolled noisily along the roads, or planes dropped bombs and the armies made a lot of noise. Wars cause damage. There is no doubt about war; people die, cities get destroyed. It is an ugly, noisy, violent thing.

But people slept through the beginning of the Great Cyber War. It made no noise. No one was killed. Cities remained intact. Most people didn’t even realise the Great Cyber War was happening, even when it was happening to them. It began quietly in a small country in North East Europe a decade ago.


In 2007, Prime Minister Andrus Ansip of Estonia decided he would move the Bronze Soldier — a statue of the fallen Russian soldier that stood in the middle of his country’s capital, Tallinn. On the surface, this was a reasonable thing to do. Estonia had been occupied by Russia and Stalin transported Estonians to gulags in Siberia as part of a plan to suppress the Estonian language and culture.

The Medieval Old Town of Tallinn, Estonia. Photo: DEA/De Agostini/Getty Images

But Ansip was also a politician, and the Bronze Soldier was a touch-paper. Estonians, still angry about how they had been treated by Russia, wanted the statue out of the centre of their Capital. Why should they celebrate the heroes of a war that had left their country occupied and brutalised? But for Russians, the removal of the Bronze Soldier was yet another affront, a reminder that since the end of Communism they had been relegated to second-class citizens. Arguably the whole thing was handled badly, but unrest was also provoked in a way that would now be familiar to us, but then was a mystery.

Bronze Soldier Riots, Tallinn, Estonia 2007. Photo: STRINGER/AFP/Getty Images

On the night work began to move the statue, a small number of police encircled it as it was excavated to be moved to a cemetery on the edge of town. Protests turned to riots, and Tallinn fell into chaos for the night. Mysteriously, Russians across Tallinn were receiving text messages encouraging them to take to the streets. Back then, before we knew what we now know, it didn’t occur to anyone to wonder where these messages came from. At the same time, Nashi, a “youth NGO” sponsored by Putin’s Kremlin, formed violent protests outside the Estonian Embassy in Moscow, and the Estonian Ambassador was attacked as she left the building. The riots were accompanied by a serious and coordinated cyber attack against government websites, banks, and the media. Estonia considered it an act of war, and contacted NATO.

Bronze Soldier Riots, Tallinn, Estonia 2007. Photo: STRINGER/AFP/Getty Images

This was not the sort of military attack NATO was set up to repel; there were no tanks, no soldiers, no planes, no bombs. It was not grounds to trigger Article 5. Discussion developed about whether this constituted an attack, or act of war, and these led to the NATO cyber defence centre being established in Estonia, to monitor the situation and plan for future attacks.


In 2008, what appeared to be an anomaly with Estonia was repeated in Georgia. This time the cyber attacks came alongside actual military attacks. Whilst Russia denied sending their army to attack Georgia, in a move that was to set a precedent later in Ukraine, they argued that soldiers had been volunteering, and that Russia itself had only sent in Peace Keepers. This early example of hybrid warfare, mixing apparent mercenaries, cyber attacks, and military intervention legitimised under the guise of protection and peace keeping, destabilised Akbhazia and South Ossetia, and undermined Georgia, including preventing it from joining NATO. This left the area in what came to be called a “frozen conflict,” an engineered situation that prevents a region from joining organisations like NATO, thwarting progress.

One reason Estonia had been attacked back in 2007, three years after joining the EU, was because it was doing too well. Russia could not afford other satellite countries thinking that leaving Russia’s orbit, and joining Europe and NATO, would lead to stability, wealth, and democracy, as Estonia’s incredible success since 1991 had shown. Such a pattern might lead to countries still under Russia’s sway to face West and pull out of Russia’s orbit. Indeed, such successes just over the border might even lead to unrest within Russia itself, as street protests from 2011 were beginning to show.

Estonia and Belarus had both started their journey at the same time, after the fall of Communism. One had embraced capitalism, democracy, and modernity with full vigour, while the other embraced old-style Communist dictatorship with equal drive. One had joined NATO and the EU, firmly turning away from Russia, whilst the other had been shunned by the world, and remained closely tied to Russia. As Wired suggested, this provided an A-B test for the two routes for a post-Soviet country. The comparison between the two states paints a picture that contradicts Putin’s message to his people, and to the satellite states Russia still tried to control. Escaping Russia had led Estonia to take off and fly, whilst Belarus had stagnated. Putin needed countries like Estonia to be less successful, or even better to fail.

Putin’s grip on power relied on the Russian people believing in the social contract he’d propagated, that they give up their freedom in return for Putin making Russia stable and successful. His argument was that if Russia tried to become a Western style democracy it would collapse into chaos, as it had done after the fall of Communism.

Russia could not directly invade countries like Estonia, and risk taking on NATO; Russia has an economy the size of Italy, and until recently a dilapidated military, so direct confrontation wasn’t an option. So they devised a way to undermine countries, chipping away at them: Hybrid Warfare.


This experimentation with hybrid warfare, and that fear of Russian satellites turning to the West, in particular joining the EU and NATO, came to a head in 2014 when the Ukrainian people removed the Kremlin-backed leader, the deeply corrupt Viktor Yanukovych (who paid out $2 billion in bribes during his four years in office). This posed a risk both of Ukraine leaving the grips of Russia and becoming a success like its neighbour Poland, and of inspiring a similar rebellion against Putin himself. Hybrid warfare and the creation of “frozen conflicts” came of age with Ukraine.

The war in Ukraine was not just a military intervention, but the result of a coordinated information war. There was so much misinformation it is easy to forget this fact, which is the point of bamboozling people with contradictory stories and facts. Igor Strelkov, the agent sent into the Donetsk, the Eastern region of Ukraine, by Russia to foment dissent, later turned against Putin, and admitted his role. In interviews, he later said:

“Donetsk did not fight and was not preparing to…life was absolutely peaceful there, before we left Sloviansk, the Ukrainian side did not prepare to assault Donetsk, they estimated that it would return to them peacefully. No-one wanted to make war at first… but in April and May [of 2014] everything was building up, and the rebellion area was expanding. We were gradually taking control of populated areas of the Donetsk Republic.”

The campaign was directed by Putin’s strategist, Vladislav Surkov, the architect of Putin’s Russia, and of the strategy of sowing chaos and confusion that has developed into Russia’s main weapon. Alongside the military action, Ukraine also suffered then, and has continued to suffer, massive cyber attacks. Their banks, transport systems, energy grid, and more have been hit in the years after the war was started. One attack cut power to over 250,000 people, prompting Wired Magazine to suggest Ukraine had become “Russia’s test lab for cyber warfare.

Maidan Square, Kiev. Photo: NurPhoto/Corbis HIstorical/Getty Images

By now, the Great Cyber War was well established. The war had started in 2007 in Estonia, and now peaked in 2014 with a full-scale military operation in Ukraine, leading to the shooting down of a passenger airline by Russian army missiles, and the seizing of Crimea, an island off Ukraine’s Southern coast with a population similar to the European country of Latvia. It was the first time a country had taken land from another country by force since the Second World War.

Now at this point you may think I am wrong, or this didn’t happen, or it didn’t happen like that. You may question whether the MH17 was shot down by Russia, or may point out that there was a referendum in Crimea, and the people voted to leave Ukraine. You may also argue any number of conspiracy theories about the war in Ukraine.

You have been info-bombed.

Around the Ukraine war, vast amounts of misinformation, false propaganda, and conspiracies were churned out by Russia, as part of their Hybrid War. This was a system already set up years back, based on Soviet and KGB practices, and honed for the modern era in Estonia and Georgia.

A recent report published by NATO, and authored mainly by Estonian officials, looks back at what happened, and is happening in Ukraine. It concluded that:

“Russian information operations skilfully target a wide range of audiences with different beliefs and convictions. The anti-Ukrainian approach relies on a variety of stylistic forms and nuances. It can take on the form of sensationalism and blaming… or use a more restrained approach. In addition to the content of the messages, Russian technically ensures that certain messages reach specific audiences and others do not.”

They include the recommendation that we must stop taking Russia seriously as an exporter of “alternative opinions,” because it is a country where there is no democracy or freedom of speech, and therefore it is able to propagate lies unrestrained and unchallenged.

As an example, the BBC reported in November 2017 about the St Petersburg “Internet Research Agency,” the Kremlin-backed troll factory, being involved with fake videos being produced to fan the flames of the conflict in Eastern Ukraine:

“The Russian Liberation Movement videos were distributed through seven fake accounts on the Russian social network VK. The accounts used false names and photographs of random people as profile pictures. The accounts were opened just a few days before the material appeared and were largely reposting entertainment content in an effort to look ‘real.’

BBC Russian also found that a social media account sharing the material was computer generated and belonged to a botnet--a large controlled network of social media accounts.”

A mainstay of Information Warfare is that for every true story or fact, numerous other contradictory stories are pushed out. It is immaterial whether they are believable, or believed, vaguely true or total fantasy. The aim is to bury the truth in a messy pile of crap, so nobody can work out what is true at all. We shouldn’t be surprised, as this method is no secret; it’s called the Gerasimov Doctrine and was written up in 2013 by the Russian General, Valery Gerasimov, in a report. He explained:

“The very ‘rules of war’ have changed. The role of non-military means of achieving political and strategic goals has grown, and, in many cases, they have exceeded the power of force of weapons in their effectiveness. … All this is supplemented by military means of a concealed character.”

On Twitter, Putin opponent and famous chess player Garry Kasparov explained it thus:

“The point of modern propaganda isn’t only to misinform or push an agenda. It is to exhaust your critical thinking, to annihilate truth.”

We end up wasting time arguing about which story or fact is true or not, rather than addressing any core issues. That is precisely the intention of those bombarding us with conflicting facts. We are increasingly distracted by falling down rabbit holes of argument about what did or didn’t happen, all of which is designed to distract us from what is actually happening. Surkov and Gerasimov have told us this, but they are so good at it that they can explain to us in detail that they are about to con us, and still con us despite that.

And so Russia managed to get away with invading Ukraine. Putin later proudly admitted that he had in fact invaded Ukraine and seized Crimea, despite denying previously that his “little green men” and disruptive tourists were Russian soldiers. Lying to the world, then admitting to that lie, and still getting away with it is as much a reflection of his audacity as of our stupidity.

By now you’d have thought people would have noticed something was going on. If this Great Cyber War had been a conventional war, by now we’d all have been mobilised. Three countries had been attacked, a civilian airliner had been shot down, part of Ukraine had been taken by force. The thing that connected all this was Russian information and cyber warfare. We were all lost in arguing about who shot the plane down, whether Crimea chose to join Russia or was invaded, whether the CIA had been behind the protests in Kiev, whether the Russians in Eastern Ukraine were soldiers or tourists. And all the while we were distracted by the mad propaganda stories spreading over social media. We were distracted by the white noise from the simple reality that Russia had invaded Ukraine, seized Crimea by force, and shot down a passenger plane.

Some voices were trying to shout through the noise, but they were not the headlines, and anything they posted online was attacked by pro-Russian comments, which we now know were coming either from Russian “social media factories,” or from Russian bots, which automatically post again and again to swamp the other real comments.

Meanwhile, Russia stepped into a new arena, backing Syrian dictator Bashar al Assad. A popular rebellion against him threatened to remove a Russian ally from the Middle East. The ensuing war also offered Russia a chance to show its military might in full public view, and position itself as a relevant player in world affairs. It also triggered an event that was to rock Europe.

The Russian-backed war in Syria led to a humanitarian crisis that slammed into Europe, unsettling it to its very core. The flood of refugees challenged the liberal order in Europe, and cornered some of its leading politicians.

Through all this chaos we did not notice the Great Cyber War, but all of this was the Great Cyber War. These were all connected events, battles of the same war, parts of the same plan.

Read the whole story
5 days ago
9 days ago
Sydney, Australia
Share this story

Understanding Bias in Peer Review

1 Share

In the 1600’s, a series of practices came into being known collectively as the “scientific method.” These practices encoded verifiable experimentation as a path to establishing scientific fact. Scientific literature arose as a mechanism to validate and disseminate findings, and standards of scientific peer review developed as a means to control the quality of entrants into this literature. Over the course of development of peer review, one key structural question remains unresolved to the current day: should the reviewers of a piece of scientific work be made aware of the identify of the authors? Those in favor argue that such additional knowledge may allow the reviewer to set the work in perspective and evaluate it more completely. Those opposed argue instead that the reviewer may form an opinion based on past performance rather than the merit of the work at hand.

Existing academic literature on this subject describes specific forms of bias that may arise when reviewers are aware of the authors. In 1968, Merton proposed the Matthew effect, whereby credit goes to the best established researchers. More recently, Knobloch-Westerwick et al. proposed a Matilda effect, whereby papers from male-first authors were considered to have greater scientific merit that those from female-first authors. But with the exception of one classical study performed by Rebecca Blank in 1991 at the American Economic Review, there have been few controlled experimental studies of such effects on reviews of academic papers.

Last year we had the opportunity to explore this question experimentally, resulting in “Reviewer bias in single- versus double-blind peer review,” a paper that just appeared in the Proceedings of the National Academy of Sciences. Working with Professor Min Zhang of Tsinghua University, we performed an experiment during the peer review process of the 10th ACM Web Search and Data Mining Conference (WSDM 2017) to compare the behavior of reviewers under single-blind and double-blind review. Our experiment ran as follows:
  1. We invited a number of experts to join the conference Program Committee (PC).
  2. We randomly split these PC members into a single-blind cadre and a double-blind cadre.
  3. We asked all PC members to “bid” for papers they were qualified to review, but only the single-blind cadre had access to the names and institutions of the paper authors.
  4. Based on the resulting bids, we then allocated two single-blind and two double-blind PC members to each paper.
  5. Each PC member read his or her assigned papers and entered reviews, again with only single-blind PC members able to see the authors and institutions.
At this point, we closed our experiment and performed the remainder of the conference reviewing process under the single-blind model. As a result, we were able to assess the difference in bidding and reviewing behavior of single-blind and double-blind PC members on the same papers. We discovered a number of surprises.

Our first finding shows that compared to their double-blind counterparts, single-blind PC members tend to enter higher scores for papers from top institutions (the finding holds for both universities and companies) and for papers written by well-known authors. This suggests that a paper authored by an up-and-coming researcher might be reviewed more negatively (by a single-blind PC member) than exactly the same paper written by an established star of the field.

Digging a little deeper, we show some additional findings related to the “bidding process,” in which PC members indicate which papers they would like to review. We found that single-blind PC members (a) bid for about 22% fewer papers than their double-blind counterparts, and (b) bid preferentially for papers from top schools and companies. Finding (a) is especially intriguing; with no author information reviewers have less information, arguably making the job of weighing the merit of each paper more difficult. Yet, the double-blind reviewers bid for more work, not less, than their single-blind counterparts. This suggests that double-blind reviewers become more engaged in the review process. Finding (b) is less surprising, but nonetheless enlightening: In the presence of author names and institution, this information is incorporated into the reviewers’ bids. All else being equal, the odds that single-blind reviewers bid on papers from top institutions is about 15 percent above parity.

We also studied whether the actual or perceived gender of authors influenced the behavior of single-blind versus double-blind reviewers. Here the results are a little more nuanced. Compared to double-blind reviewers, we saw about a 22% decrease in the odds that a single-blind reviewer would give a female-authored paper a favorable review, but due to the smaller count of female-authored papers this result was not statistically significant. In an extended version of our paper, we consider our study as well as a range of other studies in the literature and perform a “meta-analysis” of all these results. From this larger pool of observations, the combined results do show a significant finding for the gender effect.

To conclude, we see that the practice of double-blind reviewing yields a denser landscape of bids, which may result in a better allocation of papers to qualified reviewers. We also see that reviewers who see author and institution information tend to bid more for papers from top institutions, and are more likely to vote to accept papers from top institutions or famous authors than their double-blind counterparts. This offers some evidence to suggest that a particular piece of work might be accepted under single-blind review if the authors are famous or come from top institutions, but rejected otherwise. Of course, the situation remains complex: double-blind review imposes an administrative burden on conference organizers, reduces the opportunity to detect several varieties of conflict of interest, and may in some cases be difficult to implement due to the existence of pre-prints or long-running research agendas that are well-known to experts in the field. Nonetheless, we recommend that journal editors and conference chairs carefully consider the merits of double-blind review.

Please take a look at our full paper for more details of our study.
Read the whole story
9 days ago
Sydney, Australia
Share this story
Next Page of Stories