Study suggests more than half off negative tweets about The Last Jedi may have been politically motivated.

[The Romulan Republic]

Based on an anyalisis of over 900 tweets directed at Rian Johnson over the course of about seven months. With a much smaller but significant number likely troll, bot, or sock puppet accounts, including sixteen trolls identified as likely originating in Russia, and eleven suspected bots:

https://www.theverge.com/2018/10/2/1792 ... bots-study

When it arrived in theaters last year, writer-director Rian Johnson’s Star Wars: The Last Jedi was greeted with an immediate backlash from a specific corner of its audience. As Vox’s Todd VanDerWerff pointed out in December, the criticism seemed to come from a few different angles: some felt the film was too progressive, that it was too jokey, that it was not interested in the elaborate universe of fan theories that has accreted since the original trilogy’s release, or that the characters’ journeys weren’t exactly to their liking.

None of these lines of attack are new or really surprising, especially for a beloved, 40-year-old series that many feel has defined their experience with science fiction and fan culture in general. What’s new about this round of critique is how political (and politically useful) it became — at least for those who want to use popular culture to influence voters.

A new paper from Morten Bay, a research fellow at USC’s Annenberg School for Communication and Journalism, finds that the polarized fan discourse surrounding The Last Jedi also happened to be the site of an attempted Russian political influence campaign. (The paper hasn’t yet been peer-reviewed, and may change before it’s published — so it’s a little less reliable than research that comes out in journals, where other researchers get to kick the tires.)

Bay examined a corpus of messages tweeted at Rian Johnson between December 13th, 2017, and July 20th, 2018 — a total of 967 tweets — ran a sentiment analysis (using a tweet’s language to characterize it as positive, negative, or neutral) on them, segmented the results by account, and then analyzed the Twitter accounts themselves. “Overall,” Bay concludes, “50.9% of those tweeting negatively was likely politically motivated or not even human.” (Bay also found that most fans aren’t so dissatisfied with The Last Jedi that they’re going to boycott any new Star Wars films.)

Bay notes in his abstract that the negative tweets were probably sent to get media coverage of the fandom conflict, which, in his words, was meant to “further propagat[e] a narrative of widespread discord and dysfunction in American society.” Persuading voters of that, he notes, is a goal of both the alt-right and the Russian Federation.

“Russian trolls weaponize Star Wars criticism as an instrument of information warfare with the purpose of pushing for political change, while it is weaponized by right-wing fans to forward a conservative agenda and for some it is a pushback against what they perceive as a feminist/social justice onslaught,” Bay writes.

Of those 967 tweets collected over the dates above, Bay found that 206 “expressed a negative sentiment toward the film and its director, which is 21.9% or a little more than one in five fans,” Bay wrote. Forty-four of those accounts were identified as bots, sock puppets, or trolls, and 61 of those 206 accounts showed a “clear political agend[a]” — a definition that includes real humans who tweet heavily about politics. Of the 44 bot/sock puppet/troll accounts, 33 were identified as trolls or sock puppets. Bay identified only 16 of those 33 as appearing to be Russian trolls. The trolls and bots are actually a minority of the accounts tweeting negative opinions about The Last Jedi.

These days, it’s becoming harder and harder to tell whether the fans angrily tweeting their grievances are even real, let alone politically motivated. You can’t always judge a movie by its Rotten Tomatoes score, and you might not be able to judge a fandom by its tweets, either.

Update, October 2, 6:40 p.m. Eastern Time: This story has been updated to include more details about the study’s methodology. The Verge has reached out to the study’s author for further comment; we will update the story as we hear more.

Article's from last fall, but this tallies with what I expected, and have read elsewhere. Johnson himself tweeted in response to it:

A bit of Morten's research came out awhile ago and made some headlines - here's his full paper. Looking forward to reading it, but what the top-line describes is consistent with my experience online.

[NecronLord]

Honestly, that kinda fits. Like, I think that Star Wars 8 was a bad movie, for sure, a disappointment and a fundamental misunderstanding of the source material, but I'm not going to message outrage at the creator. I just don't care that much.

In no way does it mean there's a silent majority who think the movie is good, though.

Nor is social media a replacement for test audiences, professional critics, and paid opinion polls for the filmmakers wanting to improve their product.

[Esquire]

Gosh, who could have predicted that Twitter might not present a representative sample of opinion?

[The Romulan Republic]

Honestly, that kinda fits. Like, I think that Star Wars 8 was a bad movie, for sure, a disappointment and a fundamental misunderstanding of the source material, but I'm not going to message outrage at the creator. I just don't care that much.

Now that's a fair question: do tweets at Johnson specifically represent a fair sample of general opinion on the film?

I'd say a broader study is warranted, though these results are certainly suggestive.

In no way does it mean there's a silent majority who think the movie is good, though.

Nor is social media a replacement for test audiences, professional critics, and paid opinion polls for the filmmakers wanting to improve their product.

Social media is not remotely reliable for gauging public opinion, no. Unfortunately, it seems to be wielding increasing influence on the direction of major media franchises. That bodes very badly for the future of the creative arts.

Social media is just too God damn easy for a vocal and unscrupulous minority to manipulate to make themselves appear bigger than they are.

[The Romulan Republic]

There's also the problem that the trolls are just flat-out more engaged, more motivated. I've noticed a frequent trend when I look at the Facebook posts of franchises like Doctor Who, Star Wars, etc: there are usually far more "likes", "loves", etc. than negative clicks on a post- but the actual comments, particularly the top comments, are almost invariably vile crap about how its ruined because of a "feminist agenda" or somesuch. They're motivated to post, the people who like it or don't feel strongly aren't, which means the Alt. Reich dominates every conversation, and appears to represent an overwhelming majority when it really doesn't.

[tezunegari]

I remember having read about this study, but IIRC it has been ridiculed for its methodology.

[The Romulan Republic]

One could argue that its too narrow in its focus (on just tweets directed at Johnson), I suppose, but I'm not sure what else is wrong with its methodology beyond that. Do you have a source for this "ridicule"?

[tezunegari]

Do you have a source for this "ridicule"?

Sadly none at hand, it's been 6+ months since reading about it.

[Solauren]

I have said it before, and I will say it again.

What most people think is the majority, is really the obnoxious and vocal minority, screaming because they're not the center of attention. Much like an spoiled and upset 2 year old is the loudest sound in a store.

[Formless]

I could swear we already had a thread on this back on October when it was published, but even if not, then its worth reposting this: the author of the study says the media is completely misrepresenting his work, and your thread title is the misrepresentation (hence, I'm asking right now for it to be retitled as its utterly false). Its not half, its nowhere near half. Its 1/10'th once you separate out probable bots and sockpuppets, and the study only looked at posts directed at Rian Johnson's personal twitter feed. The writer was not trying to study all Twitter posts about the film, and he very much would like people to stop acting like that was his intention. The fact that its consistent with Rian Johnson's personal experience is true literally by design. The problem is that no one in the media except CNet actually read the damn study.

[Formless]

ghetto edit: Let me add an addendum that the one in ten figure is one in ten negative tweets being Russian or bots, while half were politically motivated in general (so think homegrown American conservatives). Being politically motivated is not in itself a sign of trolling, as most people cannot separate out their politics from their enjoyment of films; what is of interest is when there is a deliberate smear campaign, which the study found no evidence of. One in ten was about what the study's author expects to be the normal ratio of Russians in any Twitter sample, and not at all unusual. Still, your article repeats both the specific misrepresentation that more than half were Russian influencers, and the general misrepresentation that the study was representative of Twitter as a whole platform. This is factually untrue, and the reason I've requested that the thread be retitled.

[The Romulan Republic]

I could swear we already had a thread on this back on October when it was published, but even if not, then its worth reposting this: the author of the study says the media is completely misrepresenting his work, and your thread title is the misrepresentation (hence, I'm asking right now for it to be retitled as its utterly false). Its not half, its nowhere near half. Its 1/10'th once you separate out probable bots and sockpuppets, and the study only looked at posts directed at Rian Johnson's personal twitter feed. The writer was not trying to study all Twitter posts about the film, and he very much would like people to stop acting like that was his intention. The fact that its consistent with Rian Johnson's personal experience is true literally by design. The problem is that no one in the media except CNet actually read the damn study.

The article title is misleading, which I why I didn't quote it.

The thread title was based on the actual numbers quoted in the article. So unless you're saying the article outright lied about the numbers (in which case I will ask you to site sources to back that up), it seems to me like you're just whining that I'm lying because you can stand the idea that an overwhelming majority doesn't agree that TLJ sucked, or that that opinion might often be driven by ulterior political motives.

And speaking of dishonesty... Your argument is "its not half once you separate out the probable bots and sock-puppets"? Well yeah, no shit. If you don't count the most obvious examples of criticism being politically motivated, then obviously most of the criticism isn't politically motivated.

[Formless]

The article title is misleading, which I why I didn't quote it.

The thread title was based on the actual numbers quoted in the article. So unless you're saying the article outright lied about the numbers

The article you posted misrepresents the numbers and what they actually mean. NO, half of the negative tweets were not Russian, but the article you posted nevertheless opens its discussion with the following lie:

"A new paper from Morten Bay, a research fellow at USC’s Annenberg School for Communication and Journalism, finds that the polarized fan discourse surrounding The Last Jedi also happened to be the site of an attempted Russian political influence campaign."

There was no such Russian campaign, the numbers don't show any evidence of such a campaign, and the study's author has explicitly told CNet that this is a misrepresentation of his actual findings. One in ten were Russian. That's statistically low enough as to be irrelevant. But that's the media's narrative, because the media likes to blame the Russians for everything, even hate for a friggen Star Wars movie apparently. As if the Russians give that much of a crap. You're a gullible dumbshit if you think otherwise.

The second misrepresentation, which you repeat in the thread title, is that this study is meant to be representative of all Twitter discourse about The Last Jedi. Half of all posts aimed at Rian Jhonson that were negative were "politically motivated". That's not half of all negative posts about The Last Jedi. If it were, then the author has committed a grave sampling error by choosing to only look at posts directed to Rian Jhonson's twitter feed. People who choose to direct their posts at him, hell people who are subscribed to his feed at all, clearly self-select, leading to a population that almost certainly does not reflect the motivations of the average Twitter user. Fortunately, that isn't what the study was supposed to be about, and he acknowledges this in the limitations section of the study (as per good practice).

Furthermore, his method of determining whether a post was "politically motivated" was to "manually" sort the posts, then after filtering them that way he searched for terms like "SJW" and example's of Trump's name coming up. Now, here's the thing: any time a researcher "manually" determines whether a post is one thing or another, there is the possibility of bias on the part of the person doing the sorting. This is well understood by researchers in fields like psychology and sociology, and we know its a problem because many studies have had issues where another judge sorts the same data manually and finds that they can't agree with the first judge. Manual sorting is kinda bullshit that way. We call it human bias, and it effects all studies that use human judges. To improve such a methodology, you have to at the very least have multiple people judging the same data set independently and compare results. The more judges agree on any given post, the less likely human bias is effecting the analysis of the data. But he didn't apparently do that. He was the sole judge.

Searching the data set for terms that correlate to political motives is a better method. "SJW" has a known history of being a conservative dogwhistle. But you have to prove the correlations between the terms and political motivation. Rigor demands a higher standard than just common experience among liberal leaning readers. Furthermore, there are contexts where Trump's name might come up where it doesn't necessarily imply a political motive for why that person posted. Its hard to gauge a person's motives in 244 characters or less. Which, lets be frank, makes it hard to judge anything except whether a post was even made by a human or not.

[The Romulan Republic]

The article title is misleading, which I why I didn't quote it.

The thread title was based on the actual numbers quoted in the article. So unless you're saying the article outright lied about the numbers

The article you posted misrepresents the numbers and what they actually mean. NO, half of the negative tweets were not Russian, but the article you posted nevertheless opens its discussion with the following lie:

"A new paper from Morten Bay, a research fellow at USC’s Annenberg School for Communication and Journalism, finds that the polarized fan discourse surrounding The Last Jedi also happened to be the site of an attempted Russian political influence campaign."

I mean, Russian troll accounts could be just individual trolls. They could be more orchestrated. That claim certainly appears to be taking a leap beyond what the study proves, and its sloppy journalism- but it is not a claim that I made in either my title or my posts. So if this is the basis for implying that I am a liar and demanding that my title be edited, then you can fuck right off. I merely cited the actual numbers in the study. Don't like them? Facts don't care what you like.

There was no such Russian campaign, the numbers don't show any evidence of such a campaign, and the study's author has explicitly told CNet that this is a misrepresentation of his actual findings. One in ten were Russian. That's statistically low enough as to be irrelevant. But that's the media's narrative, because the media likes to blame the Russians for everything, even hate for a friggen Star Wars movie apparently. As if the Russians give that much of a crap. You're a gullible dumbshit if you think otherwise.

The Russian government doesn't care about Star Wars. But it does care about stirring up divisions in Western nations. And will use any tool to that end.

And blaming "the media" (as if the media is a homogenous monolith with a single view), and whining about how their out to get the poor innocent Russians, is right out of the Trumpist propaganda playbook.

The second misrepresentation, which you repeat in the thread title, is that this study is meant to be representative of all Twitter discourse about The Last Jedi. Half of all posts aimed at Rian Jhonson that were negative were "politically motivated". That's not half of all negative posts about The Last Jedi. If it were, then the author has committed a grave sampling error by choosing to only look at posts directed to Rian Jhonson's twitter feed. People who choose to direct their posts at him, hell people who are subscribed to his feed at all, clearly self-select, leading to a population that almost certainly does not reflect the motivations of the average Twitter user. Fortunately, that isn't what the study was supposed to be about, and he acknowledges this in the limitations section of the study (as per good practice).

The study may not have been representative of all Twitter discourse, but it clearly was intended to examine whether there was a political agenda against the film, because otherwise what's the fucking point? I acknowledged further study is needed, but these results are certainly suggestive, and I stand by that.

Furthermore, his method of determining whether a post was "politically motivated" was to "manually" sort the posts, then after filtering them that way he searched for terms like "SJW" and example's of Trump's name coming up. Now, here's the thing: any time a researcher "manually" determines whether a post is one thing or another, there is the possibility of bias on the part of the person doing the sorting. This is well understood by researchers in fields like psychology and sociology, and we know its a problem because many studies have had issues where another judge sorts the same data manually and finds that they can't agree with the first judge. Manual sorting is kinda bullshit that way. We call it human bias, and it effects all studies that use human judges. To improve such a methodology, you have to at the very least have multiple people judging the same data set independently and compare results. The more judges agree on any given post, the less likely human bias is effecting the analysis of the data. But he didn't apparently do that. He was the sole judge.

Source for that?

Also, I just love how you keep putting "manually" in sarcastic/scare-quotes while implying that the fact that it was reviewed by a person rather than entirely automatic (because automated allgorythms are so reliable) means it can be dismissed out of hand and I'm a liar for posting it.

Searching the data set for terms that correlate to political motives is a better method. "SJW" has a known history of being a conservative dogwhistle. But you have to prove the correlations between the terms and political motivation. Rigor demands a higher standard than just common experience among liberal leaning readers. Furthermore, there are contexts where Trump's name might come up where it doesn't necessarily imply a political motive for why that person posted. Its hard to gauge a person's motives in 244 characters or less. Which, lets be frank, makes it hard to judge anything except whether a post was even made by a human or not.

There are many things that would be very easy to judge on less than 245 characters. If someone posts "God hates F*gs", for example, it doesn't take a great deal of analysis to figure out their motives. In any case, you have to either back up your insinuation that the study's standard was "common experience among liberal leaning readers", and the implication that the author (much like reality) has a liberal bias. Or else retract it.

[NecronLord]

The thread title will not be altered by moderator action. If you think a thread is incorrect, engage with it in debate, don't try and ask for moderators to intervene in the debate on your behalf.

[Formless]

I mean, Russian troll accounts could be just individual trolls. They could be more orchestrated. That claim certainly appears to be taking a leap beyond what the study proves, and its sloppy journalism-

They are claiming that the study says something that the study's author has specifically denied, that 50% of the negative tweets were Russian. Not true in the slightest. I've read the thing myself.

That's not sloppy journalism, that's dishonesty, pure and simple. Its misleading their audience into thinking they read it, and they obviously didn't. And few if any news source retracted the inaccurate figures as well. CNet is the only source I have found that has definitely read it.

but it is not a claim that I made in either my title or my posts. So if this is the basis for implying that I am a liar and demanding that my title be edited, then you can fuck right off.

The fact that you can't see the naked sampling bias in this study says you are either gullible enough to take news sources at face value rather than reading the primary source, or you just don't know what good and bad study design looks like. The latter is fine, given most people don't have that knowledge, but when someone tells you "this is shitty study design", listen to what they have to say.

Note that at no point have I called you a liar. I've called the media dishonest, but despite your hyperventilating, I haven't said anything about your honesty.

The Russian government doesn't care about Star Wars. But it does care about stirring up divisions in Western nations. And will use any tool to that end.

And blaming "the media" (as if the media is a homogenous monolith with a single view), and whining about how their out to get the poor innocent Russians, is right out of the Trumpist propaganda playbook.

Please provide evidence for the claim that the Russian Government cares about stirring up divisions in Western Nations. And don't use this very study as evidence; its a shit study, as I've already explained, and will further elaborate on since you need it explained.

And please, take your straw man and burn it. The media is not a monolith, but it is homogeneous in many cases due to many "journalists" being too lazy to read primary sources. If you don't read a primary source, what is left for you to read? Other journalists. This clearly happened in this case because the same exact claims were repeated by so many news outlets, and anyone who has read the study knows the numbers aren't what they say they are. One does not have to be a Trump supporting conservative to have legitimate criticisms of journalism as its practiced in the Internet age.

Source for that?

Also, I just love how you keep putting "manually" in sarcastic/scare-quotes while implying that the fact that it was reviewed by a person rather than entirely automatic (because automated allgorythms are so reliable) means it can be dismissed out of hand and I'm a liar for posting it.

Source for what? The part where he said in the actual study that he used manual sorting, or the part where manual sorting is known to be prone to human bias? By the way, those aren't scare quotes, since manual sorting is a technical term not everyone will be familiar with I thought it best to leave it in quotes. Its obvious that you, for instance, don't really know what it means either.

If you want me to source the claim that manual sorting, that is, having a trained rater go through the data by hand and rate whether the archived twitter post is positive, negative, or neutral; or politically motivated or not politically motivated; is an unreliable methodology especially when there is only one rater sorting the posts, then try "Research in Psychology: Methods and Design, 7'th edition":

Experimenter Bias

As well as illustrating falsification and parsimony, the Clever Hans case (Box 3.3 in Chapter 3) is often used to show the effects of experimenter bias on the outcome of some study. [...] Similarly, experimenters testing hypotheses sometimes may inadvertently do something that leads participants to behave in ways that confirm the hypothesis. Although the stereotype of the scientist is that of an objective, disspassionate, even mechanical person, the truth is that researchers can become emotionally involved in their research. Its not difficult to see how a desire to confirm a strongly held hypothesis might lead an unwary but emotionally involved experimenter to behave (without awareness) in such a way as to influence the outcome of the study.

I've omitted the details about Clever Hans, because you can look it up for yourself online. This goes on to talk about how an experimenter's behavior influences the participants' behavior in an experiment, so it might not seem immediately relevant. However, chapter 12 clarifies that it is relevant even to observational studies such as this:

Observer Bias

A second problem for those doing observational research is experimenter bias. In Chapter 6, you learned that when experimenters expect certain outcomes to occur, the might act in ways that could bring about such a result. In observational research, observer bias means having preconceived ideas about what will be observed and having those ideas color one's observations. For example, consider what might happen if someone is studying aggression in preschoolers and believes from the outset that little boys will be more aggressive than little girls. For that observer, the exact same ambiguous behavior could be scored as aggressive if a boy did it but not aggressive if performed by a girl. [...] Bias can also occur because observational studies may collect huge amounts of information. Deciding which observations to report involves reducing this information to a manageable size, and the choices about what to select as relevant and what to omit can be effected by preconceived beliefs.

Biasing effects can be reduced by using good operational definitions and by training observers to identify precisely defined target behaviors. When actually making the observations, behavior checklists are normally used. These are lists of predefined behaviors that observers are trained to spot. [...]

In addition to defining behaviors with precision, anther way to control for observer bias is to have several observers present and see if their records match. This is interobserver reliability, a concept you encountered in Chapter 7. This form of reliability is usually measured in terms of percentage of times that observers agree. Of course, both observers could be biased in exactly the same way, but a combination of checklists, observer training, and agreement among several observers generally controls bias.

The omitted sections I deemed irrelevant (information on animal studies, detail on checklists you can figure out yourself, and videotaping observations). By now, you should get the point: manual sorting, i.e. having a (hopefully) trained observer sort through open ended responses in a survey or archived posts in "natural" online environment like Twitter and make judgements about the behaviors they see in the writing, is just as prone to experimenter and observer bias as any other study, and the way you correct for it is exactly the same as in any other study. Have multiple observers sort through the data with behavior checklists and good operational definitions for (in this case) political motivation. The checklist should not be applied after someone has given it a pass with no checklist, but that's essentially what he did here. Moreover, this study appears to have only one researcher doing all of the work as a PhD student, so by definition it basically has no inter-rater reliability. Hence why I do not consider it a good study.

And plenty of professional researchers submit bad studies, so this is just something that you have to be wary of any time you read one. Its just a fact of life in the social sciences.

Satisfied, Rom?

There are many things that would be very easy to judge on less than 245 characters. If someone posts "God hates F*gs", for example, it doesn't take a great deal of analysis to figure out their motives. In any case, you have to either back up your insinuation that the study's standard was "common experience among liberal leaning readers", and the implication that the author (much like reality) has a liberal bias. Or else retract it.

Not my claim, Rom. Read again. I didn't say that the writer of the study has a liberal lean, I said the two of us lean liberal, and biases us to believing his methods correct. Except, I have been trained not to rely on my intuitions, and instead to look at whether the study has anti-bias measures built in. It doesn't. That's a problem. And whats more, the study makes untenable claims of being representative. That's also a problem.

Though, if you want to know whether the writer of the study leans liberal himself, read it yourself. His central argument that Conservatives in this country are picking up on Russian propaganda tactics and the negative tone he takes towards this development is quite telling, IMO.