3. Methodology

This page is part of a graduate thesis. Learn more.

This study examines the quality of argument in journalism using standards from informal logic and critical thinking. The ideal form of such an examination would set out a definitive line past which a journalistic argument could be declared “good” or “bad,” “acceptable” or “unacceptable.”

This study does not attempt to provide a line. Whether to accept a journalistic argument is perhaps best left to individuals using their assumptions, evidence requirements, and so on. Instead, this study asks research questions that provide the raw material, so to speak, for choosing whether to accept an argument:

  1. What are the issues and conclusions offered in a sample of Pulitzer-Prize winning stories?

  2. What reasons and evidence to support those conclusions are offered in the stories?

  3. What important ambiguities do the stories contain? Important ambiguities here means undefined terms in stories that (a) could be reasonably defined in several ways and (b) would not support a story’s conclusion under at least one of those alternate, reasonable definitions.

  4. What important descriptive assumptions do the stories contain, especially assumptions not addressed by previous research on, for example, journalism’s “bias” toward moderate politics or the economic status quo?

  5. What fallacies are present in the stories?

  6. Given the reasons and evidence for the conclusions journalists offer, as well as the important ambiguities, descriptive assumptions, and fallacies in their stories, how do Pulitzer Prize-winning stories fulfill or not fulfill journalists’ stated goals of telling something about the way the world is, was, or will be, so that citizens can make decisions in a democracy. A shorter version of this question would be, “how well do the reasons and evidence support the conclusion?”

Critical case sampling, a subset of purposive sampling, will be used to select the stories under review. Critical case samples are used to examine unusually important instances of a class. The conclusions drawn from the critical cases can lead to “logical generalizations” that “‘if it happens here, it will happen anywhere,’ or … ‘if that group is having problems, then we can be sure all of the groups are having problems’” (Patton, 1990, p. 174).

This study will analyze the 2011 Pulitzer Prize winners in the Public Service, Investigative Reporting, and Explanatory Reporting categories, or a total of 28 stories. To minimize the risk of analyzing a different version of a story from the one the Pulitzer committee considered, the analysis of each story is based on the versions available on the Pulitzer Prize website, except where otherwise noted.[1] Each winning series of stories is introduced more fully and its inclusion in this study defended at the end of this section.

The Pulitzer Prizes denote what the profession considers unusually powerful journalism (see Bates, 1991). A Pulitzer Prize should indicate that the reporting will present a strong argument: clear statements of conclusions and key terms, good reasoning and evidence, and few harmful fallacies.

Purposive samples have been used to study Pulitzer Prize-winning journalism at small newspapers (Hatcher, 2007); Pulitzer-winning reporting itself has been previously tested for differences in source diversity and research techniques between it and other investigative work (Hansen, 1990). Critical case samples have been used to study relationships between news media and democracy through editorials (Lee & Lin, 2006), letters to the editor (Nielsen, 2010), and media impact in new democracies (Tworzecki & Semetko, 2010). Critical case studies have also focused on coverage of specific topics, including labor movements (Martin, 2003) and the environment (Lester, 2010; Lester & Hutchins, 2009).

A critical case sample of Pulitzer-winning reporting, then, provides a window into whether audiences could expect everyday journalism to provide quality arguments: If the elite reporting is having problems, then it seems more likely that the everyday reporting has problems, too. And the more certain there are to be problems in everyday reporting, the stronger the argument is for doing something else than reading it.

Both argumentation and informal logic still maintain links to their parent field of rhetoric (van Eemeren et al., 1996, p. 29). Accordingly, this study will draw on rhetorical criticism to study the news articles in question. Rhetorical criticism is appropriate for this study because it shares a concern for human-constructed messages and the way in which those messages can influence the views of the audience (Foss, 2004, p. 4). It also is comfortable with drawing even conclusions — tentative ones — from small samples (Foss, 2004, p. 8). Finally, there does not appear to be any other methodology dominant within either argumentation or informal logic; indeed, few articles from those fields contain methodology sections at all.

Validity

There are at least three ways to respond to the idea of applying theories of argumentation and informal logic to journalism.

One response is that it is improper to consider journalists as providing “arguments.” Under this response, journalists provide perhaps information or facts, or they relay opinions of others, but they do not offer a coherent “argument,” nor do they mean to. The above sections have attempted to answer this response. They described the ways journalists do actually put forward descriptive arguments, and they described the consonance between journalists’ intent in reporting and the application of argumentation and informal logic.

A second response is suggested by Postman’s (1988) eloquent critique of broadcast journalism.

Postman argued that the nature of television makes its content immune to analysis as propositions, especially regarding the political news that sustains journalism’s theory of democracy. “Exposition, explanation, and argument — the instruments of rational discourse — are less and less used as a means of expressing political ideas,” Postman said. “The use of extended and complex language is being rapidly replaced by the gestures, images, and formats of the arts of show business” (p. 14).

Postman’s critique would call into question the validity of this thesis if it could be shown that non-broadcast media now, too, do not express political issues through “exposition, explanation, and argument.”

Such a demonstration does not seem implausible. As noted, for example, Barnhurst and Mutz (1997) brought forward evidence of a long-term shift in the nature of “news” in print journalism. More recently, Nick Denton, the leader of the Gawker blog network, told Fallows (2011), “Liberals love to talk about the erosion of logic and the scientific method … But what if the answer to a false narrative isn’t fact? … Maybe the answer to a flawed narrative is a different narrative. You change the story” (p. 46). The danger of Postman’s critique for the present research is that journalism cannot fairly be judged using tools for arguments if journalism has ceased to be argumentative.

The third response to the validity of this thesis was suggested by a note from Scriven (1976):

Reasoning is the best guide we have to the truth. That doesn’t mean you should never listen to your “inner voice,” your instincts, or to authorities; it just means that you ought to use reason to decide when to listen to them. … Reason is always the ultimate court of appeal — which is not to say that explicit direct reasoning is always the best basis for judgment. (pp. 3–4)

Just as this thesis argues that journalists need a point at which they would admit that journalism is no longer worth consuming, critical thinkers must know when to concede that their skills are not usefully applied. Journalism might be one such context.

One reason to think journalism and reason are not synonymous was stated just above: Journalism of all sorts might have ceased to offer anything representing propositions and arguments. Indeed, as noted, May (1988) and Marlin (1984) have argued that journalism attempts to persuade by “suggestion,” not reason.

A second reason to doubt reason’s applicability to journalism is if journalists follow what Richards (1980) called the “received theory of reasoning.” People who hold the “received theory” tend to think that their views are not “positions,” which can be backed by reason, but that instead they are indisputable “opinions.” These proponents, Richards says, inevitably conclude that the only means of persuasion is through empathy. Journalists who hold the received theory, then, wouldn’t think to attempt to convince someone using argument as conceived by this paper.

Finally, a third reason to doubt the connection between reason and journalism is that journalists might be considered experts in their field, and that to accept what they say without much critical thought is merely a reasonable appeal to authority.

To the extent that any of these potential descriptions of journalism check out, the case for using tools of argument analysis against them weakens. But, interestingly, the case for almost any critique of journalism based in logic would also have to be abandoned. Who cares if there are more than “two sides” to an issue (Govier, 1988) or that consolidation has restricted access to the “marketplace of ideas” in the press (Barron, 1966) if we are not meant to use reason to approach the content of journalism?

The stories

The Los Angeles Times, Sarasota Herald-Tribune, and Milwaukee Journal Sentinel captured the 2011 Pulitzer Prizes in the three categories under study.

The Pulitzer Prize in Public Service is awarded for “a distinguished example of meritorious public service by a newspaper or news site through the use of its journalistic resources.” The Los Angeles Times won the 2011 award for “its exposure of corruption in the small California city of Bell where officials tapped the treasury to pay themselves exorbitant salaries, resulting in arrests and reforms” (The Pulitzer Prizes, 2011c). The paper also won the George Polk Award for Local Reporting and Public Service Award from the Los Angeles Press Club for its work.

In all, the Times published more than 200 stories about Bell (Santo, 2011). Sixteen stories dated July 15 to December 28, 2010, were submitted as Pulitzer entries. Whereas each winning entry from the Herald-Tribune and Journal Sentinel tended to be longer, investigative stories, the entries from the Times included shorter, events-driven stories. The range of types of stories available from the Times allowed for testing argumentation and critical thinking techniques against complex and simpler forms of journalism.

Paige St. John of the Sarasota Herald-Tribune won the Pulitzer Prize in Investigative Reporting. The award goes to “a distinguished example of investigative reporting by an individual or team, presented as a single article or series, using any available journalistic tool.” St. John examined “weaknesses in the murky property-insurance system vital to Florida homeowners, providing handy data to assess insurer reliability and stirring regulatory action” (The Pulitzer Prizes, 2011b).

Although St. John’s stories usually cited an investigation lasting “more than a year,” she reportedly spent three years on the project (Santo, 2011, though Masters (2011) puts it at two years). She traveled to Bermuda and Monte Carlo to report, and used her computer-assisted reporting training to build online applications that made use of the data on insurers she collected (Masters, 2011).

Lastly, the Pulitzer Prize in Explanatory Reporting, honoring “a distinguished example of explanatory reporting that illuminates a significant and complex subject, demonstrating mastery of the subject, lucid writing and clear presentation,” went to Mark Johnson, Kathleen Gallagher, Gary Porter, Lou Saldivar and Alison Sherwood of the Milwaukee Journal Sentinel. They won for their reporting on “an epic effort to use genetic technology to save a 4-year-old boy imperiled by a mysterious disease” (The Pulitzer Prizes, 2011a). Only three stories were submitted to the committee, in contrast to the 16 and nine articles of the Public Service and Investigative categories, respectively.

This study has its basis in journalism’s attempt to benefit democracy, and the benefit to democracy that would result from strong arguments presented in these three series is generally clear.

For the Los Angeles Times, an exposure of corruption by local officials seems transparently beneficial to democracy. The Herald Tribune series, while nominally about the insurance industry, also has implications for democratic life. It examines the relationship between what is legal for Florida insurers to do and the actions of state elected officials and regulators, both of which St. John often questions.

The Journal Sentinel series is trickier to classify, and admittedly much of it consists of a well-written tale about an issue confronting only one family, not one confronting a citizenry. But the reporters provide the link to democracy in their attempt to contextualize the family’s struggle and doctors’ choices. They write, for example, that gene sequencing has the potential to begin a “new era in medicine,” and that “reading Nicholas Volker’s genes could change the way doctors treat patients, especially those whose symptoms don’t match any known disease.”

The connection of the Journal Sentinel series to medicine and health care also provides the key to its democratic importance. Reporting on developments in medicine and health care can have short and long-term effects on life for citizens. Michelle Bachmann’s criticism of a cervical cancer vaccine during competition for the 2012 Republican presidential nomination affected the process of selecting the person who could lead the nation, not to mention actual vaccination rates (Grady, 2011). But although political controversies fade, laws tend to stick around, and reporting on medical controversies can help inspire laws. For example, the late–1980s reporting about Alar, a chemical sprayed on apples that was suspected of increasing cancer risk, contributed to the introduction of laws “forbidding libel and slander against fruits, vegetables and food products” that exert a chilling effect on the press to this day (Friedman et al., 1996, p. 2).

The Journal Sentinel‘s coverage of the doctors’ techniques, and the resulting laudatory attention from the likes of NBC’s “Today” show (Inbar, 2011),[2] could have effects extending much further than the publication of the series. So it has a place in the present analysis’ focus on journalism’s democratic implications.


  1. Non-text material from each series included on the Pulitzer website, such as illustrations, was also left out of the analysis.  ↩

  2. Almost too laudatory. The rousing subheadline on the “Today” website, “Groundbreaking DNA sequencing solved medical mystery,” is plainly false, according to the Journal Sentinel. The newspaper’s December 26 story, referring to a transplant Nicholas received “as a result” of the sequencing, read: “It is not clear what the transplant will mean for the mysterious illness that ravaged his intestine.”  ↩

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s