The following is an analysis that is part of a graduate thesis. Learn more.
St. John’s dove back into a specific company and its impact in this story. Specifically, she investigated a firm called Risk Management Solutions, which provided software to insurers and reinsurers that estimated the likelihood and severity of hurricanes. The companies, in turn, used those estimates to price their services.
St. John explored two issues: How did RMS calculate its hurricane forecast after Hurricane Katrina, and what was the effect of its forecast? She reached three descriptive conclusions. First, that RMS generated its forecast through a meeting with four hand-picked scientists. Second, that the meeting and resulting storm model “created an $82 billion gap between the money insurers had and what they needed, a hole they spent the next five years trying to fill with rate increases and policy cancellations” (5). Third, that RMS’s model was unscientific and not based on scientific consensus.
Her first conclusion, that the meeting between RMS and the scientists took place, was strongly documented through interviews. She paraphrased and quoted extensively from one of the scientists, Jim Elsner (23–35), and somewhat less on two other scientists (36–39), and said she unsuccessfully attempted to contact the fourth (36). Adding credibility to the conclusion was the fact that even the response from RMS, at least that which was quoted, attempted only to endorse, not contest, the meeting (“RMS defended its new model by suggesting it had brought scientists together for a formal, structured debate” (50)).
The second conclusion was a little more difficult to parse. St. John provided three descriptive claims, with varying levels of support for each.
She said that the changes in hurricane modeling by RMS “created an $82 billion gap between the money insurers had and what they needed” (5). In other words, under RMS’s new model, which forecast more hurricanes over the following five years, with a higher likelihood that those storms would be catastrophic, insurers would need more money to pay claims than they had previously prepared for, to the tune of $82 billion.
From where did St. John get this figure? Its origin was shadowy. She wrote:
The yet-unpublished five-year model did not become an industry standard until December 2005, when it was embraced by A.M. Best, the Chicago firm that provides financial ratings for insurance investors.
Best said it would determine an insurer’s soundness by simulating its performance in back-to-back 100-year hurricanes as calculated by the five-year model. …
According to a confidential presentation one of its officers gave an industry think tank, RMS calculated its new hurricane model raised the expected cost of a major U.S. hurricane by $55 billion.
Plugging that model into A.M. Best’s stress test meant the industry as a whole would need to raise $82 billion to remain solvent. (69–70, 73–74)
So the key transition was from the confidential presentation by RMS, about which little was said in the rest of the story, to “plugging in” the presentation data into A.M. Best’s test. Unfortunately, it was not clear how the Best model worked, who conducted it, or how one plugged in to the other. Was it a figure arrived at by RMS, Best, or St. John? Simply doubling $55 billion, as might have been the supposed method (“back-to-back 100-year hurricanes”) does not achieve the expected result. How, then, did the stress test work? All of these were unanswered questions that made it difficult for readers to accept this part of the conclusion.
There appeared to be clearer evidence for the claim that insurers tried to fill the $82-billion gap with rate increases. St. John cited, among other data, “comments made in quarterly earnings calls” and Allstate’s “4,000-page request for a 22 percent rate hike” (79, 81). These data came from only the first couple of years after RMS introduced the new model, meaning they didn’t quite justify the full “next five years” claim, although they came reasonably close.
However, no data were presented in support of the claim that policy cancellations were used as part of the recovery effort. In sum, then, the combined claim about the financial difficulty insurers faced after RMS’s model and how they reacted was partially justified, but not strongly enough to accept as a whole, at least under the evidence presented.
The third conclusion suffered from some harmful ambiguity at the outset; it was difficult to determine what the conclusion actually was. St. John wrote:
RMS said the change that drove Florida property insurance bills to record highs was based on “scientific consensus.”
The reality was quite different. (6–7)
This could be interpreted as either “it is not the case that RMS based its change on scientific consensus” or “it is not the case that there was scientific consensus.” The latter interpretation encompasses the former, though not vice versa (there might have been a consensus, but not one that RMS intentionally considered).
St. John did not clear the confusion, and indeed it appeared that at times she tried to demonstrate both points: She provided reasons to think both that that RMS’s methodology was suspect, and that the scientific community was more fractured than RMS would acknowledge. In both cases, readers would have needed to use a little imagination to attach the reasons and evidence to the conclusion, though not a troubling amount.
The paraphrasing of some scientists that “the industry skipped the rigors of scientific method [and] ignored contradictory evidence and dissent” (9) was somewhat useful for demonstrating that RMS did not base its conclusion on scientific consensus. It did not necessarily show that there was no consensus, though. RMS might have used foul methods while happening upon what was a consensus about hurricanes anyway.
More useful was her reporting of the concerns of a state commission that must approves hurricane models before they may be used to set insurance rates. Those reviewers “planned to reject the model,” St. John said (85).
A draft report shows the objections centered largely on how RMS had determined its new hurricane rates.
The panel said the model change failed to meet credibility and bias tests, and it questioned how RMS had picked its four scientists and why so few were invited. (86–87)
Additionally, it took only a small step for readers to justifiably conclude that RMS’s conclusions were not based on scientific consensus given St. John’s detailed descriptions of how the sessions with scientists were conducted. Based on interviews with participants, St. John described an abrupt first meeting with the four scientists:
The RMS modelers believed Florida would remain the target of most hurricane activity. Elsner’s research showed storm activity shifted through time and that it was due to move north toward the Carolinas.
But RMS’ facilitator said there was not enough time to debate the matter, Elsner said. There were planes to catch.
In the end, the four scientists came up with four hurricane estimates — similar only in that they were all above the historic average.
RMS erased that difference with a bit of fifth-grade math. It calculated the average. (41–44)
To even the most casual reader, “planes to catch” and averages probably would not have spelled “scientific consensus.” Nor would, as St. John described, the use of Tiddlywinks to rank different RMS-picked climate models, as seven scientists were asked to do in 2008 to update RMS’s software (94). She also quoted the criticism written on the blog of a scientist involved in the 2008 session, who came up with nearly the same ranking of models with a random number generator in his office as did the scientists in the room (110).
So if St. John meant to say “it is not the case that RMS based its change on scientific consensus,” she demonstrated that point quite well. Complaints of a few scientists, though, does not a consensus break. St. John’s evidence to suggest “it is not the case that there was scientific consensus” was a little weaker. She paraphrased Karen Clark, the former chief executive of AIR Worldwide, “an RMS competitor,” as saying RMS’s model “lacked sufficient scientific support” (17). Later she paraphrased AIR Worldwide and Eqecat, another competitor, as arguing the same to A.M. Best (75), although “the warnings were not heeded” (78). The obvious hangup to these data was that as RMS competitors with financial interests in seeing RMS falter, their credibility as experts on the matter was poisoned.