Nate Silver speeds through his keynote.[/caption] AUSTIN, TX—Nate Silver feels a little odd about his fame. That’s not to say that he hasn’t worked to get to his enviable position. Thanks to his savvy with predictive models, he managed to forecast the most recent presidential election results in all 50 states. That accuracy, combined with the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, transformed him into a rare breed: a statistician with a household name. But onstage at this year’s SXSW conference, Silver termed his fame “strange” and “out of proportion,” and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. “It bothered me that this was such a big deal,” he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn’t good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. Silver has been plugging his new book, “The Signal and the Noise,” in which he does his level best to demonstrate how Big Data, despite peoples’ best intentions, often leads businesses and societies awry. By means of example, he takes things back several centuries, to the invention of the printing press in Europe. While printing presses gave the world cheaper books and the means of reproducing information more quickly, Silver argued that the resulting spike in available data led to chaos—the century after its invention was one of the bloodiest ever. Religious factions and opposing kingdoms could put their grievances in a pamphlet and spread them far and wide, sparking arguments and worse. “In the short term, the more information produced, the more conflict, and the more problems,” he said. The amount of information in the world has only grown in the centuries since, spiking in the past twenty years with the invention of cable news and the World Wide Web. “You have people consuming and using media in ways that are different from one another,” he said. “You definitely can’t consume all of it.” As a result, people only cherry-pick the information that appeals to their beliefs and interests: Liberals tend to watch MSNBC, for example, while conservatives stick to Fox News. The data they absorb might be robust in some aspects, but weak in areas they refuse to see. Problems with how we ingest and use information extend from politics to academia and other fields. “You have a gap between what we think we know and what we really know,” he said. “We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.” As an example, he pointed to the famous series of chess matches between Garry Kasparov and Deep Blue in the 1990s. At one point, Deep Blue made a nonsensical move, shifting a rook several spaces forward in defiance of sound endgame strategy. Kasparov, a human being with a brain designed for logic, assumed that Deep Blue had some strategy he couldn’t even begin to comprehend; as a result, he began playing very defensively for the rest of the match. But Deep Blue’s strange move had been the result of a coding glitch, one that had the computer make random legal moves rather than run out of time on the match clock. Kasparov had misinterpreted the data—and while it didn’t cost him that particular match, it shows how even geniuses can make wrong inferences based on the information in front of them. “In theory, having more data can never hurt you,” Silver said. “In practice, if you start to cherry pick your data, you get in trouble.” He also suggested that the awe over his political predictions—which he insisted repeatedly was just a case of averaging polls—demonstrates that the whole field of predictions and statistics still has quite some distance to evolve. “The idea that we’re on the verge of some Big Data singularity is a bit naïve,” he said. But that hasn’t stopped various entities from approaching him with project proposals, all of them hoping to harness what they see as his oracle-like abilities to predict the future: one group wanted to pull a Moneyball on an Indian cricket league, for example, while another asked him to figure out which foods will prove a popular hit. “I think this is a type of science,” he said, “a very new type of science.” Lastly, for those who want to follow in his footsteps, he recommended exploring fields “that have not been thought of in an analytic way.” He also cautioned that, while businesses can gain a lot of insight from statistical tools, the use of analytics “does not absolve you of the responsibility to have a good business culture.” In other words, someone running a company can’t simply keep statisticians or data analysts locked away in a room somewhere; they need to interact with others throughout the organization. SXSW: Nate Silver Discusses Data Bias, Strangeness of Fame
[caption id="attachment_8374" align="aligncenter" width="618"] Nate Silver speeds through his keynote.[/caption] AUSTIN, TX—Nate Silver feels a little odd about his fame. That’s not to say that he hasn’t worked to get to his enviable position. Thanks to his savvy with predictive models, he managed to forecast the most recent presidential election results in all 50 states. That accuracy, combined with the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, transformed him into a rare breed: a statistician with a household name. But onstage at this year’s SXSW conference, Silver termed his fame “strange” and “out of proportion,” and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. “It bothered me that this was such a big deal,” he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn’t good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. Silver has been plugging his new book, “The Signal and the Noise,” in which he does his level best to demonstrate how Big Data, despite peoples’ best intentions, often leads businesses and societies awry. By means of example, he takes things back several centuries, to the invention of the printing press in Europe. While printing presses gave the world cheaper books and the means of reproducing information more quickly, Silver argued that the resulting spike in available data led to chaos—the century after its invention was one of the bloodiest ever. Religious factions and opposing kingdoms could put their grievances in a pamphlet and spread them far and wide, sparking arguments and worse. “In the short term, the more information produced, the more conflict, and the more problems,” he said. The amount of information in the world has only grown in the centuries since, spiking in the past twenty years with the invention of cable news and the World Wide Web. “You have people consuming and using media in ways that are different from one another,” he said. “You definitely can’t consume all of it.” As a result, people only cherry-pick the information that appeals to their beliefs and interests: Liberals tend to watch MSNBC, for example, while conservatives stick to Fox News. The data they absorb might be robust in some aspects, but weak in areas they refuse to see. Problems with how we ingest and use information extend from politics to academia and other fields. “You have a gap between what we think we know and what we really know,” he said. “We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.” As an example, he pointed to the famous series of chess matches between Garry Kasparov and Deep Blue in the 1990s. At one point, Deep Blue made a nonsensical move, shifting a rook several spaces forward in defiance of sound endgame strategy. Kasparov, a human being with a brain designed for logic, assumed that Deep Blue had some strategy he couldn’t even begin to comprehend; as a result, he began playing very defensively for the rest of the match. But Deep Blue’s strange move had been the result of a coding glitch, one that had the computer make random legal moves rather than run out of time on the match clock. Kasparov had misinterpreted the data—and while it didn’t cost him that particular match, it shows how even geniuses can make wrong inferences based on the information in front of them. “In theory, having more data can never hurt you,” Silver said. “In practice, if you start to cherry pick your data, you get in trouble.” He also suggested that the awe over his political predictions—which he insisted repeatedly was just a case of averaging polls—demonstrates that the whole field of predictions and statistics still has quite some distance to evolve. “The idea that we’re on the verge of some Big Data singularity is a bit naïve,” he said. But that hasn’t stopped various entities from approaching him with project proposals, all of them hoping to harness what they see as his oracle-like abilities to predict the future: one group wanted to pull a Moneyball on an Indian cricket league, for example, while another asked him to figure out which foods will prove a popular hit. “I think this is a type of science,” he said, “a very new type of science.” Lastly, for those who want to follow in his footsteps, he recommended exploring fields “that have not been thought of in an analytic way.” He also cautioned that, while businesses can gain a lot of insight from statistical tools, the use of analytics “does not absolve you of the responsibility to have a good business culture.” In other words, someone running a company can’t simply keep statisticians or data analysts locked away in a room somewhere; they need to interact with others throughout the organization.
Nate Silver speeds through his keynote.[/caption] AUSTIN, TX—Nate Silver feels a little odd about his fame. That’s not to say that he hasn’t worked to get to his enviable position. Thanks to his savvy with predictive models, he managed to forecast the most recent presidential election results in all 50 states. That accuracy, combined with the huge readership platform provided by The New York Times hosting his FiveThirtyEight blog, transformed him into a rare breed: a statistician with a household name. But onstage at this year’s SXSW conference, Silver termed his fame “strange” and “out of proportion,” and described his model as little more than averaging the state and national polls, spiced a bit with his algorithms. “It bothered me that this was such a big deal,” he told the audience. In politics, he added, most of the statistical analysis being conducted simply isn’t good, which lets someone like him stand out; same as in baseball, where he made his start in predictive modeling. In fields with better analytics, the competition for someone like him would be much fiercer. Silver has been plugging his new book, “The Signal and the Noise,” in which he does his level best to demonstrate how Big Data, despite peoples’ best intentions, often leads businesses and societies awry. By means of example, he takes things back several centuries, to the invention of the printing press in Europe. While printing presses gave the world cheaper books and the means of reproducing information more quickly, Silver argued that the resulting spike in available data led to chaos—the century after its invention was one of the bloodiest ever. Religious factions and opposing kingdoms could put their grievances in a pamphlet and spread them far and wide, sparking arguments and worse. “In the short term, the more information produced, the more conflict, and the more problems,” he said. The amount of information in the world has only grown in the centuries since, spiking in the past twenty years with the invention of cable news and the World Wide Web. “You have people consuming and using media in ways that are different from one another,” he said. “You definitely can’t consume all of it.” As a result, people only cherry-pick the information that appeals to their beliefs and interests: Liberals tend to watch MSNBC, for example, while conservatives stick to Fox News. The data they absorb might be robust in some aspects, but weak in areas they refuse to see. Problems with how we ingest and use information extend from politics to academia and other fields. “You have a gap between what we think we know and what we really know,” he said. “We tend to be oversensitive to random fluctuations in the data and mistake the fluctuations for real relationships.” As an example, he pointed to the famous series of chess matches between Garry Kasparov and Deep Blue in the 1990s. At one point, Deep Blue made a nonsensical move, shifting a rook several spaces forward in defiance of sound endgame strategy. Kasparov, a human being with a brain designed for logic, assumed that Deep Blue had some strategy he couldn’t even begin to comprehend; as a result, he began playing very defensively for the rest of the match. But Deep Blue’s strange move had been the result of a coding glitch, one that had the computer make random legal moves rather than run out of time on the match clock. Kasparov had misinterpreted the data—and while it didn’t cost him that particular match, it shows how even geniuses can make wrong inferences based on the information in front of them. “In theory, having more data can never hurt you,” Silver said. “In practice, if you start to cherry pick your data, you get in trouble.” He also suggested that the awe over his political predictions—which he insisted repeatedly was just a case of averaging polls—demonstrates that the whole field of predictions and statistics still has quite some distance to evolve. “The idea that we’re on the verge of some Big Data singularity is a bit naïve,” he said. But that hasn’t stopped various entities from approaching him with project proposals, all of them hoping to harness what they see as his oracle-like abilities to predict the future: one group wanted to pull a Moneyball on an Indian cricket league, for example, while another asked him to figure out which foods will prove a popular hit. “I think this is a type of science,” he said, “a very new type of science.” Lastly, for those who want to follow in his footsteps, he recommended exploring fields “that have not been thought of in an analytic way.” He also cautioned that, while businesses can gain a lot of insight from statistical tools, the use of analytics “does not absolve you of the responsibility to have a good business culture.” In other words, someone running a company can’t simply keep statisticians or data analysts locked away in a room somewhere; they need to interact with others throughout the organization. 