If Then: How the Simulmatics Corporation Invented the Future by Jill Lepore.
Jill Lepore is a Harvard University professor who writes about history, broadly defined. She wrote These Truths, a best-selling overview of American history, as well as The Secret History of Wonder Woman, along with nine other books.
If Then is about Simulmatics, a long-forgotten company that in the 1960s gathered and organized data from a variety of sources, stored the information on punch cards, and used a state-of-the-art IBM mainframe computer – as big as a small house – to organize and retrieve the material. Simulmatics was way ahead of the curve in understanding that mining and analyzing data was the wave of the future.
The company was founded by Edward Greenfield, who was born in Chicago in 1927. His resume was impressive – University of Chicago undergrad and Yale Law School. It was also completely fabricated. Ed did attend Yale after a fashion -– he sat in on a seminar.
Greenfield loved politics and Madison Avenue advertising and was involved in trying to get Adlai Stevenson elected president in 1952 and 1956. He was a classic 1950s liberal. He also was a huckster who could be very persuasive.
In the 1950s both political parties were liberal, with the Republicans the party of business and the Democrats the party of labor. During the decade, political advertising became a major part of campaigns. While in 1948 Harry Truman thought ad agencies were bunko, in 1952 Dwight Eisenhower used television to run ads. His opponent, Adlai Stevenson, seen as a cerebral Hamlet-like figure, thought that being on TV was beneath the dignity of a presidential candidate. In the 1952 campaign Republicans spent $1.3 million on TV while the Democrats spent $77,000, mostly televising Adlai’s boring speeches. That campaign also marked the first time a computer was used to call the election. With only 3 million votes in, UNIVAC predicted that Ike would win 438 electoral votes to Stevenson's 93. The actual results were 442 for Eisenhower and 89 for Stevenson, with Ike winning the popular vote 55% to 45%.
Eugene Burdick was a brilliant academic who thought he was James Bond. He was a California surfer who went to Stanford and ended up as a Rhodes Scholar. That got him a job as a professor at the University of California at Berkeley. He was interested in and studied psychology and ended up spending a year at the Center for Advanced Study in the Behavioral Sciences in Palo Alto, CA. The Ford Foundation funded the Center and also created the RAND corporation, now a respected think tank. The Center pulled in top behavioral scientists from all over the country, and they worked together to figure out how to use data to predict human behavior. They came up with the idea of classifying humans into dozens and dozens of groups based on demographics, employment, religion, income, kind of car they drove, and other characteristics. They then would interview people in these groups as well as gather additional information and put it all together to predict future behavior. In 1955, Greenfield recruited Burdick to be part of the new Social Science Division of Edward L. Greenfield & Co in New York City, but Burdick was not interested in being part of the soon-to-be formed Simulmatics Corporation. He did write a best-selling novel based on it, The 480 (the number of unique human profiles that the company used as the basis for its analyses).
Despite having good contacts with Stevenson’s people, Greenfield could not get into the 1956 campaign. Adlai just didn’t believe in advertising – it was beneath him. He lost again, although defeating Ike was never really in the cards.
Greenfield agreed with Burdick and the academics from the Center for Advanced Study in the Behavioral Sciences that using data to predict human behavior was the next big thing. He got busy assembling a team of experts. He needed a math whiz and found one at MIT, Ithiel de Sola Pool, the son of a rabbi who, along with his wife, was a political activist and an admirer of communist principles. Ithiel was a genius who had a job lined up with the Hoover Institute if he could get security clearance. His parents' communist connections and Ithiel’s activities with liberal groups in his youth set off alarms in the early 1950s as Joe McCarthy beat the drum about communist infiltration of our government. Things weren’t going well for Pool in the investigation until a young congressman from California intervened – Richard Nixon. De Sola Pool had his clearance and also had a friend in Nixon.
In 1955 Ithiel joined Greenfield’s team in New York. While there, he came up with the theory of social networks that would become the foundation of Facebook some 50 years later.
Next, Ed needed someone who knew computers. Alex Bernstein was wicked smart and knew technology. In the mid 1950s, he had graduated from the City University of New York and had become a computer expert. One of his jobs was to play chess against a huge IBM 704 computer in a display window in Manhattan to publicize the machine. He also was a really good programmer. Ed hired him.
As a point of reference, the huge mainframe that Bernstein played chess against could do 12,000 floating point operations per second. A 2012 Apple iPhone can do 171,000,000 floating point operations per second.
The next thing Ed needed was a mathematician with a sociological background who could merge the quantitative (hard numbers about individual characteristics/demographics) with the qualitative data (opinions and feelings) to predict outcomes. Bill McPheewas a math savant who skipped undergrad school to go to grad school and develop a model to predict voting behavior. He was a genius who happened to be a jerk. He demeaned and mistreated his wife, Minnow, all of the time, while exploiting her to do the grunt work for his research. At one point, she had him committed to Bellevue, a mental hospital in New York. He was nuts. Ed hired him.
Now that Greenfield had his team, they worked to develop a process to analyze data in such a way to predict future behavior, specifically voting. The Macroscope utilized all of the data on punch cards and could predict outcomes reasonably well based on what a candidate said about a specific issue.
In the late 1950s Greenfield pitched the concept to various people: Newton Minow who later became the Federal Communications Commissioner under President Kennedy; author and Kennedy confidante Arthur Schlesinger; and Adlai Stevenson – all of whom hated it as being undemocratic.
Undeterred, the Simulmatics Company opened for business on February 18, 1959 with this mission: “The Company proposes to engage principally in estimating probable human behavior by the use of computer technology.” The new company collected punch cards from 100,000 surveys conducted by Gallup as the basis for its database.
In 1960, after securing the nomination against a diffident Adlai Stevenson and a ticked-off Lyndon Johnson, the Kennedy campaign contracted with the company to develop analytical reports on critical election issues, starting with looking at how to get Black voters, a group that historically supported Republicans, the party of Lincoln. The fact that many Democratic senators from the South were overt racists kept Blacks in the Republican fold. Kennedy had to get some of those votes to win. Another analysis recommended that Kennedy tackle the religion issue - he was Catholic - head-on.
The compilation program that Greenfield’s crew had developed was called the People Machine, and it basically worked pretty well. Kennedy did cater his campaign to engage more Blacks, and he did directly confront the religion issue in his famous speech about how he would not be a Catholic president but would be a president who happened to be Catholic. Kennedy won under somewhat dubious circumstances involving Chicago Mayor Daley finding enough late votes to give Jack Illinois after it looked like he had lost it.
(Bob’s Note: Nixon’s supporters urged him to contest the election but he said, “Our country can’t afford the agony of a constitutional crisis - and I damn well will not be a party to creating one just to become President or anything else.” Similarly, Al Gore gave a gracious concession speech in December of 2000 after the Supreme Court stopped the recount in Florida with George W. Bush ahead. Think about it.)
Many people in the campaign, especially Bobby Kennedy, thought that Simulmatics was at best superfluous and at worst useless. The company’s research revealed nothing that the campaign didn’t already know. In any event, Greenfield used Kennedy’s win to market his company. He got a puff piece in Harper’s Magazine, as well as massive television and radio coverage of Simulmatics as the company that won the presidency for Jack Kennedy. The downside to all of the press was that many, if not most, people were nervous about having computers call the shots in elections. There was a lot of blowback.
The company pitched its services to ad agencies and media companies as well as to the new Kennedy administration. They got a few contracts and were off and running. The People Machine was modified by adding lots of data about consumer consumption behavior. It became a product called Media-Mix for use in advertising and marketing companies.
James Coleman became famous in the late 1960s for producing the Coleman Report on unequal educational opportunity. His extensive research on over 600,000 students showed that educational opportunity varied widely based on income and parents' education level and that poor students in schools with diverse populations including higher income kids did better. He was accused of blaming the victim and racism for his findings which happened to be correct. I actually read the 600-page report which was the basis for some of my dissertation methodology on the relationship between community demographics and student achievement.
However, in the early 1960s, before many people knew who Coleman was, Ed Greenfield recruited him for his expertise in using statistics to inform the kind of sociological research that Simulmatics needed in order to refine its Media-Mix consumer marketing tool. Coleman thought that it would be very difficult to create a statistical model to account for what drove consumers to buy certain products; but, as it turned out, ad agencies loved the pitch that Ed made about his company's ability to use computerized data to analyze advertising effectiveness. But instead of hiring Simulmatics, the agencies built their own in-house data analysis departments.
The 1962 elections The New York Times hired the company to help the paper analyze the 1962 midterm elections. Given that people increasingly relied on television and radio to get news as it was breaking, newspapers shifted to doing more in-depth pieces to keep their readers happy. The Times bought a 4-ton IBM 1401, the Model T computer of the period, that Simulmatics would use to dig into the voting results. The paper was interested in the New York governor’s race as well as several congressional and senate races around the country.
It did not go well. The computers were delivered only a few days before the election, and it turned out that no one really knew how to run them or to connect them to different machines so that they could exchange information. Also, Coleman and Pool (the math whiz) were having trouble writing the program with the variables they needed for the analyses. They never quite got it right. Simulmatics didn’t come up with any insights or early predictions on the races but did bill the Times way above the agreed contract price. The paper terminated the relationship.
Undaunted, Simulmatics pitched the Advanced Research Projects Agency of the Defense Department on creating a computerized data system that could predict strife around the world. The company got a small contract to work with existing Department of Defense consultants at MIT to move the project ahead.
By 1964, Ed Greenfield’s company was in trouble, with no major clients. While the company was primarily staffed by liberal Democrats, they were contacted by the Goldwater campaign. They didn’t follow up on that but did make a pitch to work on LBJ’s campaign which was rejected.Ed did manage to get a federal contract to study Venezuela which helped a bit, but he was focused on a bigger prize: Vietnam. By the end of 1964, the US was committed to the war in Southeast Asia.
Vietnam Secretary of Defense Robert McNamara was one of the first people to rely on computer analyses when he was president of the Ford Motor Company before he went to Washington. McNamara believed that you could use data to figure out anything, including how to defeat North Vietnam. He was fascinated by body counts and kill ratios and the tonnage of bombs dropped and the number of soldiers in a specific area. As time went on, it became clear that you needed more than numbers to be effective, and that’s when Ed Greenfield pitched his Defense Department contacts to contract with Simulmatics to better understand the hearts and minds of the Vietnamese through the use of the company’s data analysis technology. It worked and the company was given the equivalent of tens of millions of dollars – in today’s dollars – to go to Vietnam and figure things out.
The company was essentially doing psychological warfare operations (psych ops) with the Vietnamese. Ed hired a lot of academics to do field interviews and reports, a lot of young women college graduates to type everything, and a lot of local interpreters to translate.
Aside from being paid a lot of money, things did not go well for the company. The academics who did the field work could not relate to the people they were interviewing. Often, they went off script so much that the results were worthless. One New York City psychologist, who was a friend of Ed’s, did Freudian analyses of his interviewees, not helpful. The hires were given extensive assistance in learning some Vietnamese, but few bothered to learn the language.
Beginning in 1965, the war ramped up quickly as did American and Vietnamese casualties. The draft increased from 10,000 a month to 30,000 a month. The military said that it would not harm civilians but since they couldn’t tell a civilian from a combatant, 3,000,000 Vietnamese died.
By 1966, many people were growing weary of the war. Ironically, almost all of Greenfield’s employees in the US and in Southeast Asia were liberal Democrats who were very anti-war. The protest movement grew on campuses and in the cities, and Simulmatic’s principals were among those called war criminals at demonstrations.
Despite being criticized for its shoddy work, the company kept working for the Defense Department. They were charged with doing research to support McNamara’s strategic hamlets program that would evaluate the loyalty of villages. The company’s presence in-country grew to 50 professionals plus support staff.
Back home things were not going well at the company’s Cambridge office at MIT where students routinely protested the war and the fact that Simulmatics was supporting the conflict in Vietnam. Pool, who was an MIT professor as well as vice president of the company, made trips to Vietnam in 1967 and, like top military leaders, wrote glowingly about how great the war was going.
Pool did not impress the Army people he worked with in the field. When the Defense Department finally looked at the questionnaire he developed to interview subjects, they were appalled at how bad it was in not being culturally compatible with the Vietnamese approach to life. On top of that, no one ever explained what the goal was to the interpreters, so they were limited as to how much value they could add to the dialogue. The conclusion of higherups in Vietnam and DC was that the field work interviews were basically useless and that the reports were couched to say what the local Army brass wanted to hear. This might not have been a problem a few years earlier, but by late 1967 things were falling apart in the war. Many top leaders in Washington and Vietnam didn’t think we could win. McNamara told Johnson to negotiate a peace and withdraw, advice that LBJ rejected.
Back in the USA Back home, there was racial unrest and major protest. There were over 150 riots in cities in the summer of 1967. The company managed to get some contracts to develop a system to predict riots. This sounds really silly, but back then most major cities were trying to develop the same capability. Simulmatics did its thing and interviewed people and wrote a computer program to predict riots. It didn’t work but they still got paid.
By 1968, things were not good in Vietnam, the US of A, and at Simulmatics. Ed laid off the entire NYC office, the main one. The Defense Department finally pulled the plug on the Vietnam operation. Creditors were barking at the door. Ed, and most of the main people in the company had gotten or were getting divorced. Ed was constantly drunk.
The principals decided to sell the company but could find no buyers. Since Ed was useless, Pool went out to try to get business. He pitched Hubert Humphrey’s and Richard Nixon’s campaigns but was turned down by both.
Pool tried to get funding to set up a national data center that would contain information on all Americans, but that notion was rejected by just about everyone. Ironically, today Google and Facebook have a lot of that information about us on their servers.
Pool predicted Google, iPads, the Internet. He did understand what the future would look like. He also became a lightning rod for protesters at MIT where he was a faculty member. They baited him and put out “Wanted for War Crimes” posters. He was willing to debate the students, but that never happened. He did debate noted anti-war protester Noam Chomsky in a classic standoff that was like a gladiator fight. Pool continued to try to defend the company’s work for the Defense Department, but to no avail. Students screamed that “MIT was the Defense Department’s whore.” If so, it was a well paid one, with about 50% of the school’s budget coming from DOD.
In the spring of 1970, a biggie in anti-war protests, a group of students marched to the building that had been the Cambridge office of the company. Alas, Simulmatics had left the building two years earlier.
Simulmatics declared bankruptcy in August of 1970. By then, Robert McNamara’s data-driven analytics approach to waging war was discredited which didn’t help Ed Greenfield’s company survive. Most of the people discussed in the book did reasonably well after the company collapsed. The main scientists and behaviorists were well credentialed and usually ended up in academia or think tanks. The central figures all moved on with their lives but not their wives. Just about everybody ended up divorced.
Ed Greenfield sort of got his act together. He reunited with his kids after a nasty divorce. He pitched a new project, the Mood Corporation, to gather and organize data on individuals and sell it to interested parties. It was Facebook before Facebook. It didn’t go anywhere. He died in 1983 of a heart attack at age fifty-six.
Daniel Ellsberg, famous for releasing Robert McNamara’s Pentagon Papers in 1971, had crossed paths with some Simulmatics employees in Vietnam in the late 1960s. The Nixon administration went after Ellsberg with a vengeance, and several Simulmatics people were called before grand juries, one of whom was briefly jailed for contempt for not answering questions about his work colleagues in Southeast Asia. Not much came of the attempted prosecutions, and Nixon left office in disgrace. Nixon routinely used the Justice Department as his personal get-even agency. Donald Trump tried that, but his attorney general did not go after people just because the boss wanted him to.
Lepore’s last chapter is the grand summary of Simulmatic’s place in the history of data mining. While the company failed, it got a lot of things right. Today, predictive analytics are the rage in academia, advertising and marketing, political campaigns, and real science and medicine. I used a predictive analytics model in my dissertation. Since the late 1990s, the methodology has become an almost routine way to analyze past behavior and actions in order to get a handle on what the future will bring. The COVID-19 vaccines that are coming into the market now heavily depend on predictive analytics to figure out what combination of variables will result in immunity from the virus.
Professor Lepore makes a pretty good case that studying the human condition and trying to predict future behavior is much more difficult than using data analytics to find a virus vaccine or plot the movement of the stars, which are hard science, not psychology. She is very concerned about the incredible power of digital media in gathering all sorts of information about people without any boundaries being set by government or the people using the services. She sees a problem with universities today that they are eager to align with commercial digital data divers that will fund academic projects. She is not wild about Facebook.
Bob’s Take
The first hundred pages of the book are a bit tedious in that the author goes into excruciating detail about the individuals who founded Simulmatics. Along the way, she does provide a solid context to explain what was going on in the United States in the late 1950s and early 1960s. Since I was there and I’m really into American history, I found it a bit much, but many readers would find it helpful.
Simulmatics was an inept company based on a breakthrough idea. None of the five principals knew anything about business or basic organizational dynamics and it showed. Often their approach to a challenge was “Ready. Shoot. Aim.” They were way ahead of the curve in recognizing that computers and in-depth data mining were the foundation for very sophisticated analyses that would give clients valuable information about their customers, be they housewives in mid-America, rich guys who golfed, or the Vietnamese people. Besides being terrible businessmen, there were some other problems.
— The computers of the mid and late 1960s were very underpowered. The 4-ton IBM monster they installed at the New York Times didn’t have close to the power of the first iPhone. The data was terrible. There wasn’t much of it and a lot of it was wrong. Today we have ways of cleaning up data sets that they didn’t have, but the major issue was that their source information was not good. <strong>
— They were building the plane while they were flying it, always a challenge. While the principal scientists of Simulmatics were leaders in their fields, they were tasked with doing things that hadn’t been done before. James Coleman was a whiz stats guru, but his job at Simulmatics was to develop programs that did much more than what he had been working on. Had he enough time, it would have worked, but, because the business side was so bad, he never had enough time.
— People in the 1960s were looking for Silver Bullets to fix intractable problems like Vietnam and poor race relations. What Ed Greenfield was pitching with his People Machine methodology sounded like it was a Silver Bullet, a one-stop, easy solution. It wasn’t.
— The US government really does a lot of dumb stuff. To be gentle, Simulmatics operation in Vietnam was a clown show, albeit one generously funded by the Defense Department. Despite throwing literally millions of dollars at the Vietnam project, there was very little oversight. Once military people started to look under the hood, they found that there was no there there. Through it all, Simulmatics kept hiring more people and moving to bigger and more plush quarters, all with no questions being asked by the funders.
