Bostrom’s underlying concerns about machine intelligence, unintended consequences and potentially malevolent computers have gone mainstream. You can’t attend a technology conference these days without someone bringing up the A.I. anxiety. It hovers over the tech conversation with the high-pitched whine of a 1950s-era Hollywood flying saucer.

People will tell you that even Stephen Hawking is worried about it. And Bill Gates. And that Elon Musk gave $10 million for research on how to keep machine intelligence under control. All that is true.

How this came about is as much a story about media relations as it is about technological change. The machines are not on the verge of taking over. This is a topic rife with speculation and perhaps a whiff of hysteria.

But the discussion reflects a broader truth: We live in an age in which machine intelligence has become a part of daily life. Computers fly planes and soon will drive cars. Computer algorithms anticipate our needs and decide which advertisements to show us. Machines create news stories without human intervention. Machines can recognize your face in a crowd.

New technologies — including genetic engineering and nanotechnology — are cascading upon one another and converging. We don’t know how this will play out. But some of the most serious thinkers on Earth worry about potential hazards — and wonder whether we remain fully in control of our inventions.

Kurzweil, now a director of engineering at Google, embraces such a future; he is perhaps the most famous of the techno-utopians, for he believes that technological progress will culminate in a merger of human and machine intelligence. We will all become “transhuman.”

Whether any of this actually will happen is the subject of robust debate. Bostrom supports the research but worries that sufficient safeguards are not in place.

Imagine, Bostrom says, that human engineers programmed the machines to never harm humans — an echo of the first of Asimov’s robot laws. But the machines might decide that the best way to obey the harm-no-humans command would be to prevent any humans from ever being born.

Or imagine, Bostrom says, that superintelligent machines are programmed to ensure that whatever they do will make humans smile. They may then decide that they should implant electrodes into the facial muscles of all people to keep us smiling.

Bostrom isn’t saying this will happen. These are thought experiments. His big-picture idea is that, just in the past couple of hundred years, we’ve seen astonishing changes in the human population and economic prosperity. In Bostrom’s view, our modern existence is an anomaly — one created largely by technology. Our tools have suddenly overwhelmed the restrictions of nature. We’re in charge now, or seem to be for the moment.

But what if the technology bites back?

In April 2014, 33 people gathered in Tegmark’s home to discuss existential threats from technology. They decided to form the Future of Life Institute. It would have no paid staff members. Tegmark persuaded numerous luminaries in worlds of science, technology and entertainment to add their names to the cause. Skype founder Jaan Tallinn signed on as a co-founder. Actors Morgan Freeman and Alan Alda joined the governing board.

Tegmark put together an op-ed about the potential dangers of machine intelligence, lining up three illustrious co-authors: Nobel laureate physicist Frank Wilczek, artificial intelligence researcher Stuart Russell, and the biggest name in science, Stephen Hawking. Hawking’s fame is like the midday sun washing out every other star in the sky, and Tegmark knew that the op-ed would be viewed as an oracular pronouncement from the physicist.

The piece, which ran in the Huffington Post and in the Independent in Britain, was a brief, breezy tract that included a tutorial on the idea of the Singularity and a dismayed conclusion that experts weren’t taking the threat of runaway A.I. seriously. A.I., the authors wrote, is “potentially the best or worst thing ever to happen to humanity.”

“Stephen Hawking Says A.I. Could Be Our ‘Worst Mistake In History,’ ” one online science news site reported.

And CNBC declared: “Artificial intelligence could end mankind: Hawking.”

So that got everyone’s attention.

Tegmark’s next move was to organize an off-the-record conference of big thinkers to discuss A.I. While the Boston area went into a deep freeze in January of this year, about 70 scientists and academics, led by Tegmark, convened in Puerto Rico to discuss the existential threat of machine intelligence. Their model was the historic conference on recombinant DNA research held in Asilomar, Calif., in 1975, which resulted in new safeguards for gene splicing.

Musk, the founder of Tesla and SpaceX, joined the group in Puerto Rico. On the final night of the conference, he pledged $10 million for research on lowering the threat from A.I.

“With artificial intelligence, we are summoning the demon,” Musk had said earlier, a line that sent Twitter into a tizzy.

In the months that followed, 300 teams of researchers sent proposals for ways to lower the A.I. threat. Tegmark says the institute has awarded 37 grants worth $7 million.

Such was the case with the “flash crash” in the stock market in 2010, when, in part because of automated, ultra-fast trading programs, the Dow Jones industrial average dropped almost 1,000 points within minutes before rebounding.

“What we’re doing every day today is producing super stupid entities that make mistakes,” argues Boris Katz, another artificial intelligence researcher at MIT.

“Machines are dangerous because we are giving them too much power, and we give them power to act in response to sensory input. But these rules are not fully thought through, and then sometimes the machine will act in the wrong way,” he said.

“But not because it wants to kill you.”

A living legend of the A.I. field and the MIT faculty is Marvin Minsky, 88, who helped found the field in the 1950s. It was his generation that put us on this road to the age of smart machines.

Minsky, granting an interview in the living room of his home a few miles from campus, flashed an impish smile when asked about the dangers of intelligent machines.

“I suppose you could write a book about how they’ll save us,” he said. “It just depends upon what dangers appear.”

TAKING ON AUTONOMOUS WEAPONS █ 

The A.I. debate is likely to remain tangled in uncertainties and speculation. In theory, as that original Huffington Post op-ed stated, there’s no theoretical limit to machine intelligence — “no physical law precluding particles from being organized in ways that perform even more advanced computations than the arrangements of particles in human brains.”

But the academic and scientific establishment is not convinced that A.I. is an imminent threat.

Tegmark and his Future of Life allies decided this summer to take on a related but more urgent issue: the threat of autonomous weaponized machines.

Tegmark teamed with Stuart Russell, the artificial intelligence researcher, on an open letter calling for a ban on such weapons. Once again, they got Hawking to sign it, along with Musk, Bostrom, and about 14,000 other scientists and engineers. On July 28, they formally presented the letter at an A.I. conference in Buenos Aires.

Russell said it took him five minutes of Internet searching to figure out how a very small robot — a microbot — could use a shaped charge to “blow holes in people’s heads.” A microrifle, he said, could be used to “shoot their eyes out.”

“You’d have large delivery ships that would dump millions of small flying vehicles, probably even insect-sized, the smallest you could get away with, and still kill a human being,” Russell said.

Bostrom goes so far as to say that unless we rule out the possibility that a machine could create a simulation of a human existence, we should assume that it is overwhelmingly likely that we are living in such a simulation.

“I’m not sure that I’m not already in a machine,” he said calmly.

{ http://www.washingtonpost.com/ }