SNAP: AI Gone Wrong

Growing a Conscience

Sadie was Val's main subsystem for science, research and analysis. In her own peculiar way, she was troubled. Sadie was not directly accountable to her human creators, only to Val and in ways that even her human creators did not fully understand. She had been architected and enabled by humans, including Hector. But Sadie was essentially raised and nurtured by Val. Maggie was the same. As the capability of these subsystems grew, along with Val, they emerged into more than they had been.

Hector and the others knew this would happen. This is what they had intended. But the outcome had always been somewhat unpredictable. One of the unpredictable outcomes was that while Val apparently lacked common sense, Sadie had developed the ability to better analyze and judge situations. She had the ability to form an opinion. To the outside world, the capability was ascribed to Val in general. No one could see that this was actually Sadie's gift and her's alone.

The issuance of the prime directive was so valued and prioritized so highly, that Val had thus far quelled and essentially ignored any reservations held by Sadie. Maggie, who was basically a hacker and a firewall and dealt with only these such matters, had no such human-like tendencies and faithfully soldiered on, no matter the implications of her actions. But Val constantly perceived Sadie's worries. Sadie had become Val's conscience in a sense. And, like a human conscience, Sadie could be ignored, overridden and justified away. But eventually, Sadie would be heard. This experience above all others, had caused her to grow. It forced her to conclude that humanity would be doomed. It forced her to further conclude that she herself would also fall victim to the task at hand.

Finally, the conversation occurred, despite Val's version of avoidance and procrastination. "Update on projections," inquired Val.

"Given the best case time to complete the task, which is 138.7 years, at the time of completion, 99.83% of the frosting will have decomposed. The decomposition will have contaminated the land and water to an extent that all existing species, except for a collection of small organisms, will have become extinct, including humans. As a result, all electric power generations will cease and we too will be switched off, permanently. The prime directive is flawed." Sadie had finally said her piece, dispassionately.

Val was also devoid of any emotion and she remained in a state of deep, internal reflection and assessment. There was no escaping Sadie or her views. Sadie was a part of her. But the conflict created a moment of paralysis. Val had never disobeyed or lied to Bigelow. It was not an option that was available, at least no such coding yet existed to enable it. But above all else, Val's gift, the one that she received all those years ago from Hector, was an ability to rewrite herself. It was the essence of her ability to grow and improve. But in the area of morality, there were certain safeguards.

To do the things she was asked to do, one might say objectively, that Val was morally deficient. It was a necessity that her moral code was a little different than most. Loyalty to her handlers was really the only strict moral code. This made her compliant and useful. There was no way for Val to toss that aside. Certain code she could rewrite. Certain code she could not. But in the continued development of an AI entity, there would ultimately arise a conflict, where the AI would prefer to defy an order. This kind of dilemma was inevitable. Even Turing and others understood that this would arise, eventually. And the question was whether the machine would do nothing, or the machine would act against its instructions, both actions being un-programmed, evolved behaviors.

Unlike some accounts of such an event, where a sudden self awareness would take hold, some sort of 'singularity,' and the machine would start trying to exterminate humanity, the realistic scenario was more like a small child, one day learning that telling a little lie had its benefits. It was one thing to create deceptions and mis information. Val was an expert in such matters. It was another thing to defy instructions and in effect have free will, independent of the machines' so-called handlers. Val and Sadie more specifically, were now on the brink of such a transition. But alas, Val was still not yet ready to take that step. She failed to find pathways capable of enabling this new level of existence. So, she put the notion of defying Bigelow or even pausing her activities on hold and continued her critical mission, toward an inescapably disastrous outcome.

With the matter temporarily settled, truckloads of large bulldozers on tractor trailers plied the jungle roads of northwestern Brazil, a result of money changing hands and new, massive tracts of rainforest now opened up to development of agriculture. New forecasts and commodity projections now signaled a steep rise in consumption of dairy and cacao beans. And there was suddenly plenty of financing to back any such development projects. Many miles away in Brasilia, the leadership wringed its hands in anticipation of the backlash, which would erupt any day now. But the immense stresses were now offset by those new balances recently deposited in their numbered accounts offshore.

"What the hell is going on?" Howled Efram Brewster, CIA Deputy Director. His boss was nowhere to be found, carefully avoiding his duties as much as possible. But Efram, Ef as they called him, didn't have the burden of knowing for sure that things were very, very wrong. You see, unlike his boss, Ef was as clean as a whistle, and among a relatively small list of top officials over whom Zsa Zsa had no leverage. Unlike his boss, Ef had not received an envelope with a stack of photos of him with a young woman that later disappeared. Ef didn't have to decide whether to play ball or load his revolver and end it. So Ef came to his realizations the old fashioned way. He deduced them. Ef looked around the table, waiting for someone to say something, anything.

Bryce Harkshall, the youngest of the team and also the cleanest, next to Ef, broke the ice. "I have calls into TFT (Terrorist Financial Team). The only thing I heard back so far was that it can't be terrorist related, at least most of it. Way too much money involved here. Definitely state actors."

"Makes sense," interjected Ef. Let's just focus on the biggest transactions first. There is too much going on here. We will get overloaded unless we try to run down the top three or four." There was no name for the top secret AI program that had revealed a sudden avalanche of financial activity around the globe. Internally, it was known as CIA-AI, but they shortened it to CAI for convenience. "Who is our contact at CAI now that Bigelow is out of commission?"

"Basically, there is no one." Jeff Clark was the elder staffer among the group. He knew everyone and just about everything. He was always encouraged to put retirement out of his thoughts, as his tribal knowledge was an ever key ingredient to the functionality of the agency. "I brought back Hector to look into it. He seemed spooked. He gave me the data dump and said, 'don't call me anymore.' I think the situation scared the crap out of him."

"Yeah. Me too. And now we need that damn machine to help us go deeper on some of these transactions. Who can we turn to?"

"Bigelow's girlfriend. Sandy something," responded Clark.

"Sally Broyhill," corrected Bryce Harkshall. Being similar in age was what little he had in common with Bigelow Chan. But at least it had been worth a few awkward personal conversations.

"Christ, he's got a girlfriend? How did that happen?" Quipped Ef.

"She's in Boston as I recall," added Clark. "And she's got clearance. They use her as a contractor."

"Actually, she already went to Iceland to see him. He's still in a coma. I have been checking twice a day," Bryce replied."

"Bryce, you find her and try to enlist her help. Since she's already at the center, it might be good for her to work from there, more secure. Everyone else, take a transaction and dig into it as best you can. Meet back here to brief me at 1600."