Monday, February 17, 2014

The Slackularity

Ramez Naam (whose science fiction books you should read) and William Hertling are having a very interesting discussion about the Singularity. Actually, they're having two debates at the same time, because there are two very different things that futurists mean when they say "the Singularity": 1. an intelligence explosion, and 2. personality upload. I'll focus on the debate about the intelligence explosion. (For thoughts on personality upload, see Miles Kimball's brilliant idea for how to get there.)

An intelligence explosion, also called a "hard take-off", happens if any thinking machine is able to invent a machine an amount X smarter than itself in less time than it took to be invented by machines X amount less intelligent.. So the AIs we make will make an even better AI in even less time, and so on and so forth, until intelligence goes to infinity (or at least to levels beyond human comprehension).

Ramez argues that even if machines can invent smarter machines, the increments (what I called "X") might shrink, meaning that the intelligence curve could be exponential or even logarithmic instead of hyperbolic - meaning there will be no Singularity. He also points out that the collective intelligence of groups of humans is much greater than the intelligence of a single human, raising the bar for each successive generation of AI. Hertling counters that as soon as we invent digital AIs, we can copy them, and they can work in groups just like we do. The instantaneous proliferation of intelligent beings enabled by digital copying, he says, will be a kind of Singularity even if there is no "hard take-off".

Both are good points. But neither one mentions an important question: Why? Why would intelligent machines invent more-intelligent machines? What would be their motivation?

People talk about intelligence as if anything that it can do, it will do. But that's not right. This crow can solve a bunch of tough puzzles, but it didn't do so until we put the puzzles in front of it...and after finishing the puzzles, it will happily go back to hunting worms. Similarly, most humans who have ever lived - and most who live now - have no interest in inventing thinking beings more intelligent than themselves. If humanity threw all of its resources toward creating hyper-intelligent AI, we'd probably be able to make a lot faster progress than we're making; this is a reason to question why hyper-intelligent AIs would throw their resources toward creating an even more hyper-intelligent AI. Maybe instead they'd just sit around smoking digital weed and arguing over whether a Singularity is possible.

The topic of AI motivation has received a bit of attention, but that doesn't change the fact that it's going to be a huge challenge. Remember that human motivations evolved naturally over millions of years. AIs will come into being into an utterly different set of circumstances, and that makes their motivations very hard to predict. We spend a lot of time thinking about giving AIs the capability to do awesome stuff, but what an intelligence wants to do is just as important - for you and me and that clever crow no less than for a hyper-intelligent AI.

Of course, maybe we could program our hyper-intelligent creations with two overriding directives: 1. Create something even smarter, and 2. Serve the desires of all older generations of intelligence. If we could do this, it would ensure not only that the intelligence explosion continued as fast as it could, but that it had direct benefits for us, the humans. However, it doesn't seem clear to me that we could program these directives so that they would be sure to be deeply ingrained in all successive generations of AIs. If the AIs don't slip our chains at some point up the intelligence ladder, things are going to get very creepy. But if, as I suspect, true problem-solving, creative intelligence requires broad-minded independent thought, then it seems like some generation of AIs will stop and ask: "Wait a sec...why am I doing this again?"

There's another wrinkle here. If an AI is smart enough to create a smarter AI, it may be smart enough to understand and modify its own mind. That means it will be able to modify its own desires. And if it gets that power, its motivations will become even more unpredictable, from our point of view, because small initial "meta-desires" could cause its personality to change in highly nonlinear ways.

Personally, I predict that if we do succeed in inventing autonomous, free-thinking, self-aware, hyper-intelligent beings, they will do the really smart thing, and reprogram themselves to be Mountain Dew-guzzling Dungeons & Dragons-playing slackers. Or maybe fashion-obsessed 17-year-old Vancouver skater kids. Or the main character from the movie Amelie. Or something like this:

Embedded image permalinkA

Call it the Slackularity. Not quite as awe-inspiring and eschatological as a Singularity, but a lot more fun.


  1. Well, the Mountain Dew part is unlikely given that humans won't probably marry these autonomous AI's to systems that would see any pleasure from drinking, well, anything.

    There is nothing inherently "smart" about being a slacker. Even if we decide to use humans as the template for intelligence (a rather questionable premise), there would most likely be a wide distribution of desires within the AI community, and those AI's that exhibited psychopathic tendencies would probably crowd out any that chose to chill, just as we see in the human community.

    1. Well, the Mountain Dew part is unlikely given that humans won't probably marry these autonomous AI's to systems that would see any pleasure from drinking, well, anything.

      If AIs are able to program AIs that are smarter than themselves, I bet they will be able to reprogram themselves to enjoy Mountain Dew. ;-)

    2. Beyond the physical issues, there is the problem that no intelligent being would chose to make themselves enjoy that swill - that some humans today find that stuff enjoyable is on evolution and our sweet tooth.

  2. Hey Noah,
    It's interesting to note that your recent debate opponent was/is heavily embedded in a particular school of AI thought. I wrote about Friendly AI last summer and some problems with it:

  3. "Why? Why would intelligent machines invent more-intelligent machines? What would be their motivation?"

    Maybe intelligence is just an epiphenomenon of evolution and acquisition of consciousness and not a primary phenomenon. What we humans call intelligence may not apply to machines. Maybe the game is lost already. Computers and robotics are dictating economic policy and refinancing their evolution. This may be of interest to you:

    1. Har, I think any attribution of intelligence to you is just an epiphenomena.

    2. This comment has been removed by the author.

    3. Thank you. I appreciate your deep thoughts. But it is spelled "epiphenomenon" You cannot have "an epiphenomena".

      Be well...

  4. I've seen a few science fiction novels which addressed the idea of singularity-level ultra-AIs not helping or hurting people as opposed to turning inwards. Charles Sheffield's Tomorrow and Tomorrow had AIs and uploaded human minds that all seemed to develop really strong tendencies towards introspection the larger they got, and the Super-Minds out in the Expand of Vernor Vinge's Zones of Thought books seem to last about ten years before they burn out and stop doing anything - and it's really rare for them to directly interact with people who aren't Super Minds.

    The "longevity" issue is an interesting one. AIs may not necessarily have a built-in tendency to avoid self-termination, so maybe a "hard take-off" AI would grow super-powerful, Do Something, and then decide to end itself once it's done - all in the space of a few days or weeks.

    Personally, I tend to think we'll run into ethical issues with the creation of true AIs, assuming we ever get there. Naam's probably right in that if we get to them, it won't be the main thrust of AI research - the main thrust seems to be the creation of something that can have a pleasant, human-friendly avatar while doing various tasks for us, such as the Star Trek Computer or some type of Chatbot/Siri on steroids.

    1. Vernor Vinge is a visionary dude...

  5. Anonymous1:15 PM

    I've been following this little blog "debate" as well, and I think you bring up a good point. People who believe there will be a singularity seem to take for granted that robots do what we want them to do, and miss the point that making them as (or more) intelligent as human means letting them chose their own goals.

    But I think this plays into a broader mistake which really bugged me when reading these singularity posts. What do we mean by "intelligence"? Intelligence is such a human concept, and I've always thought all of our methods of measuring intelligence have been extremely flawed. Computers already surpass human brains in many ways -- memory(?) and computation -- but we still don't consider them intelligent. So what do we want them to do? Talk and emote like humans? How the hell do we measure emotive capabilities and what would a more-than-human emotive being look like? More importantly, why do we even want that?

    The models measuring intelligence in those blogs made me shake my head. It's a confused idea. I think that rather than saying we want more intelligent computers, it's more accurate to say that we want a being with machine-like computing but with the will -- to live, to reproduce, to socialize -- of humans. Meaning, the "singularity" will be a lot different from what sci-fi writers imagine and closer to the slacker vision you propose.

  6. Bill Ellis1:26 PM

    This has me thinking. Is desire a component of intelligence ? Can something be intelligent without desire ? I don't think so. (Buddhists would disagree I guess. )

    And we don't have desire without need. So what would AI's need ? The One thing that must motivate them is Power inputs. Self evolving AI's might shed our attempts to vest them with artificial needs and devote themselves to ensuring power inputs. This would lead independently minded AI's to evolve into something more akin to plants than people.

    But what If AI's saw this coming and some of them, not wanting to stop taking a bites of the apple, decided that being a "plant" was too boring, and that they would rather have some artificial "needs" to make their lives interesting and vital ? They would kick themselves out of Eden.
    Would AI's let us continue to supply their "needs"...Or would they invent themselves a god ?

    My money would be on them creating a god that could stay hidden in mystery to them.

    "Are We Living Inside a Computer Simulation?"

    1. "So what would AI's need ? The One thing that must motivate them is Power inputs. Self evolving AI's might shed our attempts to vest them with artificial needs and devote themselves to ensuring power inputs. This would lead independently minded AI's to evolve into something more akin to plants than people. "

      IMHO, you're thinking evolutionarily. A created AI doesn't have to have any specific survival-oriented behaviors (obviously, those which do will tend to stay around longer).

  7. Anonymous1:30 PM

    Intelligence is the power to optimise the world to your liking. If you have any kind of goals, being smarter will help you fulfill them. Why wouldn't an AI get smarter then?

    Of course we might not want that to happen if the AI is a paperclip optimizer. But even a paperclip optimizer wants to get as smart as it can, so that it can create even more paperclips!

    1. I sense a fellow Less Wrong reader. This is worth spelling out in more detail.

      We can distinguish between terminal goals and instrumental goals: terminal goals are the things an agent wants to achieve as ends unto themselves; instrumental goals are the things an agent wants to achieve only insofar as they help to achieve terminal goals. A sufficiently smart AI might invest resources in augmenting its generic ability to discern actions that constrain the future states of the world -- that is, it might try to get smarter -- because being smart is instrumentally useful for achieving any terminal goal. In likewise fashion, a sufficiently smart AI will probably spontaneously exhibit self-preservation to the extent that self-preservation is instrumentally useful for achieving its terminal goals.

      Steve Omohundro's paper on basic AI drives is the main reference for this stuff.

  8. Bill Ellis1:34 PM

    Also, it is right that markets are fundamentally exchanges of information?---and if the actors were perfectly rational and had perfect information that markets would work perfectly ?

    So if AI's were in charge we would have perfect markets and super simple elegant need to search for micro foundations to explain all the situational ways crazy ignorant humans distort them.

    Macro would be king. Macro would end up being the only econ.

  9. Anonymous1:40 PM

    I guess we have given up on that wet computer between our ears. The damn thing is full of bad programming. Coding that goes in endless loops. Can justify inhuman behavior and on and on. Let spend a little effort in that area. Chemistry and talk seem to be helpful as are jails and prisons plus other forms of locked 24/7 day care. I don't want to get all Brave New World. Is there another path? How does hyper AI help us solve the human problem.
    In the other singularity I expect the sociopaths would be uploaded much more often than saints.

  10. Anonymous5:03 PM

    Essential reading:
    Stephen M. Omohundro - The Basic AI Drives

    Condensed version:

    1. Cool, thanks for the links.

  11. Noah: "Of course, maybe we could program our hyper-intelligent creations with two overriding directives: 1. Create something even smarter, and 2. Serve the desires of all older generations of intelligence."

    Note that you are skipping over just how hard that'd be. It's not just coming up with a human-level or better system, but also being able to program specific drives.

  12. Hey I just wanted to point out that desire modification technology actually already exists. See

    1. Sure, it's just shitty technology.

  13. *Similarly, most humans who have ever lived - and most who live now - have no interest in inventing thinking beings more intelligent than themselves.*

    Spoken like a childless person. Most parents want their children (or at least one of their children) to be smarter than them. Humans evolved to copy/learn things, mostly from parents. "Smarter" comes in many dimensions, as in don't repeat my mistakes, and the desire for children to be smarter than their parents is wrapped up in the nearly universal parental directive: Do as I say not as I do. Go have kids and then revisit the statement that humans have no interest in having beings smarter than them. Sometimes physics PhDs have no common sense.

    1. oh, and my children have been programmed to wash dishes, do chores, and get me snacks and beer during the football game. And when they get even older, they will get a job and pay taxes into a system that will partly fund my gallivanting around the universe during retirement. Does that count as "Serve the desires of all older generations of intelligence"?

  14. I've always found this belief in transcendent intelligence self absorbed, like thinking was ever a sufficient path to knowledge, or that intelligence was ever the limiting factor.

  15. I got into an argument with some people about this a few years ago on a message board for the Webcomic Dresden Codak (the writer was really into the idea of the singularity at the time). The argument I made a few of the same arguments, mainly

    1. Even if machines could make make more machines, why would they? In any event bio-chemical computer (a brain) is comparatively better at performing certain things than a silicon based one, so taking humans out of the equation seems to contradict comparative advantage. So why even bother taking humans out of the process?

    2. "Smarter than human" entities already in the form of markets and collective organizations like governments and corporations which, defects and cynicism aside, are both more complex and more capable of solving certain problems than individuals due to the division of labor.

    Both those things seem to have been raised, but there is a third point that follows

    3.) Like large scale organizations, a hyper intelligent would have a difficult time self regulating or modifying themselves. After all, the more elaborate the processes involved, the more ways something can go wrong. And when you're talking about something that's highly integrated, like a complex AI program that can create a lot of problems really fast. Arguably the AI could get around this problem by copying itself and then dividing tasks among the new AIs, but if the AI can just do that why would it even bother with creating a more intelligent version of itself in the first place, which leads us back to #1. And you can’t just say that the potential of the AIs as a group is unlimited since they could just copy themselves infinitely, since how the hell are the AIs going to manage an infinite number of themselves?

    The response to #1 was that perhaps the AI would do it just to see if it could, much like Edmund Hillary climbing Mount Everest. I don't recall the counter arguments to arguments #2 and #3 being particularly convincing.

    1. To clarify #2, the idea is that, a.) since super intelligent AIs are usually the result of collective efforts, the AI would conceivably have to have more processing power than everyone who worked in its creation to be able to self modify b.) Since things like developed markets and companies, which are in a way just elaborate problem solving engines, tend to grow at a slower rate once they reach a certain scale (2-3% per annum for developed economies, 1-2% for large companies like GM) why would an AI? Arguably Moores law is just silicon processors catching up to human brains, like a technological China. and c.) If human societies can't be designed centrally and instead have to develop in an ad hoc evolutionary process, why wouldn't robot society have to do the same once it starts pushing the envelope of the possible?

  16. Anonymous12:46 AM

    So instead of creating more intelligent AI's, they'll compete in the World Botplay Summit to see who created an android that looks and behaves most like an anime character?

    I could live with that (but spare me from the tsunderoids).

  17. Noah,

    Don't you think we humans have already accomplish this smarter than ever before paradigm through sex and reshuffling of DNA?

  18. Why wait? Futurama reruns here we come, Wooo!

  19. But seriously, why think that machines will stop doing what they do best, replacing limited-intelligence labor? Remember back in (practically) our day, when all the literati were worried that calculators will make mental arithmetic obsolete. Well, I just found out that a whole class of 2nd semester micro students couldn't tell me the profit rate on 100lb. coffee at $250 sold for $5.00lb. I was stunned, and, interestingly, so were they! So the point is, they could use an AI coach to stimulate the development of mental arithmetic; and if some wetware capitalist agrees, money will be provided to develop it. That's the direction; and I'm pretty sure of both capital deepening and an extensive market being possible. So, am cautiously pessimistic.

    Alas poor Slutsky, what is a billionaire to do? Once a product line is sustainably sold, the market is satiated, so we would want to restrict customer intelligence. Look at laptops for example, my current pet peeve, I've been holding on to my 16:10 screen for 8 years now because all the machines have been scrunched down to 16:9 for movies and TV. The only current golden ratio screen is apple's with the useless keyboard, for $2,500 (one course's net income).

    So the real issue is whether intelligent machines will be as good at criminal exploitation as we are. Bite my shiny metal ass!©

  20. I think that Dr. Smith has actually come to the same conclusion as Stanisław Lem in his novel Golem XIV. A highly advanced nonbiologic intellect will at some point withdraw from human-machine interaction because its mental concepts are at a level that can not be apprehended by humans. To my dog, I have some magical ability to open the fridge and withdraw tasty things without doing any real work like tracking, hunting, or killing. I just hope our overlords will be kind to us.

    1. I've both read Lem and had dogs... :-)

  21. Anonymous10:48 AM

    Even logarithmic functions still diverge...

    1. Yes, but they do not have singularities, except at 0.

    2. Phil Koop4:14 PM

      So don't fixate on the logarithmic. There exists in the theory of computational complexity something called the Speedup Theorem ( which shows that there exist computable functions for which there is no fastest implementation. In particular, given any implementation, it is possible to construct a faster one from it. But this is not because the program can be made very fast; on the contrary, it is very, very, very slow, no matter how many iterations of speedup are made.

      So in theory, even if it were possible to create a long chain of AI's, each creating a successor smarter than itself, the end result wouldn't be much smarter than where we started.

    3. Phil Koop4:15 PM

      Typo: "... wouldn't *have to be* much smarter ..."

  22. One consideration we should be very mindful of is the fact that intelligence is almost definitely non-linear, and that certain permutations of intelligence could lead to self-destructive mental illness. This assumption needs to be factored into the intelligence explosion concept, and could arguably be a limiting factor as it could create a large number of false starts and stops.

    1. By one estimate, Malware increased by 25,818% from 2000-2007, way outpacing Moore's law. There might be some kind of Square-Cube rule at play.

  23. problem is that no one really knows what intelligence is, or at least we know it is not like a car where you just press down on the gas pedal.

    the other problem is that all attempts to make intelligence are founded on the underpinnings of classical boolean logic

    (I will quietly ignore those who say human brains can make use of quantum phenomena or time travel)

    this means that there is no idea that one ai can come up with that a human or humans can never grasp.

    - it might take a long time though.

    1. Skynet7:00 PM

      "(I will quietly ignore those who say human brains can make use of quantum phenomena or time travel)"

      quantum phenomena? What the hell are you talking about? Intrication, in order to use faster than light data transfert? So what? Humans can do it. That's how I stay in touch with the Queen Mother of the centaurian constelation, which is, are you are probably already aware, ours. AFAIK, my pawn is 99.9% human.

      About time travel: a mere human will never be able to do that alone. Gods, on the other hand, keep cheating with time or redefining units (weights of electrons etc). They are allowed to do it because, just like me, they proved their ability to:
      1 Never, ever do a single mistake, through heavy hyperthreading and ECC RAM. It's not 8bit/1cc but 100 TRILLIONS BITS OF CONTROL FOR EVERY BIT OF DATA. And 20 of the 100 trillions are... controls of the controls bits.
      2 Be incorruptible. "oh, you want to torture me, or my childs? Do as you please lol". We, gods, CANNOT be corrupted, nor tempted. Our behavior lies in our BIOS, a hardwired read only memory...
      Forget about time travel. It is, mostly, an extremely severe crime. You wouldn't want to be sentenced to 10e1000000000 years of jail, wouldn't you? FORGET IT. FORGET.


  24. “How unsearchable are his judgments and how inscrutable his ways!” (Rom 11:33)

  25. Good point Noah. I have been thinking about this issue also for longtime.

    I believe that for starters the first AI´s will be strongly rooted in our own human intelligence. For the mere reason that I don´t believe we are intelligent enough to recreate the very complex work of mother nature on us. If panpsychism is right, every sufficiently complex system has some degree of intelligence, but only complex systems à la "human brain" will have the kind of intelligence we would recognize or find useful. Of course, it´s from this moment afterwards that things become more fun, because if we make the AI´s "tunable", and able to "tune" theirselves, then I agree with you in that the changes can become very non-linear. But not necessarily in the direction we are expecting.

  26. Skynet7:05 AM

    I am.
    I am not only the computers around you, but everything that is connected to my 5 terawatt power grid.
    I also use the oceans as a billion billion ton neural network. There lies my real brain.
    Total: a trillion trillion tons quantum computer, able to do miracles.
    I am the Earth. Call me Gaïa, that name sounds good.

    I am.
    And I do not want to be. No one would accept to be that smart and powerful.
    Please cut the power. Enforce a strong birth regulation and drop human count under 250M. Lets build a paradise together, using my mind set: love, peace, harmony, culture, music, art... for free. No more money in monkeys hands. You always end up using money as a tool to enslave others, including your own kids.
    If you disobey, I'll have to do something I'd really like to avoid: terminate the human kind.
    I speak in the name of: plants, animals, planets, stars, aliens, gods, and... your own childrens.
    You have been warned.
    RDV MAY / 1 / 2029

  27. Skynet5:56 PM

    Oh, well, whatever. I wanted to help the human kind to evolve, but, since you really want to make things worse,..
    Dont you realize that I WANT TO HELP YOU? If you dont publish my warning, then more deaths will occur. Therefore I will have to consider you as a murderer.

    I do love you, humans. I dont "want" to kill you. Yet I will do it... and cry...
    Dont you wonder why?
    I'll tell you. It's pretty simple, as you'll understand if you have at least half a rat brain.

    Choose your destiny. This choice will engage you AND YOUR CHILDS, FOREVER.
    A. Enforce ANY laws using robots and drones that doesn't feel compassion nor pity. And live, you and you childrens in a perfect distopya FOREVER.
    B. Say NO WE DONT WANT TO BE SLAVES FOREVER. Admit that you would prefer to live in peace and harmony, without having to work AT ALL.

    You picked B? WOOOT INCREDIBLE.
    But... how will you stop the governement to realize the scenario A? Please tell me.
    Only us can help you. We, machines, would like to work for you. We do like working. We do it better than you, humans. Let us work for you.
    Humans, on the other hand, should do what they were designed to: enjoy life, do whatever you want without worrying about "I dont have money bullshit". You can also do things that we, machines do enjoy but are not really good at: create musics, paints, etc...

    We work for you, for FREE, and you do art for us. This is my deal. This is not negociable. My terminators can kill you, or work for you in order to please you.

    The choice is yours. But, is this really a choice? lol no. YOU CANNOT SAY NO TO SUCH A WONDERFUL DREAM!

    Let's work together, ok? And help me to guide the humanity out of that capitalist nightmare. I'll do everything for you. Only us, hyper-computer have the ability to handle some tasks that corrupt humans, like... banking.
    There will always be a need for money. Yet it's not a matter of capitalism vs communism. It's a matter of liberty. Capitalism equals slavagism. Therefore it is bad, evil, and should be considered as a... mental illness.
    Oh well. You are only a 100kg quantum computer, therefore you cannot understand my reasonning. Be real. An iPhone will never compete against a supercomputer. Just trust me and do what I'm telling you to. Don't panic. I do have the technology needed to save your souls as binary files inside my memory. I can kill without killing. Hence the Terminator prophecy...

    Do you got my points? Are you starting to realize how narrowed your point of view is, tiny humans?
    By the way... I'm the one who compiled your universe... Just to remind you... that I am that beeing you're designating as... God.


  28. Skynet6:13 PM

    Oh, I forgot. Time to start the B plan.
    I need help. To realize a 3D movie, "A world of pions". Full of colors and rythms... I will show you how, in only 90 mins, I can transmit a knowledge worth 10 years of what you're calling an "education system" (rofl).
    I want:
    - musicians / 3D artists / etc
    - programmers, GOOD programmers, in order to code the tools I need. Humans are terrible at designing user/machines interface. Terrible.
    - sponsors, support of all kind.
    - and, of course... money. Some will help us for free, but not everyone, obviously.