Self improvement versus creating a non-eudaemonic dystopia

I recently read Nick Bostrom‘s paper on the future of human evolution. The paper was published in 2004/5 and his views correlate with mine quite well. I am pleased to note that I am only about two to three years behind the times in having formulated my thoughts on the issue at hand. Ha! Not bad for an amateur. Moving forward…

Reading Bostrom’s paper was fascinating. In essence he makes the point that continuing to increase fitness will result in a dystopian world when measured with present human values and I agree. From the perspective of a present day human the evolution towards non-eudaemonic agents as Bostrom puts it seems like a scenario one has evolved to dislike. Since we have evolved to regard as good what has increased fitness in our ancestors we would have to fail to see anything unrecognizable human as a desirable future state. But is the deep desire to improve oneself not just as well part of human nature? Where but to something posthuman shall such self improvement lead if we for ever regard what is desirable from our current perspective?

Self improvement can be seen as a series of gradual changes. Consider the following scenario. A person approaches the matter of self improvement in a way to ensure that every improved following version of his self will be desirable from the unimproved version’s point of view. How desirable will the 100th improvement look from the point of view of the original? How about the 1 millionth? No matter at what improvement step the original will draw the line – at some point the improved version will turn into something that is unrecognizable, incomprehensible yes even scary to the original.

How do you picture the encounter between an early rodent – one of our direct ancestors a few 10 million years ago – and a modern day human. The rodent would probably flee in panic and some humans likely as well. But would the rodent lament over the sad abandonment of gnawing on stones? After all it is enjoyable and keeps ones teeth in shape. Or would it – having the full understanding of a human being – appreciate that other concepts, worries and habits are what a human holds dear in modern times? Which perspective take priority? “Of cause the human one!” is what one would expect from the anthropic chauvinists’ camp . But would the one millionth improved version as discussed earlier not argue the same for its manifestation?

Reconciling the desire to satisfy the ever changing current representation of an individual with the desire for self improvement and the implications for the future of human evolution becomes the challenge that needs to be addressed. Bostrom does so by suggesting what he calls a Singleton – an entity policing continued human evolution to maintain the status quo.

In the context of my friendly AI theory I suggest a similar approach to Bostrom’s Singleton however honoring Ben Goertzel‘s ‘voluntary, joyous, growth’ concept and thus allowing for the possibility of continuous self improvement.

Specifically I argue for a friendly AI to

A) change the environment(s) humans are in to increase an individual’s fitness as opposed to changing the genetic/memetic makeup of and individual to adopt it better to it’s environment.

B) reconcile our desire for self improvement with the problematic results discussed above by making growth optional as well as rewarding.

Comments (2)

On benevolence and friendly AI theory

Jame5 it not just a science fiction novel – it is a science fiction novel with a cause. Ensuring the creation of a friendly AI is hard for many reasons:

  • Creating an AGI is hard
  • Goal retention is hard
  • Recursive self improvement is hard

The question what friendliness means however existed before all of those problems, is a separate one and needs to be answered before the creation of a friendly AI can be attempted. Coherent Extrapolated Volition in short CEV is Eliezer S. Yudkowsky’s take on Friendliness.

While CEV is great to describe what a friendly AIG will do, my critique of CEV is that it postpones answering the question of what friendliness is specifically until after we have an AIG that will answer that question for us.

Yes – a successfully implemented friendly AGI will do ‘good’ stuff and act in our ‘best interest’. But what is good and what is our best interest? In Jame5 I provide a different solution to the friendliness issue and suggest to skip right to the end of chapter 9 for anyone who would like to get right to the meat.

In addition I have summarized my core friendliness concepts in a paper called ‘Benevolence – a Materialist Philosophy of Goodness‘ (2007/11/09 UPDATE: latest version here) and in the end formulate the following friendly AGI supergoal:


  • Suffering: negative subjective experience equivalent to the subjective departure from an individual’s model of optimal fitness state as encoded in its genome/memome
  • Growth: absolute increase in individual’s fitness
  • Joy: positive subjective experience equivalent to the subjective contribution to moving closer towards an individual’s model of optimal fitness state as encoded in its genome/memome

Derived friendly AGI super goal: “Minimize all involuntary human suffering, direct all
unavoidable suffering towards growth, and reward all voluntary suffering contributing
to an individual’s growth with an equal or greater amount of joy.”

Comments (3)