





          
          
          
          
          NOT WITHOUT US
          
          A Challenge to Computer Professionals to Use Their Power to
          Bring the Present Insanity to a Halt
          
          
          by Joseph Weizenbaum, professor of computer science at MIT
          
          
          
          
          (This is an English translation of a talk given in German to
          the Association of Computer Professionals in West Germany in
          July 1986. You are welcome to reproduce and distribute it.)
          
          
          Whenever I come to Europe, especially to West Germany, I am
          amazed by the normality of everyday life: superhighways,
          "music" that assaults one in restaurants, the many parks,
          the forests of television antennas on the roofs of houses
          and so on. I am amazed because of Europe's geographic
          position and all that follows from it. In West Germany, for
          example, there is the border with the other Germany, dense
          with military installations of all sorts. There are holes in
          the street that are intended to be filled with nuclear land
          mines if Russian tanks should come. These are signs of
          Europe's physical and psychological proximity to the final
          catastrophe.
          
          We in America are, in a certain sense, no more distant from
          the catastrophe than the Europeans are. Not only Chernobyl,
          but also the threat of war is everywhere. And war is
          everyone's enemy. In case of war, regardless of whether
          unintentionally initiated by technology allegedly designed
          to avert war, or by so-called statesmen or women who thought
          it their duty to push the button, you may die ten minutes
          earlier than we in fortress America, but we shall all die.
          
          But we have no holes in our streets for atomic land mines
          that are intended to delay Soviet tank regiments. We see our
          missile silos only now and then--that is, only whenever it
          pleases someone to show them to us on television. No matter
          how passionately our government tries to convince us that
          the nasty Soviets are effectively as near to us as to
          Europeans, that they threaten us from Cuba and Nicaragua,
          Americans are, on the whole, quite unconvinced and
          untroubled by such efforts. The American experience of war
          has allowed us to develop an "it can't happen here"
          attitude, rather than a concrete fear of what appears to be
          far removed from the immediate concerns of daily life.






          Not Without Us                                          2
          
          
          
          We know that it is emotionally impossible for anyone to live
          for very long in the face of immediate threats to existence
          without bringing to bear psychological mechanisms that will
          exclude these dangers from consciousness, permitting them to
          surface only rarely. But when repression necessitates
          systematically misdirected efforts, or excludes potentially
          life-saving behavior, then it is time to replace it with a
          conscious effort to find the prod to correct action.
          
          That time has come for computer professionals. We now have
          the power radically to turn the state of the world in
          directions conducive to life.
          
          In order to gain the necessary courage--not all of us are
          saints or heroes--we have to understand that for us as
          individuals, as well as for those we love, our present
          behavior is far more dangerous, even life threatening, than
          what healthy common sense now demands of us. None of the
          weapons that today threaten every human being with murder,
          and whose design, manufacture and sale condemns countless
          people to starvation, could be developed without the earnest
          cooperation of computer professionals. Without us, the arms
          race, especially the qualitative arms race, cannot march
          another step.
          
          What does this say to us?
          
          First, that we computer experts--as well as specialists in
          many other technical domains--share in the guilt of having
          brought about the present dangerous state of the world.
          Those among us who, perhaps without being aware of it,
          devote our talents and strengths to death rather than to
          life have little right to curse politicians, statesmen and
          women for not bringing us peace. It isn't enough to make
          pretty posters that can be carried in demonstrations. Those
          who carry them must care whether their daily work helps to
          make possible the very devices the use of which they are
          protesting.
          
          At this point, the domain called Artificial Intelligence
          (AI) comes especially to mind. Many of the technical tasks
          and problems in this subdiscipline of computer science
          stimulate the imagination and creativity of technically
          oriented workers particularly strongly. Goals like making a
          thinking being out of the computer, giving the computer the
          ability to understand spoken language, making it possible
          for the computer to see, offer nearly irresistible
          temptations to those among us who have not fully sublimated
          our playful sandbox fantasies, or who mean to satisfy our
          delusions of omnipotence on the computer stage. Such tasks






          Not Without Us                                          3
          
          
          
          are extraordinarily demanding and interesting. Robert
          Oppenheimer called them sweet. Besides, research projects in
          these areas are generously funded. The required moneys
          usually come out of the coffers of the military, at least in
          America.
          
          It is enormously tempting and, in Artificial Intelligence
          work, seductively simple to lose or hide oneself in details,
          in subproblems and their subproblems and so on. The actual
          problems on which one works--and which are so generously
          supported--are disguised and transformed until their
          representations are mere fables: harmless, innocent, lovely
          fairy tales.
          
          Here is an example. A doctoral student characterized his
          projected dissertation task as follows. A child, six or
          seven years old, sits in front of a computer display that
          shows a kitten and a bear, in full color. The kitten is
          playing with a ball. The child speaks to the computer
          system: "The bear should say 'thank you' when someone gives
          him something." The system responds in a synthetic, but
          nevertheless pleasing voice: "Thank you, I understand." Then
          the child again: "Kitty, give your ball to your friend."
          Immediately we see the kitten on the computer display throw
          the ball to the bear. Then we hear the bear say: "Thank you,
          my dear kitten."
          
          This is the kernel of what the system, development of which
          is to constitute the student's doctoral work, is to
          accomplish. Seen from a technical point of view, the system
          is to understand spoken instructions--that alone is not
          simple--and translate them into a computer program which it
          is then to integrate seamlessly into its own computational
          structure. Not at all trivial, and beyond that, quite
          touching.
          
          Now a translation to reality. A fighter pilot is addressed
          by his pilot's assistant system: "Sir, I see an enemy tank
          column below. Your orders, please." The pilot: "When you see
          something like that, don't bother me, destroy the bastards
          and record the action. That's all." The system answers:
          "Yes, sir!" and the plane's rockets fly earthward.
          
          This pilot's assistant system is one of three weapons
          systems that are expressly described, mainly as a problem
          for artificial intelligence, in the Strategic Computing
          Initiative, a new major research and development program of
          the American military. Over $600,000,000 are to be spent on
          this program in the next four or five years.






          Not Without Us                                          4
          
          
          
          It isn't my intention to assail or revile military systems
          at this point. I intend this example from the actual
          practice of academic artificial intelligence research in
          America to illustrate the euphemistic linguistic
          dissimulation whose effect it is to hinder thought and,
          ultimately, to still conscience.
          
          I don't know whether it is especially computer science or
          its subdiscipline Artificial Intelligence that has such an
          enormous affection for euphemism. We speak so readily of
          computer systems that understand, that see, decide, make
          judgments, and so on, without ourselves recognizing our own
          superficiality and immeasurable naivete with respect to
          these concepts. We anesthetize our ability to evaluate the
          quality of our work and, what is more important, to identify
          and become conscious of its end use.
          
          The student mentioned above imagines his work to be about
          computer games for children, involving perhaps toy kittens,
          bears and balls. Its actual and intended end use will
          probably mean that some day a young man, quite like the
          student himself--someone with parents and possibly a girl
          friend--will be set afire by an exploding missile sent his
          way by a system shaped by his own research. The
          psychological distance between the student's conception of
          his work and its actual implications is astronomic. It is
          precisely that enormous distance that makes it possible not
          to know and not to ask if one is doing sensible work or
          contributing to the greater efficiency of murderous devices.
          
          One can't escape this state without asking, again and again:
          "What do I actually do? What is the final application and
          use of my work? Am I content or ashamed to have contributed
          to this use?"
          
          I am reminded in this context of a well known American
          journalist who, during a Middle East highjacking, suggested
          that under certain circumstances the Israelis shoot ten Arab
          prisoners and, should the circumstances not change, shoot
          ten more the next day, and so on. He should not have made
          this suggestion unless he was prepared to go personally
          among the prisoners and look into the eyes of the men, some
          of whom would hear him say, "You, you will die today." He
          should have been prepared as well to hold the pistol to the
          heads of those he selected and command his own finger to
          pull the trigger.
          
          Just so should we ask ourselves about our own work. Once we
          have abandoned the prettifying of our language, we can begin
          to speak among ourselves realistically and in earnest about






          Not Without Us                                          5
          
          
          
          our work as computer professionals.
          
          "You, colleague of many years, you are working on a machine
          consisting of two to the fifteenth and more microprocessors
          running simultaneously. With the help of such a machine one
          can first simulate then construct much more efficient,
          smaller and lighter hydrogen bombs. Imagine, for a moment,
          you were an eyewitness at Hiroshima in 1945; you saw people
          stripped of their skin die. Would you want to make this
          happen thousands of times more? Would you so torture a
          single human being with your own hands? If you would not,
          regardless of what end would be served, then you must stop
          your work."
          
          One should ask similar questions with respect to other
          branches of computer science, for example, with respect to
          attempts to make it possible for computer systems to see.
          Progress in this domain will be used to steer missiles like
          the Cruise and Pershing ever more precisely to their
          targets, where murder will be committed.
          
          Many will argue that the computer is merely a tool. As such
          it can be used for good or evil. In and of itself, it is
          value free. Scientists and technicians cannot know how the
          products of their work will be applied, whether they will
          find a good or an evil use. Hence scientists and technicians
          cannot be held responsible for the final application of
          their work.
          
          That point of view is manifested in the world famous Draper
          Laboratory, next door to the MIT building where I work.
          Draper is devoted almost entirely to missile guidance and
          submarine navigation. Many of the scientists employed there
          argue that the systems they work on can take men to the moon
          and bring them back, as well as guarantee that missiles
          aimed at Moscow will actually hit Moscow, their target. They
          cannot know in advance, they say, which of these two or
          still other goals their work will serve in the end. How then
          can they be held responsible for all the possible
          consequences of their work?
          
          So it is, on the whole, with computer professionals. The
          doctoral student I mentioned, who wishes to be able to
          converse with his computer display, does in fact believe
          that future applications of his work will be exclusively in
          innocent applications like children's games. Perhaps his
          research is not sponsored by the Pentagon's Strategic
          Computing Initiative; perhaps he never even heard of SCI.
          How then can he be held responsible if his work is put to
          anti-human use?






          Not Without Us                                          6
          
          
          
          Here is where we come to the essence of the matter. Today we
          know with virtual certainty that every scientific and
          technical result will, if at all possible, be put to use in
          military systems.
          
          The computer, together with the history of its development,
          is perhaps the key example. But we should also think in this
          connection of everything that has to do with flight, or of
          things atomic, of communications systems, satellites, space
          ships, and most of the scientific achievements of the human
          genius. We may then convince ourselves that in the concrete
          world in which we live, the burden of proof rests with those
          who assert that a specific new development is immune from
          the greed of the military.
          
          In these circumstances, scientific and technical workers
          cannot escape their responsibility to inquire about the end
          use of their work. They must then decide, once they know to
          what end it will be used, whether or not they would serve
          these ends with their own hands.
          
          I don't believe the military, in and of itself, to be an
          evil. Nor would I assert that the fact that a specific
          technology that has been adopted by the military is, on that
          ground alone, an evil. In the present state of the evolution
          of the sovereign nation-state--in other words, in the insane
          asylum in which we live--each state needs a military just as
          every city needs a fire department. But no one pleads for a
          fire station on every corner, and no one wishes for a city
          fire department that makes a side business of committing
          arson in the villages adjacent to the city.
          
          But we see our entire world, particularly its universities
          and science and engineering facilities, being more
          profoundly militarized every day. "Little" wars burn in
          almost every part of the earth. (They serve, in part, to
          test the high tech weapons of the "more advanced nations.")
          More than half of all the earth's scientists and engineers
          work more or less directly in military institutions, or in
          institutions supported by the military. That is an evil that
          must be resisted.
          
          We must also recognize that it is only our already
          internalized habit of prettifying our language, in order not
          to arouse our conscience, that permits us to speak in terms
          of weapons and weapons delivery systems at all, when we are,
          in fact, discussing atomic explosives and hydrogen bombs.
          Those aren't weapons, they are mass murder machines and mass
          murder machine delivery systems. That is how we should speak
          of them: clearly, distinctly, and without evasion. Once we






          Not Without Us                                          7
          
          
          
          recognize that a nuclear mass murder machine is nothing
          other than an instant Auschwitz--without railroads or
          Eichmanns or Dr. Mengele, but an Auschwitz just the same--
          can we continue then to work on systems that steer these
          devices to living cities?
          
          That is the question I ask. Each of us must earnestly ask
          ourselves such questions and deeply consider the responses
          we find in ourselves. Our answers must finally manifest
          themselves in our actions--concretely, in what we do every
          day.
          
          Probably the most pandemic mental illness of our time is the
          almost universally held belief that the individual is
          powerless. This self-fulfilling delusion will surely be
          offered as a counter argument to my theses. I demand, do I
          not, that a whole profession refuse to participate in the
          murderous insanity of our time. "That cannot be effective,"
          I can already hear it said," That is plainly impossible.
          After all, if I don't do it, someone else will."
          
          First, and on the most elementary level, "If I don't do it,
          someone else will" cannot serve as a basis of moral
          behavior. Every crime imaginable can be justified with those
          words. For example: If I don't steal the sleeping drunk's
          money, someone else will. But it is not at all trivial to
          ask after the meaning of effectiveness in the present
          context. Surely, effectiveness is not a binary matter, an
          either/or matter. To be sure, if what I say here were to
          induce a strike on the part of all scientists with respect
          to weapons work, that would have to be counted as effective.
          But there are many much more modest measures of
          effectiveness.
          
          I think it was George Orwell who once wrote, "The highest
          duty of intellectuals in these times is to speak the
          simplest truths in the simplest possible words." For me that
          means, first of all, to articulate the absurdity of our work
          in my actions, my writings and with my voice. I hope thereby
          to stir my students, my colleagues, everyone to whom I can
          speak directly. I hope to encourage those who have already
          begun to think similarly, and to be encouraged by them, and
          possibly rouse others out of their slumber. Courage, like
          fear is catching.
          
          Even the most modest success in such attempts has to be
          counted as effective. Beyond that, in speaking as I do, I
          put what I discuss here on the public agenda and contribute
          to its legitimation. These are modest goals that can surely
          be reached.






          Not Without Us                                          8
          
          
          
          But, finally, I want to address such larger goals as, for
          example:
          
               -- Ridding the world of nuclear mass murder devices and
                  perhaps also of nuclear power generators.
          
               -- So reordering the world that it becomes impossible
                  ever again to convince workers of one country that
                  it is a necessity of life that they feed their
                  families on the flesh and the blood and the tears of
                  people of other countries. (That is, unfortunately,
                  the fate of many workers today, and not only those
                  who earn their daily bread in armaments factories,
                  but equally those of us whose daily work is to
                  sharpen high tech weapons.)
          
               -- So reordering the world that every human being has
                  available to him or herself all material goods
                  necessary for living in dignity. (I have often heard
                  well-meaning people say that, if we apply
                  technology, especially computer and communications
                  technology wisely, we may reach this goal in perhaps
                  50 to 100 years. But we can reach it sooner, and
                  without waiting for technological advances. For the
                  obstacle is not the absence of technology, it is the
                  absence of political will.)
          
          I once heard Elie Wiesel say: "We must believe the
          impossible is possible." I understood that in two different
          ways. First, had we been able to believe that "the land of
          the poets and the thinkers" could give birth to human
          extermination factories, we might not have had to experience
          Bergen Belsen. The impossible horror proved possible and
          became reality.
          
          But there is a more hopeful interpretation. It seemed
          impossible in the America of only 150 years ago ever to
          abolish the slavery of the black people. The entire economy
          of America's south was built on cotton. Cotton could neither
          be planted nor harvested, it was believed, without the
          unpaid toil of thousands of human beings out of whose
          wretchedness the plantation master could squeeze his profit.
          Nevertheless, at first only a few farseeing men and women,
          dreamers all, in Massachusetts, later many more citizens,
          came to believe the impossible was possible, that the slaves
          could be freed and slavery ended.
          
          The impossible goals I mention here are possible, just as it
          is possible that we will destroy the human race. I alone can
          neither achieve the one nor prevent the other. But neither






          Not Without Us                                          9
          
          
          
          can it be done without me, without us.
          
          I have no right to demand anything from my colleagues. But
          they must know that we have the power either to increase the
          efficiency of the mass murder instruments we have and
          thereby make the murder of our children more likely, or to
          bring the present insanity to a halt, so that we and our
          children have a chance to live in human dignity.
          
          Let us think about what we actually accomplish in our work,
          about how it will be used, and whether we are in the service
          of life or death.

