Author: Bushnell, Nolan
August, 1996

WHIMSY and fun are often the precursors to powerful tools that are used later for more serious
applications. A project at inception might not be useful because of lack of infrastructure or
incomplete refinements. The automobile and airplane were considered toys before they were
reliable and safe. Now they represent a dominant force in the transportation business. The
computer game has filled a similar role in being the incubator for many innovations that drive the
usefulness of the computer. This article will describe this role-developing a set of technologies,
interaction styles, and social transformation that contributes to the computer field. We begin with a
brief historical discussion and then consider some future possibilities.

The following technologies started or at least were popularized with the computer game business:

* Interactive storylines

* Collaborative computing

* Anthropomorphism

* Widespread use of raster scan monitors

* Sprites

* Real-time graphics

* Graphical user interfaces

* Three-dimensional graphics

* Publicly available computing resources

* Trackballs, joysticks

* Sound feedback

Computer-generated Imagery

We have come far in the relationship between man and computer. To put it into context, let's
remember 1971 and the advent of Computer Space (the first video game, not Pong). Monitors
were virtually unknown in the computer business other than in some laboratories with vector
displays. The interface of choice was punched cards, or paper tape from mechanical teletype
machines, or from output of large line printers.

IN the early days of computer games the television was inextricably linked to the idea of presented
electronic displays. In fact there was no other source than television broadcasts for electronically
displayed imagery. When the computer game industry was being invented it was much cheaper to
create computer games using actual televisions from which the tuner would be striped out rather
than buying CRTs. The early games used standard production model television sets that were
modified. At Atari we generated dumpsters of scrap television tuners that were stripped out of the
sets. It was not until the mid-1970s that actual CRT monitors designed for displaying
computer-generated data and images were available to the industry. People in the bars and
restaurants that were playing the games slowly got used to the thought that a signal could be
generated locally (in the beginning it was common for people to ask how the TV station knew
what the knobs on the machine were doing) since in most minds the TV and the TV station were
totally linked. Later, that concept was further enhanced with the local modulator that allowed the
home game to be connected to the living room television set. Could the Apple II be as successful if
it had been bundled with its own expensive monitor?

Computers in the Entertainment Field

The game industry, like the business industry, moved from mechanical to electronics to computers.
The early coin-operated video games were not really computers at all, but were state machines
that used TTL logic and counters. Remember that the first microprocessor was not yet invented in
1971. The calculations that were necessary to play a game were not possible at an acceptable
price at that time. It seems quaint now, but there was actually a four-bit TTL counter for every
scoring digit that appeared on the screen.

The Introduction of Scenario and Storyline

Video games further conditioned players in real-time sequence manipulation and other activities
that had only before been touched on in sports. The game had an ability to simplify and make
repeatable processes and skills that trained a whole generation into seeking and finding hidden
goals, processes, and systems.

The rise of Nintendo and the Japanization of the game business that occurred in the late 1980s
created perhaps the most extensive import of foreign culture that has happened in America other
than through immigration. A large portion of the games we play are scripted, designed, and
authored in the orient. It is generally viewed as a national strength that the U.S. exports its culture
to the world through its movie and television production capabilities. The converse fact is that
many distinctly Asian concepts have entered the world culture through video games (the Ninja
Turtles and other distinctive oriental fighting games, for example). The video-game business
affected many facets of the computer revolution, many of which have changed the basics of
human-computer relationships. They will also continue to change other aspects as networks
become an increasing part of game play.

The Game Industry Invented User Interfaces

Many user interfaces were developed for the game business. The user interface demands of games
are undeniable. From the leisure/entertainment definition games must be easy to interact with and
learn. Particularly in the coin-operated sector, which was unprotected from the crutch of training or
user manuals, it was simply a fact that a confusing user interface failed to attract enough quarters
from customers for monetary success, and that good user communication resulted in more income
for the machine. Controls and placement of controls further enhanced the body of knowledge of
how people and their machines interact.

The use of a non-keyboard interface was perfected in the game industry. The dependence on
direct manipulation of on-screen programming, and the use of anthropomorphic characters to
create a sense of relationship started in the game industry. Steve Mayer talked about sprites in the
early 1970s as a way to create cheap images-we called them stamps. In the 1970s memory was
very expensive (both ROM and RAM) so anything that allowed you to save bits was a good idea.

The microworld in which users immersed themselves in an "alternative reality" was developed into
an art form in the game industry. Who can forget the experience in "Colossal Cave" on the early
game machines?

Technology Expectations Created by the Game Industry

Technology expectations have driven many systems. Would people have embraced automatic
teller machines as quickly without prior knowledge of games' touch screens or buttons that affect
outcomes on the screen? Computer-generated images have also been made commonplace in the
everyday world. Today it seems a normal part of the entertainment world to see characters and
whole worlds that started in games. Software and hardware innovation structures such as sprites,
blitter objects, and certain aspects of direct memory access were all created because of the intense
graphics requirements for games. Many new forms of games were made possible--goal-directed
behavior has really been enabled in powerful ways. Is it possible that one must know the laws of
physics in order to "save the princess"? Perhaps this metaphor holds the keys for many new
training, business, and communication structures.

The Game Industry Created the Consumer Computer Industry

Economics of scale can be a major factor in the diffusion of a technology into a society. The game
business used so much silicon that it funded huge amounts of foundry building and process
refinements. Atari was the largest consumer of N-channel LSI for most of the 1970s and early
1980s. Its consumption of ROM and RAM created powerful incentives for manufacturers to
produce memory chips in previously unheard-of quantities and prices. Without the lower prices
these high volumes supported, the personal computer would have been priced several thousand
dollars more per unit. Monitors would have been significantly more costly without the increased
volume of the game business.

The Computer Game Industry's Anti-educational Underbelly

The social dynamics of games have been not all positive. Games by their nature represent
challenge and conflict. The rules of sports and war all have winners and losers. The very existence
of a score implies challenge. The concept of win-win is a much more difficult concept of
communication. The increasing appetite for violence in movies and television is perhaps a result of
graphic, detailed conflict from the game-playing experiences that each child has. Can some of the
more useful concepts of life such as cooperation, charity, and shared goals be as important in
future games as conflict? It is probably extremely difficult but worth a try.

Where Will the Computer Game Industry Take Us?

As with any genera, the game industry is not static; possibly more than any other genera, it
contributes to and provides momentum for the technology of the future. We need to think of where
this can take us in social interaction, entertainment, education, and in our work.

Any discipline that has grown to be as big a part of our lives as software needs to develop
categories to help in the understanding as well as the creation of the products. Let's consider a
classification system for software that focuses on the environment of the user and the way a person
reacts with the software.

Deskware. We are aware of this class since it tends to be the dominant software. It is meant to be
viewed approximately from 18 inches away and is manipulated with a keyboard and pointing
device. It is expected that one can read large amounts of text if necessary and modify text as well
as manipulate small objects on the screen. This medium presupposes a one-on-one relationship of
human to computer. It also presupposes a hard flat surface upon which to place and use the
control devices.

Couchware. This is what is put on the television set in the family room and will become more
important as the television becomes a two-way communication device. It is meant to be viewed
from eight feet away, plus or minus four feet. The control for couchware is always supposed to be
hand-held in free space and can be wired or remote. The best examples are videogame controllers
or wireless remote control units. I believe the dominant pointing device for couchware will be much
like today's remote control with some new functions. Couchware cannot presuppose a great deal
of reading or text manipulation and small object control is out of the question. Scrolling and
"detent" access to buttons will dominate.

Kitchenware. This piece of software is in fact the nerve center of the family. It is viewed from 18
inches but it is presumed the user is standing and the display screen is mounted on the wall. The
user interface is in all cases a touch screen. The information is all button-accessed, and has to be
instantaneous and very simple. This software has to actually be used by family members as young
as four years old, even if a chair must be used to reach the device.

Bedroomware and Bathroomware are very close in function and form factor to kitchenware, the
focus being scheduling and communications. The homecontrol features are all accessible and for
bathroomware games are also very important. There are those that presuppose the bedroom has a
TV and that modified couchware is the operative mode. Location certitude is a necessary feature
of all these and a remote control that has any kind of pointing feature is sure to lack that certainty
of location.

Autoware is very different if thinking is based on one's assumptions about its use for the driver or
passenger. In fact many believe there should be no visual software for automobile drivers. I believe
that for the driver the visual interface has been in place for years, just not in a computer GUI
mode. Clearly the dashboard and instrument cluster should have been using computer screens
years ago-I can remember looking at the cost-benefit tradeoffs for electronic dashboards 10 years
ago and being frustrated because of the unwillingness of the automobile manufacturers to change.
We did the analysis for the automobile navigation company ETAK and hoped we would be in the
software business selling maps and travel guides to the Autoware interface that was in all new cars.

Storeware is the stuff of the electronic checkout counters and product information material. Some
of its unique features are language flexibility and extremely high graphic content. Touch-screen
input and ruggedized construction of other controls are also presumed.

Bankware. We have all seen the terrible job that the automatic teller machines have done in
making the process of bank transactions about following instructions to press keys. Clearly new
software is needed.

Gameware. The coin-operated video game and the user interface was in some ways the first user
interface used by ordinary people. We learned many things that I think are true today more than
ever in all forms of software. One: People will not and hate to read instructions. Two: If people
cannot get up the learning curve in 15 seconds they will not spend the second quarter. Three: If
you must give instructions make them short, direct, and in as large a font as possible.

With the game Pong over 20 years ago the instructions we used were very simple-there are those
that think they could be a metaphor for life. The guidelines for use were: Insert Quarter.

Ball Will Serve Automatically.

Avoid Missing Ball for High Score.

Is there a piece of game software that when played well will create a business plan for the player?
Is there a game that trains a sales force on a newly released product? Are there games that can
become the answer to declining scores and capabilities of the nation's school children?
Goal-directed software (games) may have many uses that are just beginning to be understood and
that may be the key to surviving in the explosive area of communication requirements in which an
individual in an increasingly complex world is swimming in a sea of undigested information.

Electronic Pets

Several years ago I started a company to build and market electronic pets. It was and still is my
feeling that a good electronic pet properly designed would be significantly better than the flesh and
blood kind. The people I present this idea to are almost uniformly horrified at the thought that a
mechanical device could replace Rover, Mittens, or Spot.

Even when I describe releasing them from the obligatory trip to the pound it doesn't seem to help.
They wonder how could I create the cold wet nose. I reply that Mattel has marketed slime for
years and that small ultrasonic refrigeration works great and is very cheap. Then they ask if the pet
will come when called: of course, and with much better reliability than before.

Others think the real bond between owner and pet comes from care and feeding, creating a feeling
that it is dependent. To which I respond that it is a simple thing to replace the automatic charging
system with one in which the batteries are recharged in a charger and manually placed in the pet
dish. The pet loads the battery aboard while discharging another. When the first bank is totally
discharged it drops the battery on the carpet wherever it happens to be. The battery can then be
picked up with the super-duper battery scooper, just like now. Signs would appear in New York
City to battery curb your pet or risk receiving a ticket.

What about the safety aspects of a good guard dog? The sensor technology allows the robotic pet
to sense sounds at very low levels, and equipped with the Pit Bull JawsTM option can be quite
fearsome to intruders. An additional advantage is the lack of distraction from sides of beef or other
dogs in heat. This pet is on the job until a password is heard.

The real reason for an electronic pet is for the things the pet can do that the normal flesh and blood
(bio types) cannot do. Suppose a flight attendant for an airline wants to have a pet. It is very
difficult for that person to have one, the schedule is erratic and the long stretches away from home
make a pet for a single person almost impossible. With an electronic version, the pet can stand
guard for weeks at a time. Barking at any significant sound, making periodic inspections of the
other rooms to check for fire or a leaky faucet. If anything is detected, the pet can call the
appropriate agency with the location and nature of the problem.

Upon return the pet, in this case Max, greets the master by saying "Hi, Barbara, have I ever
missed you! Have you had a hard trip?" or other such caring and dependent message. To show
excitement, the mechanical animal will run around in small circles making short, high-pitched
beeping sounds.

The real fun with an electronic pet comes with the games. The easy games are the
question-and-answer games. These are structured with the pet asking the questions and the person
answering with yes or no. There are many scoring variations on this basic concept. The classic
hide-and-seek for small children is a lot of fun. Because the pet has very accurate sensing
mechanisms, there can be many games based on monitoring the motion and location of the players
by the pet. In the childhood game of red light/green light the players can attempt to sneak up on the
pet as long as the pet is green (if there are no lights there can be audible sounds). As soon as the
light turns red, the player must freeze until the light turns green again. The player that moves while
the red light is on is sent back to the beginning to start all over. The idea is to trade off the slow
and steady approach to game playing since the timid player is not likely to be moving if they move
little during any green cycle.

Other games such as 'Jump on the Monster" where the pet spins around and then declares that
there is an invisible monster six feet in front of him. The player then tries to estimate how far six feet
is and jump there as fast as possible. If the player is too slow or in the wrong place the monster
moves to another place and no score for jumping on it is given. A new form of musical chairs in
which there are arbitrary "chairs" defined on the floor. The pet names a person and they must
move to the new place on the floor. The pet keeps track of who is sitting where and starts calling
out names and locations that get more and more confusing until there are conflicting orders and the
game stops. There will be at least one person in the wrong spot.

The pet can be programmed to speak and understand only a foreign language and act as a great
tutor. The nightly bedtime story for the kids will quickly become part of the family ritual. All the
neat things that the pet can do will make the robotic pet an important part of our future.

The electronic pet's territory might comprise an electronically controlled house, offering many
possibilities for interactivity and cooperative efforts between the two.

Playing Games With Your House

Home control has many definitions. My definition is a house in which all the electrical outlets,
appliances, and lights have an address and are controlled by computer intelligence. Each room has
a sensor pod and stereo speakers. It can also have light switches at any point attached to any
surface with double-backed tape. The switches work with the control computer to control a
specific function, though in most cases they are not necessary. The sensor pods have infrared
motion detectors, microphones, light sensors, and temperature detectors.

Using the power of the computer, the house soon learns that based on the time and other factors it
turns on the lights in rooms that people are in and turns out the lights in rooms that are empty. It
listens for commands and knows who the person in the room is through "signature characteristics"
so that when the phone rings, the house simply asks the specific person if the call should be
answered. The call then becomes a speaker phone conversation unless the handset in the room is
lifted. These are just some of the interesting thing smart houses can do, along with all sorts of
interesting energy management tasks.

PLAYING games with your house is fun. Some of the simple games are based on the fact that the
house knows where you are. It can choose a location in the house at random and then give you
"colder" or "warmer" hints to guide you to the right spot, the score being higher when the fewest
hints are needed to find the spot. The reverse game is one in which the house agrees to cover its
collective eyes while you hide. The house then prompts you to respond and only listens with its
stereo microphones in one room. The house's score is based on the smallest number of guesses it
takes to find you (provided the house doesn't cheat).

Another large class of games is word games that ask a series of yes or no questions about all sorts
of things. In the early stages the house will probably have a somewhat limited speaker-independent
vocabulary. Thus the yes/no restriction, which is removed in later years as the house's vocabulary
expands.

The house can be a great partier. Think of dancing to music that moves from room to room. It can
pull the party around the house in a rock-and-roll samba line and add an additional dimension. The
house as a musical instrument concept is also fun. The house could play a note whose pitch is a
function of the right/left position of the musician in the room and the amplitude a function of the
forward/back position in the room. Another algorithm could be for a clap to start a note and the
forward/back position be the duration of the note. People could be jamming in different rooms
with their voices being blended and augmented. Of course, the whole thing can be recorded for
later playback once all the musicians have had a rest. Who knows, it may be the aerobics of the
future.

If you haven't had a murder mystery dinner party yet, I highly recommend it. There are also several
interactive plays that have been produced. The best one, Tamara, played in New York several
years ago. The idea is that all the people at the party are characters in a story. Someone is
murdered and through questions and answers clues are learned about the people (characters) that
lead to a solution. It is a great time and having the house be the narrator and host would add a
great deal to the evening. With the house's ability to add sound effects, to adapt the scripts and to
direct the group to other rooms, the implications are marvelous.

For kids, the house can be a friend with counting games and imaginary friends that talk to the child
but can't be seen. The house can be used as a storyteller and a watcher to make sure the child
keeps away from the front door or workshop. The house becomes not just an inanimate container
but an active and interesting member of the family: reminding, assuring, entertaining, and playing
with the family.

The Virtual Room

The virtual room is an old concept that is almost reality, meaning that it is possible to build one if
you have enough money. The idea is to have a room that is totally changeable. All the walls would
be active flatscreen panels that would be operated by computer. The first ideas were for video
"wallpaper" in which the color and/or pattern could be changed to match the mood. With
multimedia, the thought of being able to create the feeling of being beside a brook in Yosemite, on
the seashore of Hawaii, on the streets of Manhattan, on the top of Mt. Everest, on the center grass
of the Kentucky Derby, or floating in space somewhere between the earth and moon becomes
possible.

The flat screen panels are much too costly, but I calculate that by using frosted plexiglass on the
ceiling and four walls and using some of the very good liquid crystal projectors that are made by
Sharp and Panasonic, the technology for the room could be had for under $20,000. This may
sound like a lot, and it is, but in the world of private interior decorators, it would be rated on the
low end of the scale. Like with so many other things, the rich will pay for the low-volume
engineering for things that in the future become available to everyone.

Imagine the hardware is installed by the end of the year in 10,000 homes. Where does the
software come from? How much does it cost? Can there be a business selling it? If you want to be
surrounded by stationary objects or graphics, the price could be very low, but if the idea is to have
surround sound and motion effects, the price could be quite high and complicated to produce.

You can also add smell quite cheaply by buying a multichambered smell creator. It consists of
individual chambers that contain aroma pellets that are used in the amusement industry. On
computer command, a small quantity of air is passed over the pellet and that air is then injected
into the air conditioning ducts.

Disney, with its circlerama technology, found that simultaneously filming in 360 degrees
represented an interesting challenge. Where does the cameraman stand? They solved the problem
by clustering the cameras and shooting down with mirrors. They often shot with the pod being
suspended from a helicopter or crane. At other times the unit, which is rather ungainly, was
mounted on a truck and the people running it were located below. The software for the virtual
room will not have that advantage. Who knows, perhaps we have stumbled onto the first real
postulate of the virtual world. How does one record (observe) the world without affecting it?
Sounds a little like a non-subatomic version of Heisenberg's uncertainty principle, which states that
you cannot observe any phenomenon without affecting it.

The proper virtual room should have a small room next to it to store the different furniture or props
that may be needed to round out and complete the experience. The couch on which one would
like to fly through space is very different than the one used to view the sunset over the Golden
Gate Bridge.

While the virtual room as a living space is interesting and visionary, the virtual room in interactive
mode comes alive with possibilities. Some of the best work in this area has been done by Myron
Krueger and chronicled in his book Artificial Reality. I highly recommend this book: it offers some
fascinating ways to combine people images and sensors. To give you an idea of how
forward-thinking he was, his work was done nearly 15 years ago using mainframe computers and
links to process the sensor data and create some of the images in wireframe. The whole thing
could be run now on a simple 486-based machine but I always respect the real pioneers.

Imagine there are sensors in the room that detect your position in the room. Suppose further they
can tell where your head is. By modifying the images based on personal location some interesting
possibilities begin to unfold. As an example, the floor is considered solid as you look down but as
you walk closer to the edge of the room, the floor occludes less of the view below. Like walking
to the window in a skyscraper, you can only see the street below if you look down when you are
next to the window. If you ever want to believe you are standing on a platform in free space, just
go into a virtual room that has image modification based on head position. This exceeds the
perception in the best eye phone VR system I have seen to date.

Now let's fly to the room (platform) from the couch and, using joystick and computers, have a
great shared experience. Your associates can be part of the fun by being tailgunners, etc. Or if it's
driving the hook-and-ladder fire engine through the streets of Manhattan (I've always wanted to
drive the back wheels), two steering wheels can be used.

Of course mazes and dungeons and dragons are a "must have" for the room with the specter of the
huge malevolent dragon actually standing 30 feet above you. It might really be scary. Interactive
party games can be played in which several people enter the room, each person is sensed and
placed as an icon on one of the walls. The computer then creates the problem or contest. First,
each person needs to see how to control his or her character. It can be straight translation of
position on the floor to position on the screen. If you walk forward, your icon moves up. If you
walk to the right, your icon moves to the right. But it may be more complex such as your location
being the sum of your motion and someone else's or the difference. Krueger found that very
complex interactions of groups can be created and solved by groups in this type of facility.

It may happen that the virtual room at the turn of the century may be to parties what the hot tub
was in the 1970s but no matter what, it will be a great place to just hang out. 3