From: rick@cs.arizona.edu (Rick Schlichting) Subject: Kahaner Report: Remarks on VR activities in Japan Date: 26 Oct 92 16:59:02 GMT [Dr. David Kahaner is a numerical analyst on sabbatical to the Office of Naval Research-Asia (ONR Asia) in Tokyo from NIST. The following is the professional opinion of David Kahaner and in no way has the blessing of the US Government or any agency of it. All information is dated and of limited life time. This disclaimer should be noted on ANY attribution.] [Copies of previous reports written by Kahaner can be obtained using anonymous FTP from host cs.arizona.edu, directory japan/kahaner.reports.] To: Distribution From: David K. Kahaner US Office of Naval Research Asia (From outside US): 23-17, 7-chome, Roppongi, Minato-ku, Tokyo 106 Japan (From within US): Unit 45002, APO AP 96337-0007 Tel: +81 3 3401-8924, Fax: +81 3 3403-9670 Email: kahaner@cs.titech.ac.jp Re: Remarks on VR activities in Japan 26 Oct 1992 This file is named "vr-10.92" ABSTRACT. A summary of recent Japanese activities in Virtual Reality (VR). I have written several reports on Virtual Reality (VR), Artificial Reality, and Tele-existence, see for example ["icat.92", 5 Aug 1992, "vr.991", 9 Oct 1991]. In these reports I mentioned that most of the hardware in use has been imported from the US, and that the first large scale commercial applications are thought to be for games. The current report is a combination of (mostly) newspaper and trade journal articles. Much of the text is puffy and has an obvious "spin". As far as I can tell, it describes technology that will already be well know to the VR research community. Nevertheless, I have combined and distributed it here to emphasize the following points. (1) Japanese researchers are developing VR hardware of their own, as well as developing interesting applications outside the game arena. (2) Japanese companies are becoming seriously interested in this technology and seem intent on making use of it as a problem solving tool. (3) Japanese companies see the VR market as potentially lucrative for hardware that will be developed in Japan, both unilaterally and as joint projects with US companies. (4) Japanese companies realize that the ability to operate devices intuitively by ordinary gesture-like motions can be exploited in many practical fields. They also realize that present man-machine interfaces could be made easier, even if these interfaces fall short of outright VR. ----------------- 1. APPLICATIONS CAD Systems (NEC) Gesture recognition (ATR Labs) Software analysis and development (Tokyo Electric Power) Construction Robots (Tokyu Construction, Musashi Inst of Technology, Fujita Construction) Power Plant Monitoring (Hitachi) Molecular Modelling 2. FORCE FEEDBACK EXPERIMENTS Iwata (Tsukuba) Sato (Tokyo Institute of Technology) 3. BUSINESS RELATED Nissho Electronics, Ltd. (Japan) agreement with VPL (US) Media Int. Corp. (MICO) (Japan) agreement with Telepresence Res (US) Japan Tech Transfer Assoc (JTTAS) sets us AR and Tele-Exis Res Committee -------------------------------------- 1. APPLICATIONS Virtual Reality applications to CAD Systems, Construction Robots, Software. From: Tokyo Nikkei Mechanical 20 Apr 92 pp 56-66. Researchers are making progress on the use of virtual reality (VR) in equipment operation. Unlike devices which perform two-dimensional movements like mice or joy-sticks, VR seeks to employ ordinary human motions and spatial movements in the operation of equipment. Nippon Electric Co., Ltd. (NEC) is doing research on using VR in a 3D CAD system, while Tokyo Electric Power Co., Ltd., is working on a system in which VR is used to analyze software. Tokyu Construction Co., Ltd., is seeking to adopt a VR-based system for the remote-control operation of construction robots. Fujita is doing research jointly with the U.S. company VPL on a VR application in operating construction robots via communications lines. VR technology uses Computer Graphics and three-dimensional images, etc., so as to make what does not actually exist then and there appear to the operator as though it does, so that the operator can use hand or finger movements to move an object around, or alter its shape, in virtual space. Progress is being made in research which aims to such objects inside virtual space in place of 3D CAD shape models or actual construction robots. A robot, of course, moves in three dimensions, and 3D CAD obviously uses three-dimensional space. VR enables all kinds of operations to be done via human movement. A "hand" or "arm" inside "space" is linked to the three-dimensional movements of the machine or model being manipulated, thereby facilitating intuitive comprehension of what is happening. Researchers hope to make this technology useful in handling 3D CAD and manipulating construction robots. Take CAD system input, for instance. If we are talking about entering a flat drawing, then a mouse is perfectly adequate. It is a different story, however, when it comes to entering a 3D object. You can move a mouse around easily on a flat surface, but you cannot move it around freely in 3D space. A mouse simply cannot give the operator a true sense that he or she is working with a three-dimensional representation. Research efforts are now being directed toward the use of VR in incorporating 3D movements and developing input interfaces which allow intuitive operation. Transforming 3D CAD Models With 'Data-Gloves' --------------------------- NEC's C&C System Research Laboratory has developed a VR system in which a 3D CAD shape model can be manipulated with a feeling of actually touching it with the hands. With this system, it is easy to perform operations in which it appears that a model is being manipulated as though it were made of clay. The shape of the model can be changed, it can be sculpted, and parts can be added or deleted using "one's own hands ('agents') appearing on the screen." The movement of these "agents" is linked to those of data-gloves which the operator actually wears. This gives the feeling of actually handling the model directly with one's hands. This feeling cannot be realized by manipulating a mouse. The input device used in the NEC system are data-gloves made by the US company VPL. This is a glove-shaped device that permits hand-position and finger-bend data to be input through the use of magnetic sensors and optical fiber. Also used is a system made by the US company Solidray Laboratory which liquid-crystal shutter glasses to produce a three-dimensional solid image. The 3D image is produced by changing between left-eye and right-eye images with the liquid-crystal shutters. The model, the hands that manipulate the model, and the operator's hand agents appear on the screen. Every operator wears data-gloves on both hands, so two hand agents are described on the screen for each operator. The model is gripped, is moved about, and its attitude is changed with the agents. When two terminals are linked together and joint manipulations are performed, four agents appear on the screen. There are triangular and rectangular icons at the top of the screen for calling functions to shift the perspective or change the color, etc. The general shape of the model is changed by means of a planar cursor. When the shapes of the hands wearing the data-gloves are altered, the agents are transformed from hand-shape to panel shape. This panel is the planar cursor. The planar cursor can be moved to any position or attitude by moving the hands. The model is cut or sculpted in this plane at suitable locations and thereby shaped. The planar cursor represents positions in three-dimensional space and also the inclination of the plane in that space, so it is very difficult to manipulate the planar cursor with a mouse that can only move in two dimensions. With the data-gloves, however, one need only tilt one's hands to put the model in the attitude in which it needs to be cut. The advantage is that manipulations can be made intuitively without learning any special procedures. When a yellow plate is touched by an agent, the "toolbox" is called up. The toolbox contains prefabricated shape models and parts. A shape model can be pulled out with the agent and compared with the model being designed, or a part can be added. Suppose, for instance, that you want to add tires to the model body of a car. "Three-dimensional computer graphics alone is not adequate for getting the right positions," says Shoji Kawagoe, a research section chief in the Terminal System Research Department of NEC's C&C System Research Laboratory. "You need something which gives you a 3D view." Here again, we are talking about incorporating a sense of intuitiveness into the work, similar to what you have when building a plastic model. It is also possible to change the agents to a pointer shape by altering one's hand gestures. This pointer is used to designate a specific part of a model that is being studied by more than one person. A coloring function called a "color ring" can be called up by touching the green plate. The model is placed in the middle of the color ring, and the model is colored by designating the desired color in the ring by means of the pointer. Need High Positional Precision in Input Unit for Serious CAD Work ----------- "We developed this system for the purpose of using three-dimensional CAD in a network environment," explains Kawagoe. The idea is to connect remote sites via communications lines in a system wherein a number of people can simultaneously study the same CAD model. The peculiar sense of immediate "presence" afforded by VR is employed to make it seem like the model is actually there where the operator is when he or she is manipulating it. "We plan to move our development work in the direction of further enhancing CAD functions," said Kawagoe. For serious use, however, the data-gloves do not provide adequate precision. They can detect positions in 3D space by means of magnetic sensors. The precision of this detection is only to within several centimeters, and is readily influenced by metal objects located in the vicinity. It is therefore very time-consuming to adjust the system. The data-glove is a simple device in which optical fibers and sensors are installed in a glove. That is why it is now used so widely in research on VR application systems. According to Kawagoe, in terms of future practical applications, "the poor precision afforded by 'data-gloves' is a system bottleneck relative to CAD needs." What is needed is a new spatial positioning input device. Recognizing Gestures with Image Processing------------------------------ Even though various manipulations can be performed merely by moving one's hands, it is nevertheless troublesome to use a special device like data-gloves which must be worn. It would be much better if VR environments could be used without this annoyance. Researchers at ATR Communication System Research Institute (Kyoto) are working on a system which uses image processing to recognize facial expressions and gestures. Such a system would provide a more natural interface which could be used without wearing any kind of special gear. The ATR system employs images captured by two overhead CCD cameras. An image of the hand is abstracted, and the positions of the center of gravity and of the tips of the fingers are determined. In this way the shape and position of the hands are discovered. It is possible to tell whether the fingers are bent or not by the distance from the center of gravity to the finger tips. The positional precision depends on the broadness of the space in which the positions must be detected. If the hands are to be moved within a cubic area 1 meter on a side, then a precision of about 5 cm, i.e., several percent of one side, can be achieved. "Currently the positional precision is about the same as with the data-gloves," says Haruo Takemura, a researcher working in the Knowledge Processing Laboratory at ATR Communications System Research Institute. Recognition based on multiple signs are built up by combinations of finger-number and finger-bending patterns. It is difficult to distinguish between individual fingers, however. The system can tell a thumb from a forefinger, but not a forefinger from a middle finger. When a SUN 3/260 system is used as the host, four recognitions can be made every second. The system has more recently been improved to speed up this recognition speed, however, and a speed of 15 recognitions per second has been achieved. This sign recognition is all based on static signs at the current time, but Takemura speaks of a "desire to move development efforts toward recognizing gestures" from continuously changing "dynamic" sign patterns. If continuous changes can be recognized, ATR researchers believe that it will be possible to distinguish individual fingers. ATR is also doing active research on line-of-sight detection and on detecting facial characteristics. The goal is to combine these techniques so that an object can be manipulated in virtual space by image processing alone, without the operator having to don special gear. 3D Imaging of Software Structures for Analysis, Design ---------------------- VR input is also being studied in applications involving computer program design. The Systems Research Laboratory under Tokyo Electric Power Company's Technology Development Headquarters is doing research on a system in which inter-task relationships during program execution are represented as 3D images. The inter-task relationships are studied as one manipulates the 3D images with the hands while wearing data-gloves. The features in this system are divided between "visual analysis" and "visual design." The visual-analysis features are used to extract bugs and analyze task structures. The visual-design features are used to convert the results of modifying the program to source code. The intuitive manipulation made possible by VR plays a major role in the system. In visual analysis, trace data is first extracted from a program which is running and rendered visible as a solid three-dimensional image. The trace data consists of data on the internal operating conditions when a program is running: when a particular task started and when it stopped, what files or tasks were called up, etc. The image is displayed on a large back-projection display and made three-dimensionally visible with liquid-shutter glasses. The tasks or files represented as rectangular parallelepipeds in three-dimensional space are gripped and moved with the hands while wearing data-gloves to analyze the program, extract bugs, and adjust the program. By appealing to the operator's intuition, this method makes it easy to analyze or modify programs. The operational status of the control programs used in Tokyo Power's central control facility is rendered three-dimensionally visible based on actual trace data. A thick white line in the middle of the display represents the time axis. Files and tasks are called up in the order in which they are connected by the other lines, thereby indicating how the program is operating. By watching this screen, it is easy to see the timing with which tasks are run and files are called up while the program is executing. In order to modify a program, one can use the data-gloves to move or add internal structures inside the program as it is being represented in 3D. The company has provided the system with 13 different modes in which the data-gloves can pose. They are used to enlarge or reduce the image on the screen and to move the figure in the screen, etc., in conjunction with commands for opening menus and making selections therefrom. These features are used in making modifications in a program. Tasks can be gripped with an agent, for example, and moved back and forth along the time line to alter the operational timing of those tasks. It is also possible to insert a completely new task. The system immediately performs a simulation to determine the effects of changing the task positions and updates the screen display. In the past, numerical trace-data outputs had to be read by somebody and a line diagram (like a train routing diagram) created before a control program could be studied. When studying the effect of altering the call-up timing, the lines had to be redrawn. A transformer-station control program contains more than 20 tasks and 20 files each, so a line diagram on paper was very complicated. It is difficult to look at such a diagram and understand it immediately, and redrawing the lines is tricky. The system used by Tokyo Electric Power Co. (Tepco) at its System Research Laboratory not only automates such analysis work as this, but also three-dimensionalizes the analysis by means of VR technology and facilitates manual manipulation. These are very attractive features. By imparting a virtual shape to a program which is not a physical object, it is possible to employ human intuition with a sense that the object "itself" is being moved with the hands. It is hoped that this approach will make such analysis work more efficient. Mikio Okada, a lead researcher working in the laboratory's Control Research Lab said the following. "We will finish the interface portion this year. We then hope to get to the practical level within 2 more years." With the visual-analysis functions of the same system, complex relationships between tasks and files within a program can be represented in a form that humans can readily comprehend. This technology should have applications in process control systems in production lines as well as in software research. Research has also moved ahead to the next stage, namely that of visual design. Visual design, according to Okada, is "a feature for expanding a picture in software." It is visual design that takes the 3D figures or diagrams analyzed by visual analysis and converts them to source code. The goal is to achieve a system which takes the results of manipulating the figures or diagrams and directly reflects these in programs. In the past, research was conducted on the generation of program text code from symbols. This, however, was limited to lower command levels, even within a program structure. There has evidently been little research done on taking the large program structures constituted by task relationships and transforming these from symbols to source code. We have now reached the level where the call relationships between tasks, or between tasks and files, can be converted to code. The next step is now to do research on encoding chronological relationships and call-up timing. "It is very hard to properly reflect wait times," says one researcher working in the Control Research Lab. When used in visual analysis, the precision of the data-gloves has little effect. When doing the visual design, however, this precision does become a problem because time relationships must be expressed very exactly at positions in three-dimensional space. Researchers hope to be able to reflect time relationships in conjunction with the visual-design features by the end of next year. VR Brings On-Site Feel to Remote Control of Construction Robots--------------- Research on input systems employing VR technology is not limited to the field of CAD or other program software. It is also being conducted in hardware fields like robotics. Tokyu Construction is studying a VR system with the objective of using it in the remote control of deep-foundation work robots (a type of construction robot). Researchers are seeking to control construction robots with 3D images and hand movements. This is a sub-field within VR that is called tele-existence. When the smallness of the diameter of an opening prevents the entry of construction equipment, human workers dig holes and put the foundations in place. These are called deep foundations. Robots have been developed to replace humans in this deep-foundation work. After the hole is dug out, the excavated surfaces are lined with steel plates called liner plates, and the foundation concrete is poured in. Accordingly, the diameter of the excavated hole must be made larger than the diameter of the foundation being put in place by the thickness of the liner plate. In a remote-control situation, the positional relationship between the robot bucket and the excavated surface must be known accurately. Tokyu compared the work precision in three cases, namely (1) when using images from one camera, (2) when using three-dimensional visualization employing two cameras, and (3) when relying on the naked eye. It was found that the sense of positional relationships in space with 2D screen images taken with one camera is inadequate, and that 3D visualization is necessary. It was also demonstrated that the 3D visualization of images taken with two cameras provide roughly the same degree of work precision as is provided by the naked eye. With the remote-control system now being developed, a stereo-visual image is taken using two CCD cameras to achieve 3D visualization. Two CRT screens equipped with polaroid filters are then synthesized with half-mirrors to achieve this 3D realization. This permits a 3D visualization system to be configured much more simply than when using a head-mounted display or the liquid-crystal shutter technique which switches the image between the two eyes at high speed. "We can obtain good pictures at low cost," says Masayuki Takasu, director of the Mechatronics Development Lab under Tokyu Construction's Technology Headquarters concerning the reason this method was adopted. Electro-Hydraulic Bilateral Control gives 'Feel' During Manipulation--------- A dedicated position input device is used to move a bucket when excavating. For this position input device, the arm of a small experiment-type shovel is shortened to 1/5 normal size. To move the shovel bucket to the position aimed at and perform excavation work, the operator moves the input device which he or she holds while watching a three-dimensional image. With the position input device and the small shovel, an electro-hydraulic bilateral control system is configured which has an electrical system on the input side and a hydraulic system on the output side. Research is being done on this project with the cooperation of Miroku Sato, a professor at Musashi Institute of Technology working in the Control Engineering Research Laboratory, Mechanical Engineering Department, School of Engineering, Musashi Institute of Technology. With conventional lever-type control structures, the valves of hydraulic actuators are controlled. However, there was no immediately apparent relationship between the vertical and horizontal movements of the levers, on the one hand, and the movements of the arm, on the other, so considerable skill was required to operate the arm and bucket correctly with the levers. With this new method, however, it is claimed that a beginner can excavate in the desired location. With lever operation, considerable skill is required to accurately determine the excavation position. With the new method, the position is said to be easily determined with great precision. This system is equipped with load cells on both the input and output sides to give the operator "feel" when he or she is excavating and thereby appeal to the tactile sense. Very little research is being done on the use of VR to realize this sense of touch. "It's a big help on the job just to be able to feel some kind of feedback when a force acts on the tip of the bucket," explains Takasu. With "feel," the operator can tell when he or she has struck a hard layer, so this sensory feedback is absolutely necessary for deep-foundation robot operating systems. The difficulty in using force feedback is said to be the adjustment of the load cells. The full force acting on the bucket cannot be returned to the operator, such force would be too strong. Some appropriate fraction of this force must be returned, after a proportional reduction is made. "It is very difficult to find the best proportionality because of the differences between individual operators," says Takasu. "The fatigue factor will be too great if the force returned is too large. But there will not be enough 'feel' if the force returned is too weak." Now that researchers see their way clear to realizing some degree of positional precision, they are now working on ways to determine the optimal operating-force feedback proportion while increasing the number of samples. Long Communications Line Linked to Construction Site------------------------ The construction company Fujita is conducting joint research with VPL (which developed the data-glove system) on applications of VR technology for construction robots. The basic goal here again is to develop remote-control systems. More specifically, however, the two companies want to be able to connect domestic or overseas construction sites via ISDN (Integrated Service Digital Network) lines to control construction robots from a control center and to monitor and control site conditions. Fujita is primarily responsible for developing the hardware for the construction robots and communications technology, while VPL is responsible for developing the software for the computer graphics and VR technology used with the construction robots. In the system now being tested experimentally, camera images and computer graphics are synthesized and viewed on a three-dimensional display which the operator wears. The video images give the operator a sense of being right there at the construction site, and he or she can manipulate the robot easily with cursors or pointers displayed with computer graphics. The overall system is divided between Japan (Fujita) and the United States (VPL) so that researchers can experiment with remote-control operations over extremely great distances. "We have created a situation in which a construction robot in America can be controlled from Japan," says Kenichi Kawamura, a vice president with Fujita Research (subsidiary of the Japanese parent company) who is directly in charge of the project. "The biggest task problem facing us now is the time delay involved in long-distance communications," says Kawamura. This is not much of a problem when both the construction site and the control center are in Japan. When one is in Japan and the other in the United States, however, there is a delay of a second or more from the time the command is given until the robot actually moves. The same is true when sending a camera image from the robot. The company hopes to cope with the delay problem by employing computer simulations of the robot movements and displaying predicted motions on the operator screen. Other needs include technology for sending 3D video data at high speeds and input devices having high positional precision. Fujita hopes to solve these problems before the end of 1992. "We hope to use this technology to create monitoring and control systems to reduce manpower requirements at construction sites and facilitate better remote management and control," says Kawamura with reference to future goals. Exploiting Sense of 'Being There' Provided by Camera Images------------------ There are other possibilities for creating easy-to-understand interfaces besides (VR). It is also possible to impart a sense of "being there" and thereby enhance operability by means of real camera images. The Hitachi Research Institute of Hitachi Ltd. has developed a prototype of a plant monitoring system which uses camera images to provide such a sense of "being there." This system does not use 3D visualization or other VR technology, but does give a real sense of plant work-site conditions and thereby seeks to enhance operability. The prototype system is a model of a monitoring system for a thermal power plant. The condition of the plant can be monitored and controlled by directly manipulating the camera images. These direct manipulations of the camera images are conducted in the following way. Suppose, for example, that there is a need to check on the condition of the fire under a boiler. The operator uses a mouse to move a pointer and clicks it over the image of a boiler peephole. A frame will then appear on the screen, indicating that a peephole has been selected as the device. When the peephole is clicked again, the screen will switch to an expanded image of the peephole. When the mouse is clicked a third time, the screen will change to the image of the flames which a monitoring camera is taking. The object which one wishes to view can be seen merely by indicating it on the screen, making operation extremely simple. When changing the operational status, furthermore, the operating panel image can be called up by using the mouse to designate the image of the device. Once the image of the control panel appears, the dials thereof can be designated with the mouse, and these dials can be tuned by moving the mouse. The dials on the control panel at the work site will move as the mouse is moved, and hence the operational status of the equipment will be changed. It is also possible to listen to the sounds being made at the work site and make decisions on the status of the system based on these sounds. ------------- 2. FORCE FEEDBACK EXPERIMENTS Iwata (Tsukuba University) ------------------------------------------- From: Tokyo TRIGGER May 92 pp 20-25. "Until recently most interaction research has been focused on the visual and auditory senses," explains Nadao Iwata, a lecturer in the Structural Engineering Department at Tsukuba University. "Research on the tactile senses of touch and pressure have lagged behind. In order to enhance human manipulational sensation, however, we need to study the physical interactions involving immediate bodily sensations as well as indirect sensations." If we run into a wall, we bounce off of it. How far can we go in reproducing these sensations of touch and force, so common in our everyday experience-in the realm of VR? According to Iwata, there are four ways in which dynamic feedback can be expressed currently in VR technology. a. Master-Manipulator Technique Using a manipulator based on mechatronics technology, a physical reaction is communicated back to the operator as he or she performs actions in virtual space. This kind of equipment usually involves large-scale hardware and is very expensive. Hence it is thought to be unsuitable for human interfaces. b. Wire (Tension) Utilization With this method positions are detected from the length of wires and while those wire lengths are being controlled, and physical reactions are produced. Small movements such as those made with the tips of the fingers can be used to good effect. The range of movement is limited, however, because the wires that stretch in all directions will become tangled. c. Joy Stick This is an input/output system made up of a control stick which can be freely tilted longitudinally and laterally, and equipment which detects the angle, direction, and force of the tilts. The advantage is desk-top compactness. The disadvantage is that the number of degrees of freedom is limited. d. Data Gloves Progress is being made in research on feedback from gloves in which shape-memorizing alloys and air-pressure cylinders are used. This technique is limited to what can be felt with the hands, however, and cannot express the sensation of running into and bouncing off of a wall. Each of the four approaches summarized above has its own advantages and shortcomings, and no definitive system seems to have been found. For the foreseeable future, therefore, it seems likely that these methods will be used either individually or in combinations to suit the application. One of Iwata's experiments involves a feedback system that is being developed, based on the master manipulator approach. One immediately thinks of mammoth system at the mention of a robot arm, but the apparatus under development at the lwata laboratory is a very compact desk-top system. The system consists of a manipulator unit which follows hand or arm movements with six degrees of freedom and three actuators which follow finger movements. The operator can move his or her hands and fingers independently. When the position and movement of the hand is detected by sensors, a virtual hand in a monitor screen moves accordingly. When the hand in the screen strikes an object, the manipulator motor is mechanically controlled so that the movement of the operator's hand is restricted to produce a real sense of resistance. Similarly, when a virtual rubber ball is grasped, one senses elasticity in one's hand, and when an object is lifted, the operator senses the `weight' of the object. As to the visual sense, instead of using an HMD (head-mounted display), this laboratory says it will project images just as they are on a high-resolution screen. This would result in the operator's hands fixed to the manipulator coming into the field of vision, so a mirror is placed in front of the operator's face at an angle of 45 degrees and the computer screen is projected there. Thus the mirror hides the hands from view. "If we can recreate the trial-and-error environment of a design process in a VR environment, the time and effort spent making a mock-up could be sharply reduced," explains Iwata. "Take the case of automobile design, for example. The designer has to make small models out of clay. In giving form to an inspiration, not only must the work be seen, but the sensation of the hands is also very important. If this kind of intuitive expression could be input, it would be possible to let the computer handle the tedious work of the design process." At the lwata laboratory, experiments are being conducted in which a single-lens reflex camera created on a computer-graphics screen is handled with virtual hands to determine its weight balance and how it feels when operated. This manipulator also has its limitations, however. The biggest one is its "particularity". No matter how the apparatus is set up, dead angles are always found where work cannot be done. An example might be the inability to reach an object immediately below the manipulator. This will no doubt result in having to configure the system to suit the nature and purpose of each job. Feel of Going Up and Down Stairs------------------------------------------ Another large-scale VR is under development at Iwata's Tsukuba laboratory. This is the virtual perambulator. lwata wanted to build an apparatus that would take the previously separated bodily sensations and handle them together. With this apparatus, the walker has his or her upper body fixed in place and an HMD placed on his or her head. The position of the head is detected with image sensors. Ultrasonic wave generators are attached to the walker's toes, and the positions of the walker's feet are determined by measuring the time required for the ultrasonic waves to reach receiver units placed in three different locations. These motion data are sent to a computer which generates a virtual space, and the view through the HMD changes in real time in response to the movements of the walker. The perambulator is made so that the sense of reaction or resistance associated with climbing or descending stairs is produced by tension on wires attached to the feet. When ascending a step, the wire length is regulated so that the take-off foot feels the force of resistance. In order to represent the reaction force in the case of opening a virtual door, a manipulator having six degrees of freedom was fabricated. Once this system is perfected, designers of large structures, city developers, and public park constructors can perform simulations to find out ahead of time what it feels like to walk around in such facilities. "The demand for view assessment is bound to continue to grow," says lwata. "Computer graphics are already being used, but, with this apparatus, bodily movement is coordinated with visual images, so that a more lifelike model can be experienced. I think it might have useful applications in simulating living environments. A new house could be experienced while it is still in the design stage, for example." Subtle Sensations of Craftman's Fingertips Reproducible---------------------- Yukio Fukui, a researcher with the Product Science Research Institute operated by the Agency of Industrial Science and Technology under the Ministry of International Trade and Industry (MITI), has developed a force feedback apparatus which uses an XY recorder. XY recorders--which record changes in coordinates relative to vertical and horizontal axes--have been used for some time to record vibrations or temperature fluctuations. Fukui decided to use these recorders in the field of VR. His research falls into the "joy stick" category. The head unit in the XY recorder is equipped with four-directional strain gauges. When a head is moved by hooking a fingertip into a depression in the middle of a gauge, a cursor moves on a screen. Immediately one tries to follow the object in the screen. The operating feeling is no different than with a conventional mouse, but when the cursor tried to bite into the object, the XY recorder stops dead in its tracks. "The force and direction of the fingers are measured with the strain gauge, information on the position of the cursor on the screen is fed back and into the XY recorder, and the finger movement is restricted as though someone said, 'That's as far as you go'", explained Fukuui. This apparatus has three advantages. One is that it can move freely, without any constraints of particularity, so long as movement is limited to the two XY dimensions. Another is that force control and computation is simpler than with robot arms and wire tension because the positions are determined by two XY coordinates. The third advantage is that the recorders used are commercially available. Where would it be helpful to follow subtle shapes in a virtual environment? "Sometimes you have an object with a curved surface which is only slightly deformed in one place. The deformation might be undetectable to the eye, but readily discernible to the touch. When making a metal mold or die, a skilled craftsman might feel it with his or her fingers and detect subtle curves which he or she then corrects. This kind of trial-and-error operation can be reproduced in a VR environment." In other words, it is said to be possible to use a computer to feel subtle tactile sensations which cannot be expressed in words or discerned visually. "We can think of many applications in the field of education and training," says Fukui. "Students of shodo (Japanese calligraphy) could use this to trace out the subtle pen strokes of the master. Wouldn't that be great?" Maybe this portends a future in which educational or training software will be sold not just for the visual (video) and verbal (audio) information it imparts, but also for the "feeling" of the master which it enables the user to experience. Fukui says that he will start development work on other 3D feedback devices. "We are now developing a system with which a stick which moves longitudinally, laterally, and vertically is operated to move something like a virtual spatula to trace out or transform shapes." Iwata's comment on how the history of the human race could be likened to a series of attempts to broaden the "virtual world" (imaginative world) and thereby expand human awareness made a lasting impression on me. "Take the ordinary telephone," says Iwata. "You truly experience this expansion of the virtual when you listen to the voice of someone speaking on the other side of the planet. The same can be said of sitting in your parlor and watching things happening on the surface of the moon. In like manner, VR has the potential for explosively expanding our world of awareness." It is amazing to see the world of data inside a computer spread tangibly before your eyes by VR, and thrilling to experience the initial moment of entering the imaginary world that is created. VR technology continues to make all kinds of "impacts" on us. And now, the sensory information that is fed back to us in real time from the world of VR presents new world vistas to us. We can visit the interiors of virtual buildings, or we can delve into the world of molecules, and experience the feeling of taking action and doing work. And in the course of this process we gain intuitive understanding of a kind that heretofore had been inaccessible. "What to a human being is the sense of reality?" As we work out the answer to this question, we may be incorporating not only the tactile senses into the world of VR, but the senses of smell and taste as well. Technology will continue to expand the world of our experience and awareness, without limit, until it seems that another world, a new earth, has fallen into our hands. Sato (Tokyo Institute of Technology)------------------------------------- Makoto Sato, a professor at the Precision Engineering Research Laboratory of the Tokyo Institute of Technology, is the developer of SPIDAR (SPace Interface Device for Artificial Reality). This acronym is also suggestive of a "spider." One first sticks one's index fingers into a thimble-like rings. Each ring is suspended from all directions by wires. If you yank down on the ring, the length of the wires changes. These changes are measured with a rotary encoder to detect the position of the finger. For the visual input, anaglyph-based stereoscopic vision is used. When red and green glasses are looked into, the operator sees a 3D wire-frame jar in a large screen. When I tried pushing the virtual jar with my finger while it was still in the thimble-like ring, the cylindrical jar caved in and took on the form of a large squarish peanut or gourd. My finger, meanwhile, was locked into place and could not be moved. It really felt as though I had bumped into something. "When the finger reaches the position occupied by the object, the wire is restrained and your motion is restricted," explained Sato. "This reproduces the sensation of touching something." Next I tried pushing on the bulges in the jar with my finger. The bulges gradually caved in, and the jar returned smoothly to its cylindrical shape. The jar has a constant volume, so pushing in on one place will bring the depressions back out. It feels like making a clay jar on a potter's wheel. This unique apparatus developed from a very simple wish. Sato explains. "Seven years ago I saw the Fujitsu pavillion's 3D video at the science expo. I was extremely impressed. In one scene a DNA (deoxyribonucleic acid) double helix twists its way up toward the ceiling. It was so real; I wanted to reach out and touch it." Sato thought how great it would be if only one could "touch" a virtual object. If one could only use his hands to move and change the shape of a physical object. Sato's idea became a reality just 3 years later (in 1988) at the science expo. He unveiled what might be called the predecessor of SPIDAR. In this prototype apparatus, a ball is suspended from all directions by wires. When this ball is struck with the tip of a billiard stick, the motion of the ball is detected from changes in the length of the wires, these data are input into a computer, the virtual ball rolls and strikes another ball. The result might be called a "virtual billiard table." The dream of "touching" an imaginary object in VR was realized with this "virtual billiard table." The next problem is that of controlling the reaction feedback from the object touched. It took 4 more years to create SPIDAR. With SPIDAR II, currently under development, both the index finger and thumb are used. Sato calls his SPIDAR II the "world of virtual building blocks." The system has two rings in which the fingers are inserted. Each ring is suspended from four wires, making eight wires in all. When the operator looks through 3D glasses equipped with liquid-crystal shutters, he or she sees a stack of building blocks with holes in them and a stick passing through their centers. "Try picking up a block with the thumb and index finger and moving it to another stick," said Sato. At first one can't judge the distance well and waves in thin air. The blocks were hard to grasp. When one does grasp a block, the operator can immediately "feel" the object from the restraint on the wires. I had a difficult time transferring the block to the another stick, and was relieved when I finally got the hole in the block on the stick and let the block drop. "SPIDAR is most suitable for work calling for supple movements with the fingertips," Sato explained further. "In the future, we might develop applications of this technology in the field of tele-existence if we can perfect `robot nurses'- or other robots which perform delicate operations. "As to the future, we want to increase the number of fingers used, make it possible to employ both hands, make the work performed more complicated, and thus heighten the sense of reality. We also hope to develop 3D input devices that are as compact and easy to use as a mouse." Artists and illustrators have recently begun to make extensive use of computer graphics as tools for expressing their creations. Sato spoke enthusiastically about the need to develop a "3D mouse" which could be used easily not only by specialists working in VR and robotics, but also by everyone who uses computer graphics. VR Used in Molecular Design------------------------------------------------ The range of applications for such technology is very broad, extending past the CAD (computer aided design) field to education, medicine, and the amusement industry. "It is also useful in molecular design in such fields as pharmaceuticals and protein synthesis," said lwata. "Suppose, for instance, that you wish to make a new molecule by means of molecular bonding. You have to study the attraction and the potential that exists between molecules and find out where the low-potential, readily bondable locations are. In the world of giant molecules which have complex structures, however, the researcher is faced with myriads of parameters and conditions which exceed the processing limitations of computers. So, what do you do? You build a molecular model with computer graphics and then use virtual hands to manipulate and join the molecules." (At the University of North Carolina, progress is already being made in applying VR technology to molecular and atomic structures. This is one example of a job in which efficiency is improved by juxtaposing the human sense of touch with VR technology.) 3. BUSINESS RELATED Nissho Electronics, Ltd., has had an agreement with VPL since 1987 to act as the sole sales agent for the latter in Japan. VPL is a pioneer in the VR field in the United States. A press conference was held on March 5 1992 at which the two companies announced a tie-up to conduct international business in this field. At the press conference, Atsushi Kato, manager of the Electronic Equipment Business Department at Nissho Electronics, called VR technology a "futuristic technology that will support Japan in the 21st century," while J. Gilmore, president of VPL, said that the purpose of the tie-up was "internationalization based on a new stage, new staff, and world strategy." Since entering into the dealership agreement in 1987, Nissho Electronics has been marketing VPL's VR systems and VR-related products-including data-gloves, eyephones, and the RB2 system-to some 100 Japanese users (companies). The two main elements in the business tie-up are the beginning of Japanese production and the formation of the "VR Club." Nissho Electronics has pursued the VR business as an import agent. This historical development has highlighted the necessity of implementing domestic production. What is most important in practical implementation of VR is not the hardware (the product itself) so much as the "soft" issues of how the user can use the hardware and how the user wants to use the hardware. Each individual user tends to look at a VR system or product and say something like this: "We at ABC Ltd. want to use your product for such and such particular purpose; can't we change or enhance this or that feature to better facilitate that purpose?" The "VR Club" was formed in response to this diversified demand. The idea is to make it possible for member companies to use basic VPL patents to produce their own products in Japan. Nozomi Kikuchi, assistant manager of the Application Electronics Department at Nissho Electronics put it this way. "Companies apply for membership in the 'VR Club' through Nissho Electronics. Final versions of the membership agreement and fee structure have yet to be hammered out, but we're working on it. Member companies can use the roughly 20 basic VR-related patents possessed by VPL to develop and produce their own applications. Progressing to manufacturing in Japan should break the barrier to the full-fledged development of VR." Kikuchi said also that it would be fun to see how far products like the eyephone and data-glove would evolve with Japanese technology. "It was magnetic sensors that got us involved in VR," says Kikuchi. "We were surveying the U.S. market seeking sales routes for our magnetic sensors when we found one company that wanted to buy in large quantities. We figured there must be some kind of really good application behind the big orders, so we went to visit the company. That company, of course, was VPL. "At that time, nobody had ever heard of VR. Our first impression was that VPL was doing research in an area bordering on the fantastic. That's what I myself thought at first. And then we discovered that the VPL people were into something really amazing. So we thought, o.k., let's try our hand at this and see what happens. That was back in 1986. Now VR has suddenly become a very hot topic. I guess we were just lucky." The market has changed a lot in the past year, however, according to Kikuchi. Most of the sales prior to the change had been to laboratories and public organizations, primarily for research use. Now most of the inquiries have to do with product development. "We get a lot of inquiries from people who say they would like to work with us on this or that application," said Kikuchi. "We may now be entering an era when the curtain goes up on VR as a serious business." VPL thinks the market potential may be in the neighborhood of 1 Trillion Yen, but the company is cautious, realizing the market is still being developed and that the new technology has not yet been put on a full commercial base. VR is attracting wide attention, and sales are definitely growing with a multiplier effect. "But the current status is just a step above outright R&D," says Kato. The market being targeted is the engineering market in such fields as robotics. The objective is to improve productivity using VR to simulate processes, time, and manufacturing steps. "Fundamentally, though, VR is a technology that can be used just about everywhere," says Kato. "In a sense, the applications are only limited by the user's imagination. So the differences in the types of jobs actually envisioned might not be such a problem." MICO: Mass-Media Revolution?--------------------------------------------- Media International Corporation (MICO) has concluded a tie-up agreement with Telepresence Research. The latter was founded by Tom Fumace, a pioneer in VR in the United States who did research at NASA's Ames Laboratory. MICO, a trading company which deals in imaging and video software, was established by NHK, with the financial backing of C. Itoh & Co., Ltd., The Seiyu, Ltd., Dai-Ichi Kangyo Bank, Ltd., and Sumitomo Bank, Ltd. VR to MICO, is "a new communications media," says Akira Mochizuki, manager of MICO's Media Development Department. MICO is working with affiliates of Telepresence Research (including Fake Space Laboratory, Nymark and Company, and Crystal River) to commercially develop such technologies as tele-robotics, local-base entertainment systems, and both 3D video and audio in virtual space. "MICO is only interested in media products," explains Mochizuki, "so we see VR is a new media that is worth being pioneered." What is interesting about MICO's VR business is that it focuses on software instead of hardware. "At the mention of VR, most people think of such 'hardware' as the head-mounted display (HMD) or data-gloves which people have to wear," continues Mochizuki. "But this isn't the way it is. VR, in the fundamental sense, is software and is media. It is software, therefore, which should come first in our thinking about VR, not hardware. To be more specific, we shouldn't be so much concerned about whether the video quality is good or bad, but rather about how to use it; that is key. It's like, in the beginning there was software. We should be successful if we make good software that capitalizes on the features of VR to transcend distance and time, and to be able to get inside our computers." This will never happen, however, just by acting as a conduit for imports from America. "MICO is not simply a trading company. We propose how to use VR, explore joint research projects, and think of software with users. We must handle everything here: coordination with American VR manufacturers, direction, and production." The key to all of this is none other than domestic production. The 3D display system "BOOM" currently being handled sells for around 10M Yen. By producing the system in Japan, this price can be cut by a third or even by half, it is believed. With NHK heavily involved in MICO, there is naturally interest in connecting VR with communications and broadcasting, but Mochizuki quickly points out that such possibilities are still very much in the future. "VR may provide the impetus for touching off a revolution in the shape of mass media. VR is not just for sending information like television does. It permits interactive, two-way communication. But we are still talking about raw potential here. Applications in communications and broadcasting are still quite a ways off." Information Collection Complete; On To Business Targets------------------- Since its formation by the Japan Technology Transfer Association (JTTAS) last October, the Artificial Reality and Tele-Existence Research Committee has already signed up some 90 participating companies. "As a research society, we're already vying for number one or two," laughs Shinji Ishikawa, a project leader. Most of the early companies to join were in the electronics, mechatronics, and game manufacturing industries. Thanks perhaps to the current boom, however, the society is being joined by more and more advertising agencies, software production companies, computer graphics production companies, and other media-related companies. While this popularity of the research society is a most welcome development, too many members can prove disruptive of the society's activities, so the structure of the society has been reorganized around subcommittees. There are currently four subcommittees, namely the Simulation, Design, and Amusement Subcommittee, the Human Factor Subcommittee, the Robot Control and Communications Subcommittee, and the Internationalization Subcommittee. "All of the member companies have evidently completed their information collection activities regarding VR," reports Ishikawa. "Now they are at the stage of figuring out what to do next, based on the information in hand. We are starting to see actual action. At this stage, however, we are not yet seeing any products realized. "In making VR a viable business, a decision must be made whether to regard it as a brand new technology or to handle it as a technology to be applied in the context of existing technologies. We are not yet seeing much in the way of products, but I think maybe that we could possibly see something like that in another 2 years or so." What sort of businesses will have developed out of VR 2 years from now? Who knows. But you can be sure that, behind the scenes, each company is working feverishly to identify and define specific targets. -----------------------------END OF REPORT-----------------------------