Picture a computer company's sales person pitching his best delivery to a group of prospects. He runs through a list of features of the new WhizBang 4000 product, meticulously describing each with impeccable terminology and demonstrating with charts and tables how the product will change the course of his prospects' business. And he finishes his speech by turning his back to the slides on the wall, facing his audience and stating,
"... and it all happens in real time!"
Of course, if this were a late-night infomercial, the audience would react with ooohs, aahhs and wow! Of course, no one would think to raise their hand and say, "What do you mean by real-time?"
Interestingly, we all tend to throw the term real-time around quite a bit. There seems to be an innate understanding of what it means, but not a formal definition that is widely agreed upon. Whatever the case, it does not stop us from using it to impress our listeners. It has, after all, a bit of a macho ring to it. But the use of it such in diverse contexts often can be confusing. Read on for an attempt at an applicable definition.
Strictly speaking, it is impossible to produce real-time computer programs. After all, real-time means immediate. The term is used in a reference of causality, that is, the time between when an action occurs and its ensuing reaction. In the real world, reactions occur simultaneous to the initial action. In computer programs, however, there is always a delay between when an action occurs and when the resulting reaction manifests itself. To put it in classical computer terminology, actions are a form of input after which there is a processing stage, followed by the resultant output. The delay between when the action occurs and the result is often referred to as latency.
It is in this latency that we will find the working definition for the term real-time. Lets explore some real world examples.
A pilot flying an aircraft moves the controls and then experiences the reaction of the aircraft. The initial action occurs when he moves the controls with his hands, the reaction is when his equilibrium experiences a change in forces and the horizon begins to bank. In real life there can be some perceived latency due to cables stretching, inertial resistance and aerodynamic forces. However, each of these occurances are immediate responses to input. The latency is caused by a chain of real-time reactions not a "delay in reality". A highly accurate simulator would be able to simulate the response latency by simulating each of the elements in the chain of reactions.
A surgeon performing open-heart surgery, applies a scalpel to her patient's skin. The skin gives and the surgen skillfully adjusts her pressure on the scalpel to carefully manage the depth at which the scalpel is allowed to penetrate the skin. Action and reaction are tigthly coupled and the surgeon's response must be tuned to react to the feedback of the feel of the knife against the patient's skin. Research being done in today's surgery simulations with haptic devices must take this feedback loop into account.
A company storing large stockpile of a highly sought-after product experiences an earthquake and 75% of the product in stock is destroyed. The market reaction is to raise the price of the product at the retail outlets. The latency on this action/reaction can be as long as a day or as short as 3 seconds, depending on the level of communication between the warehouse and the retail outlet.
A "smart" missle is fired and achieves 2280 km/hr on route to its target. at this speed, it must make small adjustments to its aerodynamic control surfaces to stay on course. At this speed the latency between when it experiences a drift off-course and it responds to that drift by adjusting the control surfaces must be minute so as to not diverge out of control.
All of the above examples use computers that claim real-time performance. However, the latency varies from sub-millisecond, as in the case of the smart missile, to one-day, as in the case of the stock price change. Often, we think the term real-time means real-fast, but we see in these examples that real-fast is a highly relative term.
Simulations often use the term, "man in the loop" or "machine in the loop". The prior paragraphs provide two examples of "man in the loop" with the pilot and the surgeon, one example of "machine in the loop" with the missile and one, I guess we could term as, "market in the loop". We'll refer to these as the "subject in the loop" where the subject is man, machine, or market.
Interestingly, the measurement of latency can be closely associated with the subject's perception of delay in action/reaction. While a 3 second delay would be considered very fast for an update on a product's price, a 3 second delay response time would make an aircraft unflyable. Likewise, a 16 millisecond latency to a human is imperceptable, a 16 millisecond delay would put the missile into an out-of-control tumble through the air.
|
An example of the difference between man-in-the-loop and machine-in-the-loop is demonstrated in the X29 aircraft, employing a forward-swept wing design. The aircraft is highly unstable and requires super-human capability to respond to drifts in instability with the controls to keep the aircraft upright. While human perception of change can be measured in milliseconds, it requires a computer which can sample the flight process at sub-millisecond precision to keep the aircraft from tumbling end-over-end, out of control. |
In our search for a usable definition for the much-abused term real-time, we must include a statement about the system's capability to respond to a given action with a reaction latency that is consequentially imperceptable to the subject. That is, the pilot does not feel a delay in reaction, the surgen has feedback that allows her to interact with the scalpel, the retail store will not loose money, and the missile will react quickly enough to stay the course.
Our real-time computer process must then sample the system at a frame rate high enough to make the subject's perception of latency non-existent (or tolerant for the task at hand - note that it is impossible to make a machine in the loop simulation imperceptable to latency, when it is the machine itself that is measuring latency).
Is latency the only thing to consider? In the real world, our perception is also subject to a perception of linear time. That is, perceptibly immediate response to an action occurs every time. There is never a case where the response is later than at other times.
Likewise, the computer simluation of reality must be constant. While sampling the system at a constant frame rate, a "dropped" frame will produce twice the latency and may change it from being imperceptible to being perceptible.
It should be noted that, theoretically, it is possible to create a real-time system that does not "sample" at a given frame rate. The system would be reactive to change. As long as the reaction time were within the imperceptible range for the subject, the system could be considered "real-time". However, in practice we see that this is actually impossible. All computers sample, wether it be in the space of the application itself or in the operating system that must keep tabs on all of its devices at a regular rate so that it can deliver those "changes".
So, it is safe to say that a real-time system requires a frame schedule with a constant and consistent rate.
Using this discussion, the following definition of real-time should be employable by the majority of real-time applications.
A real-time system is one that samples the process at consistent and constant frame rates high enough to cause latency to be inconsequentially imperceptible to the subject.
Note that there is no explicit statement about how fast a real-time system needs to be, only that it be sampled consistently, regularly, and with low latency. The gating factor for any particular real-time system is in the definition of the latency time.
Now that we understand that real-time doesn't necessarily just mean real-fast, let's explore what this means in the world of 3D graphics. In past articles I have alluded to the difference between real-time 3D graphics and interactive 3D graphics. Perhaps, in light of this article, we can see, more clearly, what the difference is.
OK, let see. For a real-time system, we'll need a frame rate (constant and consistent). Interestingly, it is not a random or arbitrary decision. The makers of computer monitors know that human perception of "flicker" occurs somewhere below 60 Hz. Most computer monitors will update at 60 to 85 frames/second.
Also intersting is the fact that the movie industry has placed human perception of motion at 24 frames per/second. Understand, however, that perception of motion and visual perception of light are two different matters. While the mind can be aware of "flicker" at rates below 60 hz, the eye will "tune in" to this frequency. When motion occurs at rates below the visual refresh rate of the monitor anomalies occur. Next time you are at the movies, sit close to the screen and watch when the scene pans. You'll experience double vision. There is a whole article on why this occurs here.
In computer graphics, we use double buffering for visual quality, by drawing the newly updated scene to one buffer and displaying the other. We then swap buffers when moving from one frame to the next. On quality systems, we synchronize the swapping of buffers to the moment just before which the monitor begins refreshing (sometimes referred to as "blank time"). Our frame rate, therefore, is dictated by the rate at which the monitor is updating.
Human perception of real-time 3D graphics, then, is dictated by the rate at which the monitor updates. Latency between when the frame of a scene is drawn and a human can then see the scene is set by the 16 millisecond (or so) frame time of the monitor's refresh rate. In other words, at this rate, transition between frames is humanly imperceptible.
It is interesting to note here that the above statement may not necessarily be entirely accurate. It could very well be that human perception of a monitor refreshing at 60 hz adjusts to be tolerant of the update rate, rather than the rate being imperceptible. Take for example, the experience of running your monitor at 85 hz for some time, then returning to 60 hz. Suddenly it seems that you can 'see' the flicker.
Lucky for us, a monitor's frame rate is highly constant and consistent. However, there is only one mechanism we have in OpenGL programs to synchronize our visuals to this frame rate: SwapBuffers. And this executes only on the GPU. On most modern-day systems, the rest of our application, running on the CPU, runs non-deterministically, subject to I/O latencies, OS scheduling parameters, and bus contentions.
For the record, it is this author's position that higher quality real-time applications could be achievable on modern-day day PC systems if the graphics hardware vendors would provide a user-space interrupts occuring at vertical retrace intervals, upon which a frame scheduler could be written.
The mainstream PC application programming approach is an event-driven model. The system is required to respond to a user's event in a timely manner to provide the user with manageable latency perception. That is, when the user clicks a button, the time between when the mouse button is clicked and the visual button collapses should provide the user that the system is responsive. Latencies for this type of application can be measured in the dozens of milliseconds, even fractions of a second before it starts to "feel" unresponsive.
This response time needs to be considered differently than the visual response time of an out-the-window cockpit view of a flight simulator, for example. While the response time of a user manipulating a GUI driven tool can tolerate relatively high latencies, a latency of more than one frame on a visual system will render the user physically ill.
An event-driven system is one that is destined to encounter problems in a real-time application. When frame rates are made lower priority to user events, or low-resolution timers, not synchronized with the visual system are used to determine sampling rate, frames will be dropped and the system will loose its constancy and consistency.