From: "Dr. Guam's Agony" Subject: APPS: 'Biofeedback' and VR Date: Mon, 11 Aug 1997 23:53:54 -0700 (PDT) Here's something I have been thinking about off and on for a long time. I am presuming that this LIST will accomodate a rough-draft description of something which I feel is implementable in many ways, in pre-1997 software and hardware technologies. Summary: "Imagine an evolvable computer-mediated 'biofeedback' system." Assume: 1 a sufficiently-rich telemetry system for INPUT FROM one or more humans (monophonic microphones to human voices is perhaps capable of providing upwards of 5 INDEPENDENT PARAMETERS changing faster than 1/10 second timebase) 2 a sufficiently-rich visual OUTPUT TO the humans (INDEPENDENTLY VARIABLE PARAMETERS might include number of 'separate' 'objects' relative positions of 'objects' relative velocities and spins of each 'object' degree of independance of objects (brownian versus ballet motion) roundness/blockiness of a given 'object or of entire ensemble use of color, shading, subtle tinting to encode parameters ) 3 The Software: 3.1 Analysis of Human Input Channels into Independant Parameters "Factor Analysis": Pitch, Vibrato, Shrillness, Stress, Silibance, Rhythm-Pulse, Loudness, ... Speech would be another possible parameter-rich dimension. Analysis techniques might include: Fuzzy Classification, Statistical Factor Analysis, Predictableness/Novelty Detection, Neural Net 'Monkey See', Standard Models Of The Input System (in my simple case of microphones: pitch, loudness, timbre (FFT), amplitude envelope) *Hardware* Pitch/AmplitudeEnvelope/FFT/Silence Extraction 3.2 Recording of as much as possible (Both raw datastreams, and "stuff derived by the Feedback System", and "coincidental datastreams like temperature and what's on TV...") 3.3 MAPPING "Switchboards" (and other models like "Metaphors") FROM the processed-into-independent parameters input and into the graphical metaphorization of those parameters (above, into simplistic abstract geometrical 'objects') ***** * ***** * ***** The next level would be to make the Metaphorizing Mapping ITSELF be manipulable BY THE Human, through the Graphics he is watching, and through the Input system. In other words, I want the Human Pilot to be able to bootstrap-jacobsladder his way into an interface which he is happy with SOLELY BY USING THE FEEDBACK SYSTEM ITSELF, WITHOUT A LOT OF 'META' COMMANDS *AND* WITHOUT RELINQUISHING THE ROLE OF INTERFACE DESIGNER TO ANOTHER HUMAN. However, "Meta-Commands" and "Special Modes" are certainly nice things to have... it's just that if he creates a TREEHOUSE kind of place, he doesn't want some ugly "outside frame" to be on the screen, if that is too ugly. (Maybe he brings a girl into the treehouse, and he doesn't want her to see the "controls". In other words, I want to be able to dive into an immersive and expressive environment (((which might be quite 'alien' and 'abstract'... just as long as the Human Pilot's Mind has co-evolved with it and KNOWS HOW TO GET (TO) WHAT IT WANTS, AND 'KNOWS' (at least unconsciously) WHAT THE INTERFACE 'MEANS' WHEN IT DOES A GIVEN THING (allowing for a bit of 'new and open terrain' (room to grow) in the metaphor-mapping, so long as the basic parameters are stable and dominant (for example, the linkage between major basic input-from-human parameters and basic major output-to-human parameters should be something the human can "return to and trust" if more subtle parameters are less "nailed".) Hopefully the "loop lag time" would be within the "realtime" range (although I believe that the human can adapt to all kinds of "echo lag" and use of "echoes" would actually be a very good output parameter). ***** * ***** * ***** Use Of This Thing: 1. ((This is line 88 of this file! --> Pianos usu. have 88 keys...)) Imagine I gave you a PIANO. That is a pretty good analogy to what I am trying to do here, except this is a piano which can be "slid" into being an EASEL with a rack of oilpaints or into being a CAR driving through a forest (or a minisub in tropical fishtank). The idea is to have a (probably abstract) articulately expressive realtime VR environment which the Human Pilot can evolve to his own satisfaction, or which a team (or possibly unrelated teams) of humans, which is evolvable and represents the state of the mind of the Pilot(s). A universal translator, perhaps, if there are more than one Pilot, in that the sub-parameters controlled by each SubFlock of Pilots are presented in an integrated environment. For a lone human Pilot, this system is a way to make visible both conscious parameters and unconscious parameters of his mind. IGNORE ---> ((101lines 4300chars at 22:00)) An archiving system including a library of Starter "Metaphoric Environments" might include: 1 Tropical Fish in "Octopus's Garden" 2 Flocks of Flocks of (different) birds 3 Tree with bubble-creatures "inhabiting it" 4 'Cooking' 5 Garden 6 Single Tree or Rosebush 7 House/Human/Cat/Mouse/Furniture Cartoon 8 "Gyros, Pulleys and Gears" in hierarchies, a "silly factory" perhaps 9 A Dancing (Bear) 5 Background / Foreground 6 "Style" / "Zeitgeist" / "Gestalt" / "Themes" / "Basic Givens" 7 Meta-Meta-Linking between the Metaphor Models created: Pilot (or system) can decide to 'morph over' into another, perhaps more appropriate metaphor-interface "valley". 8 Structured Parameters: Hierarchies, Neighborhoods, Parameter Systems... 9 What if Pilot Forgets An Archived Interface ("What makes the elevator go UP ? Why is the cat frowning now ?") System then has to re-establish a mapping to the variables somehow... Here then is a "problem" (and a major opportunity...): I would just allow the pilot to "unconnect" 'objects' which he doesn't understand (The Cat would freeze in a neutral state) and then allow the system, once it finds a new parameter to display, and then can re-animate the given 'object'. Perhaps there is some kind of 'Foregrounding' of the NEW PARAMETER AND ITS FEEDBACK-OBJECT, which would increase the focus on the new sub-metaphoric 'handle', and make the feedback more articulate. (Like learning how to wiggle your ears, or how to whistle, starting with a lack of awareness that these things are even possibilities... and gradually developing the ability to do wiggle each ear separately or whistle with birdsong-level complexity). 10 Different, evolved interfaces for each Pilot. This indeed is when we have a universal translator type of thing. 11 Inclusion of circum-Pilot ancillary input streams (Like, the MUSIC which the pilot is listening to. The VIEW out his window, ...) 12 Creation of an Ultramodel linking all the Metaphoric Interfaces This would simplify the database used for all the metaphor-models, for one thing... 13 PREDICTIVE "PRESPONSIVENESS": Smooth ALREADYNESS 1 Keeps possible futures on hand anticipating Pilot's 'next move' 2 Predicts "issues" (see 15 below) far in advance 3 Predicts actual Pilot "Footsteps" slightly in advance "Just In Time" 4 14 "The Clutch" Pilot can maybe press on foot-lever which pushes the interface-metaphor towards the UltraModel (which is probably kind of abstract and 'overloadedly symbolic'). When in the UltraModel environment, a sort of Grand Central Station, the System is mainly oriented towards selecting the most appropriate interface-metaphor to "slip into". Thus, a light but constant use of "the Clutch" (much like how the "reverb pedal" on a piano is used to "slur" notes and chords across transitions). 15 "Issues" Let's use a chess-game metaphor, with Pilot "playing against" the System. While the Pilot's specific 'next move' (like a step out into a void) is not determinable when the Situation is in a (relative) "zone of FURCATION" (bi-, tri-, multi-, noninteger-... a "saddle region"), the "issues" being dealt with by the programmer ARE identifiable (in a manner which is meta-analogous to how "parameters" are identified and assigned to 'metaphorical objects'). What is an "issue"? It is something which shifts the Situational Metaphor-'Style' ("Zeitgeist") in some way: it might add a new parameter, it might change the emphasis of the parameters (Foreground and Background; the 'Central' parameters; what is considered 'Subtle', what is considered 'Secondary' or 'Ignorable' or 'Inconsequential' and what is considered to be 'Important'; what is considered 'Crisply Articulated' and what is considered 'Vague', 'Ambiguated' or 'Not Quite Under Control'. 'Issues' have to do with 'Decisions Pending'. Sometimes these are borderline implications of already-made 'Decisions'... and thus Issues ramify into further future issues. "... And that raises the Question of ..." 16 Reverb, echoes, "working memory", English Roundabouts, and "NOW-ness" Relates also to "Foregrounding and Backgrounding". 17 Invisibility, Observers, 'Operating Theaters', Concentric Layers Relates also to recording the whole shebang in its "MIDI" form: recording all the individual event-streams AND the 'derived-stuff'. A full fly-through can be "reanimated" and enhanced, even to the point of producing spin-offs of the original 'central Pilot(s) datastreams', either in subtle ways, preserving all the original events and timings... or in radical ways, branching off into "roads not taken" and "roads not even thought of" by the Original Pilot(s). But also we can have "one-way walls" allowing a Pilot's experience to be SOLO *for him* while also allowing Other Pilots (of various kinds and levels) to have a Situation *based on* the 'Central' Situation, but separate from it, and not affecting it. The word "Downstream" is appropriate for these "outer shellular involvements", because while they are co-simultaneous with the 'Core Pilots', they are not 'competing' with it or 'confusing' it. Ian. 23:23 11august97. 212lines. unicode:2714. 214lines semifinal 23:33. Basically I wrote all this off the top of my head... and would like to do a sort of "postal chess" with VR LIST people, and turn this TOP DOWN SKETCHING-UNTO-DETAIL approach to something I deeply want to 'manifest' into a reality (eventually), and probably the first implementation of what I am 'aiming at' will be as TEXT AND IMAGINATION, not as electronic hardware and coded/debugged C/JAVA/SCHEME/whatever software. 23:53. 262lines 11179chars finalfianl. "Dr. Guam's Agony"