US20110202934A1 - Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces - Google Patents
Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces Download PDFInfo
- Publication number
- US20110202934A1 US20110202934A1 US13/026,097 US201113026097A US2011202934A1 US 20110202934 A1 US20110202934 A1 US 20110202934A1 US 201113026097 A US201113026097 A US 201113026097A US 2011202934 A1 US2011202934 A1 US 2011202934A1
- Authority
- US
- United States
- Prior art keywords
- user interface
- signals
- input
- finger
- focus control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
Definitions
- the invention relates to user interface devices that provide additional user interface control signals beyond those of a traditional mouse, touchpad, or track ball, and relates also to window manager input focus control, and in particular to the direction of signals from such a user interface device to particular applications responsive to input focus control provided by a windowing system, operating system, or both.
- the present invention addresses the routing of additional user interface signals provided by the High Dimensional Touchpad “HTPD” (for example as taught in 1999 filings of U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S. patent applications), Advanced Mice (for example as taught in U.S. Pat. No. 7,557,797 pending U.S. patent application Ser. Nos. 12/619,678, 13/025,129, 13/024,569, and related pending U.S.
- HPD High Dimensional Touchpad
- a method for routing signals from a user interface device providing traditional user interface device signals and additional user interface signals to an application includes receiving traditional user interface device signals and additional user interface signals from a user interface device, routing the traditional user interface device signals to a selected application according to a first input focus selection, and routing the traditional [additional?] user interface device signals to the selected application according to a second input focus selection.
- the first and second input focus selection made by at least one focus control element.
- the at least one focus control element comprises a window manager, a window system, or an operating system. Further, the first and second input focus selection is made by the same focus control element, or the first and second input focus selection are made by a first focus control element and the second input focus selection is made by a second focus control element.
- the user input device may a computer mouse comprising a first and second scroll wheel, a touchpad, or a High Definition Touch Pad (HDTP).
- the touch user interface is responsive to gestures and the additional user-adjustable input comprises at least one gesture.
- the touch user interface may be responsive to one of the yaw angle, roll angle, or pitch angle of a finger in contact with the touch user interface.
- the additional user-adjustable input is responsive to the yaw angle, roll angle, or pitch angle.
- the touch user interface also may be responsive at least two angles of a finger in contact with the touch user interface, and the additional user-adjustable input is responsive to a measurement of each of the two angles.
- FIG. 1 depicts a plurality of windows, one or more of which can be a hypermedia application window such as a browser, and hierarchies of visually displayed and other objects within or associated with these windows.
- a hypermedia application window such as a browser
- FIG. 2 illustrates the side view of a finger lightly touching the surface of a tactile sensor array.
- FIG. 3 depicts a popularly accepted model of a typical cell phone or PDA capacitive proximity sensor implementation.
- FIG. 4 is a graphical representation of a tactile image produced by contact of a human finger on a tactile sensor array.
- FIG. 5 provides a graphical representation of a tactile image produced by contact with multiple human fingers on a tactile sensor array of lesser spatial resolution than that depicted in FIG. 4 .
- FIG. 6 depicts a signal flow in an HDTP implementation.
- FIGS. 7 a - 7 f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology.
- FIG. 8 suggests general ways in which two or more of these independently adjustable degrees of freedom adjusted simultaneously.
- FIG. 9 demonstrates a few two-finger multi-touch postures and/or gestures from the many that can be readily recognized by HDTP technology.
- FIG. 10 shows an example of how raw measurements of the six quantities of FIGS. 7 a - 7 f, together with shape recognition for distinguishing contact with various parts of the hand and the touchpad, can be used to create a rich information flux of parameters, rates, and symbols.
- FIG. 11 shows an approach for incorporating posture recognition, gesture recognition, and other functions to create a rich human/machine tactile interface system capable of additionally supporting or incorporating syntax and grammars.
- FIGS. 12 a - 12 d depict operations acting on various parameters, rates, and symbols to produce other parameters, rates, and symbols, including operations such as sample/hold, interpretation, context, etc.
- FIG. 13 depicts a user interface input arrangement incorporating one or more HDTPs that provides user interface input event and quantity routing for focus control.
- FIGS. 14 a - 14 g depict a number of arrangements and embodiments employing HDTP technology.
- FIGS. 15 a - 15 e depict various integrations of an HDTP into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797 and pending U.S. patent application Ser. No. 12/619,678.
- FIGS. 16 a and 16 b illustrate examples of conventional scroll-wheel mouse provided with an added left-right scroll-wheel as taught in U.S. patent application Ser. No. 13/024569.
- FIGS. 17 a - 17 c illustrate examples where a single trackball is incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797.
- FIGS. 18 a - 18 c illustrate examples where two trackballs are incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797, some of these ( FIGS. 18 b - 18 c ) comprising yet other additional sensors.
- FIG. 18 d depicts a mouse provided with a trackball and a small touchpad as taught in U.S. Pat. No. 7,557,797.
- FIG. 18 e depicts a mouse provided with a plurality of slider controls as taught in U.S. Pat. No. 7,557,797.
- FIGS. 19 a - 19 c depicts exemplary embodiments providing HDTP technologies with a HID device abstraction for interfacing to applications.
- FIGS. 20 a - 20 d depict arrangements for directing additional user interface parameter signals to applications.
- Windowing systems are well known with foundational principles dating back decades (see for example F. R. Hopgood, et al., Methodology of Window Management, Springer-Verlag, Berlin, 1986, ISBN 0387161163) and are known at least as an operational level to virtually all users of these devices. Without getting into the many well-known aspects of windowing systems, one skilled in the art is reminded that:
- FIG. 1 depicts a visual display screen area 100 displaying a plurality of representative windows—here 101 , 102 , 110 .
- windows 101 , 102 , 110 are shown as overlapping so as to streamline the discussion; one skilled in the art will understand the appearance of overlapping of one or more of these windows.
- One or more of these windows can be a hypermedia window such as a browser (here 110 ), and hierarchies of objects ( 111 and 111 . 1 ; 112 and 112 . 1 , 112 . 2 ) rendered within or superimposed over the display area of the browser window 110 .
- the hypermedia (browser or application) window 110 of FIG. 1 also depicts a toolbar 110 .
- Such vertical and horizontal scrollbars typically appear when the display area within the window is smaller than the vertical and/or horizontal span of the visual content, allowing control of the vignette displayed within the aperture created by the display, this control responsive to the positions of scrollbar(s) within the degrees of possible travel.
- the position of a scrollbar is in turn controlled by one or more input aspects from the pointing device.
- More contemporary computer mice additionally provide a scrollwheel along with the traditional components and features of the traditional mouse. Some scrollwheels allow the wheel to be depressed downward to operate a spring-loaded switch that provides a third class of button events.
- the scrollwheel provided by a contemporary computer mouse is solely directed to the operation of the vertical scrollbar (for example, the scrollbar 110 . vs of FIG. 1 ), if displayed, of the currently selected window.
- computer mice providing “2-way scrolling” (sometimes called “4-way scrolling”) features wherein the scrollwheel, in addition to conventional forward-back rotation, can be tilted left or right with the resulting signal directed to the control the horizontal scrollbar (for example, the scrollbar 110 . hs of FIG. 1 ), if displayed, of the currently selected window.
- touch screens have recently received tremendous attention with the addition of array tactile imaging capabilities.
- touch screen technologies permit multi-touch sensing, metaphors, and gestures.
- touch screen technologies have obtained great commercial success from there defining role in the iPhone and subsequent adaptations in PDAs and other types of cell phones and hand-held devices, these were in fact taught in the 1999 filings of U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978.
- These more advanced user interface pointing devices provide additional user control capabilities that can be used in hypermedia applications, and in particular in web-based applications rendered in a browser.
- a known example of this is the aforementioned use of the scrollwheel in controlling the degree of zoom in the web-based Google Maps application.
- a touchpad used as a pointing and data entry device can comprise an array of sensors.
- the array of sensors is used to create a tactile image of a type associated with the type of sensor and method of contact by the human hand.
- the tactile image comprises and array of data elements such as an array of pressure measurements, and array of proximity measurements, an array of reflective optical measurements, etc.
- the tactile image can be or comprise a pressure image, proximity image, reflective optical image, etc.
- each data element comprises a scalar numerical value corresponding to a measurement from an associated sensor.
- at least one data element comprises a plurality of scalar numerical values.
- each data element comprises one or more scalar values produced from signal processing, image processing, and/or other operations applied to measurements provided by an array of sensors.
- the individual sensors in the sensor array are pressure sensors and a direct pressure-sensing tactile image is generated by the sensor array.
- the individual sensors in the sensor array are proximity sensors and a direct proximity tactile image is generated by the sensor array. Since the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the sensor array comprised of proximity sensors also provides an indirect pressure-sensing tactile image.
- the individual sensors in the sensor array can be optical sensors.
- an optical image is generated and an indirect proximity tactile image is generated by the sensor array.
- the optical image can be observed through a transparent or translucent rigid material and, as the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the optical sensor array also provides an indirect pressure-sensing tactile image.
- the array of sensors can be transparent or translucent and can be provided with an underlying visual display element such as an alphanumeric and/or graphics and/or image display.
- the underlying visual display can comprise, for example, an LED array display, a backlit LCD, etc.
- Such an underlying display can be used to render geometric boundaries or labels for soft-key functionality implemented with the tactile sensor array, to display status information, etc.
- the touchpad can comprise a tactile sensor array obtains or provides individual measurements in every enabled cell in the sensor array that provides these as numerical values.
- the numerical values can be communicated in a numerical data array, as a sequential data stream, or in other ways.
- the numerical data array can be regarded as representing a tactile image.
- the tactile sensor array should not be confused with the “null/contact” touchpad which, in normal operation, acts as a pair of orthogonally responsive potentiometers. These “null/contact” touchpads do not produce pressure images, proximity images, or other image data but rather, in normal operation, two voltages linearly corresponding to the location of a left-right edge and forward-back edge of a single area of contact. Such “null/contact” touchpads, which are universally found in existing laptop computers, are discussed and differentiated from tactile sensor arrays in issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978 (pre-grant publication U.S. 2007/0229477).
- a tactile sensor array is a pressure sensor array.
- Pressure sensor arrays discussed in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978. These typically operate by measuring changes in electrical (resistive, capacitive) or optical properties of an elastic material as the material is compressed.
- Prominent manufacturers and suppliers of pressure sensor arrays include Tekscan, Inc. (307 West First Street, South Boston, Mass., 02127, www.tekscan.com), Pressure Profile Systems (5757 Century Boulevard, Suite 600, Los Angeles, Calif. 90045, www.pressureprofile.com), Sensor Products, Inc. (300 Madison Avenue, Madison, N.J. 07940 USA, www.sensorprod.com), and Xsensor Technology Corporation (Suite 111, 319-2nd Ave SW, Calgary, Alberta T2P 0C5, Canada, www.xsensor.com).
- a proximity sensor array or effective equivalents can be used as a tactile sensor array.
- a tactile proximity sensor array suitable for use with the present invention can be implemented in a wide variety of ways using any number of techniques or physical effects. The only requirement is that the tactile proximity sensor array produce a multi-level gradient measurement image as a finger, part of hand, or other pliable object varies is proximity in the immediate area of the sensor surface.
- FIG. 2 illustrates a representative side view of a finger 1001 lightly touching the surface 1002 of a tactile sensor array.
- the finger 1001 contacts the tactile sensor surface in a relatively small area 1003 .
- the finger curves away from the region of contact 1003 , where the non-contacting yet proximate portions of the finger grow increasingly far 1004 a, 1005 a, 1004 b, 1005 b from the surface of the sensor 1002 .
- These variations in physical proximity of portions of the finger with respect to the sensor surface should cause each sensor element in the tactile proximity sensor array to provide a corresponding proximity measurement varying responsively to the proximity, separation distance, etc.
- the tactile proximity sensor array advantageously comprises enough spatial resolution to provide a plurality of sensors within the area occupied by the finger (for example, the area comprising width 1006 ).
- the region of contact 1003 grows as the more and more of the pliable surface of the finger conforms to the tactile sensor array surface 1002 , and the distances 1004 a, 1005 a, 1004 b, 1005 b contract.
- the finger is tilted, for example by rolling in the user viewpoint counterclockwise (which in the depicted end-of-finger viewpoint clockwise 1007 a ) the separation distances on one side of the finger 1004 a, 1005 a will contract while the separation distances on one side of the finger 1004 b, 1005 b will lengthen.
- the finger is tilted, for example by rolling in the user viewpoint clockwise (which in the depicted end-of-finger viewpoint counterclockwise 1007 b ) the separation distances on the side of the finger 1004 b, 1005 b will contract while the separation distances on the side of the finger 1004 a, 1005 a will lengthen.
- Capacitive proximity sensors can be used in various handheld devices with touch interfaces (see for example, among many, http://electronics.howstuffworks.com/iphone2.htm, http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf).
- Prominent manufacturers and suppliers include Balda AG (Bergmüner Str. 228, 32549 Bad Oeynhausen, Del., www.balda.de), Cypress (198 Champion Ct., San Jose, Calif. 95134, www.cypress.com), and Synaptics (2381 Bering Dr., San Jose, Calif. 95131, www.synaptics.com).
- the region of finger contact is detected by variations in localized capacitance resulting from capacitive proximity effects induced by a nearly-adjacent finger. More specifically, the electrical field at the intersection of orthogonally-aligned conductive buses is influenced by the vertical distance or gap between the surface of the sensor array and the skin surface of the finger.
- the capacitive proximity sensor technology is low-cost, reliable, long-life, stable, and can readily be made transparent.
- FIG. 3 (adapted from http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf with slightly more functional detail added) shows a popularly accepted model of a typical cell phone or PDA capacitive proximity sensor implementation.
- the present invention can use the same spatial resolution as current capacitive proximity touchscreen sensor arrays.
- a higher spatial resolution is advantageous.
- the touch of a fingertip can be comprised within the physical dimensions of one sensor element or one sensor-separation spacing.
- the touch of a fingertip can span the physical dimensions of many sensor elements and sensor-separation spacing, for example as in the higher resolution example depicted in (soon to be discussed) FIGS. 4-5 .
- each LED in an array of LEDs can be used as a photodetector as well as a light emitter, although a single LED can either transmit or receive information at one time.
- Each LED in the array can sequentially be selected to be set to be in receiving mode while others adjacent to it are placed in light emitting mode.
- a particular LED in receiving mode can pick up reflected light from the finger, provided by said neighboring illuminating-mode LEDs.
- the invention provides for additional systems and methods for not requiring darkness in the user environment in order to operate an LED array as a tactile proximity sensor.
- potential interference from ambient light in the surrounding user environment can be limited by using an opaque pliable and/or elastically deformable surface covering the LED array that is appropriately reflective (directionally, amorphously, etc. as can be advantageous in a particular design) on the side facing the LED array.
- Such a system and method can be readily implemented in a wide variety of ways as is clear to one skilled in the art.
- potential interference from ambient light in the surrounding user environment can be limited by employing amplitude, phase, or pulse width modulated circuitry and/or software to control the underlying light emission and receiving process.
- the LED array can be configured to emit modulated light modulated at a particular carrier frequency or variation waveform and respond to only modulated light signal components extracted from the received light signals comprising that same carrier frequency or variation waveform.
- a flat or curved translucent panel can be used as sensor surface.
- light applied to the opposite side of the translucent panel reflects light in a distinctly different manner than in other regions where there is no finger or other tactile contact.
- the image captured by an associated video camera will provide gradient information responsive to the contact and proximity of the finger with respect to the surface of the translucent panel.
- the parts of the finger that are in contact with the surface will provide the greatest degree of reflection while parts of the finger that curve away from the surface of the sensor provide less reflection of the light.
- Gradients of the reflected light captured by the video camera can be arranged to produce a gradient image that appears similar to the multilevel quantized image captured by a pressure sensor. By comparing changes in gradient, changes in the position of the finger and pressure applied by the finger can be detected.
- the tactile sensor array can be connected to interface hardware that sends numerical data responsive to tactile information captured by the tactile sensor array to a processor.
- this processor will process the data captured by the tactile sensor array and transform it various ways, for example into a collection of simplified data, or into a sequence of tactile image “frames” (this sequence akin to a video stream), or into highly refined information responsive to the position and movement of one or more fingers and/or other parts of the hand.
- a “frame” can refer to a 2-dimensional list comprising a number of rows and a number of columns forming an array, the array comprising tactile measurement value(s) for every sensor in a tactile sensor array at a given instance.
- each data element comprises a scalar numerical value corresponding to a measurement from an associated sensor.
- at least one data element comprises a plurality of scalar numerical values.
- each data element comprises one or more scalar values produced from signal processing, image processing, and/or other operations applied to measurements provided by an array of sensors. The time interval between one frame and the next one depends on the frame rate of the system and the number of frames in a unit time (usually frames per second).
- FIG. 4 is a graphical representation of a tactile image produced by contact with the bottom surface of the most outward section (between the end of the finger and the most nearby joint) of a human finger on a tactile sensor array.
- this example tactile array there are 24 rows and 24 columns; other realizations can have significantly more (hundreds or thousands) of rows and columns. Tactile measurement values of each cell are indicated by the numbers and shading in each cell. Darker cells represent cells with higher tactile measurement values.
- FIG. 5 provides a graphical representation of an example tactile image produced by contact with multiple human fingers on a tactile sensor array. In other embodiments, there can be a larger or smaller number of pixels for a given images size, resulting in varying resolution.
- Individual sensor elements in a tactile sensor array can vary sensor-by-sensor when presented with the same stimulus.
- the invention provides for each sensor to be individually calibrated in implementations where that can be advantageous. Sensor-by-sensor measurement value scaling, offset, and/or nonlinear warpings can be invoked for all or selected sensor elements during data acquisition scans. Similarly, the invention provides for individual noisy or defective sensors to be tagged for omission of their flawed measurements during data acquisition scans and/or post-scan data processing.
- FIG. 6 depicts an example realization wherein a tactile sensor array is provided with real-time or near-real-time data acquisition capabilities.
- the captured data reflects spatially distributed tactile measurements (such as pressure, proximity, etc.).
- the tactile sensory array and data acquisition stage provides this real-time or near-real-time tactile measurement data to a specialized image processing arrangement for the production of parameters, rates of change of those parameters, and symbols responsive to aspects of the hand's relationship with the tactile or other type of sensor array. In some applications, these measurements can be used directly.
- the real-time or near-real-time derived parameters can be directed to mathematical mappings (such as scaling, offset, and/or nonlinear warpings) in real-time or near-real-time into real-time or near-real-time application-specific parameters or other representations useful for applications.
- general purpose outputs can be assigned to variables defined or expected by the application.
- FIGS. 7 a - 7 f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology. The depiction in these figures is from the side of the touchpad.
- FIGS. 7 a - 7 c show actions of positional change (amounting to applied pressure in the case of FIG. 7 c ) while FIGS. 7 d - 7 f show actions of angular change.
- Each of these can be used to control a user interface parameter, allowing the touch of a single fingertip to control up to six simultaneously-adjustable quantities in an interactive user interface.
- FIG. 8 suggests general ways in which two or more of these independently adjustable degrees of freedom adjusted at once with a single finger 800 :
- HDTP More advanced implementations of the HDTP provide for multi-touch capabilities that can be far more sophisticated that those popularized by the Apple iPhone, NYU, and others.
- FIG. 9 demonstrates a few representative two-finger multi-touch postures and/or gestures from the hundreds that can be readily recognized by HDTP technology.
- HDTP technology can also be configured to recognize and measure postures and/or gestures involving three or more fingers, various parts of the hand, the entire hand, multiple hands, etc., as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605
- FIG. 10 shows an example of how raw measurements of the six quantities of FIGS. 7 a - 7 f, together with shape recognition for distinguishing contact with various parts of the hand and the touchpad, can be used to create a rich information flux of parameters, rates, and symbols, as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605.
- FIG. 11 shows a representative approach for incorporating posture recognition, gesture recognition, state machines, and parsers to create an even richer human/machine tactile interface system capable of incorporating syntax and grammars, as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605.
- sequence of symbols can be directed to a state machine, as shown in FIG. 12 a , to produce other symbols that serve as interpretations of one or more possible symbol sequences.
- one or more symbols can be designated the meaning of an “Enter” key, permitting for sampling one or more varying parameter, rate, and/or symbol values and holding the value(s) until, for example, another “Enter” event, thus producing sustained values as illustrated in FIG. 12 b .
- one or more symbols can be designated as setting a context for interpretation or operation and thus control mapping and/or assignment operations on parameter, rate, and/or symbol values as shown in FIG. 12 c . The operations associated with FIGS.
- FIG. 12 d shows mapping and/or assignment operations that feed an interpretation state machine which in turn controls mapping and/or assignment operations.
- the invention provides for both context-oriented and context-free production of parameter, rate, and symbol values. The parallel production of context-oriented and context-free values can be useful to drive multiple applications simultaneously, for data recording, diagnostics, user feedback, and a wide range of other uses.
- FIG. 13 depicts a representative user arrangement incorporating one or more HDTP system(s) or subsystem(s) that provide(s) user interface input event and routing of HDTP produced parameter values, rate values, symbols, etc. to a variety of applications.
- these parameter values, rate values, symbols, etc. can be produced for example by utilizing one or more of the individual systems, individual methods, and/or individual signals described above in conjunction with the discussion of FIGS. 10 , 11 , and 12 a - 12 b. As discussed later, such an approach can be used with other rich multiparameter user interface devices in place of the HDTP.
- An arrangement similar to that of FIG. 13 is also taught in pending U.S. patent application Ser. No.
- FIG. 13 is adapted from FIG. 6 e of that pending application (U.S. patent application Ser. No. 12/502,230) for further expansion here.
- the Focus Control element uses a selected subset of the information stream provided by the HDTP or other user interface device providing traditional user-adjustable inputs supplemented by additional user-adjustable inputs.
- the Focus Control element uses a selected subset of of the information stream to interpret the user's intention for the direction of focus among several windows, applications, etc.
- the figure shows only applications, but some of these can be replaced with application child windows, operating system, background window, etc.
- focus may be controlled by an ⁇ x,y ⁇ location threshold test and a “select” symbol event, although other information may be used in its place.
- At least two parameters are used for navigation of the cursor when the overall interactive user interface system is in a mode recognizing input from cursor control. These can be, for example, the left-right (“x”) parameter and forward/back (“y”) parameter provided by the touchpad.
- the arrangement of FIG. 13 includes a representative implementation of this.
- these two cursor-control parameters can be provided by another user interface device, for example another touchpad or a separate or attached mouse (the latter to be discussed shortly in the context of FIGS. 15 a - 15 e ).
- another user interface device for example another touchpad or a separate or attached mouse (the latter to be discussed shortly in the context of FIGS. 15 a - 15 e ).
- control of the cursor location can be implemented by more complex means.
- This is the control of location of a 3D cursor wherein a third parameter must be employed to specify the depth coordinate of the cursor location.
- the arrangement of FIG. 13 would be modified to include a third parameter (for use in specifying this depth coordinate) in addition to the left-right (“x”) parameter and forward/back (“y”) parameter described earlier.
- focus control is used to interactively routing user interface signals among applications.
- focus is determined by either the current cursor location or a previous cursor location when a selection event was made.
- this selection event typically involves the user interface providing an event symbol of some type (for example a mouse click, mouse double-click touchpad tap, touchpad double-tap, etc).
- the representative arrangement of FIG. 13 includes an implementation wherein a select event generated by the touchpad system is directed to the focus control element.
- the focus control element in this arrangement in turn controls a focus selection element that directs all or some of the broader information stream from the HDTP system to the currently selected application. (In FIG. 13 , “Application K” has been selected as indicated by the thick-lined box and information-flow arrows.)
- each application that is a candidate for focus selection provides a window displayed at least in part on the screen, or provides a window that can be deiconified from an icon tray or retrieved from beneath other windows that may be obfuscating it.
- focus selection element that directs all or some of the broader information stream from the HDTP system to the operating system, window system, and/or features of the background window.
- the background window can be in fact regarded as merely one of the applications shown in the right portion of the arrangement of FIG. 13 .
- the background window can be in fact regarded as being separate from the applications shown in the right portion of the arrangement of FIG. 13 . In this case the routing of the broader information stream from the HDTP system to the operating system, window system, and/or features of the background window is not explicitly shown in FIG. 13 .
- FIG. 14 a illustrates a HDTP as a peripheral that can be used with a desktop computer (shown) or laptop) not shown).
- FIG. 14 b depicts an HDTP integrated into a laptop in place of the traditional touchpad pointing device.
- the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen.
- FIG. 14 c depicts an HDTP integrated into a desktop computer display so as to form a touchscreen.
- FIG. 14 d shows the HDTP integrated into a laptop computer display so as to form a touchscreen.
- FIG. 14 e depicts an HDTP integrated into a cell phone, smartphone, PDA, or other hand-held consumer device.
- FIG. 14 f shows an HDTP integrated into a test instrument, portable service-tracking device, portable service-entry device, field instrument, or other hand-held industrial device.
- the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen.
- FIG. 14 g depicts an HDTP touchscreen configuration that can be used in a tablet computer, wall-mount computer monitor, digital television, video conferencing screen, kiosk, etc.
- FIGS. 14 a , 14 c , 14 d , and 14 g or other sufficiently large tactile sensor implementation of the HDTP, more than one hand can be used and individually recognized as such.
- FIGS. 15 a - 15 e depict various representative integrations of an HDTP into the back of a conventional computer mouse.
- the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen.
- Such configurations have very recently become popularized by the product release of Apple “Magic MouseTM” although such combinations of a mouse with a tactile sensor array on its back responsive to multitouch and gestures were taught earlier in pending U.S. patent application Ser. No. 12/619,678 (priority date Feb. 12, 2004) entitled “User Interface Mouse with Touchpad Responsive to Gestures and Multi-Touch.”
- more than two touchpads can be included in the advance mouse embodiment, for example as suggested in the arrangement of FIG. 15 e .
- one or more of the plurality of HDTP tactile sensors or exposed sensor areas of arrangements such as that of FIG. 15 e can be integrated over a display so as to form a touchscreen.
- the HDTP in the above examples is used to supply more than the traditional two user interface parameters provided by a conventional user interface input device such as a conventional computer mouse, trackball, touchpad, etc.
- a conventional user interface input device such as a conventional computer mouse, trackball, touchpad, etc.
- the present invention provides for the use of other user interface input arrangements and devices as alternatives to or in conjunction with one or more HDTPs.
- Focus control can be implemented in a manner completely or nearly analogous with FIG. 20 , as well as other approaches (for example as will be presented later in the contexts of FIGS. 19 a - 19 d ).
- the scroll-wheel of a scroll-wheel mouse is used to provide a third simultaneously adjustable user interface parameter.
- a second or yet more additional scroll-wheels can be added to a conventional scroll-wheel mouse.
- the resultant collection of scroll-wheels can be relatively positioned in parallel, oriented at orthogonal angles so as to support a coordinate-metaphor, positioned on the sides of the mouse body, etc.
- FIGS. 16 a and 16 b illustrate examples of conventional scroll-wheel mouse provided with an added left-right scroll-wheel 1622 as taught in U.S. patent application Ser. No. 13/024569.
- Such arrangements can employ a connecting cable, or the device can be wireless.
- FIGS. 17 a - 17 c illustrate examples where a single trackball is incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797.
- FIGS. 18 a - 18 c illustrate examples where two trackballs are incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797.
- 18 a - 18 c can be the conventional two degree of freedom type (roll left-right, roll away-towards) or can provide three to six degrees of freedom as taught in U.S. Pat. No. 7,557,797; U.S. patent application Ser. No. 10/806,694. Such arrangements can employ a connecting cable, or the device can be wireless.
- Another example Advanced Mice arrangements include the trackball/touchpad/mouse combinations of FIGS. 18 c and 18 d and the multiple slider configuration of FIG. 18 e , each taught in U.S. Pat. No. 7,557,797.
- Other example Advanced Mice arrangements include those with two or more scroll wheels (for example as in pending U.S. patent application Ser. No. 13/024,569), a multiple-parameter joystick providing three or more simultaneously adjustable user interface inputs on the back of a mouse (for example as in pending U.S. patent application Ser. No. 13/025,129), and such a multiple-parameter joystick combined with a trackball (for example as also in pending U.S. patent application Ser. No. 13/025,129).
- Each of these arrangements can employ a connecting cable, or the device can be wireless.
- images of the human hand as captured by video cameras can be used as an enhanced multiple-parameter interface responsive to hand positions and gestures, for example as taught in pending U.S. patent application Ser. No. 10/683,915 and more specifically in paragraphs [314], [321]-[332], [411], [653], and (in view of paragraph [325]) also paragraphs [241 ]-[263] of that pending application's pre-grant publication U.S. 2004/0118268.
- the types of human-machine geometric interaction between the hand and the HDTP facilitate many useful applications within a visualization environment.
- a few of these include control of visualization observation viewpoint location, orientation of the visualization, and controlling fixed or selectable ensembles of one or more of viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, simulation control parameters, etc.
- the 6D orientation of a finger can be naturally associated with visualization observation viewpoint location and orientation, location and orientation of the visualization graphics, etc.
- the 6D orientation of a finger can be naturally associated with a vector field orientation for introducing synthetic measurements in a numerical simulation.
- the 6D orientation of a finger can be naturally associated with the orientation of a robotically positioned sensor providing actual measurement data.
- the 6D orientation of a finger can be naturally associated with an object location and orientation in a numerical simulation.
- the large number of interactive parameters can be abstractly associated with viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, numeric simulation control parameters, etc.
- the “x” and “y” parameters provided by the HDTP can be used for focus selection and the remaining parameters can be used to control parameters within a selected GUI.
- the “x” and “y” parameters provided by the HDTP can be regarded as a specifying a position within an underlying base plane and the roll and pitch angles can be regarded as a specifying a position within a superimposed parallel plane.
- the yaw angle can be regarded as the rotational angle between the base and superimposed planes.
- the finger pressure can be employed to determine the distance between the base and superimposed planes.
- the base and superimposed plane can not be fixed as parallel but rather intersect as an angle associated with the yaw angle of the finger.
- either or both of the two planes can represent an index or indexed data, a position, pair of parameters, etc. of a viewing aspect, visualization rendering aspect, pre-visualization operations, data selection, numeric simulation control, etc.
- the USB HID device class provides an open interface useful for both traditional computer pointing devices such as the standard computer mouse and other user interface devices such as game controllers.
- the USB HID device class has also been used to interface with the Logitech 3DConnexion SpaceNavigatorTM.
- the USB HID device class is currently specified at the time of this patent application by at least the Device Class Definition for HID 1.11, currently available at http://www.usb.org/developers/devclass docs/HID1 11.pdf.
- the invention provides for the USB HID device class to be used for at least additional user interface signals (user interface parameters) provided by the High Dimensional Touchpad (HTPD), Advanced Mice, and other multidimensional or rich parameter user interfaces that generate additional user interface signals above those found in traditional computer mice, touchpads, and trackballs.
- user interface parameters user interface signals
- HTPD High Dimensional Touchpad
- Advanced Mice Advanced Mice
- other multidimensional or rich parameter user interfaces that generate additional user interface signals above those found in traditional computer mice, touchpads, and trackballs.
- a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor that is connected to the computer via a USB interface.
- the exemplary HDTP signal processing and HDTP gesture processing are implemented on the computer or other device.
- the HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
- FIG. 19 a depicts an exemplary implementation of such an embodiment.
- a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor and one or more associated processor(s) which in turn is/are connected to the computer via a USB interface.
- the exemplary HDTP signal processing and HDTP gesture detection are implemented on the one or more processor(s) associated with HDTP sensor.
- the HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
- FIG. 19 b depicts an exemplary implementation of such an embodiment.
- a USB HID device abstraction is used as a software interface even though no USB port is actually used.
- the HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
- FIG. 19 c depicts an exemplary implementation of such an embodiment.
- ADPs can interface to a computer or other device in yet other ways.
- a special purpose interface can be used.
- the Bluetooth networking standard can be used.
- the additional interactively-controlled user input parameters provided by an HDTP (such as that taught in the 1999 filings of issued U.S. Pat. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S. patent applications), Advanced Mice (such as that Mice taught in the 2004 filings of issued U.S. Pat. No. 7,557,797 and related pending U.S. patent applications such as Ser. Nos. 12/619,678, 13/025,129, 13/024,569), and other rich multiparameter user interface devices supply more interactively-controlled parameters than the established number supported by conventional window and operating systems.
- HDTP such as that taught in the 1999 filings of issued U.S. Pat. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,
- the invention provides for additional user interface parameter signals provided by the not only the High Dimensional Touchpad (HTPD) and Advanced Mice, but also other multidimensional or rich parameter user interfaces providing additional user interface signals above those found in traditional computer mice, touchpads, and trackballs.
- HTPD High Dimensional Touchpad
- Advanced Mice other multidimensional or rich parameter user interface devices providing additional user interface signals above those found in traditional computer mice, touchpads, and trackballs
- APDs Advanced Pointing Devices
- the entire (interactively-controlled) information flux provided by an APD is carried over the same framework used to carry the traditional computer mouse/touchpad user interface signals from conventional pointing devices.
- only the driver for the APD need be added and recognized by the window system, window manager, operating system, or combination of these.
- the window system, window manager, operating system, or combination of these then distributes the entire (interactively-controlled) information flux to the application selected according to focus control implemented by the operating system.
- window systems, window managers, and operating systems such an approach can be implemented without modification.
- window managers, operating systems, or combination of these such an approach can require a modification to the window and/or operating system(s). Should a particular existing window system, window manager, operating system, or combination of these resident on a computing device require such modification, the invention provides for the modification to be implemented via a downloadable patch or other form of an update (for example, using a data-storage media).
- FIGS. 19 a and 19 b depict a representative rendering of this approach.
- the driver for the APD presents traditional computer mouse/touchpad user interface signals from conventional pointing devices (thin straight arrowed lines) to the window system, window manager, operating system, or combination of these as well as additional computer mouse/touchpad user interface signals (thick straight arrowed lines) from the APD.
- the window system, window manager, operating system, or combination of these comprises a focus control functionality used to selectively route the traditional user interface signals and additional user interface signals.
- the focus control can be responsive to at least the position of a displayed cursor with respect to a displayed application window, the cursor and application window displayed on a display screen.
- merely positioning the cursor within the window of an application makes a focus selection to that application.
- positioning the cursor within the window of an application is not alone sufficient to make a focus selection to that application; instead the focus stays with the last selection until a user-provided selection event is made, for example a mouse click or double click, a touchpad tap or double-tap, a trackball button click or double click, etc.
- focus control (for example, as defined by cursor location with respect to one or more displayed application windows) is responsive traditional (computer mouse/touchpad/trackball) user interface signals (thin straight arrowed lines).
- traditional user interface signals such as a system employing a 3D desktop
- at least one additional parameter can be also directed to focus control and/or cursor location.
- there are a plurality of applications some designed to accept only traditional computer mouse/touchpad user interface signals (in the upper right of each figure) as well as other applications designed to accept these traditional signals as well as one or more of the additional user interface signals provided by the APD (in the lower right of each figure).
- the focus control routes only the traditional interface signals to a selected application designed to accept only traditional computer mouse/touchpad user interface signals.
- the focus control routes a larger collection of signals, including both traditional computer mouse/touchpad user interface signals as well as at least one additional user interface signal made available by the APD.
- the window system, window manager, operating system, or combination of these only distributes traditional computer mouse/touchpad user interface signals from conventional pointing devices and other provisions are used to direct the additional user interface parameter signals provided by the APD to selected applications.
- This can be implemented in a number of ways. In one example, depicted in FIG. 19 c , separate focus controls are used, each responsive to the traditional user interface signals provided by the APD. In another example, depicted in FIG. 19 d , the operating system focus control provides signals to the routing element for the additional user interface parameter signals provided by the APD. Other variations are anticipated and are provided for by the invention.
- user interface signals are routed to an application, the application it self can utilize or sub-route the user interface signals in various ways.
- Some applications such as data visualization, maps, simulations, CAD systems, etc. can beneficially use more than three simultaneously interactively adjustable user inputs directly.
- Other applications such as browsers and viewers, can support such applications indirectly as taught and discussed for example in pending U.S. patent application Ser. No. 12/875,119.
- Browsers, viewers, and hypermedia documents can also be provided with advanced hypermedia objects that generalize the notion of hyperlinks, rollovers, sliders, buttons, etc. that are configured to utilize additional user interface signals; such advanced hypermedia objects taught and discussed for example in pending U.S. Patent Application 61/435,395.
Abstract
A method for routing signals from a user interface device providing traditional user interface device signals and additional user interface signals to an application is described. Traditional user interface device signals and additional user interface signals are received from a user interface device. Routing of traditional user interface device signals and additional user interface signals from such a user interface device to particular applications can be made responsive to input focus control provided by a windowing system, window manager, operating system, or combination. In one approach, input focus control is provided a single focus control element. In another approach, separate focus control elements are used for traditional user interface device signals and additional user interface signals.
Description
- Pursuant to 35 U.S.C. §119(e), this application claims benefit of priority from Provisional U.S. Patent application Ser. No. 61/303898, filed Feb. 12, 2010, the contents of which are incorporated by reference.
- Certain marks referenced herein may be common law or registered trademarks of the applicant, the assignee or third parties affiliated or unaffiliated with the applicant or the assignee. Use of these marks is for providing an enabling disclosure by way of example and shall not be construed to exclusively limit the scope of the disclosed subject matter to material associated with such marks.
- 1. Field of the Invention
- The invention relates to user interface devices that provide additional user interface control signals beyond those of a traditional mouse, touchpad, or track ball, and relates also to window manager input focus control, and in particular to the direction of signals from such a user interface device to particular applications responsive to input focus control provided by a windowing system, operating system, or both.
- 2. Overview of the Invention
- The present invention addresses the routing of additional user interface signals provided by the High Dimensional Touchpad “HTPD” (for example as taught in 1999 filings of U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S. patent applications), Advanced Mice (for example as taught in U.S. Pat. No. 7,557,797 pending U.S. patent application Ser. Nos. 12/619,678, 13/025,129, 13/024,569, and related pending U.S. patent applications), and other multidimensional or rich parameter user interfaces providing additional user interface signals above those found in traditional computer mice, touchpads, and trackballs. The routing of signals from such a user interface device to particular applications can be made responsive to input focus control provided by a windowing system, operating system, or both.
- For purposes of summarizing, certain aspects, advantages, and novel features are described herein. Not all such advantages may be achieved in accordance with any one particular embodiment. Thus, the disclosed subject matter may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages without achieving all advantages as may be taught or suggested herein.
- In one aspect of the invention, a method for routing signals from a user interface device providing traditional user interface device signals and additional user interface signals to an application includes receiving traditional user interface device signals and additional user interface signals from a user interface device, routing the traditional user interface device signals to a selected application according to a first input focus selection, and routing the traditional [additional?] user interface device signals to the selected application according to a second input focus selection. The first and second input focus selection made by at least one focus control element.
- Another aspect of the invention is that the at least one focus control element comprises a window manager, a window system, or an operating system. Further, the first and second input focus selection is made by the same focus control element, or the first and second input focus selection are made by a first focus control element and the second input focus selection is made by a second focus control element.
- The user input device may a computer mouse comprising a first and second scroll wheel, a touchpad, or a High Definition Touch Pad (HDTP). When the user input device comprises a touch user interface, the touch user interface is responsive to gestures and the additional user-adjustable input comprises at least one gesture.
- The touch user interface may be responsive to one of the yaw angle, roll angle, or pitch angle of a finger in contact with the touch user interface. The additional user-adjustable input is responsive to the yaw angle, roll angle, or pitch angle. The touch user interface also may be responsive at least two angles of a finger in contact with the touch user interface, and the additional user-adjustable input is responsive to a measurement of each of the two angles.
- The above and other aspects, features and advantages of the present invention will become more apparent upon consideration of the following description of preferred embodiments taken in conjunction with the accompanying drawing figures.
-
FIG. 1 depicts a plurality of windows, one or more of which can be a hypermedia application window such as a browser, and hierarchies of visually displayed and other objects within or associated with these windows. -
FIG. 2 illustrates the side view of a finger lightly touching the surface of a tactile sensor array. -
FIG. 3 depicts a popularly accepted model of a typical cell phone or PDA capacitive proximity sensor implementation. -
FIG. 4 is a graphical representation of a tactile image produced by contact of a human finger on a tactile sensor array. -
FIG. 5 provides a graphical representation of a tactile image produced by contact with multiple human fingers on a tactile sensor array of lesser spatial resolution than that depicted inFIG. 4 . -
FIG. 6 depicts a signal flow in an HDTP implementation. -
FIGS. 7 a-7 f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology. -
FIG. 8 suggests general ways in which two or more of these independently adjustable degrees of freedom adjusted simultaneously. -
FIG. 9 demonstrates a few two-finger multi-touch postures and/or gestures from the many that can be readily recognized by HDTP technology. -
FIG. 10 shows an example of how raw measurements of the six quantities ofFIGS. 7 a-7 f, together with shape recognition for distinguishing contact with various parts of the hand and the touchpad, can be used to create a rich information flux of parameters, rates, and symbols. -
FIG. 11 shows an approach for incorporating posture recognition, gesture recognition, and other functions to create a rich human/machine tactile interface system capable of additionally supporting or incorporating syntax and grammars. -
FIGS. 12 a-12 d depict operations acting on various parameters, rates, and symbols to produce other parameters, rates, and symbols, including operations such as sample/hold, interpretation, context, etc. -
FIG. 13 depicts a user interface input arrangement incorporating one or more HDTPs that provides user interface input event and quantity routing for focus control. -
FIGS. 14 a-14 g depict a number of arrangements and embodiments employing HDTP technology. -
FIGS. 15 a-15 e depict various integrations of an HDTP into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797 and pending U.S. patent application Ser. No. 12/619,678. -
FIGS. 16 a and 16 b illustrate examples of conventional scroll-wheel mouse provided with an added left-right scroll-wheel as taught in U.S. patent application Ser. No. 13/024569. -
FIGS. 17 a-17 c illustrate examples where a single trackball is incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797. -
FIGS. 18 a-18 c illustrate examples where two trackballs are incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797, some of these (FIGS. 18 b-18 c) comprising yet other additional sensors. -
FIG. 18 d depicts a mouse provided with a trackball and a small touchpad as taught in U.S. Pat. No. 7,557,797. -
FIG. 18 e depicts a mouse provided with a plurality of slider controls as taught in U.S. Pat. No. 7,557,797. -
FIGS. 19 a-19 c depicts exemplary embodiments providing HDTP technologies with a HID device abstraction for interfacing to applications. -
FIGS. 20 a-20 d depict arrangements for directing additional user interface parameter signals to applications. - In the following description, reference is made to the accompanying drawing figures which form a part hereof, and which show by way of illustration specific embodiments of the invention. It is to be understood by those of ordinary skill in this technological field that other embodiments may be utilized, and structural, electrical, as well as procedural changes may be made without departing from the scope of the present invention.
- In the following, numerous specific details are set forth to provide a thorough description of various embodiments. Certain embodiments may be practiced without these specific details or with some variations in detail. In some instances, certain features are described in less detail so as not to obscure other aspects. The level of detail associated with each of the elements or features should not be construed to qualify the novelty or importance of one feature over the others.
- Desktop, laptop, tablet, web, and other types of contemporary computers provide for a plurality of active software applications to share visual display and user input devices by means of some form of windowing system. Windowing systems are well known with foundational principles dating back decades (see for example F. R. Hopgood, et al., Methodology of Window Management, Springer-Verlag, Berlin, 1986, ISBN 0387161163) and are known at least as an operational level to virtually all users of these devices. Without getting into the many well-known aspects of windowing systems, one skilled in the art is reminded that:
-
- A plurality of windows can be displayed simultaneously on the screen of the (desktop, laptop, tablet, or web) computer;
- Multiple windows can overlap one another;
- The windowing system also provides a visually-rendered cursor whose position is determined by left-right/forward-back operation provisions of a pointing device (mouse, touchpad, trackball, etc.);
- Windows are typically selected by “clicking” a discrete-event provision (button operation, touchpad tap, etc.) of the pointing device—windows can also be selected by default in some cases, such as when the initialization of a previously inactive application displays, updates, or pops-up a new window;
- A selected window remains selected until the user selects a different window or a window is selected by default;
- User keyboard input and other types of pointing device input is typically directed to aspects of an application associated with the window that is currently selected.
-
FIG. 1 depicts a visualdisplay screen area 100 displaying a plurality of representative windows—here 101, 102, 110. In this figure., none of thewindows browser window 110. The hypermedia (browser or application)window 110 ofFIG. 1 also depicts a toolbar 110.tb as well as a vertical scrollbar 110.vs and a horizontal scrollbar 110.hs. Such vertical and horizontal scrollbars typically appear when the display area within the window is smaller than the vertical and/or horizontal span of the visual content, allowing control of the vignette displayed within the aperture created by the display, this control responsive to the positions of scrollbar(s) within the degrees of possible travel. The position of a scrollbar is in turn controlled by one or more input aspects from the pointing device. - Turning now to traditional user interface pointing devices, the traditional mouse, traditional trackball, and traditional touchpad, and traditional touchscreen typically are used to provide the following user inputs:
-
Traditional User Input Traditional Traditional Touchpad or Type Mouse Trackball Touchscreen Cursor “X” Left-right position Left-right rotation Left-right position position of housing of trackball of finger/stylus Cursor “Y” Forward-back Forward-back Left-right position position position of rotation of of finger/stylus housing trackball Left click Left button Left button Left button and/or tap Right click Right button Right button Right button Double (left) Double operation Double operation Double operation click of Left button of Left button of Left button and/or double-tap - More contemporary computer mice additionally provide a scrollwheel along with the traditional components and features of the traditional mouse. Some scrollwheels allow the wheel to be depressed downward to operate a spring-loaded switch that provides a third class of button events. As mentioned just above, typically the scrollwheel provided by a contemporary computer mouse is solely directed to the operation of the vertical scrollbar (for example, the scrollbar 110.vs of
FIG. 1 ), if displayed, of the currently selected window. More recently, computer mice providing “2-way scrolling” (sometimes called “4-way scrolling”) features wherein the scrollwheel, in addition to conventional forward-back rotation, can be tilted left or right with the resulting signal directed to the control the horizontal scrollbar (for example, the scrollbar 110.hs ofFIG. 1 ), if displayed, of the currently selected window. - Providing an additional scroll control to a scrollwheel mouse that can be used to operate the horizontal scrollbar with a left-right operation was taught several years prior to the appearance of such products in the specification of issued U.S. Pat. No. 7,557,797 (priority date Feb. 12, 2004) and is to be addressed in a pending continuation patent application from that specification subject to that same priority date.
- Additionally, touch screens have recently received tremendous attention with the addition of array tactile imaging capabilities. Such touch screen technologies permit multi-touch sensing, metaphors, and gestures. Although such touch screen technologies have obtained great commercial success from there defining role in the iPhone and subsequent adaptations in PDAs and other types of cell phones and hand-held devices, these were in fact taught in the 1999 filings of U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978.
- These more advanced user interface pointing devices provide additional user control capabilities that can be used in hypermedia applications, and in particular in web-based applications rendered in a browser. A known example of this is the aforementioned use of the scrollwheel in controlling the degree of zoom in the web-based Google Maps application.
- Further, there remains a wide range of additional control capabilities that can be provided by further enhanced user interface technologies. A number of representative enhanced user interface technologies are described next, specifically:
-
- (a) the HDTP taught in the 1999 filings of U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S. patent applications; and
- (b) the Advanced Mice taught in the 2004 filings of issued U.S. Pat. No. 7,557,797 and related pending U.S. patent applications such as Ser. Nos. 12/619,678, 13/025,129, and 13/024,569. The capabilities of these, or to a more limited extent, the capabilities of contemporary generation user interface pointing devices can be used to enhance the capabilities of traditional hypermedia objects (such as the hyperlink, button, rollover, menu, and slider) as well as defining new types of hypermedia objects.
- In an embodiment, a touchpad used as a pointing and data entry device can comprise an array of sensors. The array of sensors is used to create a tactile image of a type associated with the type of sensor and method of contact by the human hand. The tactile image comprises and array of data elements such as an array of pressure measurements, and array of proximity measurements, an array of reflective optical measurements, etc. Thus the tactile image can be or comprise a pressure image, proximity image, reflective optical image, etc. In an embodiment, each data element comprises a scalar numerical value corresponding to a measurement from an associated sensor. In another embodiment, at least one data element comprises a plurality of scalar numerical values. In an embodiment, each data element comprises one or more scalar values produced from signal processing, image processing, and/or other operations applied to measurements provided by an array of sensors.
- In one embodiment, the individual sensors in the sensor array are pressure sensors and a direct pressure-sensing tactile image is generated by the sensor array.
- In another embodiment, the individual sensors in the sensor array are proximity sensors and a direct proximity tactile image is generated by the sensor array. Since the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the sensor array comprised of proximity sensors also provides an indirect pressure-sensing tactile image.
- In another embodiment, the individual sensors in the sensor array can be optical sensors. In one variation of this, an optical image is generated and an indirect proximity tactile image is generated by the sensor array. In another variation, the optical image can be observed through a transparent or translucent rigid material and, as the contacting surfaces of the finger or hand tissue contacting a surface typically increasingly deforms as pressure is applied, the optical sensor array also provides an indirect pressure-sensing tactile image.
- In some embodiments, the array of sensors can be transparent or translucent and can be provided with an underlying visual display element such as an alphanumeric and/or graphics and/or image display. The underlying visual display can comprise, for example, an LED array display, a backlit LCD, etc. Such an underlying display can be used to render geometric boundaries or labels for soft-key functionality implemented with the tactile sensor array, to display status information, etc.
- In an embodiment, the touchpad can comprise a tactile sensor array obtains or provides individual measurements in every enabled cell in the sensor array that provides these as numerical values. The numerical values can be communicated in a numerical data array, as a sequential data stream, or in other ways. When regarded as a numerical data array with row and column ordering that can be associated with the geometric layout of the individual cells of the sensor array, the numerical data array can be regarded as representing a tactile image.
- The tactile sensor array should not be confused with the “null/contact” touchpad which, in normal operation, acts as a pair of orthogonally responsive potentiometers. These “null/contact” touchpads do not produce pressure images, proximity images, or other image data but rather, in normal operation, two voltages linearly corresponding to the location of a left-right edge and forward-back edge of a single area of contact. Such “null/contact” touchpads, which are universally found in existing laptop computers, are discussed and differentiated from tactile sensor arrays in issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978 (pre-grant publication U.S. 2007/0229477). Before leaving this topic, it is pointed out that these the “null/contact” touchpads nonetheless can be inexpensively adapted with simple analog electronics to provide at least primitive multi-touch capabilities as taught in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978 (therein, paragraphs [0022]-[0029] of its pre-grant publication U.S. 2007/0229477, for example).
- One implementation of a tactile sensor array is a pressure sensor array. Pressure sensor arrays discussed in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978. These typically operate by measuring changes in electrical (resistive, capacitive) or optical properties of an elastic material as the material is compressed. Prominent manufacturers and suppliers of pressure sensor arrays include Tekscan, Inc. (307 West First Street, South Boston, Mass., 02127, www.tekscan.com), Pressure Profile Systems (5757 Century Boulevard, Suite 600, Los Angeles, Calif. 90045, www.pressureprofile.com), Sensor Products, Inc. (300 Madison Avenue, Madison, N.J. 07940 USA, www.sensorprod.com), and Xsensor Technology Corporation (
Suite 111, 319-2nd Ave SW, Calgary, Alberta T2P 0C5, Canada, www.xsensor.com). - In lieu of a pressure sensor array, a proximity sensor array or effective equivalents (for example, as can be accomplished with a video camera as described in issued U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978) can be used as a tactile sensor array. In general, a tactile proximity sensor array suitable for use with the present invention can be implemented in a wide variety of ways using any number of techniques or physical effects. The only requirement is that the tactile proximity sensor array produce a multi-level gradient measurement image as a finger, part of hand, or other pliable object varies is proximity in the immediate area of the sensor surface.
- More specifically,
FIG. 2 illustrates a representative side view of afinger 1001 lightly touching thesurface 1002 of a tactile sensor array. In this example, thefinger 1001 contacts the tactile sensor surface in a relativelysmall area 1003. In this situation, on either side the finger curves away from the region ofcontact 1003, where the non-contacting yet proximate portions of the finger grow increasingly far 1004 a, 1005 a, 1004 b, 1005 b from the surface of thesensor 1002. These variations in physical proximity of portions of the finger with respect to the sensor surface should cause each sensor element in the tactile proximity sensor array to provide a corresponding proximity measurement varying responsively to the proximity, separation distance, etc. The tactile proximity sensor array advantageously comprises enough spatial resolution to provide a plurality of sensors within the area occupied by the finger (for example, the area comprising width 1006). In this case, as the finger is pressed down, the region ofcontact 1003 grows as the more and more of the pliable surface of the finger conforms to the tactilesensor array surface 1002, and thedistances finger finger finger finger - Capacitive proximity sensors can be used in various handheld devices with touch interfaces (see for example, among many, http://electronics.howstuffworks.com/iphone2.htm, http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf). Prominent manufacturers and suppliers include Balda AG (Bergkirchener Str. 228, 32549 Bad Oeynhausen, Del., www.balda.de), Cypress (198 Champion Ct., San Jose, Calif. 95134, www.cypress.com), and Synaptics (2381 Bering Dr., San Jose, Calif. 95131, www.synaptics.com). In these sensors, the region of finger contact is detected by variations in localized capacitance resulting from capacitive proximity effects induced by a nearly-adjacent finger. More specifically, the electrical field at the intersection of orthogonally-aligned conductive buses is influenced by the vertical distance or gap between the surface of the sensor array and the skin surface of the finger. The capacitive proximity sensor technology is low-cost, reliable, long-life, stable, and can readily be made transparent.
FIG. 3 (adapted from http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf with slightly more functional detail added) shows a popularly accepted model of a typical cell phone or PDA capacitive proximity sensor implementation. In some embodiments the present invention can use the same spatial resolution as current capacitive proximity touchscreen sensor arrays. In other embodiments of the present invention, a higher spatial resolution is advantageous. For example, in many contemporary capacitive proximity sensors, the touch of a fingertip can be comprised within the physical dimensions of one sensor element or one sensor-separation spacing. In higher resolution implementations, the touch of a fingertip can span the physical dimensions of many sensor elements and sensor-separation spacing, for example as in the higher resolution example depicted in (soon to be discussed)FIGS. 4-5 . - As a first example of an optical array sensor, Forrest M. Mims is credited as showing that a conventional LED can be used as a light detector as well as a light emitter. Recently, light-emitting diodes have been used as a tactile proximity sensor array (for example, as depicted in the video available at http://cs.nyu.edu/˜jhan/ledtouch/index.html). Such tactile proximity array implementations typically need to be operated in a darkened environment (as seen in the video in the above web link). In one embodiment provided for by the invention, each LED in an array of LEDs can be used as a photodetector as well as a light emitter, although a single LED can either transmit or receive information at one time. Each LED in the array can sequentially be selected to be set to be in receiving mode while others adjacent to it are placed in light emitting mode. A particular LED in receiving mode can pick up reflected light from the finger, provided by said neighboring illuminating-mode LEDs. The invention provides for additional systems and methods for not requiring darkness in the user environment in order to operate an LED array as a tactile proximity sensor. In one embodiment, potential interference from ambient light in the surrounding user environment can be limited by using an opaque pliable and/or elastically deformable surface covering the LED array that is appropriately reflective (directionally, amorphously, etc. as can be advantageous in a particular design) on the side facing the LED array. Such a system and method can be readily implemented in a wide variety of ways as is clear to one skilled in the art. In another embodiment, potential interference from ambient light in the surrounding user environment can be limited by employing amplitude, phase, or pulse width modulated circuitry and/or software to control the underlying light emission and receiving process. For example, in an implementation the LED array can be configured to emit modulated light modulated at a particular carrier frequency or variation waveform and respond to only modulated light signal components extracted from the received light signals comprising that same carrier frequency or variation waveform. Such a system and method can be readily implemented in a wide variety of ways as is clear to one skilled in the art.
- As a second example of an optical array sensor, use of video cameras for gathering control information from the human hand in various ways is discussed in U.S. Pat. No. 6,570,078 and Pending U.S. patent application Ser. No. 11/761,978. In another video camera tactile controller embodiment, a flat or curved translucent panel can be used as sensor surface. When a finger is placed on the translucent panel, light applied to the opposite side of the translucent panel reflects light in a distinctly different manner than in other regions where there is no finger or other tactile contact. The image captured by an associated video camera will provide gradient information responsive to the contact and proximity of the finger with respect to the surface of the translucent panel. For example, the parts of the finger that are in contact with the surface will provide the greatest degree of reflection while parts of the finger that curve away from the surface of the sensor provide less reflection of the light. Gradients of the reflected light captured by the video camera can be arranged to produce a gradient image that appears similar to the multilevel quantized image captured by a pressure sensor. By comparing changes in gradient, changes in the position of the finger and pressure applied by the finger can be detected.
- In many various embodiments, the tactile sensor array can be connected to interface hardware that sends numerical data responsive to tactile information captured by the tactile sensor array to a processor. In various embodiments, this processor will process the data captured by the tactile sensor array and transform it various ways, for example into a collection of simplified data, or into a sequence of tactile image “frames” (this sequence akin to a video stream), or into highly refined information responsive to the position and movement of one or more fingers and/or other parts of the hand.
- As to further representative detail of the latter example, a “frame” can refer to a 2-dimensional list comprising a number of rows and a number of columns forming an array, the array comprising tactile measurement value(s) for every sensor in a tactile sensor array at a given instance. In an embodiment, each data element comprises a scalar numerical value corresponding to a measurement from an associated sensor. In another embodiment, at least one data element comprises a plurality of scalar numerical values. In an embodiment, each data element comprises one or more scalar values produced from signal processing, image processing, and/or other operations applied to measurements provided by an array of sensors. The time interval between one frame and the next one depends on the frame rate of the system and the number of frames in a unit time (usually frames per second).
FIG. 4 is a graphical representation of a tactile image produced by contact with the bottom surface of the most outward section (between the end of the finger and the most nearby joint) of a human finger on a tactile sensor array. In this example tactile array, there are 24 rows and 24 columns; other realizations can have significantly more (hundreds or thousands) of rows and columns. Tactile measurement values of each cell are indicated by the numbers and shading in each cell. Darker cells represent cells with higher tactile measurement values. Similarly,FIG. 5 provides a graphical representation of an example tactile image produced by contact with multiple human fingers on a tactile sensor array. In other embodiments, there can be a larger or smaller number of pixels for a given images size, resulting in varying resolution. Additionally, there can be larger or smaller area with respect to the image size resulting in a greater or lesser potential measurement area for the region of contact to be located in or move about. (Note the sensor array ofFIG. 3 has less spatial resolution than that associated withFIG. 5 , which in turn has less spatial resolution than that associated withFIG. 4 . - Individual sensor elements in a tactile sensor array can vary sensor-by-sensor when presented with the same stimulus. The invention provides for each sensor to be individually calibrated in implementations where that can be advantageous. Sensor-by-sensor measurement value scaling, offset, and/or nonlinear warpings can be invoked for all or selected sensor elements during data acquisition scans. Similarly, the invention provides for individual noisy or defective sensors to be tagged for omission of their flawed measurements during data acquisition scans and/or post-scan data processing.
-
FIG. 6 depicts an example realization wherein a tactile sensor array is provided with real-time or near-real-time data acquisition capabilities. The captured data reflects spatially distributed tactile measurements (such as pressure, proximity, etc.). The tactile sensory array and data acquisition stage provides this real-time or near-real-time tactile measurement data to a specialized image processing arrangement for the production of parameters, rates of change of those parameters, and symbols responsive to aspects of the hand's relationship with the tactile or other type of sensor array. In some applications, these measurements can be used directly. In other situations, the real-time or near-real-time derived parameters can be directed to mathematical mappings (such as scaling, offset, and/or nonlinear warpings) in real-time or near-real-time into real-time or near-real-time application-specific parameters or other representations useful for applications. In some embodiments, general purpose outputs can be assigned to variables defined or expected by the application. -
FIGS. 7 a-7 f illustrate the six independently adjustable degrees of freedom of touch from a single finger that can be simultaneously measured by the HDTP technology. The depiction in these figures is from the side of the touchpad.FIGS. 7 a-7 c show actions of positional change (amounting to applied pressure in the case ofFIG. 7 c) whileFIGS. 7 d-7 f show actions of angular change. Each of these can be used to control a user interface parameter, allowing the touch of a single fingertip to control up to six simultaneously-adjustable quantities in an interactive user interface. In more detail: -
-
FIG. 7 a depicts variation of the left/right position (“x”) of the finger contact; -
FIG. 7 b depicts variation of the forward/back position (“y”) of the finger contact; -
FIG. 7 c depicts variation of the up/down position or downward pressure (“p”) of the finger contact; -
FIG. 7 d depicts variation of the clockwise/counterclockwise (yaw) angle (“ψ”) of the finger contact; -
FIG. 7 e depicts variation of the left/right tilt (roll) angle (“φ”) of the finger contact; -
FIG. 7 f depicts variation of the forward/back (pitch) angle (“θ”) of the finger contact.
-
-
FIG. 8 suggests general ways in which two or more of these independently adjustable degrees of freedom adjusted at once with a single finger 800: -
- left/right position (“x”) of the
finger contact 811; - forward/back position (“y”) of the
finger contact 812; - up/down position or downward pressure (“p”) of the
finger contact 816; - clockwise/counterclockwise (yaw) angle (“ψ”) of the
finger contact 815; - left/right tilt (roll) angle (“φ”) of the
finger contact 813; - forward/back (pitch) angle (“θ”) of the
finger contact 814.
- left/right position (“x”) of the
- More advanced implementations of the HDTP provide for multi-touch capabilities that can be far more sophisticated that those popularized by the Apple iPhone, NYU, and others.
-
FIG. 9 demonstrates a few representative two-finger multi-touch postures and/or gestures from the hundreds that can be readily recognized by HDTP technology. HDTP technology can also be configured to recognize and measure postures and/or gestures involving three or more fingers, various parts of the hand, the entire hand, multiple hands, etc., as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605 -
FIG. 10 shows an example of how raw measurements of the six quantities ofFIGS. 7 a-7 f, together with shape recognition for distinguishing contact with various parts of the hand and the touchpad, can be used to create a rich information flux of parameters, rates, and symbols, as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605. -
FIG. 11 shows a representative approach for incorporating posture recognition, gesture recognition, state machines, and parsers to create an even richer human/machine tactile interface system capable of incorporating syntax and grammars, as taught for example in U.S. Pat. No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605. - The HDTP affords and provides for yet further capabilities. For example, sequence of symbols can be directed to a state machine, as shown in
FIG. 12 a, to produce other symbols that serve as interpretations of one or more possible symbol sequences. In an embodiment, one or more symbols can be designated the meaning of an “Enter” key, permitting for sampling one or more varying parameter, rate, and/or symbol values and holding the value(s) until, for example, another “Enter” event, thus producing sustained values as illustrated inFIG. 12 b. In an embodiment, one or more symbols can be designated as setting a context for interpretation or operation and thus control mapping and/or assignment operations on parameter, rate, and/or symbol values as shown inFIG. 12 c. The operations associated withFIGS. 12 a-12 c can be combined to provide yet other capabilities. For example, the example arrangement ofFIG. 12 d shows mapping and/or assignment operations that feed an interpretation state machine which in turn controls mapping and/or assignment operations. In implementations where context is involved, such as in arrangements such as those depicted inFIGS. 12 b-12 d, the invention provides for both context-oriented and context-free production of parameter, rate, and symbol values. The parallel production of context-oriented and context-free values can be useful to drive multiple applications simultaneously, for data recording, diagnostics, user feedback, and a wide range of other uses. -
FIG. 13 depicts a representative user arrangement incorporating one or more HDTP system(s) or subsystem(s) that provide(s) user interface input event and routing of HDTP produced parameter values, rate values, symbols, etc. to a variety of applications. In an embodiment, these parameter values, rate values, symbols, etc. can be produced for example by utilizing one or more of the individual systems, individual methods, and/or individual signals described above in conjunction with the discussion ofFIGS. 10 , 11, and 12 a-12 b. As discussed later, such an approach can be used with other rich multiparameter user interface devices in place of the HDTP. An arrangement similar to that ofFIG. 13 is also taught in pending U.S. patent application Ser. No. 12/502,230 “Control of Computer Window Systems, Computer Applications, and Web Applications via High Dimensional Touchpad User Interface” by Seung Lim, andFIG. 13 is adapted fromFIG. 6 e of that pending application (U.S. patent application Ser. No. 12/502,230) for further expansion here. - In an implementation approach or modality of operation for an arrangement such as the one of
FIG. 13 , the Focus Control element uses a selected subset of the information stream provided by the HDTP or other user interface device providing traditional user-adjustable inputs supplemented by additional user-adjustable inputs. The Focus Control element uses a selected subset of of the information stream to interpret the user's intention for the direction of focus among several windows, applications, etc. The figure shows only applications, but some of these can be replaced with application child windows, operating system, background window, etc. In this example, focus may be controlled by an {x,y} location threshold test and a “select” symbol event, although other information may be used in its place. - In an arrangement such as the one of
FIG. 13 , or in other implementations, at least two parameters are used for navigation of the cursor when the overall interactive user interface system is in a mode recognizing input from cursor control. These can be, for example, the left-right (“x”) parameter and forward/back (“y”) parameter provided by the touchpad. The arrangement ofFIG. 13 includes a representative implementation of this. - Alternatively, these two cursor-control parameters can be provided by another user interface device, for example another touchpad or a separate or attached mouse (the latter to be discussed shortly in the context of
FIGS. 15 a-15 e). - In some situations, control of the cursor location can be implemented by more complex means. One example of this is the control of location of a 3D cursor wherein a third parameter must be employed to specify the depth coordinate of the cursor location. For such situations, the arrangement of
FIG. 13 would be modified to include a third parameter (for use in specifying this depth coordinate) in addition to the left-right (“x”) parameter and forward/back (“y”) parameter described earlier. - In an embodiment, focus control is used to interactively routing user interface signals among applications. In most current systems, there is at least some modality wherein the focus is determined by either the current cursor location or a previous cursor location when a selection event was made. In the user experience, this selection event typically involves the user interface providing an event symbol of some type (for example a mouse click, mouse double-click touchpad tap, touchpad double-tap, etc). The representative arrangement of
FIG. 13 includes an implementation wherein a select event generated by the touchpad system is directed to the focus control element. The focus control element in this arrangement in turn controls a focus selection element that directs all or some of the broader information stream from the HDTP system to the currently selected application. (InFIG. 13 , “Application K” has been selected as indicated by the thick-lined box and information-flow arrows.) - In some embodiments, each application that is a candidate for focus selection provides a window displayed at least in part on the screen, or provides a window that can be deiconified from an icon tray or retrieved from beneath other windows that may be obfuscating it. In some embodiments, if the background window is selected, focus selection element that directs all or some of the broader information stream from the HDTP system to the operating system, window system, and/or features of the background window. In some embodiments, the background window can be in fact regarded as merely one of the applications shown in the right portion of the arrangement of
FIG. 13 . In other embodiments, the background window can be in fact regarded as being separate from the applications shown in the right portion of the arrangement ofFIG. 13 . In this case the routing of the broader information stream from the HDTP system to the operating system, window system, and/or features of the background window is not explicitly shown inFIG. 13 . -
FIGS. 14 a-14 g and 15 a-15 e depict a number of representative arrangements and embodiments employing the HDTP technology.FIG. 14 a illustrates a HDTP as a peripheral that can be used with a desktop computer (shown) or laptop) not shown).FIG. 14 b depicts an HDTP integrated into a laptop in place of the traditional touchpad pointing device. InFIGS. 14 a-14 b the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen.FIG. 14 c depicts an HDTP integrated into a desktop computer display so as to form a touchscreen.FIG. 14 d shows the HDTP integrated into a laptop computer display so as to form a touchscreen. -
FIG. 14 e depicts an HDTP integrated into a cell phone, smartphone, PDA, or other hand-held consumer device.FIG. 14 f shows an HDTP integrated into a test instrument, portable service-tracking device, portable service-entry device, field instrument, or other hand-held industrial device. InFIGS. 14 e-14 f the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen. -
FIG. 14 g depicts an HDTP touchscreen configuration that can be used in a tablet computer, wall-mount computer monitor, digital television, video conferencing screen, kiosk, etc. - In at least the arrangements of
FIGS. 14 a, 14 c, 14 d, and 14 g, or other sufficiently large tactile sensor implementation of the HDTP, more than one hand can be used and individually recognized as such. -
FIGS. 15 a-15 e depict various representative integrations of an HDTP into the back of a conventional computer mouse. InFIGS. 15 a-15 d the HDTP tactile sensor can be a stand-alone component or can be integrated over a display so as to form a touchscreen. Such configurations have very recently become popularized by the product release of Apple “Magic Mouse™” although such combinations of a mouse with a tactile sensor array on its back responsive to multitouch and gestures were taught earlier in pending U.S. patent application Ser. No. 12/619,678 (priority date Feb. 12, 2004) entitled “User Interface Mouse with Touchpad Responsive to Gestures and Multi-Touch.” - In another embodiment taught in the specification of issued U.S. Pat. No. 7,557,797 and associated pending continuation applications more than two touchpads can be included in the advance mouse embodiment, for example as suggested in the arrangement of
FIG. 15 e. As with the arrangements ofFIGS. 15 a-15 d, one or more of the plurality of HDTP tactile sensors or exposed sensor areas of arrangements such as that ofFIG. 15 e can be integrated over a display so as to form a touchscreen. - The HDTP in the above examples is used to supply more than the traditional two user interface parameters provided by a conventional user interface input device such as a conventional computer mouse, trackball, touchpad, etc. The present invention provides for the use of other user interface input arrangements and devices as alternatives to or in conjunction with one or more HDTPs. In this section the features and capabilities of Advanced Mice are briefly reviewed and set up for their use in embodiments of the invention. Focus control can be implemented in a manner completely or nearly analogous with
FIG. 20 , as well as other approaches (for example as will be presented later in the contexts ofFIGS. 19 a-19 d). - In a simple example, the scroll-wheel of a scroll-wheel mouse is used to provide a third simultaneously adjustable user interface parameter. In another example, a second or yet more additional scroll-wheels can be added to a conventional scroll-wheel mouse. The resultant collection of scroll-wheels can be relatively positioned in parallel, oriented at orthogonal angles so as to support a coordinate-metaphor, positioned on the sides of the mouse body, etc.
FIGS. 16 a and 16 b illustrate examples of conventional scroll-wheel mouse provided with an added left-right scroll-wheel 1622 as taught in U.S. patent application Ser. No. 13/024569. Such arrangements can employ a connecting cable, or the device can be wireless. - In another example of Advanced Mice, one or more trackballs can be added to a conventional computer mouse, for example on the back of the mouse.
FIGS. 17 a-17 c illustrate examples where a single trackball is incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797.FIGS. 18 a-18 c illustrate examples where two trackballs are incorporated into the back of a conventional computer mouse as taught in U.S. Pat. No. 7,557,797. The trackballs in the arrangements ofFIGS. 17 a-17 c andFIGS. 18 a-18 c can be the conventional two degree of freedom type (roll left-right, roll away-towards) or can provide three to six degrees of freedom as taught in U.S. Pat. No. 7,557,797; U.S. patent application Ser. No. 10/806,694. Such arrangements can employ a connecting cable, or the device can be wireless. - Another example Advanced Mice arrangements include the trackball/touchpad/mouse combinations of
FIGS. 18 c and 18 d and the multiple slider configuration ofFIG. 18 e, each taught in U.S. Pat. No. 7,557,797. Other example Advanced Mice arrangements include those with two or more scroll wheels (for example as in pending U.S. patent application Ser. No. 13/024,569), a multiple-parameter joystick providing three or more simultaneously adjustable user interface inputs on the back of a mouse (for example as in pending U.S. patent application Ser. No. 13/025,129), and such a multiple-parameter joystick combined with a trackball (for example as also in pending U.S. patent application Ser. No. 13/025,129). - Each of these arrangements can employ a connecting cable, or the device can be wireless.
- Additionally, images of the human hand as captured by video cameras can be used as an enhanced multiple-parameter interface responsive to hand positions and gestures, for example as taught in pending U.S. patent application Ser. No. 10/683,915 and more specifically in paragraphs [314], [321]-[332], [411], [653], and (in view of paragraph [325]) also paragraphs [241 ]-[263] of that pending application's pre-grant publication U.S. 2004/0118268.
- The types of human-machine geometric interaction between the hand and the HDTP facilitate many useful applications within a visualization environment. A few of these include control of visualization observation viewpoint location, orientation of the visualization, and controlling fixed or selectable ensembles of one or more of viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, simulation control parameters, etc. As one example, the 6D orientation of a finger can be naturally associated with visualization observation viewpoint location and orientation, location and orientation of the visualization graphics, etc. As another example, the 6D orientation of a finger can be naturally associated with a vector field orientation for introducing synthetic measurements in a numerical simulation.
- As yet another example, at least some aspects of the 6D orientation of a finger can be naturally associated with the orientation of a robotically positioned sensor providing actual measurement data. As another example, the 6D orientation of a finger can be naturally associated with an object location and orientation in a numerical simulation. As another example, the large number of interactive parameters can be abstractly associated with viewing parameters, visualization rendering parameters, pre-visualization operations parameters, data selection parameters, numeric simulation control parameters, etc.
- In yet another example, the “x” and “y” parameters provided by the HDTP can be used for focus selection and the remaining parameters can be used to control parameters within a selected GUI.
- In still another example, the “x” and “y” parameters provided by the HDTP can be regarded as a specifying a position within an underlying base plane and the roll and pitch angles can be regarded as a specifying a position within a superimposed parallel plane. In a first example extension of the previous two-plane example, the yaw angle can be regarded as the rotational angle between the base and superimposed planes. In a second example extension of the previous two-plane example, the finger pressure can be employed to determine the distance between the base and superimposed planes. In a variation of the previous two-plane example, the base and superimposed plane can not be fixed as parallel but rather intersect as an angle associated with the yaw angle of the finger. In the each of these, either or both of the two planes can represent an index or indexed data, a position, pair of parameters, etc. of a viewing aspect, visualization rendering aspect, pre-visualization operations, data selection, numeric simulation control, etc.
- A large number of additional approaches are possible as is appreciated by one skilled in the art. These are provided for by the invention.
- The USB HID device class provides an open interface useful for both traditional computer pointing devices such as the standard computer mouse and other user interface devices such as game controllers. The USB HID device class has also been used to interface with the Logitech 3DConnexion SpaceNavigator™. The USB HID device class is currently specified at the time of this patent application by at least the Device Class Definition for HID 1.11, currently available at http://www.usb.org/developers/devclass docs/HID1 11.pdf. More generally, the invention provides for the USB HID device class to be used for at least additional user interface signals (user interface parameters) provided by the High Dimensional Touchpad (HTPD), Advanced Mice, and other multidimensional or rich parameter user interfaces that generate additional user interface signals above those found in traditional computer mice, touchpads, and trackballs. This can be done in a number ways, for example as taught in pending U.S. patent application Ser. No. 61/435,401 and as described below in material adapted from that pending U.S. Patent Application.
- In a first exemplary embodiment, a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor that is connected to the computer via a USB interface. Here the exemplary HDTP signal processing and HDTP gesture processing are implemented on the computer or other device. The HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
FIG. 19 a depicts an exemplary implementation of such an embodiment. - In another exemplary embodiment, a USB HID device abstraction is employed to connect a computer or other device with an HDTP sensor and one or more associated processor(s) which in turn is/are connected to the computer via a USB interface. Here the exemplary HDTP signal processing and HDTP gesture detection are implemented on the one or more processor(s) associated with HDTP sensor. The HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
FIG. 19 b depicts an exemplary implementation of such an embodiment. - In another exemplary embodiment, a USB HID device abstraction is used as a software interface even though no USB port is actually used. The HDTP signal processing and HDTP gesture processing implementation can be realized via one or more of CPU software, GPU software, embedded processor software or firmware, and/or a dedicated integrated circuit.
FIG. 19 c depicts an exemplary implementation of such an embodiment. Alternatively, ADPs can interface to a computer or other device in yet other ways. For example, a special purpose interface can be used. As another example, the Bluetooth networking standard can be used. - There are a number of ways conventional window systems, window managers, and operating systems can be used or adapted to support the additional interactively-controlled parameters provided by an APD. A few examples are provided here, and other approaches are anticipated by the invention.
- The additional interactively-controlled user input parameters provided by an HDTP (such as that taught in the 1999 filings of issued U.S. Pat. 6,570,078 and pending U.S. patent application Ser. No. 11/761,978, pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S. patent applications), Advanced Mice (such as that Mice taught in the 2004 filings of issued U.S. Pat. No. 7,557,797 and related pending U.S. patent applications such as Ser. Nos. 12/619,678, 13/025,129, 13/024,569), and other rich multiparameter user interface devices supply more interactively-controlled parameters than the established number supported by conventional window and operating systems. Provisions to support the use of additional interactively-controlled parameters provided by HDTP, Advanced Mice, and other rich multiparameter user interface devices with existing or extended operating systems has been taught in pending U.S. patent application Ser. No. 12/875,128. Some material from pending U.S. patent application Ser. No. 12/875,128 is directly adapted in this section for convenience. Additionally, images of the human hand as captured by video cameras can be used as an enhanced multiple-parameter interface responsive to hand positions and gestures, for example as taught in pending U.S. patent application Ser. No. 10/683,915 and more specifically in paragraphs [314], [321]-[332], [411], [653], and (in view of paragraph [325]) also paragraphs [241]-[263] of that pending application's pre-grant publication U.S. 2004/0118268.
- More generally, the invention provides for additional user interface parameter signals provided by the not only the High Dimensional Touchpad (HTPD) and Advanced Mice, but also other multidimensional or rich parameter user interfaces providing additional user interface signals above those found in traditional computer mice, touchpads, and trackballs. This fuller collection (HDTP, Advanced Mice, other multidimensional or rich parameter user interface devices providing additional user interface signals above those found in traditional computer mice, touchpads, and trackballs) will be collectively referred to as Advanced Pointing Devices (APDs).
- In one approach, the entire (interactively-controlled) information flux provided by an APD is carried over the same framework used to carry the traditional computer mouse/touchpad user interface signals from conventional pointing devices. In one version of this approach, only the driver for the APD need be added and recognized by the window system, window manager, operating system, or combination of these. The window system, window manager, operating system, or combination of these then distributes the entire (interactively-controlled) information flux to the application selected according to focus control implemented by the operating system. For some window systems, window managers, and operating systems, such an approach can be implemented without modification. In other implementations of window systems, window managers, operating systems, or combination of these, such an approach can require a modification to the window and/or operating system(s). Should a particular existing window system, window manager, operating system, or combination of these resident on a computing device require such modification, the invention provides for the modification to be implemented via a downloadable patch or other form of an update (for example, using a data-storage media).
-
FIGS. 19 a and 19 b depict a representative rendering of this approach. In each figure, the driver for the APD presents traditional computer mouse/touchpad user interface signals from conventional pointing devices (thin straight arrowed lines) to the window system, window manager, operating system, or combination of these as well as additional computer mouse/touchpad user interface signals (thick straight arrowed lines) from the APD. In each of these approaches, as well as other variations clear to one skilled in the art, the window system, window manager, operating system, or combination of these comprises a focus control functionality used to selectively route the traditional user interface signals and additional user interface signals. The focus control can be responsive to at least the position of a displayed cursor with respect to a displayed application window, the cursor and application window displayed on a display screen. In some approaches or operating modes, merely positioning the cursor within the window of an application makes a focus selection to that application. In other approaches or operating modes, positioning the cursor within the window of an application is not alone sufficient to make a focus selection to that application; instead the focus stays with the last selection until a user-provided selection event is made, for example a mouse click or double click, a touchpad tap or double-tap, a trackball button click or double click, etc. - In the suggestive rendering of
FIGS. 19 a and 19 b, focus control (for example, as defined by cursor location with respect to one or more displayed application windows) is responsive traditional (computer mouse/touchpad/trackball) user interface signals (thin straight arrowed lines). In other arrangements, such as a system employing a 3D desktop, at least one additional parameter can be also directed to focus control and/or cursor location. In the suggestive rendering ofFIGS. 19 a and 19 b, there are a plurality of applications, some designed to accept only traditional computer mouse/touchpad user interface signals (in the upper right of each figure) as well as other applications designed to accept these traditional signals as well as one or more of the additional user interface signals provided by the APD (in the lower right of each figure). The applicable portions of the description applies even if there are fewer applications of either or both types, or if there is only one type or only one application overall. In the case ofFIG. 19 a, the focus control routes only the traditional interface signals to a selected application designed to accept only traditional computer mouse/touchpad user interface signals. In the case ofFIG. 19 b, the focus control routes a larger collection of signals, including both traditional computer mouse/touchpad user interface signals as well as at least one additional user interface signal made available by the APD. - In another approach, the window system, window manager, operating system, or combination of these only distributes traditional computer mouse/touchpad user interface signals from conventional pointing devices and other provisions are used to direct the additional user interface parameter signals provided by the APD to selected applications. This can be implemented in a number of ways. In one example, depicted in
FIG. 19 c, separate focus controls are used, each responsive to the traditional user interface signals provided by the APD. In another example, depicted inFIG. 19 d, the operating system focus control provides signals to the routing element for the additional user interface parameter signals provided by the APD. Other variations are anticipated and are provided for by the invention. - Once user interface signals are routed to an application, the application it self can utilize or sub-route the user interface signals in various ways. Some applications, such as data visualization, maps, simulations, CAD systems, etc. can beneficially use more than three simultaneously interactively adjustable user inputs directly. Other applications, such as browsers and viewers, can support such applications indirectly as taught and discussed for example in pending U.S. patent application Ser. No. 12/875,119. Browsers, viewers, and hypermedia documents can also be provided with advanced hypermedia objects that generalize the notion of hyperlinks, rollovers, sliders, buttons, etc. that are configured to utilize additional user interface signals; such advanced hypermedia objects taught and discussed for example in pending U.S. Patent Application 61/435,395.
- While the invention has been described in detail with reference to disclosed embodiments, various modifications within the scope of the invention will be apparent to those of ordinary skill in this technological field. It is to be appreciated that features described with respect to one embodiment typically can be applied to other embodiments.
- The invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Therefore, the invention properly is to be construed with reference to the claims.
- Although exemplary embodiments have been provided in detail, it should be understood that various changes, substitutions and alternations could be made thereto without departing from spirit and scope of the disclosed subject matter as defined by the appended claims. Variations described for exemplary embodiments may be realized in any combination desirable for each particular application. Thus particular limitations, and/or embodiment enhancements described herein, which may have particular advantages to a particular application, need not be used for all applications. Also, not all limitations need be implemented in methods, systems, and/or apparatuses including one or more concepts described with relation to the provided exemplary embodiments.
Claims (14)
1. A method for routing signals from a user interface device to an application, the user interface device providing traditional user interface device signals and additional user interface signals, the method comprising:
receiving traditional user interface device signals and additional user interface signals from a user interface device;
routing the traditional user interface device signals to a selected application according to a first input focus selection;
routing the additional user interface device signals to the selected application according to a second input focus selection;
wherein the first and second input focus selection made by at least one focus control element.
2. The method of claim 1 wherein the at least one focus control element comprises a window manager.
3. The method of claim 1 wherein the at least one focus control element comprises a window system.
4. The method of claim 1 wherein the at least one focus control element comprises an operating system.
5. The method of claim 1 wherein both the first and second input focus selection is made by the same focus control element.
6. The method of claim 1 wherein the first and second input focus selection made by a first focus control element and second input focus selection made by a second focus control element.
7. The method of claim 1 wherein the user input device is a computer mouse comprising a first and second scroll wheel.
8. The method of claim 1 wherein the user input device is a computer mouse comprising a touchpad.
9. The method of claim 1 wherein the user input device is a computer mouse comprising a High Definition Touch Pad (HDTP).
10. The method of claim 1 wherein the user input device comprises a touch user interface responsive to gestures and the at least one additional user-adjustable input comprises at least one gesture.
11. The method of claim 1 wherein the user input device comprises a touch user interface responsive to the yaw angle of a finger in contact with the touch user interface and the at least one additional user-adjustable input is responsive to a measurement of the yaw angle.
12. The method of claim 1 wherein the user input device comprises a touch user interface responsive to the roll angle of a finger in contact with the touch user interface and the at least one additional user-adjustable input is responsive to a measurement of the roll angle.
13. The method of claim 1 wherein the user input device comprises a touch user interface responsive to the pitch angle of a finger in contact with the touch user interface and the at least one additional user-adjustable input is responsive to a measurement of the pitch angle.
14. The method of claim 1 wherein the user input device comprises a touch user interface responsive to at least two angles of a finger in contact with the touch user interface and the at least one additional user-adjustable input is responsive to a measurement of each of the two angles.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/026,097 US20110202934A1 (en) | 2010-02-12 | 2011-02-11 | Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US30389810P | 2010-02-12 | 2010-02-12 | |
US13/026,097 US20110202934A1 (en) | 2010-02-12 | 2011-02-11 | Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces |
Publications (1)
Publication Number | Publication Date |
---|---|
US20110202934A1 true US20110202934A1 (en) | 2011-08-18 |
Family
ID=44370515
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/026,097 Abandoned US20110202934A1 (en) | 2010-02-12 | 2011-02-11 | Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces |
US13/026,248 Expired - Fee Related US9830042B2 (en) | 2010-02-12 | 2011-02-12 | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice |
US15/815,446 Abandoned US20180074680A1 (en) | 2010-02-12 | 2017-11-16 | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (hdtp), other advanced touch user interfaces, and advanced mice |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/026,248 Expired - Fee Related US9830042B2 (en) | 2010-02-12 | 2011-02-12 | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice |
US15/815,446 Abandoned US20180074680A1 (en) | 2010-02-12 | 2017-11-16 | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (hdtp), other advanced touch user interfaces, and advanced mice |
Country Status (1)
Country | Link |
---|---|
US (3) | US20110202934A1 (en) |
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110210943A1 (en) * | 2010-03-01 | 2011-09-01 | Lester F. Ludwig | Curve-fitting approach to hdtp parameter extraction |
US8477111B2 (en) | 2008-07-12 | 2013-07-02 | Lester F. Ludwig | Advanced touch control of interactive immersive imaging applications via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8509542B2 (en) | 2009-03-14 | 2013-08-13 | Lester F. Ludwig | High-performance closed-form single-scan calculation of oblong-shape rotation angles from binary images of arbitrary size and location using running sums |
US8604364B2 (en) | 2008-08-15 | 2013-12-10 | Lester F. Ludwig | Sensors, algorithms and applications for a high dimensional touchpad |
US8702513B2 (en) | 2008-07-12 | 2014-04-22 | Lester F. Ludwig | Control of the operating system on a computing device via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8717303B2 (en) | 1998-05-15 | 2014-05-06 | Lester F. Ludwig | Sensor array touchscreen recognizing finger flick gesture and other touch gestures |
US8754862B2 (en) | 2010-07-11 | 2014-06-17 | Lester F. Ludwig | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces |
US8797288B2 (en) | 2011-03-07 | 2014-08-05 | Lester F. Ludwig | Human user interfaces utilizing interruption of the execution of a first recognized gesture with the execution of a recognized second gesture |
US8826113B2 (en) | 2009-09-02 | 2014-09-02 | Lester F. Ludwig | Surface-surface graphical intersection tools and primitives for data visualization, tabular data, and advanced spreadsheets |
US9019237B2 (en) | 2008-04-06 | 2015-04-28 | Lester F. Ludwig | Multitouch parameter and gesture user interface employing an LED-array tactile sensor that can also operate as a display |
US9052772B2 (en) | 2011-08-10 | 2015-06-09 | Lester F. Ludwig | Heuristics for 3D and 6D touch gesture touch parameter calculations for high-dimensional touch parameter (HDTP) user interfaces |
US20150312617A1 (en) * | 2012-11-29 | 2015-10-29 | Zte Corporation | Method, apparatus and system for controlling focus on TV interface |
US9605881B2 (en) | 2011-02-16 | 2017-03-28 | Lester F. Ludwig | Hierarchical multiple-level control of adaptive cooling and energy harvesting arrangements for information technology |
US9626023B2 (en) | 2010-07-09 | 2017-04-18 | Lester F. Ludwig | LED/OLED array approach to integrated display, lensless-camera, and touch-screen user interface devices and associated processors |
US9632344B2 (en) | 2010-07-09 | 2017-04-25 | Lester F. Ludwig | Use of LED or OLED array to implement integrated combinations of touch screen tactile, touch gesture sensor, color image display, hand-image gesture sensor, document scanner, secure optical data exchange, and fingerprint processing capabilities |
US9823781B2 (en) | 2011-12-06 | 2017-11-21 | Nri R&D Patent Licensing, Llc | Heterogeneous tactile sensing via multiple sensor types |
US9830042B2 (en) | 2010-02-12 | 2017-11-28 | Nri R&D Patent Licensing, Llc | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice |
US9950256B2 (en) | 2010-08-05 | 2018-04-24 | Nri R&D Patent Licensing, Llc | High-dimensional touchpad game controller with multiple usage and networking modalities |
US10430066B2 (en) | 2011-12-06 | 2019-10-01 | Nri R&D Patent Licensing, Llc | Gesteme (gesture primitive) recognition for advanced touch user interfaces |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101027971B1 (en) * | 2010-12-10 | 2011-04-13 | (주)헬릭스테크 | Mobile communication terminal capable of testing application and method thereof |
JP2012252642A (en) * | 2011-06-06 | 2012-12-20 | Sony Corp | Information processor, information processing method and program |
US20130074003A1 (en) * | 2011-09-21 | 2013-03-21 | Nokia Corporation | Method and apparatus for integrating user interfaces |
US9877699B2 (en) * | 2012-03-26 | 2018-01-30 | Teratech Corporation | Tablet ultrasound system |
US10552767B2 (en) * | 2012-06-15 | 2020-02-04 | Ebay Inc. | Method and a system for allocating ticket inventory on buyer side |
KR20130143160A (en) * | 2012-06-20 | 2013-12-31 | 삼성전자주식회사 | Apparatus and method for scrolling a information of terminal equipment having touch device |
US20140052580A1 (en) * | 2012-08-17 | 2014-02-20 | Kallidus, Inc. | Product explorer page for use with interactive digital catalogs and touch-screen devices |
EP3033692A4 (en) * | 2013-08-15 | 2017-03-01 | Nokia Technologies Oy | Apparatus and method for facilitating browser navigation |
CN104461306B (en) * | 2013-09-23 | 2017-12-01 | 博雅网络游戏开发(深圳)有限公司 | rolling assembly control method and device |
US20150121314A1 (en) * | 2013-10-24 | 2015-04-30 | Jens Bombolowsky | Two-finger gestures |
US20150178844A1 (en) * | 2013-12-20 | 2015-06-25 | Bank Of America Corporation | Customized Retirement Planning |
US10262335B2 (en) * | 2014-02-21 | 2019-04-16 | Ebay Inc. | Systems and methods for real time upgrades |
KR101623495B1 (en) * | 2014-10-08 | 2016-05-23 | 주식회사 맥키스컴퍼니 | Apparatus for selling item |
GB2543779B (en) * | 2015-10-27 | 2018-07-25 | Imagination Tech Ltd | Systems and methods for processing images of objects |
GB2543778B (en) * | 2015-10-27 | 2018-08-08 | Imagination Tech Ltd | Systems and methods for processing images of objects |
GB2543777B (en) * | 2015-10-27 | 2018-07-25 | Imagination Tech Ltd | Systems and methods for processing images of objects |
KR101749933B1 (en) * | 2015-11-12 | 2017-06-22 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
CN107016010A (en) * | 2016-06-30 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The treating method and apparatus of hyperlinked object |
CN107992259A (en) * | 2016-10-27 | 2018-05-04 | 中兴通讯股份有限公司 | The definite method and device of kinematic parameter |
US11216150B2 (en) * | 2019-06-28 | 2022-01-04 | Wen-Chieh Geoffrey Lee | Pervasive 3D graphical user interface with vector field functionality |
Citations (91)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4748676A (en) * | 1986-05-16 | 1988-05-31 | Fuji Electric Co., Ltd. | Pattern rotation angle detecting system |
US4899137A (en) * | 1987-09-26 | 1990-02-06 | Aeg Olympia Aktiengesellschaft | Arrangement for the input and processing of characters and/or graphic patterns |
US5237647A (en) * | 1989-09-15 | 1993-08-17 | Massachusetts Institute Of Technology | Computer aided drawing in three dimensions |
US5270711A (en) * | 1989-05-08 | 1993-12-14 | U.S. Philips Corporation | Touch sensor array systems and display systems incorporating such |
US5292999A (en) * | 1991-08-14 | 1994-03-08 | Fernandes Co., Ltd. | Electric stringed instrument having a device for sustaining the vibration of the string |
US5341133A (en) * | 1991-05-09 | 1994-08-23 | The Rowland Institute For Science, Inc. | Keyboard having touch sensor keys for conveying information electronically |
US5347295A (en) * | 1990-10-31 | 1994-09-13 | Go Corporation | Control of a computer through a position-sensed stylus |
US5357048A (en) * | 1992-10-08 | 1994-10-18 | Sgroi John J | MIDI sound designer with randomizer function |
US5378850A (en) * | 1992-01-14 | 1995-01-03 | Fernandes Co., Ltd. | Electric stringed instrument having an arrangement for adjusting the generation of magnetic feedback |
US5386219A (en) * | 1991-10-16 | 1995-01-31 | International Business Machines Corp. | Touch overlay for improved touch sensitivity |
US5420936A (en) * | 1992-10-16 | 1995-05-30 | International Business Machines Corporation | Method and apparatus for accessing touch screen desktop objects via fingerprint recognition |
US5440072A (en) * | 1992-09-25 | 1995-08-08 | Willis; Raymon A. | System for rejuvenating vintage organs and pianos |
US5442168A (en) * | 1991-10-15 | 1995-08-15 | Interactive Light, Inc. | Dynamically-activated optical instrument for producing control signals having a self-calibration means |
US5459282A (en) * | 1992-09-25 | 1995-10-17 | Willis; Raymon A. | System for rejuvenating vintage organs and pianos |
US5471008A (en) * | 1990-11-19 | 1995-11-28 | Kabushiki Kaisha Kawai Gakki Seisakusho | MIDI control apparatus |
US5475214A (en) * | 1991-10-15 | 1995-12-12 | Interactive Light, Inc. | Musical sound effects controller having a radiated emission space |
US5565641A (en) * | 1994-03-28 | 1996-10-15 | Gruenbaum; Leon | Relativistic electronic musical instrument |
US5585588A (en) * | 1991-10-24 | 1996-12-17 | Fernandes Co., Ltd. | Electric stringed instrument having a device for sustaining the vibration of a string and an electromagnetic driver for the device |
US5592572A (en) * | 1993-11-05 | 1997-01-07 | The United States Of America As Represented By The Department Of Health And Human Services | Automated portrait/landscape mode detection on a binary image |
US5592752A (en) * | 1993-05-13 | 1997-01-14 | Industrial Technology Research Institute | Process and an apparatus for producing teas |
US5659145A (en) * | 1995-04-27 | 1997-08-19 | Weil; Robert P. | Foot operated audio signal controller with lighted visual reference |
US5659466A (en) * | 1994-11-02 | 1997-08-19 | Advanced Micro Devices, Inc. | Monolithic PC audio circuit with enhanced digital wavetable audio synthesizer |
US5665927A (en) * | 1993-06-30 | 1997-09-09 | Casio Computer Co., Ltd. | Method and apparatus for inputting musical data without requiring selection of a displayed icon |
US5668338A (en) * | 1994-11-02 | 1997-09-16 | Advanced Micro Devices, Inc. | Wavetable audio synthesizer with low frequency oscillators for tremolo and vibrato effects |
US5675100A (en) * | 1993-11-03 | 1997-10-07 | Hewlett; Walter B. | Method for encoding music printing information in a MIDI message |
US5717939A (en) * | 1991-11-18 | 1998-02-10 | Compaq Computer Corporation | Method and apparatus for entering and manipulating spreadsheet cell data |
US5719347A (en) * | 1994-12-07 | 1998-02-17 | Yamaha Corporation | Keyboard apparatus for electronic musical instrument with key depression detection unit |
US5719561A (en) * | 1995-10-25 | 1998-02-17 | Gilbert R. Gonzales | Tactile communication device and method |
US5724985A (en) * | 1995-08-02 | 1998-03-10 | Pacesetter, Inc. | User interface for an implantable medical device using an integrated digitizer display screen |
US5741993A (en) * | 1995-08-01 | 1998-04-21 | Kabushiki Kaisha Kawai Gakki Seisakusho | Electronic keyboard having a discrete pitch bender |
US5748184A (en) * | 1996-05-28 | 1998-05-05 | International Business Machines Corporation | Virtual pointing device for touchscreens |
US5763806A (en) * | 1996-07-15 | 1998-06-09 | Willis; Raymon A. | Method and apparatus for midifying vintage organs and pianos |
US5786540A (en) * | 1996-03-05 | 1998-07-28 | Westlund; Robert L. | Controller apparatus for music sequencer |
US5801340A (en) * | 1995-06-29 | 1998-09-01 | Invotronics Manufacturing | Proximity sensor |
US5805137A (en) * | 1991-11-26 | 1998-09-08 | Itu Research, Inc. | Touch sensitive input control device |
US5824930A (en) * | 1995-06-09 | 1998-10-20 | Yamaha Corporation | Keyboard musical instrument having key monitor exactly discriminating key motion |
US5827989A (en) * | 1997-06-23 | 1998-10-27 | Microsoft Corporation | System and method for representing a musical event and for converting the musical event into a series of discrete events |
US5841428A (en) * | 1993-11-05 | 1998-11-24 | Intertactile Technologies Corporation | Rotary circuit control devices with changeable graphics |
US5850051A (en) * | 1996-08-15 | 1998-12-15 | Yamaha Corporation | Method and apparatus for creating an automatic accompaniment pattern on the basis of analytic parameters |
US5852251A (en) * | 1997-06-25 | 1998-12-22 | Industrial Technology Research Institute | Method and apparatus for real-time dynamic midi control |
US5889236A (en) * | 1992-06-08 | 1999-03-30 | Synaptics Incorporated | Pressure sensitive scrollbar feature |
US5932827A (en) * | 1995-01-09 | 1999-08-03 | Osborne; Gary T. | Sustainer for a musical instrument |
US5969283A (en) * | 1998-06-17 | 1999-10-19 | Looney Productions, Llc | Music organizer and entertainment center |
US5977466A (en) * | 1996-07-11 | 1999-11-02 | Yamaha Corporation | Keyboard musical instrument equipped with small simple economical key touch generator |
US5986224A (en) * | 1995-04-19 | 1999-11-16 | Elo Touchsystems, Inc. | Acoustic condition sensor employing a plurality of mutually non-orthogonal waves |
US6005545A (en) * | 1995-01-17 | 1999-12-21 | Sega Enterprise, Ltd. | Image processing method and electronic device |
US6037937A (en) * | 1997-12-04 | 2000-03-14 | Nortel Networks Corporation | Navigation tool for graphical user interface |
US6047073A (en) * | 1994-11-02 | 2000-04-04 | Advanced Micro Devices, Inc. | Digital wavetable audio synthesizer with delay-based effects processing |
US6051769A (en) * | 1998-11-25 | 2000-04-18 | Brown, Jr.; Donival | Computerized reading display |
US6100461A (en) * | 1998-06-10 | 2000-08-08 | Advanced Micro Devices, Inc. | Wavetable cache using simplified looping |
US6107997A (en) * | 1996-06-27 | 2000-08-22 | Ure; Michael J. | Touch-sensitive keyboard/mouse and computing device using the same |
US6140565A (en) * | 1998-06-08 | 2000-10-31 | Yamaha Corporation | Method of visualizing music system by combination of scenery picture and player icons |
US6204441B1 (en) * | 1998-04-09 | 2001-03-20 | Yamaha Corporation | Method and apparatus for effectively displaying musical information with visual display |
US6225975B1 (en) * | 1997-03-18 | 2001-05-01 | Alps Electric Co., Ltd. | Control device for game machine |
US6285358B1 (en) * | 1993-04-30 | 2001-09-04 | Microtouch Systems, Inc. | Method of and apparatus for the elimination of the effects of inertial interference in force measurement systems, including touch-input computer and related displays employing touch force location measurement techniques |
US6288317B1 (en) * | 1998-05-29 | 2001-09-11 | Raymon A. Willis | Real time transmission of keyboard musical performance |
US6310279B1 (en) * | 1997-12-27 | 2001-10-30 | Yamaha Corporation | Device and method for generating a picture and/or tone on the basis of detection of a physical event from performance information |
US6310610B1 (en) * | 1997-12-04 | 2001-10-30 | Nortel Networks Limited | Intelligent touch display |
US20010036299A1 (en) * | 1998-05-15 | 2001-11-01 | Andrew William Senior | Combined fingerprint acquisition and control device |
US6320112B1 (en) * | 2000-05-19 | 2001-11-20 | Martin Lotze | Procedure and device for the automatic selection of musical and/or tonal compositions |
US6323846B1 (en) * | 1998-01-26 | 2001-11-27 | University Of Delaware | Method and apparatus for integrating manual input |
US20020005108A1 (en) * | 1998-05-15 | 2002-01-17 | Ludwig Lester Frank | Tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting |
US6360019B1 (en) * | 1996-03-28 | 2002-03-19 | Microsoft Corporation | Table-based compression with embedded coding |
US6363159B1 (en) * | 1993-11-18 | 2002-03-26 | Digimarc Corporation | Consumer audio appliance responsive to watermark data |
US6373475B1 (en) * | 1997-04-25 | 2002-04-16 | Michael Challis | Converter for resistive touchscreens |
US6392705B1 (en) * | 1997-03-17 | 2002-05-21 | Microsoft Corporation | Multimedia compression system with additive temporal layers |
US6392636B1 (en) * | 1998-01-22 | 2002-05-21 | Stmicroelectronics, Inc. | Touchpad providing screen cursor/pointer movement control |
US6408087B1 (en) * | 1998-01-13 | 2002-06-18 | Stmicroelectronics, Inc. | Capacitive semiconductor user input device |
US20020093491A1 (en) * | 1992-06-08 | 2002-07-18 | David W. Gillespie | Object position detector with edge motion feature and gesture recognition |
US20030003976A1 (en) * | 2001-06-19 | 2003-01-02 | Sony Corporation | Memory card, personal digital assistant, information processing method, recording medium, and program |
US20030151592A1 (en) * | 2000-08-24 | 2003-08-14 | Dieter Ritter | Method for requesting destination information and for navigating in a map view, computer program product and navigation unit |
US6703552B2 (en) * | 2001-07-19 | 2004-03-09 | Lippold Haken | Continuous music keyboard |
US6793619B1 (en) * | 1999-06-09 | 2004-09-21 | Yaacov Blumental | Computer-implemented method and system for giving a user an impression of tactile feedback |
US20040251402A1 (en) * | 2001-09-25 | 2004-12-16 | Gerd Reime | Circuit with an opto-electronic display unit |
US20050179651A1 (en) * | 2004-02-13 | 2005-08-18 | Ludwig Lester F. | Mouse-based user interface device providing multiple parameters and modalities |
US20060001914A1 (en) * | 2004-06-30 | 2006-01-05 | Mesmer Ralph M | Color scanner display |
US7030860B1 (en) * | 1999-10-08 | 2006-04-18 | Synaptics Incorporated | Flexible transparent touch sensing system for electronic devices |
US20060252530A1 (en) * | 2003-01-08 | 2006-11-09 | Igt | Mobile device for providing filtered casino information based on real time data |
US20070044019A1 (en) * | 2003-05-23 | 2007-02-22 | Byung-Ro Moon | Multi-campaign assignment apparatus considering overlapping recommendation problem |
US20070063990A1 (en) * | 2005-09-21 | 2007-03-22 | Samsung Electronics Co., Ltd. | Touch sensitive display device and driving apparatus thereof, and method of detecting a touch |
US20080010616A1 (en) * | 2006-07-06 | 2008-01-10 | Cherif Atia Algreatly | Spherical coordinates cursor, mouse, and method |
US20080034286A1 (en) * | 2006-07-19 | 2008-02-07 | Verizon Services Organization Inc. | Intercepting text strings |
US7348946B2 (en) * | 2001-12-31 | 2008-03-25 | Intel Corporation | Energy sensing light emitting diode display |
US20080143690A1 (en) * | 2006-12-15 | 2008-06-19 | Lg.Philips Lcd Co., Ltd. | Display device having multi-touch recognizing function and driving method thereof |
US20080150848A1 (en) * | 2006-12-26 | 2008-06-26 | Lg. Philips Lcd Co., Ltd. | Organic light-emitting diode panel and touch-screen system including the same |
US20080164076A1 (en) * | 2007-01-04 | 2008-07-10 | Timothy James Orsley | Capacitive sensing and absolute position mapping in displacement type pointing devices |
US20080259053A1 (en) * | 2007-04-11 | 2008-10-23 | John Newton | Touch Screen System with Hover and Click Input Methods |
US20080300055A1 (en) * | 2007-05-29 | 2008-12-04 | Lutnick Howard W | Game with hand motion control |
US20080297482A1 (en) * | 2007-05-30 | 2008-12-04 | Microsoft Corporation | Recognizing selection regions from multiple simultaneous inputs |
US20080309634A1 (en) * | 2007-01-05 | 2008-12-18 | Apple Inc. | Multi-touch skins spanning three dimensions |
US20100110025A1 (en) * | 2008-07-12 | 2010-05-06 | Lim Seung E | Control of computer window systems and applications using high dimensional touchpad user interface |
Family Cites Families (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0574213B1 (en) | 1992-06-08 | 1999-03-24 | Synaptics, Inc. | Object position detector |
US6433801B1 (en) * | 1997-09-26 | 2002-08-13 | Ericsson Inc. | Method and apparatus for using a touch screen display on a portable intelligent communications device |
US6278443B1 (en) * | 1998-04-30 | 2001-08-21 | International Business Machines Corporation | Touch screen with random finger placement and rolling on screen to control the movement of information on-screen |
US7611409B2 (en) | 2001-09-20 | 2009-11-03 | Igt | Method and apparatus for registering a mobile device with a gaming machine |
US7961909B2 (en) | 2006-03-08 | 2011-06-14 | Electronic Scripting Products, Inc. | Computer interface employing a manipulated object with absolute pose detection component and a display |
FR2869723A1 (en) * | 2004-04-29 | 2005-11-04 | Thomson Licensing Sa | NON-CONTACT TRANSITION ELEMENT BETWEEN A WAVEGUIDE AND A MOCRORUBAN LINE |
US7598949B2 (en) | 2004-10-22 | 2009-10-06 | New York University | Multi-touch sensing light emitting diode display and method for using the same |
US8678901B1 (en) | 2005-09-07 | 2014-03-25 | Bally Gaming | System gaming |
US7864160B2 (en) | 2005-10-05 | 2011-01-04 | 3M Innovative Properties Company | Interleaved electrodes for touch sensing |
US8161530B2 (en) * | 2006-07-11 | 2012-04-17 | Identity Metrics, Inc. | Behaviormetrics application system for electronic transaction authorization |
US8125455B2 (en) | 2007-01-03 | 2012-02-28 | Apple Inc. | Full scale calibration measurement for multi-touch surfaces |
US7835999B2 (en) * | 2007-06-27 | 2010-11-16 | Microsoft Corporation | Recognizing input gestures using a multi-touch input device, calculated graphs, and a neural network with link weights |
US20090124348A1 (en) * | 2007-11-09 | 2009-05-14 | Yoseloff Mark L | Electronic dice control in gaming |
JP2009140368A (en) * | 2007-12-07 | 2009-06-25 | Sony Corp | Input device, display device, input method, display method, and program |
US9170649B2 (en) * | 2007-12-28 | 2015-10-27 | Nokia Technologies Oy | Audio and tactile feedback based on visual environment |
US9019237B2 (en) | 2008-04-06 | 2015-04-28 | Lester F. Ludwig | Multitouch parameter and gesture user interface employing an LED-array tactile sensor that can also operate as a display |
US8362883B2 (en) * | 2008-06-10 | 2013-01-29 | Design Interactive, Inc. | Method and system for the presentation of information via the tactile sense |
US9870629B2 (en) | 2008-06-20 | 2018-01-16 | New Bis Safe Luxco S.À R.L | Methods, apparatus and systems for data visualization and related applications |
US8345014B2 (en) | 2008-07-12 | 2013-01-01 | Lester F. Ludwig | Control of the operating system on a computing device via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8604364B2 (en) | 2008-08-15 | 2013-12-10 | Lester F. Ludwig | Sensors, algorithms and applications for a high dimensional touchpad |
US20100079385A1 (en) | 2008-09-29 | 2010-04-01 | Smart Technologies Ulc | Method for calibrating an interactive input system and interactive input system executing the calibration method |
US8529345B2 (en) | 2008-10-02 | 2013-09-10 | Igt | Gaming system including a gaming table with mobile user input devices |
US9104311B2 (en) | 2008-10-09 | 2015-08-11 | Lenovo (Singapore) Pte. Ltd. | Slate computer with tactile home keys |
JP4766101B2 (en) | 2008-11-10 | 2011-09-07 | ソニー株式会社 | Tactile behavior recognition device, tactile behavior recognition method, information processing device, and computer program |
US20100177118A1 (en) | 2009-01-09 | 2010-07-15 | Sytnikov Vadim Vasilievich | Path creation utility for image editor |
US8170346B2 (en) | 2009-03-14 | 2012-05-01 | Ludwig Lester F | High-performance closed-form single-scan calculation of oblong-shape rotation angles from binary images of arbitrary size using running sums |
US8274536B2 (en) | 2009-03-16 | 2012-09-25 | Apple Inc. | Smart keyboard management for a multifunction device with a touch screen display |
US8154529B2 (en) | 2009-05-14 | 2012-04-10 | Atmel Corporation | Two-dimensional touch sensors |
US8836648B2 (en) | 2009-05-27 | 2014-09-16 | Microsoft Corporation | Touch pull-in gesture |
US8217912B2 (en) | 2009-06-17 | 2012-07-10 | Broadcom Corporation | Graphical authentication for a portable device and methods for use therewith |
US20100328032A1 (en) | 2009-06-24 | 2010-12-30 | Broadcom Corporation | Security for computing unit with femtocell ap functionality |
US8179376B2 (en) | 2009-08-27 | 2012-05-15 | Research In Motion Limited | Touch-sensitive display with capacitive and resistive touch sensors and method of control |
US8313377B2 (en) | 2009-10-14 | 2012-11-20 | Sony Computer Entertainment America Llc | Playing browser based games with alternative controls and interfaces |
US20110285648A1 (en) | 2010-01-22 | 2011-11-24 | Lester Ludwig | Use of fingerprint scanning sensor data to detect finger roll and pitch angles |
US20110202934A1 (en) | 2010-02-12 | 2011-08-18 | Ludwig Lester F | Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces |
US20120056846A1 (en) | 2010-03-01 | 2012-03-08 | Lester F. Ludwig | Touch-based user interfaces employing artificial neural networks for hdtp parameter and symbol derivation |
US8686960B2 (en) | 2010-04-23 | 2014-04-01 | Lester F. Ludwig | Piecewise-linear and piecewise-affine transformations for high dimensional touchpad (HDTP) output decoupling and corrections |
US8754862B2 (en) | 2010-07-11 | 2014-06-17 | Lester F. Ludwig | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces |
US9950256B2 (en) | 2010-08-05 | 2018-04-24 | Nri R&D Patent Licensing, Llc | High-dimensional touchpad game controller with multiple usage and networking modalities |
US20120192119A1 (en) | 2011-01-24 | 2012-07-26 | Lester F. Ludwig | Usb hid device abstraction for hdtp user interfaces |
US8797288B2 (en) | 2011-03-07 | 2014-08-05 | Lester F. Ludwig | Human user interfaces utilizing interruption of the execution of a first recognized gesture with the execution of a recognized second gesture |
US20120280927A1 (en) | 2011-05-04 | 2012-11-08 | Ludwig Lester F | Simple touch interface and hdtp grammars for rapid operation of physical computer aided design (cad) systems |
US20130009896A1 (en) | 2011-07-09 | 2013-01-10 | Lester F. Ludwig | 3d finger posture detection and gesture recognition on touch surfaces |
US9052772B2 (en) | 2011-08-10 | 2015-06-09 | Lester F. Ludwig | Heuristics for 3D and 6D touch gesture touch parameter calculations for high-dimensional touch parameter (HDTP) user interfaces |
-
2011
- 2011-02-11 US US13/026,097 patent/US20110202934A1/en not_active Abandoned
- 2011-02-12 US US13/026,248 patent/US9830042B2/en not_active Expired - Fee Related
-
2017
- 2017-11-16 US US15/815,446 patent/US20180074680A1/en not_active Abandoned
Patent Citations (99)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4748676A (en) * | 1986-05-16 | 1988-05-31 | Fuji Electric Co., Ltd. | Pattern rotation angle detecting system |
US4899137A (en) * | 1987-09-26 | 1990-02-06 | Aeg Olympia Aktiengesellschaft | Arrangement for the input and processing of characters and/or graphic patterns |
US5270711A (en) * | 1989-05-08 | 1993-12-14 | U.S. Philips Corporation | Touch sensor array systems and display systems incorporating such |
US5237647A (en) * | 1989-09-15 | 1993-08-17 | Massachusetts Institute Of Technology | Computer aided drawing in three dimensions |
US5347295A (en) * | 1990-10-31 | 1994-09-13 | Go Corporation | Control of a computer through a position-sensed stylus |
US5471008A (en) * | 1990-11-19 | 1995-11-28 | Kabushiki Kaisha Kawai Gakki Seisakusho | MIDI control apparatus |
US5341133A (en) * | 1991-05-09 | 1994-08-23 | The Rowland Institute For Science, Inc. | Keyboard having touch sensor keys for conveying information electronically |
US5292999A (en) * | 1991-08-14 | 1994-03-08 | Fernandes Co., Ltd. | Electric stringed instrument having a device for sustaining the vibration of the string |
US5475214A (en) * | 1991-10-15 | 1995-12-12 | Interactive Light, Inc. | Musical sound effects controller having a radiated emission space |
US5442168A (en) * | 1991-10-15 | 1995-08-15 | Interactive Light, Inc. | Dynamically-activated optical instrument for producing control signals having a self-calibration means |
US5386219A (en) * | 1991-10-16 | 1995-01-31 | International Business Machines Corp. | Touch overlay for improved touch sensitivity |
US5585588A (en) * | 1991-10-24 | 1996-12-17 | Fernandes Co., Ltd. | Electric stringed instrument having a device for sustaining the vibration of a string and an electromagnetic driver for the device |
US5717939A (en) * | 1991-11-18 | 1998-02-10 | Compaq Computer Corporation | Method and apparatus for entering and manipulating spreadsheet cell data |
US5805137A (en) * | 1991-11-26 | 1998-09-08 | Itu Research, Inc. | Touch sensitive input control device |
US5378850A (en) * | 1992-01-14 | 1995-01-03 | Fernandes Co., Ltd. | Electric stringed instrument having an arrangement for adjusting the generation of magnetic feedback |
US20020093491A1 (en) * | 1992-06-08 | 2002-07-18 | David W. Gillespie | Object position detector with edge motion feature and gesture recognition |
US5889236A (en) * | 1992-06-08 | 1999-03-30 | Synaptics Incorporated | Pressure sensitive scrollbar feature |
US5459282A (en) * | 1992-09-25 | 1995-10-17 | Willis; Raymon A. | System for rejuvenating vintage organs and pianos |
US5440072A (en) * | 1992-09-25 | 1995-08-08 | Willis; Raymon A. | System for rejuvenating vintage organs and pianos |
US5357048A (en) * | 1992-10-08 | 1994-10-18 | Sgroi John J | MIDI sound designer with randomizer function |
US5420936A (en) * | 1992-10-16 | 1995-05-30 | International Business Machines Corporation | Method and apparatus for accessing touch screen desktop objects via fingerprint recognition |
US6285358B1 (en) * | 1993-04-30 | 2001-09-04 | Microtouch Systems, Inc. | Method of and apparatus for the elimination of the effects of inertial interference in force measurement systems, including touch-input computer and related displays employing touch force location measurement techniques |
US5592752A (en) * | 1993-05-13 | 1997-01-14 | Industrial Technology Research Institute | Process and an apparatus for producing teas |
US5665927A (en) * | 1993-06-30 | 1997-09-09 | Casio Computer Co., Ltd. | Method and apparatus for inputting musical data without requiring selection of a displayed icon |
US5675100A (en) * | 1993-11-03 | 1997-10-07 | Hewlett; Walter B. | Method for encoding music printing information in a MIDI message |
US5841428A (en) * | 1993-11-05 | 1998-11-24 | Intertactile Technologies Corporation | Rotary circuit control devices with changeable graphics |
US5592572A (en) * | 1993-11-05 | 1997-01-07 | The United States Of America As Represented By The Department Of Health And Human Services | Automated portrait/landscape mode detection on a binary image |
US6404898B1 (en) * | 1993-11-18 | 2002-06-11 | Digimarc Corporation | Method and system for encoding image and audio content |
US6363159B1 (en) * | 1993-11-18 | 2002-03-26 | Digimarc Corporation | Consumer audio appliance responsive to watermark data |
US5565641A (en) * | 1994-03-28 | 1996-10-15 | Gruenbaum; Leon | Relativistic electronic musical instrument |
US5659466A (en) * | 1994-11-02 | 1997-08-19 | Advanced Micro Devices, Inc. | Monolithic PC audio circuit with enhanced digital wavetable audio synthesizer |
US6047073A (en) * | 1994-11-02 | 2000-04-04 | Advanced Micro Devices, Inc. | Digital wavetable audio synthesizer with delay-based effects processing |
US5668338A (en) * | 1994-11-02 | 1997-09-16 | Advanced Micro Devices, Inc. | Wavetable audio synthesizer with low frequency oscillators for tremolo and vibrato effects |
US5719347A (en) * | 1994-12-07 | 1998-02-17 | Yamaha Corporation | Keyboard apparatus for electronic musical instrument with key depression detection unit |
US5932827A (en) * | 1995-01-09 | 1999-08-03 | Osborne; Gary T. | Sustainer for a musical instrument |
US6005545A (en) * | 1995-01-17 | 1999-12-21 | Sega Enterprise, Ltd. | Image processing method and electronic device |
US5986224A (en) * | 1995-04-19 | 1999-11-16 | Elo Touchsystems, Inc. | Acoustic condition sensor employing a plurality of mutually non-orthogonal waves |
US5659145A (en) * | 1995-04-27 | 1997-08-19 | Weil; Robert P. | Foot operated audio signal controller with lighted visual reference |
US5824930A (en) * | 1995-06-09 | 1998-10-20 | Yamaha Corporation | Keyboard musical instrument having key monitor exactly discriminating key motion |
US5801340A (en) * | 1995-06-29 | 1998-09-01 | Invotronics Manufacturing | Proximity sensor |
US5741993A (en) * | 1995-08-01 | 1998-04-21 | Kabushiki Kaisha Kawai Gakki Seisakusho | Electronic keyboard having a discrete pitch bender |
US5724985A (en) * | 1995-08-02 | 1998-03-10 | Pacesetter, Inc. | User interface for an implantable medical device using an integrated digitizer display screen |
US5719561A (en) * | 1995-10-25 | 1998-02-17 | Gilbert R. Gonzales | Tactile communication device and method |
US5786540A (en) * | 1996-03-05 | 1998-07-28 | Westlund; Robert L. | Controller apparatus for music sequencer |
US6360019B1 (en) * | 1996-03-28 | 2002-03-19 | Microsoft Corporation | Table-based compression with embedded coding |
US5748184A (en) * | 1996-05-28 | 1998-05-05 | International Business Machines Corporation | Virtual pointing device for touchscreens |
US6107997A (en) * | 1996-06-27 | 2000-08-22 | Ure; Michael J. | Touch-sensitive keyboard/mouse and computing device using the same |
US5977466A (en) * | 1996-07-11 | 1999-11-02 | Yamaha Corporation | Keyboard musical instrument equipped with small simple economical key touch generator |
US5763806A (en) * | 1996-07-15 | 1998-06-09 | Willis; Raymon A. | Method and apparatus for midifying vintage organs and pianos |
US5850051A (en) * | 1996-08-15 | 1998-12-15 | Yamaha Corporation | Method and apparatus for creating an automatic accompaniment pattern on the basis of analytic parameters |
US6392705B1 (en) * | 1997-03-17 | 2002-05-21 | Microsoft Corporation | Multimedia compression system with additive temporal layers |
US6225975B1 (en) * | 1997-03-18 | 2001-05-01 | Alps Electric Co., Ltd. | Control device for game machine |
US6373475B1 (en) * | 1997-04-25 | 2002-04-16 | Michael Challis | Converter for resistive touchscreens |
US5827989A (en) * | 1997-06-23 | 1998-10-27 | Microsoft Corporation | System and method for representing a musical event and for converting the musical event into a series of discrete events |
US5852251A (en) * | 1997-06-25 | 1998-12-22 | Industrial Technology Research Institute | Method and apparatus for real-time dynamic midi control |
US6310610B1 (en) * | 1997-12-04 | 2001-10-30 | Nortel Networks Limited | Intelligent touch display |
US6037937A (en) * | 1997-12-04 | 2000-03-14 | Nortel Networks Corporation | Navigation tool for graphical user interface |
US6310279B1 (en) * | 1997-12-27 | 2001-10-30 | Yamaha Corporation | Device and method for generating a picture and/or tone on the basis of detection of a physical event from performance information |
US6408087B1 (en) * | 1998-01-13 | 2002-06-18 | Stmicroelectronics, Inc. | Capacitive semiconductor user input device |
US6392636B1 (en) * | 1998-01-22 | 2002-05-21 | Stmicroelectronics, Inc. | Touchpad providing screen cursor/pointer movement control |
US6323846B1 (en) * | 1998-01-26 | 2001-11-27 | University Of Delaware | Method and apparatus for integrating manual input |
US6204441B1 (en) * | 1998-04-09 | 2001-03-20 | Yamaha Corporation | Method and apparatus for effectively displaying musical information with visual display |
US20010036299A1 (en) * | 1998-05-15 | 2001-11-01 | Andrew William Senior | Combined fingerprint acquisition and control device |
US7408108B2 (en) * | 1998-05-15 | 2008-08-05 | Ludwig Lester F | Multiple-paramenter instrument keyboard combining key-surface touch and key-displacement sensor arrays |
US20020005108A1 (en) * | 1998-05-15 | 2002-01-17 | Ludwig Lester Frank | Tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting |
US20040118268A1 (en) * | 1998-05-15 | 2004-06-24 | Ludwig Lester F. | Controlling and enhancing electronic musical instruments with video |
US6610917B2 (en) * | 1998-05-15 | 2003-08-26 | Lester F. Ludwig | Activity indication, external source, and processing loop provisions for driven vibrating-element environments |
US6400836B2 (en) * | 1998-05-15 | 2002-06-04 | International Business Machines Corporation | Combined fingerprint acquisition and control device |
US20070229477A1 (en) * | 1998-05-15 | 2007-10-04 | Ludwig Lester F | High parameter-count touchpad controller |
US20040074379A1 (en) * | 1998-05-15 | 2004-04-22 | Ludwig Lester F. | Functional extensions of traditional music keyboards |
US6570078B2 (en) * | 1998-05-15 | 2003-05-27 | Lester Frank Ludwig | Tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting |
US6288317B1 (en) * | 1998-05-29 | 2001-09-11 | Raymon A. Willis | Real time transmission of keyboard musical performance |
US6140565A (en) * | 1998-06-08 | 2000-10-31 | Yamaha Corporation | Method of visualizing music system by combination of scenery picture and player icons |
US6100461A (en) * | 1998-06-10 | 2000-08-08 | Advanced Micro Devices, Inc. | Wavetable cache using simplified looping |
US5969283A (en) * | 1998-06-17 | 1999-10-19 | Looney Productions, Llc | Music organizer and entertainment center |
US6051769A (en) * | 1998-11-25 | 2000-04-18 | Brown, Jr.; Donival | Computerized reading display |
US6793619B1 (en) * | 1999-06-09 | 2004-09-21 | Yaacov Blumental | Computer-implemented method and system for giving a user an impression of tactile feedback |
US7030860B1 (en) * | 1999-10-08 | 2006-04-18 | Synaptics Incorporated | Flexible transparent touch sensing system for electronic devices |
US6320112B1 (en) * | 2000-05-19 | 2001-11-20 | Martin Lotze | Procedure and device for the automatic selection of musical and/or tonal compositions |
US20030151592A1 (en) * | 2000-08-24 | 2003-08-14 | Dieter Ritter | Method for requesting destination information and for navigating in a map view, computer program product and navigation unit |
US20030003976A1 (en) * | 2001-06-19 | 2003-01-02 | Sony Corporation | Memory card, personal digital assistant, information processing method, recording medium, and program |
US6703552B2 (en) * | 2001-07-19 | 2004-03-09 | Lippold Haken | Continuous music keyboard |
US20040251402A1 (en) * | 2001-09-25 | 2004-12-16 | Gerd Reime | Circuit with an opto-electronic display unit |
US7348946B2 (en) * | 2001-12-31 | 2008-03-25 | Intel Corporation | Energy sensing light emitting diode display |
US20060252530A1 (en) * | 2003-01-08 | 2006-11-09 | Igt | Mobile device for providing filtered casino information based on real time data |
US20070044019A1 (en) * | 2003-05-23 | 2007-02-22 | Byung-Ro Moon | Multi-campaign assignment apparatus considering overlapping recommendation problem |
US20050179651A1 (en) * | 2004-02-13 | 2005-08-18 | Ludwig Lester F. | Mouse-based user interface device providing multiple parameters and modalities |
US20060001914A1 (en) * | 2004-06-30 | 2006-01-05 | Mesmer Ralph M | Color scanner display |
US20070063990A1 (en) * | 2005-09-21 | 2007-03-22 | Samsung Electronics Co., Ltd. | Touch sensitive display device and driving apparatus thereof, and method of detecting a touch |
US20080010616A1 (en) * | 2006-07-06 | 2008-01-10 | Cherif Atia Algreatly | Spherical coordinates cursor, mouse, and method |
US20080034286A1 (en) * | 2006-07-19 | 2008-02-07 | Verizon Services Organization Inc. | Intercepting text strings |
US20080143690A1 (en) * | 2006-12-15 | 2008-06-19 | Lg.Philips Lcd Co., Ltd. | Display device having multi-touch recognizing function and driving method thereof |
US20080150848A1 (en) * | 2006-12-26 | 2008-06-26 | Lg. Philips Lcd Co., Ltd. | Organic light-emitting diode panel and touch-screen system including the same |
US20080164076A1 (en) * | 2007-01-04 | 2008-07-10 | Timothy James Orsley | Capacitive sensing and absolute position mapping in displacement type pointing devices |
US20080309634A1 (en) * | 2007-01-05 | 2008-12-18 | Apple Inc. | Multi-touch skins spanning three dimensions |
US20080259053A1 (en) * | 2007-04-11 | 2008-10-23 | John Newton | Touch Screen System with Hover and Click Input Methods |
US20080300055A1 (en) * | 2007-05-29 | 2008-12-04 | Lutnick Howard W | Game with hand motion control |
US20080297482A1 (en) * | 2007-05-30 | 2008-12-04 | Microsoft Corporation | Recognizing selection regions from multiple simultaneous inputs |
US20100110025A1 (en) * | 2008-07-12 | 2010-05-06 | Lim Seung E | Control of computer window systems and applications using high dimensional touchpad user interface |
Cited By (38)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8878810B2 (en) | 1998-05-15 | 2014-11-04 | Lester F. Ludwig | Touch screen supporting continuous grammar touch gestures |
US8866785B2 (en) | 1998-05-15 | 2014-10-21 | Lester F. Ludwig | Sensor array touchscreen recognizing finger flick gesture |
US8878807B2 (en) | 1998-05-15 | 2014-11-04 | Lester F. Ludwig | Gesture-based user interface employing video camera |
US9304677B2 (en) | 1998-05-15 | 2016-04-05 | Advanced Touchscreen And Gestures Technologies, Llc | Touch screen apparatus for recognizing a touch gesture |
US8743068B2 (en) | 1998-05-15 | 2014-06-03 | Lester F. Ludwig | Touch screen method for recognizing a finger-flick touch gesture |
US8743076B1 (en) | 1998-05-15 | 2014-06-03 | Lester F. Ludwig | Sensor array touchscreen recognizing finger flick gesture from spatial pressure distribution profiles |
US8717303B2 (en) | 1998-05-15 | 2014-05-06 | Lester F. Ludwig | Sensor array touchscreen recognizing finger flick gesture and other touch gestures |
US9019237B2 (en) | 2008-04-06 | 2015-04-28 | Lester F. Ludwig | Multitouch parameter and gesture user interface employing an LED-array tactile sensor that can also operate as a display |
US8638312B2 (en) | 2008-07-12 | 2014-01-28 | Lester F. Ludwig | Advanced touch control of a file browser via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8894489B2 (en) | 2008-07-12 | 2014-11-25 | Lester F. Ludwig | Touch user interface supporting global and context-specific touch gestures that are responsive to at least one finger angle |
US8477111B2 (en) | 2008-07-12 | 2013-07-02 | Lester F. Ludwig | Advanced touch control of interactive immersive imaging applications via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8702513B2 (en) | 2008-07-12 | 2014-04-22 | Lester F. Ludwig | Control of the operating system on a computing device via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8643622B2 (en) | 2008-07-12 | 2014-02-04 | Lester F. Ludwig | Advanced touch control of graphics design application via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8542209B2 (en) | 2008-07-12 | 2013-09-24 | Lester F. Ludwig | Advanced touch control of interactive map viewing via finger angle using a high dimensional touchpad (HDTP) touch user interface |
US8604364B2 (en) | 2008-08-15 | 2013-12-10 | Lester F. Ludwig | Sensors, algorithms and applications for a high dimensional touchpad |
US8509542B2 (en) | 2009-03-14 | 2013-08-13 | Lester F. Ludwig | High-performance closed-form single-scan calculation of oblong-shape rotation angles from binary images of arbitrary size and location using running sums |
US8639037B2 (en) | 2009-03-14 | 2014-01-28 | Lester F. Ludwig | High-performance closed-form single-scan calculation of oblong-shape rotation angles from image data of arbitrary size and location using running sums |
US8826114B2 (en) | 2009-09-02 | 2014-09-02 | Lester F. Ludwig | Surface-curve graphical intersection tools and primitives for data visualization, tabular data, and advanced spreadsheets |
US8826113B2 (en) | 2009-09-02 | 2014-09-02 | Lester F. Ludwig | Surface-surface graphical intersection tools and primitives for data visualization, tabular data, and advanced spreadsheets |
US9665554B2 (en) | 2009-09-02 | 2017-05-30 | Lester F. Ludwig | Value-driven visualization primitives for tabular data of spreadsheets |
US9830042B2 (en) | 2010-02-12 | 2017-11-28 | Nri R&D Patent Licensing, Llc | Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice |
US10146427B2 (en) * | 2010-03-01 | 2018-12-04 | Nri R&D Patent Licensing, Llc | Curve-fitting approach to high definition touch pad (HDTP) parameter extraction |
US20110210943A1 (en) * | 2010-03-01 | 2011-09-01 | Lester F. Ludwig | Curve-fitting approach to hdtp parameter extraction |
US9632344B2 (en) | 2010-07-09 | 2017-04-25 | Lester F. Ludwig | Use of LED or OLED array to implement integrated combinations of touch screen tactile, touch gesture sensor, color image display, hand-image gesture sensor, document scanner, secure optical data exchange, and fingerprint processing capabilities |
US9626023B2 (en) | 2010-07-09 | 2017-04-18 | Lester F. Ludwig | LED/OLED array approach to integrated display, lensless-camera, and touch-screen user interface devices and associated processors |
US8754862B2 (en) | 2010-07-11 | 2014-06-17 | Lester F. Ludwig | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces |
US9950256B2 (en) | 2010-08-05 | 2018-04-24 | Nri R&D Patent Licensing, Llc | High-dimensional touchpad game controller with multiple usage and networking modalities |
US9605881B2 (en) | 2011-02-16 | 2017-03-28 | Lester F. Ludwig | Hierarchical multiple-level control of adaptive cooling and energy harvesting arrangements for information technology |
US8797288B2 (en) | 2011-03-07 | 2014-08-05 | Lester F. Ludwig | Human user interfaces utilizing interruption of the execution of a first recognized gesture with the execution of a recognized second gesture |
US9442652B2 (en) | 2011-03-07 | 2016-09-13 | Lester F. Ludwig | General user interface gesture lexicon and grammar frameworks for multi-touch, high dimensional touch pad (HDTP), free-space camera, and other user interfaces |
US10073532B2 (en) | 2011-03-07 | 2018-09-11 | Nri R&D Patent Licensing, Llc | General spatial-gesture grammar user interface for touchscreens, high dimensional touch pad (HDTP), free-space camera, and other user interfaces |
US9052772B2 (en) | 2011-08-10 | 2015-06-09 | Lester F. Ludwig | Heuristics for 3D and 6D touch gesture touch parameter calculations for high-dimensional touch parameter (HDTP) user interfaces |
US9823781B2 (en) | 2011-12-06 | 2017-11-21 | Nri R&D Patent Licensing, Llc | Heterogeneous tactile sensing via multiple sensor types |
US10042479B2 (en) | 2011-12-06 | 2018-08-07 | Nri R&D Patent Licensing, Llc | Heterogeneous tactile sensing via multiple sensor types using spatial information processing |
US10429997B2 (en) | 2011-12-06 | 2019-10-01 | Nri R&D Patent Licensing, Llc | Heterogeneous tactile sensing via multiple sensor types using spatial information processing acting on initial image processed data from each sensor |
US10430066B2 (en) | 2011-12-06 | 2019-10-01 | Nri R&D Patent Licensing, Llc | Gesteme (gesture primitive) recognition for advanced touch user interfaces |
US9532098B2 (en) * | 2012-11-29 | 2016-12-27 | Zte Corporation | Method, apparatus and system for controlling focus on TV interface |
US20150312617A1 (en) * | 2012-11-29 | 2015-10-29 | Zte Corporation | Method, apparatus and system for controlling focus on TV interface |
Also Published As
Publication number | Publication date |
---|---|
US20180074680A1 (en) | 2018-03-15 |
US20110202889A1 (en) | 2011-08-18 |
US9830042B2 (en) | 2017-11-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20110202934A1 (en) | Window manger input focus control for high dimensional touchpad (htpd), advanced mice, and other multidimensional user interfaces | |
US10216399B2 (en) | Piecewise-linear and piecewise-affine subspace transformations for high dimensional touchpad (HDTP) output decoupling and corrections | |
US10664156B2 (en) | Curve-fitting approach to touch gesture finger pitch parameter extraction | |
US20120192119A1 (en) | Usb hid device abstraction for hdtp user interfaces | |
US8754862B2 (en) | Sequential classification recognition of gesture primitives and window-based parameter smoothing for high dimensional touchpad (HDTP) user interfaces | |
US20120056846A1 (en) | Touch-based user interfaces employing artificial neural networks for hdtp parameter and symbol derivation | |
US9448645B2 (en) | Digitizer using multiple stylus sensing techniques | |
US8982051B2 (en) | Detecting touch on a surface | |
US9823781B2 (en) | Heterogeneous tactile sensing via multiple sensor types | |
US10430066B2 (en) | Gesteme (gesture primitive) recognition for advanced touch user interfaces | |
KR101408620B1 (en) | Methods and apparatus for pressure-based manipulation of content on a touch screen | |
US9317140B2 (en) | Method of making a multi-touch input device for detecting touch on a curved surface | |
US20120280927A1 (en) | Simple touch interface and hdtp grammars for rapid operation of physical computer aided design (cad) systems | |
US20090184939A1 (en) | Graphical object manipulation with a touch sensitive screen | |
US20090128516A1 (en) | Multi-point detection on a single-point detection digitizer | |
US20110227947A1 (en) | Multi-Touch User Interface Interaction | |
EP3410280A1 (en) | Object orientation detection with a digitizer | |
US20130147743A1 (en) | Spherical Touch Sensors and Signal/Power Architectures for Trackballs, Globes, Displays, and Other Applications | |
TWI396123B (en) | Optical touch system and operating method thereof | |
US20120274596A1 (en) | Use of organic light emitting diode (oled) displays as a high-resolution optical tactile sensor for high dimensional touchpad (hdtp) user interfaces | |
US9310851B2 (en) | Three-dimensional (3D) human-computer interaction system using computer mouse as a 3D pointing device and an operation method thereof | |
Morrison | A camera-based input device for large interactive displays |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: NRI R&D PATENT LICENSING, LLC, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LUDWIG, LESTER F;REEL/FRAME:042745/0063 Effective date: 20170608 |
|
AS | Assignment |
Owner name: PBLM ADVT LLC, NEW YORK Free format text: SECURITY INTEREST;ASSIGNOR:NRI R&D PATENT LICENSING, LLC;REEL/FRAME:044036/0254 Effective date: 20170907 |