Journal Articles
-
Randomized Control Trial for Evaluation of a Hands-Free Pointer for Surgical Instruction During Laparoscopic Surgery
, Surgical Endoscopy, vol. 29, no. 12, pp. 3655–3665, December 2015. AbstractBackground: Training surgeons to perform Minimally Invasive Surgery (MIS) requires surgical residents to operate under the direction and supervision of a consultant. However, the inability of the instructing surgeon to point at the laparoscopic monitor without releasing the surgical instruments remains a barrier to effective instruction in the operating room. The Wireless Hands-Free Surgical Pointer (WHaSP) has been developed to aid instruction during MIS.
Methods: The objective of this study was to evaluate the effectiveness and likeability of the WHaSP as an instructional tool as compared with conventional instructing methods during MIS. Data were successfully collected during 103 laparoscopic cholecystectomy procedures, which had been randomized to use or not use the WHaSP as a teaching tool. Audio and video from the surgeries were recorded and periods in which instruction was taking place were extracted and analyzed. Instructing surgeons, operating surgeons, and camera assistants provided feedback through a post-operative questionnaire that used a five-level Likert scale. The questionnaire results were analyzed using a Mann-Whitney U test.
Results: There were no negative effects on surgery completion time or instruction practice due to the use of the WHaSP. The number of times an instructor surgeon pointed to the laparoscopic screen with their hand was significantly reduced when the WHaSP was utilized (p < 0.001). The questionnaires showed that WHaSP users found it to be comfortable, easy to use and easy to control. Compared to when the WHaSP was not used, users found communication to be more effective (p = 0.002), locations were easier to communicate (p < 0.001), and instructions were easier to follow (p = 0.005).
Conclusions: The WHaSP system was successfully used in surgery. It integrated seamlessly into existing equipment within the operating room and did not affect flow. Surgeons perceived the WHaSP as an aid to communication and liked using it. Positive outcomes of utilizing the WHaSP were: improved communication in the OR, improved efficiency and safety of the surgery, it was easy to use, comfortable to wear, and surgeons showed a preference for utilizing the WHaSP during surgery if given a choice.
-
A Chance-Constraint Programming Approach To Preoperatve Planning of Robotic Cardiac Surgery Under Task-Level Uncertainty
, IEEE Journal of Biomedical and Health Informatics, vol. 19, no. 2, pp. 612–622, March 2015. Abstract | IEEE XploreIn this paper, a novel formulation for robust surgical planning of robotics-assisted minimally invasive cardiac surgery based on patient-specific preoperative images is proposed. In this context, robustness is quantified in terms of the likelihood of intraoperative collisions and of joint limit violations. The proposed approach provides a more accurate and complete formulation than existing deterministic approaches in addressing uncertainty at the task level. Moreover, it is demonstrated that the dexterity of robotic arms can be quantified as a cross-entropy term. The resulting planning problem is rendered as a chanceconstrained entropy maximization problem seeking a plan with the least susceptibility towards uncertainty at the task level, while maximizing the dexterity (cross-entropy term). By such treatment of uncertainty at the task level, spatial uncertainty pertaining to mismatches between the patient-specific anatomical model and that of the actual intraoperative situation is also indirectly addressed. As a solution method, the Unscented Transform is adopted to efficiently transform the resulting chance-constrained entropy maximization problem into a constrained nonlinear program without resorting to computationally expensive particlebased methods.In this paper, a novel formulation for robust surgical planning of robotics-assisted minimally invasive cardiac surgery based on patient-specific preoperative images is proposed. In this context, robustness is quantified in terms of the likelihood of intraoperative collisions and of joint limit violations. The proposed approach provides a more accurate and complete formulation than existing deterministic approaches in addressing uncertainty at the task level. Moreover, it is demonstrated that the dexterity of robotic arms can be quantified as a cross-entropy term. The resulting planning problem is rendered as a chanceconstrained entropy maximization problem seeking a plan with the least susceptibility towards uncert- inty at the task level, while maximizing the dexterity (cross-entropy term). By such treatment of uncertainty at the task level, spatial uncertainty pertaining to mismatches between the patient-specific anatomical model and that of the actual intraoperative situation is also indirectly addressed. As a solution method, the Unscented Transform is adopted to efficiently transform the resulting chance-constrained entropy maximization problem into a constrained nonlinear program without resorting to computationally expensive particlebased methods.
-
A Semi-Infinite Programming Approach to Preoperative Planning of Robotic Cardiac Surgery Under Geometric Uncertainty
, IEEE Journal of Biomedical and Health Informatics, vol. 17, no. 1, pp. 172–182, January 2013. Abstract | IEEE Xplore -
The WHaSP: A Wireless Hands-Free Surgical Pointer for Minimally Invasive Surgery
, IEEE/ASME Transactions on Mechatronics, vol. 17, no. 3, pp. 434–442, June 2012. Abstract | IEEE XploreTo address the challenges of surgical instruction during minimally invasive surgery, a wireless hands-free pointer system has been developed. The WHaSP system incorporates infrared and inertial tracking technologies to address the need for hands-free pointing during minimally invasive surgery. The combination of these technologies allows for optimal movement of the pointer and excellent accuracy while the user is located at a realistic distance from the surgical monitor. Several experimental evaluations were performed to optimize the settings of the sensors, and to validate the system when compared to a commercially available hands-free pointing system. The results show improved performance with the proposed technology as measured by the total trajectory travelled by the pointer and the smoothness of the curve. The technology presented has the potential to significantly improve surgical instruction and guidance during minimally invasive surgery.
-
A Compact 3 Degree of Freedom Spherical Joint
, ASME Journal of Mechanisms and Robotics, vol. 3, no. 3, pp. 1–9, August 2011. Abstract | ASME Digital LibrarySpherical joints have evolved into a critical component of many robotic systems, often used to provide dexterity at the wrist of a manipulator. In this work, a novel 3 degree of freedom spherical joint is proposed, actuated by tendons that run along the surface of the sphere. The joint is mechanically simple and avoids mechanical singularities. The kinematics and mechanics of the joint are modeled and used to develop both open and closed-loop control systems. Simulated and experimental assessment of the joint performance demonstrates that it can be successfully controlled in 3 degree of freedom. It is expected that the joint will be a useful option in the development of emerging robotic applications, particularly those requiring miniaturization.
-
Toward Construct Validity for a Novel Sensorized Instrument-Based Minimally Invasive Surgery (SIMIS) Simulation System
, Surgical Endoscopy, vol. 25, no. 5, pp. 1439–1445, May 2011. Abstract | SpringerLinkMinimally invasive surgical training is complicated by the constraints imposed by the surgical environment. Sensorized laparoscopic instruments capable of sensing force in 5 degrees of freedom and position in 6 degrees of freedom were evaluated. Novice and expert laparoscopists performed the complex minimally invasive surgical task of suturing using the novel instruments. Their force and position profiles were compared. The novel minimally invasive surgical instrument is construct-valid and capable of detecting differences between novices and experts in a laparoscopic suturing task with respect to force and position. Further evaluation is mandated to better understand the ability to predict performance based on force and position as well as the potential for new metrics in minimally invasive surgical education.
-
Force Sensing in Natural Orifice Transluminal Endoscopic Surgery
, Surgical Endoscopy, vol. 25, no. 1, pp. 186–192, January 2011. Abstract | SpringerLinkBackground Natural orifice transluminal endoscopic surgery (NOTES) may represent the next frontier for therapeutic minimally invasive surgery; however, its feasibility is currently limited by the lack of suitable instruments. Identifying the forces required to manipulate tissue during NOTES is a necessary first step in the development of better instrumentation.
Methods Sensorized instruments were used to measure the forces acting at the tip of the instruments during transgastric and transperineal NOTES procedures performed in two female pigs. The maximum and average forces when handling tissue were determined and compared.
Results The results show that, for the transgastric approach, the average forces required are significantly less than in the transperineal approach (43% less), and that the maximum forces required are almost 8 and 16 N in the transgastric and transperineal approaches, respectively. The forces were higher than 5 N in 1.6% of the measurements in the transgastric approach and 2.9% in the transperineal approach.
Conclusions This study presents an experimental measurement of tissue manipulation forces in a NOTES procedure. This information may be valuable for research groups interested in developing NOTES instruments and devices. It is recommended that NOTES instruments be designed to easily handle forces as high as 16 N.
-
Initial Evaluation of a Tactile/Kinesthetic Force Feedback System for Minimally Invasive Tumor Localization
, IEEE/ASME Transactions on Mechatronics, vol. 15, no. 6, pp. 925–931, December 2010. Abstract | IEEE XploreMinimally Invasive Surgery (MIS), while beneficial to patients, leads to new challenges for surgeons and prevents tumors from being localized using finger palpation. A Tactile Sensing System (TSS), consisting of a hand-held Tactile Sensing Instrument (TSI) with a visualization interface, was developed to assist in intra-operative tumor localization. This paper presents the calibration of the TSI and its integration with a visualization interface that allows palpation forces to be displayed. Experiments were conducted to determine whether providing Visual Force Feedback (VFF) to the user would significantly benefit TSS performance when attempting to locate 10 mm hemispherical tumors in ex vivo bovine liver. The TSS with VFF realized a 33% and 21% relative reduction in average and maximum applied forces, respectively, and a 53% relative increase in detection accuracy when compared to the use of the TSS without VFF. Thus, VFF improves the performance of a tactile sensing system and has the potential to help surgeons identify tumors intra-operatively during MIS.
-
Preoperative Evaluation of Patient Anatomy to Increase Success of Robotics-assisted Bypass Surgery
, Innovations: Technology and Techniques in Cardiothoracic and Vascular Surgery, vol. 5, no. 5, pp. 335–340, September–October 2010. Abstract | Lippincott Williams & WilkinsObjective: Robotics-assisted endoscopic atraumatic coronary artery bypass has been shown to be effective in reducing surgical morbidity and length of hospital stay. Unfortunately, the criteria for selecting eligible patients for this procedure is still primitive. This has motivated the use of preoperative CT scans to establish patient eligibility. The objective of this study is to establish which image measurements can be correlated to procedure success.
Methods: A retrospective study was performed in 144 patients that underwent robotics-assisted coronary bypass surgery. After an initial set of 55 patients, preoperative CT scans of the other patients were used to obtain patient specific measurements: The lateral distance between the midline of the sternum to the Left Anterior Descending coronary artery, as well as its depth from the skin surface, anteroposterior diameter of the thoracic cavity, and the transverse diameter of the thoracic cavity. The procedures were rated as successful if completed in a minimally invasive manner. Different combinations of the variables were evaluated and correlated with success.
Results: A strong correlation was found between success rate and the ratio of the lateral distance to the transverse diameter in the female patients only (0.532, p = 0.006). A ratio of less than 0.20 significantly increased the occurrence of conversion during this procedure in female cases.
Conclusions: The lateral distance of the Left Anterior Descending coronary artery from the midline divided by the transverse thoracic width of a female patient shows a significant correlation with procedure success. No significant correlations were found for male patients.
-
A Software Architecture for Adaptive Modular Sensing Systems
, Sensors, vol. 10, no. 8, pp. 7514–7560, August 2010. Abstract | MDPIBy combining a number of simple transducer modules, an arbitrarily complex sensing system may be produced to accommodate a wide range of applications. This work outlines a novel software architecture and knowledge representation scheme that has been developed to support this type of flexible and reconfigurable modular sensing system. Template algorithms are used to embed intelligence within each module. As modules are added or removed, the composite sensor is able to automatically determine its overall geometry and assume an appropriate collective identity. A virtual machine-based middleware layer runs on top of a real-time operating system with a pre-emptive kernel, enabling platform-independent template algorithms to be written once and run on any module, irrespective of its underlying hardware architecture. Applications that may benefit from easily reconfigurable modular sensing systems include flexible inspection, mobile robotics, surveillance, and space exploration.
-
Force Sensing and its Applications in Minimally Invasive Surgery and Therapy: A Survey
, Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, vol. 244, no. 7, pp. 1435–1454, July 2010. Abstract | PE PublishingThe reduced access conditions of minimally invasive surgery and therapy (MIST) impair or completely eliminate the feel of tool-tissue interaction forces. Many researchers have been working actively on the development of force sensors and sensing techniques to address this problem. The goal of this survey paper is to summarize the state of the art in force sensing techniques for medical interventions in order to identify existing limitations and future directions. A literature search was performed from January to July 2009 using a combination of keywords relevant to the area, including: force, sensor, sensing, haptics, minimally invasive surgery.
The literature search resulted in 126 papers with valuable content. This paper presents a summary of force sensing technologies, design specifications for force sensors in clinical applications, force sensors and sensing instruments that have been developed for MIST, and experiments performed to determine the need for force information. Open areas of research include: force sensor design, development of alternative methods of sensing, assessment of the impact of force information on performance, determination of the benefits of haptic information, and evaluation of the human factors involved in the processing and use of force information.
-
New Tactile Sensing System for Minimally Invasive Surgical Tumour Localization
, The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 6, no. 2, pp. 211–220, June 2010. Abstract | Wiley InterScienceBackground Minimally invasive surgery (MIS) suffers from the inability to directly palpate organs for tumour localization. A Tactile Sensing System (TSS), consisting of a probe and a visualization interface, was developed to present an active pressure map of the contact surface to locate tumours during MIS.
Methods The TSS performance was compared to MIS graspers to locate occult 10 mm phantom tumours in ex vivo bovine liver and ex vivo porcine lung. Performance assessment included applied pressure, localization distance, and accuracy.
Results The TSS realized a relative 71% reduction in maximum applied pressure and a 31% increase in detection accuracy in liver tissue (when compared to MIS graspers) and demonstrated no significant difference in performance when palpating lungs tissue.
Conclusions The TSS may help surgeons identify occult tumours during surgery by restoring some of the haptic information lost during MIS.
-
Suitability of Three Saws for Minimally Invasive Bone Cutting
, Surgical Innovation, vol. 17, no. 1, pp. 5–10, March 2010. Abstract | SageThis study compares 3 different saw types to determine which is best suited for integration into a minimally invasive bone saw. A handheld electric jigsaw, a coping saw, and a Gigli saw were used to cut into porcine ilium. Heat generated was measured using a thermocouple, and forces applied during cutting were recorded using a force/torque sensor. The coping saw generated an average maximum temperature that was 26 °C less than that generated using the jigsaw (P < .001) and 14 °C less than that for the Gigli saw (P < .001). On average, the maximum force applied through the coping saw was 14 N less than that through the jigsaw (P < .001) and 18 N less than that through the Gigli saw (P < .001). Out of the 3 saws tested, the coping saw is optimal for cutting bone based on heat generation and required force.
-
A Sensorized Instrument for Skills Assessment and Training in Minimally Invasive Surgery
, ASME Journal of Medical Devices, vol. 3, no. 4, pp. 1–12, December 2009. Abstract | ASME Digital LibraryMinimally invasive surgery (MIS) is carried out using long, narrow instruments and significantly reduces trauma to the body, postoperative pain, and recovery time. Unfortunately, the restricted access conditions, limited instrument motion, and degraded sense of touch inherent in MIS result in new perceptual-motor relationships, which are unfamiliar to the surgeon and require training to overcome. Current training methods do not adequately address the needs of surgeons interested in acquiring these skills. Although a significant amount of research has been focused on the development of sensorized systems for surgery, there is still a need for a system that can be used in any training scenario (laparoscopic trainer, animal laboratories, or real surgery) for the purpose of skills assessment and training. A sensorized laparoscopic instrument has been designed that is capable of noninvasively measuring its interaction with tissue in the form of forces or torques acting in all five degrees-of-freedom (DOFs) available during MIS. Strain gauges attached to concentric shafts within the instrument allow the forces acting in different directions to be isolated. An electromagnetic tracking system is used for position tracking. Two prototypes of the sensorized instrument were constructed. Position calibration shows a maximum root mean square (RMS) error of 1.3 mm. The results of the force calibration show a maximum RMS error of 0.35 N for the actuation force, 0.07 N in the x and y directions, and 1.5 N mm for the torque calibration with good repeatability and low hysteresis. Axial measurements were significantly affected by drift, noise, and coupling leading to high errors in the readings. Novel sensorized instruments for skills assessment and training have been developed and a patent has been filed for the design and operation. The instruments measure forces and torques acting at the tip of the instrument corresponding to all five DOFs available during MIS and provide position feedback in six DOFs. The instruments are similar in shape, size, and weight to traditional laparoscopic instruments allowing them to be used in any training environment. Furthermore, replaceable tips and handles allow the instruments to be used for a variety of different tasks.
-
Robot-Assisted Tactile Sensing for Minimally Invasive Tumor Localization
, The International Journal of Robotics Research, vol. 28, no. 9, pp. 1118–1133, September 2009. Abstract | SageBackground Minimally invasive surgery (MIS) is carried out using long, narrow instruments and significantly reduces trauma to the body, post-operative pain and recovery time. Unfortunately, the restricted access conditions, limited instrument motion, and degraded sense of touch inherent to MIS result in new perceptual-motor relationships that are unfamiliar to the surgeon and require training to overcome. Current training methods do not adequately address the needs of surgeons interested in acquiring these skills. Although significant research has focused on the development of sensorized systems for surgery, there is still a need for a system that can be used in any training scenario (laparoscopic trainer, animal labs or real surgery) for the purpose of skills assessment and training.
Method of Approach A sensorized laparoscopic instrument has been designed that is capable of non-invasively measuring its interaction with tissue in the form of forces or torques acting in all five degrees of freedom (DOFs) available during MIS. Strain gauges attached to concentric shafts within the instrument allow the forces acting in different directions to be isolated. An electromagnetic tracking system is used for position tracking.
Results Two prototypes of the sensorized instrument were constructed. An experimental evaluation was performed to assess the effectiveness of using these laparoscopic instruments to assess trainee performance during a minimally invasive knot tying task. The results indicate that the forces exerted by expert surgeons remain low for the majority of the task, followed by a characteristic peak at the end of the force profile that corresponds to the tightening of the knot. The results also show that expert surgeons perform more deliberate actions in their motion profiles resulting in smoother and less-convoluted trajectories.
Conclusions Novel sensorized instruments for skills assessment and training have been developed. The instruments measure forces and torques acting at the tip of the instrument, corresponding to all 5 DOFs available during MIS, and provide position feedback in 6 DOFs. The instruments are similar in shape, size and weight to traditional laparoscopic instruments, allowing them to be used in any training environment. Furthermore, replaceable tips and handles make the instruments versatile without limiting their commercialization potential. The results of the experimental evaluation show that there are clear differences in both the force and position profiles of trainees and surgeons with different levels of experience.
-
Feasibility of Locating Tumours in Lung via Kinesthetic Feedback
, The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 4, no. 1, pp. 58–68, March 2008. Abstract | Wiley InterScience | PDF 784.4 KBBackground Localizing lung tumours during Minimally Invasive Surgery is difficult since restricted access precludes manual palpation, and pre-operative imaging cannot map directly to the intra-operative lung. This study analyzes the force sensing performance that would allow an instrumented kinesthetic probe to localize tumours based on stiffness variations of the lung parenchyma.
Methods Agar injected into ex-vivo porcine lungs produced a model approximating commonly encountered tumours. Force-deformation data were collected from multiple sites at various palpation depths and velocities, before and after the tumours were injected.
Results Analysis showed an increase in force after the tumours were injected, ranging from 0.07 to 0.16 N at 7 mm, p<0.0001. A 2 mm/s palpation velocity minimized exponential stress decay at constant depths, facilitating easier comparisons between measurements.
Conclusion A sensing range of 0 to 1 N, with 0.01 N resolution should allow a kinesthetic palpation probe to resolve local tissue stiffness changes that suggest an underlying tumour.
-
Active-Vision Based Multi-Sensor Surveillance—An Implementation
, IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, vol. 36, no. 5, pp. 668–680, September 2006. Abstract | IEEE Xplore | PDF 1.0 MBIn this paper, a novel reconfigurable surveillance system that incorporates multiple active vision sensors is presented. The proposed system has been developed for visual-servoing and other similar applications, such as tracking and state estimation, which require accurate and reliable target surveillance data. In the specific implementation case discussed herein, the position and orientation of a single target are surveyed at predetermined time instants along its unknown trajectory. The principles of dispatching, typically used in the operation of service vehicles (e.g., taxicabs and ambulances), form the basis of an effective approach to real-time sensing-system reconfiguration. Dispatching is used to select an optimal subset of dynamic sensors, to be used in a data-fusion process, and manoeuvre them in response to the motion of the object. The goal is to provide information of increased quality for the task at hand, while ensuring adequate response to future object manoeuvres.
Our experimental system is composed of a static overhead camera to predict the object’s gross motion and four mobile cameras to provide surveillance of a feature on the object (i.e., target). Object motion was simulated by placing it on an x-y table and pre-programming a path that is unknown to the surveillance system. The (future) pose predictions of the object are used by the dispatching algorithm to determine the optimal positions and bearings of the four mobile cameras. The selected cameras are independently positioned to estimate the target’s pose (a circular marker in our case) at the desired time instant. The target data obtained from the cameras, together with their own position and bearing, are fed to a fusion algorithm, where the final assessment of the target’s pose is determined. Experiments have shown that the use of dynamic sensors, together with a dispatching algorithm, tangibly improves the performance of a surveillance system.
-
Coordinated Dispatching of Proximity Sensors for the Surveillance of Manoeuvring Targets
, Journal of Robotics and Computer Integrated Manufacturing, vol. 19, no. 3, pp. 283–299, June 2003. Abstract | ScienceDirect | PDF 954.9 KBThe surveillance of a manoeuvring target with multiple sensors in a coordinated manner requires a method for selecting and positioning groups of sensors in real time. Herein, the principles of dispatching, as used for the effective operation of service vehicles, are considered. The object trajectory is first discretized into a number of demand instants (data acquisition times), to which groups of sensors are assigned, respectively. Heuristic rules are used to determine the composition of each sensor group by evaluating the potential contribution of each sensor. In the case of dynamic sensors, the position of each sensor with respect to the target is also specified. Our proposed approach aims to improve the quality of the surveillance data in three ways: (1) The assigned sensors are manoeuvred into “optimal” sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that the surveillance system can react to future object manoeuvres. If a priori target trajectory information is available, the system performance maybe further improved by optimizing the initial pose of each sensor off-line. The advantages of dispatching dynamic sensors over similar static-sensor systems are demonstrated through comprehensive simulations.
-
ELSA: A Multisensor Integration Architecture for Industrial Grading Tasks
, Mechatronics, vol. 10, no. 1–2, pp. 19–51, February–March 2000. Abstract | ScienceDirect | PDF 314.7 KBThis paper presents the topology of the Extended Logical Sensor Architecture (ELSA) for multisensor integration and the methodology for constructing industrial sensor integration systems based on this architecture. ELSA has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. It addresses a number of issues specific to industrial inspection. The system must be modular and scalable to accommodate new processes and changing customer demands. It must be easy to understand so that non-expert users can construct, modify, and maintain the system. Furthermore, a data representation scheme which allows for the quantification of product deviations from an ideal model is required.
To address these needs, the sensors are encapsulated by a logical sensor model, providing robustness and flexibility. The construction methodology is based upon the object model which represents object classifications through combinations of primary features weighted by fuzzy membership functions. The features guide the selection of sensors and processing routines; the classifications determine the rulebase used by the inference engine for process decisions. Although inspection is the focus of this work, it is intended to be applicable to a variety of automation tasks which may benefit from a multiple sensor perception system.
Conference Articles
-
A Multi-Sensory Mechatronic Device for Localizing Tumors in Minimally Invasive Interventions
, in Proceedings of the IEEE International Conference on Robotics and Automation, Seattle, Washington, pp. 4742–4747, May 26–30, 2015. Abstract | IEEE XploreTumor localization in traditional lung resection surgery requires manual palpation of the deflated lung through a thoracotomy. It is a painful procedure that is not suitable for many patients. Therefore, a multisensory mechatronic device was designed to localize tumors using a minimally invasive approach. The device is sensorized with tactile, ultrasound and position sensors in order to obtain multimodal data of soft tissue in real time. This paper presents the validation of the efficiency and efficacy of this device via an ex vivo experimental study. Tumor pathology was simulated by embedding iodine-agar phantom tumors of varying shapes and sizes into porcine liver tissue. The device was then used to palpate the tissue to localize and visualize the simulated tumors. Markers were then placed on the location of the tumors and fluoroscopic imaging was performed on the tissue in order to determine the localization accuracy of the device. Our results show that the device localized 87.5% of the tumors with an average deviation from the tumor center of 3.42 mm.
-
Articulating Minimally Invasive Ultrasonic Tool for Robotics-Assisted Surgery
, in Proceedings of the IEEE International Conference on Robotics and Automation, Seattle, Washington, pp. 573–578, May 26–30, 2015. Abstract | IEEE Xplore -
Multimodal Noncontact Tracking of Surgical Instruments
, in Proceedings of the IEEE International Conference on Robotics and Automation, Seattle, Washington, pp. 4729–4735, May 26–30, 2015. Abstract | IEEE Xplore -
Low-Cost Force-Sensing Arthroscopic Tool Using Threaded Fiber Bragg Grating Sensors
, in Proceedings of the IEEE RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, São Paolo, Brazil, pp. 28–33, August 12–15, 2014. Abstract | IEEE XploreMinimally-invasive surgery has revolutionized many medical procedures; however, it also impedes the ability to feel the interaction between the surgical tool and the anatomical part being operated on. In order to address this problem, it is necessary to obtain accurate measurements of the interaction forces exerted on the surgical tools during surgery. These forces can then be manifested to the surgeon via a haptic device or presented visually (visual-force feedback). This paper describes the use of a fiber optic device to measure and display to the surgeon interaction forces acting on an arthroscopic tool. The sensorization of the tool involves a simple, highly efficient and robust design and is ideally suited for use in a surgical training environment aimed at narrowing the gap between trainees and expert surgeons before the trainees proceed to their first surgery in vivo. The major advantages of using fiber optics include their small size, their local simplicity, their ease of sterilization, and their high sensitivity. In this paper, a complete low-cost sensing solution is described, including 1) fiber Bragg grating sensors, 2) high resolution electronic signal processing, 3) fabrication of the tool using a wire electrical discharge machine (EDM) and 3D metal sintering technologies. Experimental results demonstrate the accuracy and performance of the sensorized tool.
-
A Parallel Remote Center of Motion Mechanism for Needle-Based Medical Interventions
, in Proceedings of the IEEE RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, São Paolo, Brazil, pp. 1–6, August 12–15, 2014. Abstract | IEEE XploreA novel parallel Remote Center of Motion (RCM) mechanism is proposed for a surgical robot designed to perform minimally invasive needle-based interventions for lung cancer diagnosis and treatment. The proposed robot provides four degrees of freedom (DOFs) to orient and move a surgical needle within a spherical coordinate system. The RCM is beneath the skin surface to minimize the invasiveness of the surgical procedure while providing the required workspace. This compact, patient-mounted robot benefits from a design capable of measuring the pure interaction forces between the needle and the tissue. In this paper, the mechanism design and its specifications are described. The kinematic analysis is presented and isotropy of the mechanism for targeting tumors is studied. Finally, the performance of the proposed robot is evaluated experimentally.
-
A Sterilizable Force-Sensing Instrument for Laparoscopic Surgery
, in Proceedings of the IEEE RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, São Paolo, Brazil, pp. 157–162, August 12–15, 2014. Abstract | IEEE XploreAlthough some technologies have been developed to measure tool-tissue interaction forces during minimally invasive surgery (MIS), none of these technologies have been approved for use in humans. The primary factor preventing the use of sensorized instruments in humans is their inability to withstand the stringent conditions present during cleaning and sterilization. This paper presents a series of experiments that were performed to develop a sterilizable instrument capable of measuring tool-tissue interaction forces in three degrees of freedom using strain gauges. The experiments provided an appropriate choice of cables and connectors, as well as an optimal combination of strain gauge adhesives and coatings that allow the sensors to withstand autoclave sterilization. A prototype of the sensorized instruments was developed and tested. The final prototype was able to withstand a sterilization cycle with excellent results (0.10-0.21 N accuracy, 0.05-0.20 N repeatability and 0.06-0.21 N hysteresis depending on the measurement direction). This work shows that autoclave sterilization is possible for a strain-gauge instrumented device.
-
Analysis of Needle–Tissue Friction during Vibration-Assisted Needle Insertion
, in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, pp. 4099–4104, November 3–7, 2013. Abstract | IEEE XploreIn this paper, a vibration-assisted needle insertion technique has been proposed in order to reduce needle-tissue friction. The LuGre friction model was employed as a basis for the current study and the model was extended and analyzed to include the impact of high-frequency vibration on translational friction. Experiments were conducted to evaluate the role of insertion speed as well as vibration frequency on frictional effects. In the experiments conducted, an 18 GA brachytherapy needle was vibrated and inserted into an ex-vivo soft tissue sample using a pair of amplified piezoelectric actuators. Analysis demonstrates that the translational friction can be reduced by introducing a vibratory low-amplitude motion onto a regular insertion profile, which is usually performed at a constant rate.
-
A Knee Arthroscopy Simulator: Design and Validation
, accepted for publication in Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Osaka, Japan, pp. 5715–5718, 2013. Abstract | IEEE XploreMany challenges exist when teaching and learning arthroscopic surgery, carrying a high risk of damaging the joint during the learning process. To minimize risk, the use of arthroscopy simulators allows trainees to learn basic skills in a risk-free environment before entering the operating room. A high-fidelity physical knee arthroscopy simulator is proposed to bridge the gap between surgeons and residents. The simulator is composed of modular and replaceable elements and can measure applied forces, instrument position and hand motion, in order to assess performance in real time. A construct validity study was conducted in order to assess the performance improvement of novices after practicing with the simulator. In addition, a face validity study involving expert surgeons indicated that the simulator provides a realistic scenario suitable for teaching basic skills. Future work involves the development of better metrics to assess user performance.
-
The Application of Force Sensing to Skills Assessment in Minimally Invasive Surgery
, in Proceedings of the 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, pp. 4355–4360, May 6–10, 2013. Abstract | IEEE XploreThe reduced access conditions present in Minimally Invasive Surgery (MIS) affect the feel of interaction forces between the instruments and the tissue being treated. This loss of haptic information compromises the safety of the procedure and must be overcome through training. Determining the skill level of trainees is critical for ensuring patient safety. The objective of this work was to evaluate the usefulness of force information for skills assessment during MIS. Experiments were performed using a set of sensorized instruments capable of measuring instrument position and tissue interaction forces. The results show that experience level has a strong correlation with force- based metrics. The proposed metrics can be automatically computed, are completely objective, and measure important aspects of performance.
-
Robot-Assisted Lung Motion Compensation during Needle Insertion
, in Proceedings of the 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, pp. 1674–1679, May 6–10, 2013. Abstract | IEEE XploreIn this paper, a robotic solution is proposed to deal with the challenges caused by lung motion during needle insertion. To accomplish this goal, a macro-micro robotic tool is designed to compensate for tissue motion using the macro part, while performing the needle insertion independently with the micro part. The main application of this work is for robotics-assisted lung tumor biopsy, where the combined motions of respiration and heartbeat may compromise success. An impedance-based controller keeps the macro reference coordinate in contact with the moving soft tissue using measurements from small pressure sensors mounted at the tip of the macro shaft. The micro part, mounted at the end of the macro robot, manipulates the needle in the harmonized reference coordinate system. Preoperative identification of ex vivo soft tissue is performed to estimate the dynamic behavior of the tissue. The controller is then synthesized based on the identified model. The effects of identification error and high frequency uncertainty are addressed in the control design. A prototype was built to evaluate the proposed approach using: 1) two Mitsubishi PA-10 robots, one for manipulating the macro part and the other for mimicking tissue motion, 2) one motorized linear stage to handle the micro part, and 3) a Phantom Omni haptic device for remote manipulation. Experimental results demonstrate the performance of the motion compensation system.
-
Design of a Minimally Invasive Lung Tumor Localization Device
, in Proceedings of the 2012 ASME Annual Dynamic Systems and Control Conference, Fort Lauderdale, Florida, DSCC2012, October 17–19, 2012. Abstract | ASME Digital CollectionThis paper describes the design, analysis, and experimental validation of a novel minimally invasive instrument for lung tumor localization. The instrument end effector is a two-degree of freedom lung tissue palpator. It allows for optimal tissue palpation to increase useful sensor feedback by ensuring sensor contact, and prevents tissue damage by uniformly distributing pressure on the tissue. Finite element analysis was used to guide the design process, resulting in a final design that could achieve a factor of safety of 4 for a 20 N force acting on the end effector–the approximate weight of a human lung. Validation experiments were conducted on a prototype instrument to assess its articulation and load-carrying capacity. The end effector design allows for the inclusion of ultrasound, tactile, and kinesthetic sensors. It is expected that this device will form the basis for robotics-assisted palpation and increase the likelihood of positive tumor localization.
-
Development of a Hands-Free Pointer for Instruction during Minimally Invasive Surgery
, in Proceedings of the 4th Biennial IEEE RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Roma, Italy, pp. 1211–1216, June 24–27, 2012. Abstract | IEEE XploreThe WHaSP, a Wireless Hands-free Surgical Pointer system has been developed to address the challenges of providing instruction during minimally invasive surgery. A headset worn by the instructing surgeon, and tracked using infrared and inertial technologies, is used to control a pointer that is overlaid on the surgical video. The combination of inertial and infrared tracking provides optimal control of the pointer and high accuracy. Experiments have been performed to validate the performance of the WHaSP prototype as compared to a commercially available hands-free pointing system. The results demonstrate improved performance, independent from the distance to the monitor. A new prototype is currently under construction that greatly improves the stability, ergonomics, and ease of use of the system. The WHaSP has the potential to significantly improve surgical instruction during minimally invasive surgery.
-
Systematic Design of an Ultrasonic Horn Profile for High Displacement Amplification
, in Proceedings of the 4th Biennial IEEE RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Roma, Italy, pp. 913–918, June 24–27, 2012. Abstract | IEEE XploreA new analytical-numerical technique is proposed to design a high displacement amplification ultrasonic horn. The profile of the horn is a NURBS curve whose parameters can be optimized to attain a high displacement amplification factor. Compared to the traditional finite element-based design approach of ultrasonic horns, which is case dependent, our approach formulates and solves the optimal horn profile problem in nondimensional space. Therefore, both the working frequency and the length of the system can be readily adjusted. Depending on the number of control points and their associated weights, the NURBS profile can be simplified to a Bezier, a rational Bezier or a B-spline curve. The optimum profile for each type of curve is then sought and the corresponding amplification factor, natural frequency and maximum strain are verified using finite element analysis. Considering a horn with a cross-sectional ratio of 16, amplification factors of 11.55, 12.12, 13.55 and 15.97 are obtained for the optimum profiles of Bezier, rational Bezier, B-spline and NURBS curves, respectively.
-
A Chance-Constrained Approach to Preoperative Planning of Robotics-Assisted Interventions
, in Proceedings of the 33rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, pp. 2127–2130, August 30–September 3, 2011. Abstract | IEEE XploreIn this paper, a planning strategy for robotics- assisted interventions is formulated in terms of uncertainty at the task level. The proposed formulation attempts to increase the chance of success by maximizing robustness with respect to the task uncertainty. It is assumed that the instrument tip pose has a Gaussian distribution in a vicinity of the desired task frame, and the planner is formulated as a chance-constrained programming problem in terms of the chance of collisions and joint limit violations based on the inverse kinematics of the arms. The proposed objective function addresses the robustness as well as the performance of the robotic arms. As an illustrative example, the planning strategy is implemented for LIMA harvesting in minimally invasive coronary artery bypass with the da Vinci robot.
-
A Framework for Preoperative Planning of Robotics-Assisted Minimally Invasive Cardiac Surgery (RAMICS) Under Geometric Uncertainty
, in Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, pp. 5018–5023, May 9–13, 2011. Abstract | IEEE XploreIn this paper, robust preoperative planning of RAMICS is formulated. The intent of the proposed planning framework is to improve surgical outcomes by contemplating the intraoperative conditions of the surgical procedure and the geometry of the patient’s thoracic anatomy. This includes improvements in target reachability, instrument dexterity for critical surgical tasks, surgical task feasibility and visibility. Given the patient’s preoperative computed tomography images of the chest, the planning framework aims to determine the optimal location of the access ports on the ribcage, along with the optimal pose of the robotic arms relative to the patient’s anatomy. To minimize susceptibility of the results to intraoperative geometric uncertainty, the planning is formulated as a Generalized Semi-Infinite Program (GSIP) with a convex lower level problem and a multi-criteria objective function. By solving the GSIP, tolerable geometric uncertainty within the task space is increased by eliminating the likelihood of collisions and joint limit violation in a neighborhood of the surgical target.
-
On Constrained Manipulation in Robotics-Assisted Minimally Invasive Surgery
, in Proceedings of the 3rd Biennial IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics, Tokyo, Japan, pp. 650–655, September 26–29, 2010. Abstract | IEEE XploreResults of a study on the restricted motion of manipulators in robotics-assisted minimally invasive surgery are presented in this paper. The governing equations of trocar kinematics are derived and the constrained manipulation is generically formulated in terms of the trocar geometry as well as the kinematics of the manipulator. The results are validated through a numerical simulation with a Mitsubishi PA10-7C arm. Following this analysis, by utilizing the constrained manipulation formulation, a modified measure of manipulability, specifically applicable to surgical tasks, is proposed. The measure is proven to be frame-invariant and dimensionally homogeneous for manipulators with similar joints (revolute or prismatic). Furthermore, the measure is shown to be capable of quantifying the impact of trocar constraints on surgical task performance when the remote center of motion is maintained through active compliant motion control.
-
Palpation System for Minimally Invasive Localization of Occult Tumors
, in Proceedings of the 3rd Biennial IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics, Tokyo, Japan, pp. 662–667, September 26–29, 2010. Abstract | IEEE XploreMinimally invasive surgery (MIS), while beneficial to patients, leads to new challenges for surgeons and prevents tumors from being localized using finger palpation. A Tactile Sensing System (TSS), consisting of a hand-held tactile sensing instrument (TSI) with a visualization interface, was developed to assist in intra-operative tumor localization. This paper presents the calibration of the TSI and its integration with a visualization interface that allows the forces applied to the tissue during palpation to be displayed. Experiments were conducted to assess the suitability of the TSS for the minimally invasive localization of phantom tumors in in vivo porcine lung during video-assisted thoracic surgery (VATS).
-
Position Control of a Spherical Joint using Feedback Linearization for SMA Wire Actuators
, in Proceedings of the 2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Montréal, Québec, pp. 1350–1355, July 6–9, 2010. Abstract | IEEE XploreShape memory alloys (SMAs) are a potent actuation modality. They possess high power densities and are easily miniaturized. However, the use of SMAs has been limited by the fact that they are difficult to control. In this work, a model of SMA behavior is presented. An I/O linearizing controller is derived and a temperature observer is formulated using this model. The efficacy of this controller is then demonstrated for position control of a single degree of freedom (DOF) joint and a novel 3 DOF spherical joint that is suitable for small-scale robotic manipulators and wrists.
-
Preoperative Planning of Robotics-Assisted Minimally Invasive Coronary Artery Bypass Grafting
, in Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, Alaska, pp. 1548–1553, May 3–8, 2010. Abstract | IEEE XploreThis paper outlines a framework for the preoperative planning of robotics-assisted minimally invasive cardiac surgery with application to coronary artery bypass grafting. The intent of the proposed framework is to improve surgical outcomes by considering the intraoperative requirements of the robotic manipulators and the anatomical geometry of the patient’s chest. This includes target reachability, instrument dexterity for critical surgical tasks and collision avoidance. Given the patient’s preoperative chest computed tomography images, the planning framework aims to determine the optimal location of the access ports on the ribcage, along with the optimal pose of the robotic arms relative to the patient’s anatomy. The proposed multi-objective optimality criteria consist of a measure of clearance as well as a new collective kinematic measure. The minimum distances among the robot arms provides a measure for the likelihood of collisions. The proposed kinematic measure is composed of two modified manipulability indices that are dimensionally homogeneous and, in contrast to previously-used measures, are more likely to yield isotropic force and torque distributions when optimized for surgical interventions. The results of a case study illustrate the compatibility of the framework with general guidelines used by experienced surgeons for port selection. Furthermore, the framework surpasses those guidelines by ensuring the feasibility of the solutions in the sense of collision avoidance and surgical target reachability.
-
Haptics-Enabled Teleoperation for Robot-Assisted Tumor Localization
, in Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, Alaska, pp. 5340–5345, May 3–8, 2010. Abstract | IEEE XploreThis paper focuses on the problem of incorporating haptics-enabled teleoperation in minimally invasive tumor localization. Since the stiffness of a tumor is higher than that of the surrounding tissue, it can be identified as a hard nodule when palpated. Using a Tactile Sensing Instrument (TSI) developed at CSTAR, the distributed pressure profiles along the contacting surface can be measured during remote tissue palpation. The tumor can be detected by using a visualization software that creates a color contour map based on the magnitude of the pressure over the palpated area. The accuracy of this method depends on the uniformity of the force applied to the tissue. A haptics-enabled teleoperation system provides the surgeon with the opportunity to feel the interaction force between the instrument and tissue during minimally invasive surgery (MIS). The objective of this research was to assess the feasibility of combining force feedback with tactile feedback in order to increase the overall performance of tumor localization. The teleoperation system used in this work consists of a Mitsubishi PA10 robot as the slave that is remotely controlled (over a dedicated network) through a 7 degree-of-freedom (DOF) haptic interface. A two-channel architecture, along with hybrid impedance control was utilized to form a bilateral teleoperation system in which the master is under force control and the slave is under position control. The experimental results confirm the effectiveness of using force feedback in robot-assisted tactile sensing for tumor detection.
-
Force/Position-based Modular System for Minimally Invasive Surgery
, in Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, Alaska, pp. 3660–3665, May 3–8, 2010. Abstract | IEEE XploreThe limitations of minimally invasive surgery include the inability to sense forces exerted by the instruments on tissue and the limited visual cues available through the endoscope. A modular laparoscopic instrument capable of measuring force and position has been designed to address these limitations. Novel image-based position tracking software has been developed and integrated within a graphical user interface. This modular system is low cost, versatile, and could be used for training, localization of critical features or for guidance during surgical procedures.
-
Design of a Novel 3 Degree of Freedom Robotic Joint
, in Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, St. Louis, Missouri, pp. 5146–5152, October 11–15, 2009. Abstract | IEEE XploreSpherical joints have evolved into a critical component of many robotic systems, often used to provide dexterity at the wrist of a manipulator. In this work, a novel 3 degree of freedom spherical joint is proposed, actuated by tendons that run along the surface of the sphere. The joint is mechanically simple and avoids mechanical singularities. The kinematics and mechanics of the joint are modeled and used to develop both open and closed loop control systems. Simulated and experimental assessment of the joint performance demonstrates that it can be successfully controlled in 3 degrees of freedom. It is expected that the joint will be a useful option in the development of emerging robotic applications, particularly those requiring miniaturization.
-
A Compact Modular Active Vision System for Multi-Target Surveillance
, in Proceedings of the 22nd IEEE Canadian Conference on Electrical and Computer Engineering, St. John's, Newfoundland, pp. 463–467, May 3–6, 2009. Abstract | IEEE XploreThis paper presents an omnidirectional active vision system that has been developed for the autonomous acquisition of detailed images of multiple targets. Omnidirectional and perspective camera technologies are integrated to create a robust vision system that combines the strengths of both camera types. A compact, inexpensive and highly modular design is presented in which system modules are stacked vertically. The vertical structure provides each module with an unobstructed 360 degree horizontal view of the surroundings and allows the omnidirectional cameras to directly guide an active camera to view a target point. The physical system design is detailed, along with a description of the system’s hardware and software architectures. The hardware architecture is scalable and fully self contained, while the software architecture is built around a user datagram protocol (UDP) network, allowing the computational load to be distributed over multiple computers.
-
Scheduling Active Camera Resources for Multiple Moving Targets
, in Proceedings of the 22nd IEEE Canadian Conference on Electrical and Computer Engineering, St. John's, Newfoundland, pp. 528–532, May 3–6, 2009. Abstract | IEEE XploreFive scheduling policies that have been developed and implemented to manage the active resources of a centralized active vision system are presented in this paper. These scheduling policies are tasked with making target-to-camera assignments in an attempt to maximize the number of targets that can be imaged with the system’s active cameras. A comparative simulation-based evaluation has been performed to investigate the performance of the system under different target and system operating parameters for all five scheduling policies. Parameters considered include: target entry conditions, congestion levels, target-to-camera speeds, target trajectories, and number of active cameras. An overall trend in the relative performance of the scheduling algorithms was observed. The Least System Reconfiguration and Future Least System Reconfiguration scheduling policies performed the best for the majority of conditions investigated, while the Load Sharing and First Come First Serve policies performed the poorest. The performance of the Earliest Deadline First policy was highly dependent on target predictability.
-
Design of a Sensorized Instrument for Skills Assessment and Training in Minimally Invasive Surgery
, in Proceedings of the 2nd Biennial IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Scottsdale, Arizona, pp. 965–970, October 19–22, 2008. Abstract | IEEE Xplore | PDF 845.3 KBThe restricted access conditions during minimally invasive surgery (MIS) result in perceptual-motor relationships that are unfamiliar to the surgeon and require training to overcome. To aid in MIS skills assessment and training, a novel sensorized instrument has been designed. Strain gauges attached to the instrument measure forces and torques acting at its tip, corresponding to all 5 degrees of freedom (DOFs) available during MIS. A position tracker provides tip motion feedback in 6 DOFs. The instrument is similar in shape, size and weight to traditional laparoscopic instruments, allowing it to be used in any training environment. Furthermore, replaceable tips and handles make the instruments highly versatile. The results of the experimental evaluation show that there are clear differences in both the force and position profiles of trainees and surgeons with different levels of experience.
-
Experimental Evaluation of Robot-Assisted Tactile Sensing for Minimally Invasive Surgery
, in Proceedings of the 2nd Biennial IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Scottsdale, Arizona, pp. 971–976, October 19–22, 2008. Abstract | IEEE Xplore | PDF 692.0 KBThe small (10 mm) incisions used in minimally invasive cancer surgery prevent direct palpation of internal organs, making intraoperative tumour localization difficult. A tactile sensing instrument (TSI), that uses a commercially available sensor to measure distributed pressure profiles along the contacting surface, has been developed to facilitate remote tissue palpation. The objective of this research is to assess the feasibility of using the TSI under robotic control to reliably locate underlying tumours. The performance of a human and a robot using the TSI to locate tumour phantoms embedded into ex vivo bovine livers is compared. An Augmented Hybrid Impedance Control scheme has been implemented on a Mitsubishi PA10-7C to perform force/position control during the trials. The results show that using the TSI under robotic control realizes a 55% decrease in the maximum forces applied, a 50% decrease in task completion times and a 40% increase in tumour detection accuracy. This demonstrates that tumour detection using tactile sensing is highly dependent on the consistent application of forces on the tactile sensing area and that robotic assistance can be of great benefit when trying to localize tumours during minimally invasive surgery.
-
Effect of Velocity Control on Kinesthetic Lung Tumour Localization
, in Proceedings of the 21st IEEE Canadian Conference on Electrical and Computer Engineering, Niagara Falls, Ontario, pp. 1337–1340, May 4–7, 2008. Abstract | IEEE Xplore | PDF 2.6 MBRestricted access during minimally invasive surgery precludes manual palpation, making the localization of lung tumours challenging. This paper investigates the force sensing performance that would allow an instrumented kinesthetic probe to localize tumours based on stiffness variations of the lung parenchyma. Agar injected into ex vivo porcine lungs produced a model approximating commonly encountered tumours. Using both constant and variable velocity approaches, force-deformation data were collected from multiple sites at various palpation depths, before and after the tumours were injected. Analysis showed an increase in force after the tumours were injected, ranging from 0.07 to 0.16 N at 7 mm. A constant palpation velocity minimized exponential stress decay at constant depths, facilitating easier comparisons between measurements. A sensing range of 0 to 2 N, with 0.01 N resolution should allow a kinesthetic palpation probe to resolve local tissue stiffness changes that suggest an underlying tumour.
-
Measuring the Angle of a Rotating Link Through Compliant Driving Tendons
, in Proceedings of the 21st IEEE Canadian Conference on Electrical and Computer Engineering, Niagara Falls, Ontario, pp. 1345–1348, May 4–7, 2008. Abstract | IEEE Xplore | PDF 708.5 KBMeasuring the position of a tendon-driven rotating link, based on the state of the tendons at the proximal end is challenging. Specifically, tendon elasticity and Coulomb friction can cause significant nonlinearities in the relationship between the motion at the proximal end of the tendon and the motion of the link. An extended Kalman filter (EKF) is proposed as a potential method to refine estimates of link position based on proximal measurements. A model link and tendon system is built to demonstrate the impact of these effects on measurement. Sensors are incorporated to allow direct measurement of the link position in addition to the motion of the tendons at their origins. A dynamic model is developed for the resulting system and used in conjunction with an EKF to generate improved estimates of the link’s motion based on proximal measurements alone.
-
Evaluation of Force Feedback Requirements for Minimally Invasive Lung Tumour Localization
, in Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, California, pp. 883–888, October 29–November 1, 2007. Abstract | IEEE Xplore | PDF 788.5 KBMinimally invasive surgery is a technique that provides numerous benefits to the patient, but presents challenges to the surgeon in that dexterity, hand-eye coordination and haptic perception are compromised. Robot-assisted minimally invasive approaches have addressed the problems of dexterity and coordination; however, the lack of kinesthetic and tactile feedback remains a significant drawback. Despite many advances in this area, little is currently known about what level of feedback performance is adequate to allow the surgeon to palpate tissue to detect an underlying tumour. This paper describes experiments that were conducted on ex-vivo porcine lung, using artificial tumours, to elucidate one measure of sensor performance required to detect the presence of a tumour. The results indicate that a force- sensitive probe with a sensing range of 0 to 10 N and a resolution of 0.01 N would allow a tumour to be localized via palpation using kinesthetic feedback.
-
A Software Architecture for Adaptive Modular Sensing Systems
, in Proceedings of the 2007 IEEE International Conference on Systems, Man and Cybernetics, Montréal, Québec, pp. 177–183, October 7–10, 2007. Abstract | IEEE Xplore | PDF 159.2 KBIn this paper, a software architecture and knowledge representation scheme that enables the combination and reconfiguration of modular sensor and actuator components is described. The proposed software architecture utilizes a realtime operating system with a pre-emptive kernel, which simplifies the implementation of the architecture itself through the modularization and concurrent execution of its various software components. A virtual machine-based middleware layer runs on top of the operating system, enabling platform-independent logical algorithms to be written once, and run on any module irrespective of its underlying hardware architecture. Logical algorithms govern the behaviour of a given set of heterogeneous modules, providing them with intelligence and enabling them to behave as a single entity known as a logical module.
-
Building Blocks for Adaptive Modular Sensing Systems
, in Proceedings of the 2007 IEEE International Conference on Systems, Man and Cybernetics, Montréal, Québec, pp. 184–189, October 7–10, 2007. Abstract | IEEE Xplore | PDF 883.8 KBThis paper describes the design and implementation of a set of building blocks for the construction of adaptive modular sensing systems. A hardware architecture is proposed which allows the development of a general set of modular components with embedded knowledge about their capabilities. These modular components are referred to as transducer interface modules (TIMs). A knowledge representation scheme enables individual TIMs to exchange information about their capabilities and form a collective identity. The overall system functionality is determined by the manner in which TIMs are connected. This functionality is supported by a distributed software architecture that allows configuration-specific algorithms to be automatically loaded into each module. These building blocks can be rapidly reconfigured to form modular systems that are expected to prove useful in many applications, including industrial control, inspection systems, mobile robotics, monitoring, and data acquisition.
-
A Centralized Omnidirectional Multi-Camera System with Peripherally-Guided Active Vision and Depth Perception
, in Proceedings of the 2007 IEEE International Conference on Networking, Sensing and Control, London, United Kingdom, pp. 662–667, April 15–17, 2007. Abstract | IEEE Xplore | PDF 1.1 MBThe growing popularity of omnidirectional vision technology has spawned numerous multi-camera designs that integrate various different camera types. This paper presents an omnidirectional vision system that combines a catadioptric camera, a fisheye camera and an active perspective camera. Aligning these cameras vertically provides a number of beneficial features, such as allowing simple peripherally-guided active vision, depth perception and a near spherical composite omnidirectional field of view. By having the active camera rotate around the outer perimeter, it can attain complete spherical access to the environment. The triangulation performance is evaluated experimentally using a target fixed to a long translation stage. Static positions of the target are estimated using a stereo pair that consists of one active perspective camera and one omnidirectional camera. Overall, the system provides sufficient accuracy to facilitate further surveillance research.
-
Calibrating an Active Omnidirectional Vision System
, in Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, Alberta, pp. 3093–3098, August 2–6, 2005. Abstract | IEEE Xplore | PDF 769.9 KBThis paper describes a straightforward process for calibrating an active vision system containing both pinhole perspective and omnidirectional cameras. The perspective cameras can be easily calibrated using standard methods. Unfortunately, these methods are not suitable for omnidirectional cameras. Methods that rely on iterative least squares optimization, using a set of known image-world correspondences, are adopted for omnidirectional cameras. To ensure unbiased estimation of camera parameters, an omnidirectional calibration rig is employed so that nearly the entire field of view contains known calibration points. Measurement uncertainties collected from each stage of calibration are then combined to estimate the overall system uncertainty. This calibration process is evaluated experimentally by estimating the location of known points using triangulation, where the results achieved are comparable with the estimated system uncertainties.
-
Developing a Modular Active Spherical Vision System
, in Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, pp. 1234–1239, April 18–22, 2005. Abstract | IEEE Xplore | PDF 1.7 MBThis paper introduces a modular, real-time, omnidirectional, active vision system, as well as a constructed prototype. By combining omnidirectional and active pan-tilt cameras, a robust vision system is created that builds on the strengths of each camera type. The system can be easily configured to provide nearly an entire spherical field of view and independently track several targets of interest within the environment. The novel design allows the camera modules to be stacked, creating a vertical sensor structure. This vertical arrangement also provides a simple solution to the epipolar geometry and triangulation for target localization. Applications for this modular system can range from simple mobile robot navigation to complex multi-target tracking and surveillance.
-
Active Vision for the Autonomous Surveillance of Dynamic, Multi-Object Environments
, in Proceedings of the 2004 ASME International Mechanical Engineering Congress and Exposition, Anaheim, California, pp. 1319–1324, November 11–15, 2004. Abstract | ASME Digital Collection | PDF 327.4 KBThis paper presents a novel method for the coordinated selection and positioning of groups of active-vision cameras for the autonomous surveillance of an object-of-interest as it travels through a multi-object workspace with an a priori unknown trajectory. Different approaches have been previously proposed to address the problem of sensor selection and control. However, these have primarily relied on off-line planning methods and only infrequently utilized on-line planning to compensate for unexpected variations in a target’s trajectory. The method proposed in this paper, on the other hand, uses a real-time dispatching algorithm, which eliminates the need for any a priori knowledge of the target’s trajectory, and, thus, is robust to unexpected variations in the environment. Experiments have shown that the use of dynamic sensors along with a dispatching algorithm can tangibly improve the performance of an active-surveillance system.
-
Sensing-System Reconfiguration: A Comparison of On-line Methods
, in Proceedings of the 14th International Conference on Flexible Automation and Intelligent Manufacturing, vol. 1, Toronto, Ontario, pp. 368–375, July 12–14, 2004. Abstract | PDF 328.5 KBThis paper investigates the performance of two dispatching approaches applied to the real-time coordination of multiple, mobile sensors. The sensing system is targeted towards the surveillance of objects in the context of autonomous manufacturing systems. Sensors are assigned and manoeuvred to collect data at specific points on the object trajectory. A technique based on reinforcement learning (RL) is compared to a heuristic dispatching method and a system that does not use dispatching at all. Through a number of simulation examples, it is shown that, on average, the RL-based dispatcher achieves very similar, if not slightly better, performance than the heuristic dispatcher. Both approaches appear to provide a benefit over non-dispatching systems, thereby validating the efficacy of the dispatching approach, despite very different underlying implementations.
-
Object Surveillance Using Reinforcement Learning Based Sensor Dispatching
, in Proceedings of the 2004 IEEE International Conference on Robotics and Automation, vol. 1, New Orleans, Louisiana, pp. 71–76, April 26–May 1, 2004. Abstract | IEEE Xplore | PDF 248.7 KBThis paper outlines an approach to the coordination of multiple mobile sensors for the surveillance of a single moving target. A real-time dispatching algorithm is used to select and position groups of sensors in response to the observed object motion. The aim is to provide robust, high-quality data while ensuring that the system can react to unexpected object manoeuvres. Sensors are assigned to collect data at specific points on the object trajectory. A dispatching strategy learned via reinforcement learning is used to control the sensor poses with respect to these points. In using the learned strategy, each sensor adopts an egocentric view of the system state to determine the most appropriate action. Simulations demonstrate the performance of the RL-based dispatcher, in comparison to similar static-sensor systems.
-
A Multi-sensor Surveillance System for Active-Vision Based Object Localization
, in Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics, vol. 1, Washington, D.C., pp. 1013–1018, October 5–8, 2003. Abstract | IEEE Xplore | PDF 742.6 KBIn this paper, the implementation of a novel surveillance system that incorporates multiple active vision sensors controlled by a real-time dispatching algorithm is presented. The proposed system improves reliability and accuracy of target surveillance – tracking systems used for visual-servoing and other similar applications.
Experiments using a dispatched system have shown that the use of dynamic sensors can improve the performance of a surveillance system, primarily, due to the following factors: (i) decrease in the uncertainty associated with the object’s estimated pose, (ii) increase in robustness of the system due to its ability to cope with a wider range of a priori unknown object trajectories, and (iii) increase in reliability through sensory fault tolerance.
-
Dispatching of Coordinated Proximity Sensors for Object Surveillance
, in Proceedings of the 2001 ASME International Mechanical Engineering Congress and Exposition, New York, New York, DSC-01-01, November 11–16, 2001. Abstract | PDF 308.7 KBThis paper presents a method of selecting and positioning groups of sensors in a coordinated manner for the surveillance of a maneuvering object. The object trajectory is discretized into a number of demand instants (data acquisition times) to which groups of sensors are assigned, respectively. Heuristic rules are used to evaluate the suitability of each sensor for servicing (observing) a demand instant, determine the composition of the sensor group, and, in the case of dynamic sensors, specify the position of each sensor with respect to the object. This approach aims to improve the quality of the surveillance data in three ways: (1) the assigned sensors are maneuvered into “optimal” sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that sensing-system can react to object maneuvers. Simulations with proximity sensors demonstrate the advantages of dispatching dynamic sensors over similar static-sensor systems.
-
Simulation-Based Sensing-System Configuration for Dynamic Dispatching
, in Proceedings of the 2001 IEEE International Conference on Systems, Man and Cybernetics, vol. 5, Tucson, Arizona, pp. 2964–2969, October 7–10, 2001. Abstract | IEEE Xplore | PDF 893.8 KBThis paper presents a methodology for determining the initial configuration of a set of sensors for a surveillance task. It serves to complement a dynamic dispatching methodology, which selects and maneuvers subsets of sensors to achieve optimal data acquisition in real-time. Specifically, given a priori information about the expected object trajectory, the initial sensor poses are determined such that the sensing-system effectiveness is maximized. This is achieved using the a constrained, non-linear, direct search method in combination with simulations of the sensing-system performance. (i.e., dynamic dispatching to adjust the sensor poses in response to the object motion.)
-
Dynamic Dispatching of Coordinated Sensors
, in Proceedings of the 2000 IEEE International Conference on Systems, Man and Cybernetics, vol. 5, Nashville, Tennessee, pp. 3318–3323, October 8–11, 2000. Abstract | IEEE Xplore | PDF 167.5 KBSensory data must be collected in-real time for the majority of autonomous decision making tasks, such as target tracking, surveillance and navigation. The use of multiple sensors may significantly improve the quality and robustness of the data. Given an environment containing a set of mobile sensors, capable of altering their position and orientation, this work addresses the problem of selecting and maneuvering subsets of these sensors for optimal data acquisition in real-time. A heuristic approach to the dispatching problem suitable for on-line implementation is illustrated by a computer-simulated example.
-
Multisensor Industrial Inspection and Grading Using ELSA
, in Proceedings of the 1999 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Atlanta, Georgia, pp. 938–943, September 19–23, 1999. Abstract | IEEE Xplore | PDF 202.1 KBThe Extended Logical Sensor Architecture (ELSA) has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. This architecture addresses a number of issues specific to industrial inspection including modularity, scalability, and design by non-expert users. To address these needs, the sensors are encapsulated by a logical sensor model, providing robustness and flexibility. The construction methodology is based upon the object model which represents object classifications through combinations of primary features weighted by fuzzy membership functions. The features guide the selection of sensors and processing routines; the classifications determine the rulebase used by the inference engine for process decisions.
-
An Open Architecture for Intelligent Multisensor Integration in Industrial Applications
, in Proceedings of the SPIE International Conference on Architectures, Networks, and Intelligent Systems for Manufacturing Integration, vol. 3203, Pittsburgh, Pennsylvania, pp. 33–44, October 13–17, 1997. Abstract | SPIE Digital Library | PDF 115.8 KBAn open architecture framework for intelligent multisensor integration in an industrial environment is being developed. This framework allows for the computational evaluation and understanding of sensor uncertainty and data validity through the comparison of sensor data in a common format.
A logical sensor model is used to represent both real and abstract sensors within the architecture. This allows for the unobtrusive addition or replacement of sensors. All logical sensor outputs are accompanied by a corresponding confidence level. These confidences are used to dynamically allocate valid sensor readings for use by higher-level sensors.
Sensory information is passed to an inference engine which uses user-selectable and adjustable fuzzy logic and/or neural network modules to provide the required decision making intelligence. This architecture may be applied to a broad range of industrial applications, especially those involving non-uniform product grading.
-
Data Representation and Organization for an Industrial Multisensor Integration Architecture
, in Proceedings of the 1997 IEEE International Conference on Systems, Man and Cybernetics, vol. 1, Orlando, Florida, pp. 821–826, October 12–15, 1997. Abstract | IEEE Xplore | PDF 165.5 KBAn open architecture for intelligent multisensor integration in an industrial environment is being developed. A logical sensor model is used to represent both real and abstract sensors within the architecture, allowing for the ready addition or replacement of sensors. Processing algorithms are also encapsulated by logical sensors. Objects are modeled using a connected graph structure wherein each node represents a salient feature of the object. Interactive training is used to determine the logical sensors required to extract desired features from objects. Extracted features are identified by the user and become part of the model. Once trained, the system can use object models for identification and classification purposes.
Abstracts
-
Validating Force-based Metrics for Computerized Assessment of Technical Skills in Laparoscopic Surgery
, accepted for presentation at the AMEE 2013 Annual Conference, Prague, Czech Republic, 2013. -
A Simulation-Based Mastery Learning Approach for Robotic Surgery
, poster presentation at the Canadian Conference on Medical Education, Québec City, Québec, April 20–23, 2013. -
Validating Force-Based Metrics for Computer-Based Assessment of Technical Skills in Laparoscopic Surgery
, podium presentation at the Canadian Conference on Medical Education, Québec City, Québec, April 20–23, 2013. -
The Effect of Visual Force Feedback on Applied Forces during a Complex Laparoscopic Procedure
, poster presentation at the SAGES Annual Meeting, Baltimore, Maryland, April 17–20, 2013. -
Validating Force-based Metrics for Computerized Assessment of Technical Skills in Laparoscopic Surgery
, poster presentation at the London Health Research Day, London, Ontario, March 19, 2013. -
Towards Precise Delivery of Needle/Probe-based Interventions for Lung Cancer Diagnosis and Treatment
, poster presentation at the London Health Research Day, London, Ontario, March 19, 2013. -
Application of Simulation Technology to Mastery Learning for da Vinci Robotic Surgical System Training
, poster presentation at the London Health Research Day, London, Ontario, March 19, 2013. -
Lung Tumour Localization and Visualization via Multisensory Data Fusion in Minimally Invasive Surgery
, poster presentation at the London Health Research Day, London, Ontario, March 19, 2013. -
Force Sensing-based Simulator for Skills Assessment for Knee Arthroscopy
, poster presentation at the London Health Research Day, London, Ontario, March 19, 2013. -
A Simulation-Based Mastery Learning Approach for Robotic Surgery
, podium presentation at the Centre for Education Research & Innovation Annual Research Conference, London, Ontario, October 9, 2012. -
Validating Force-Based Metrics for Computer-Based Assessment of Technical Skills in Laparoscopic Surgery
, podium presentation at the Centre for Education Research & Innovation Annual Research Conference, London, Ontario, October 9, 2012. -
Force-based Assessment of Skill in Minimally Invasive Surgery
, podium presentation at the 7th Minimally Invasive Robotic Association International Congress, Boston, MA, September 5–8, 2012. -
Force-based Assessment of Skill in Minimally Invasive Surgery
, poster presentation at the CIHR National Student Research Poster Competition, Winnipeg, Manitoba, June 13, 2012. -
Force-based Assessment of Skill in Minimally Invasive Surgery
, podium presentation (finalist) at the London Health Research Day, London, Ontario, March 20, 2012. -
Towards Miniaturization of Ultrasonically Activated Scalpels
, poster presentation at the London Health Research Day, London, Ontario, March 20, 2012. -
New VATS Palpator Sensing System Safely Locates Occult Tumours in vivo
, podium presentation at the Canadian Surgery Forum, London, Ontario, September 15–18, 2011. -
Force Sensing Simulator for Arthroscopic Skill Testing in Orthopaedic Knee Surgery
, podium presentation at the AMEE 2011 Annual Conference, Vienna, Austria, August 27–31, 2011. -
Increasing the Reliability of Preoperative Port Placement for Robotics-Assisted Minimally Invasive Cardiac Surgery
, poster presentation at the 14th Annual Scientific Meeting of the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS), Washington, D.C., June 8–11, 2011. -
Robotic Sample Curation, Handling, Manipulation and Analysis: The Future of Sample Return Facilities?
, podium presentation at the The Importance of Solar System Sample Return Missions to the Future of Planetary Science, The Woodlands, TX, March 5–6, 2011. -
Initial Evaluation of a Force Sensing Simulator for Orthopaedic Knee Surgery
, podium presentation at the Centre for Education Research & Innovation Annual Research Symposium, London, Ontario, October 7, 2010. -
A Hands-Free Pointer for Surgical Instruction in Minimally Invasive Surgery
, podium presentation at the Canadian Surgery Forum, Québec City, Québec, September 2–5, 2010. -
Preoperative Planning of Robotics-Assisted Minimally Invasive Cardiac Surgery
, podium presentation at the 13th Annual Scientific Meeting of the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS), Berlin, Germany, June 16–19, 2010. -
Towards a Patient Eligibility Measure for Robotics-Assisted Bypass Surgery
, poster presentation at the 13th Annual Scientific Meeting of the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS), Berlin, Germany, June 16–19, 2010. -
A Novel Sensorized Instrument-Based Minimally Invasive Surgery (SIMIS) Tool: Initial Construct Validation of Force Sensing
, podium presentation at the 12th World Congress of Endoscopic Surgery, National Harbor, MD, April 14–17, 2010. -
Sensing Forces in Natural Orifice Surgery
, podium presentation at the 12th World Congress of Endoscopic Surgery, National Harbor, MD, April 14–17, 2010. -
Multi-Screen, Hands-Free Pointer System for Training in Minimally Invasive Surgery
, poster presentation at the 12th World Congress of Endoscopic Surgery, National Harbor, MD, April 14–17, 2010. -
A Novel Sensorized Instrument-Based Minimally Invasive Surgery (SIMIS) Tool: Initial Construct Validation of Position Sensing
, poster presentation at the 12th World Congress of Endoscopic Surgery, National Harbor, MD, April 14–17, 2010. -
Multi-Screen, Hands-Free Pointer System for Training in Minimally Invasive Surgery
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2010. -
Haptics-Enabled Teleoperation for Robot-Assisted Tumour Localization
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2010. -
Preoperative Planning of Robotics-Assisted Minimally Invasive Cardiac Surgery
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2010. -
New Tactile Sensing Helps to Locate Occult Tumours during VATS
, podium presentation at the Canadian Surgery Forum, Victoria, British Columbia, September 10–13, 2009. -
Measuring Temperature and Force Outcomes with Three Different Bone Saws
, poster presentation at the 23rd International Congress and Exhibition on Computer Assisted Radiology and Surgery, Berlin, Germany, June 23–27, 2009. -
Measuring Temperature and Force Outcomes with Three Different Bone-Cutting Saws
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2009. -
A Sensorized Instrument for Skills Assessment and Training in Minimally Invasive Surgery
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2009. -
New Tactile Sensing System for Intra-operative Minimally Invasive Tumour Localization
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 23, 2009. -
New Tactile Sensing System to Locate Tumours During Minimally Invasive Robotic Surgery
, podium presentation at the 4th International Congress on Minimally Invasive Robotic Association, Québec City, Québec, January 28–31, 2009. -
Robot-Assisted Palpation Improves Minimally Invasive Tumour Localization
, podium presentation at the Canadian Surgery Forum, Halifax, Nova Scotia, September 11–14, 2008. -
A New Tactile Imaging Device to Aid with Localizing Lung Tumours during Thoracoscopic Surgery
, poster presentation at the 22nd International Congress and Exhibition on Computer Assisted Radiology and Surgery, Barcelona, Spain, June 25–28, 2008. -
Tactile Imaging System for Minimally Invasive Lung Tumour Localization
, poster presentation at the Lawson Health Research Institute Research Day, London, Ontario, March 26, 2008. -
Locating Tumours in Lung via Kinesthetic Feedback
, podium presentation (finalist) at the Lawson Health Research Institute Research Day, London, Ontario, March 26, 2008. -
Building Blocks for Flexible Inspection
, invited presentation at the 7th International Conference on Advanced Manufacturing Technologies, London, Ontario, June 5, 2007.
Patents and Intellectual Property
-
Hands-Free Pointer System
, PCT Patent Pending, no. PCT/CA2008/001690, November 2009. -
Sensorized Medical Instrument
, PCT Patent Pending, no. PCT/CA2009/00021, January 2009. -
Hands-Free Pointer System
, United States Provisional Patent, no. 61/116,675, November 2008. -
Training and Skills Assessment System for Minimally Invasive Surgery
, United States Provisional Patent, no. 61/006,443, January 2008.
Theses
-
Sensing-System Planning for the Surveillance of Moving Objects
, Doctoral thesis, Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, 2004. Abstract | PDF 1.9 MBThe surveillance of a manoeuvring target with multiple, mobile sensors in a coordinated manner requires a method for selecting and positioning groups of sensors in real time. In this context, this thesis outlines a comprehensive approach to planning sensor systems. The problem is addressed in two parts. The first part aims to optimally configure the sensing system for a particular task. This is accomplished through a simulation-based optimization procedure conducted off-line. Given a priori information about the expected object trajectories, the sensor set and initial sensor poses are determined such that the effectiveness of the sensing system is maximized. Using the optimally configured sensing system as a starting point, the second part updates the sensor poses and coordination strategies in real-time, reacting to the object motion. The principles of dispatching, as used for the effective operation of service vehicles, are adopted for this purpose. The predicted object trajectory is first discretized into a number of demand instants (data acquisition times), to which groups of sensors are assigned, respectively. Dispatching uses two complementary strategies. The coordination strategy determines which sensors will be assigned to a demand, while the positioning strategy specifies the pose of all sensors. The proposed approach aims to improve the quality of surveillance data in three ways: (1) The assigned sensors are manoeuvred into “optimal” sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that the surveillance system can react to future object manoeuvres. Two different sensor dispatching methods are developed. One uses heuristics, while the other learns appropriate strategies using reinforcement learning techniques. In addition, a method by which the two approaches may be used synergistically is introduced. Comprehensive simulations and experiments demonstrate the advantages of dispatching dynamic sensors over similar static-sensor surveillance systems.
-
ELSA: A Multisensor Integration Architecture for Industrial Grading Tasks
, Master's thesis, Mechanical Engineering, University of British Columbia, Vancouver, British Columbia, 2000. Abstract | PDF 1.2 MBThis paper presents the topology of the Extended Logical Sensor Architecture (ELSA) for multisensor integration and the methodology for constructing industrial sensor integration systems based on this architecture. ELSA has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. It addresses a number of issues specific to industrial inspection. The system must be modular and scalable to accommodate new processes and changing customer demands. It must be easy to understand so that non-expert users can construct, modify, and maintain the system. Furthermore, a data representation scheme which allows for the quantification of product deviations from an ideal model is required.
To address these needs, the sensors are encapsulated by a logical sensor model, providing robustness and flexibility. The construction methodology is based upon the object model which represents object classifications through combinations of primary features weighted by fuzzy membership functions. The features guide the selection of sensors and processing routines; the classifications determine the rulebase used by the inference engine for process decisions. Although inspection is the focus of this work, it is intended to be applicable to a variety of automation tasks which may benefit from a multiple sensor perception system.
