Mechatronics and Machine Vision in Practice 4 [1st ed.] 9783030437022, 9783030437039

Table of contents :
Front Matter ....Pages i-ix
Front Matter ....Pages 1-1
The Design of Optical Sensing Needles for Tactile Sensing and Tissue Identification at Needle Tip (Zonglai Mo, Jun Li, Weiliang Xu, Neil G. R. Broderick)....Pages 3-20
Object Detection on Train Bogies Using Structured Light Scanning (Tangwen Yang, Yantao Sun, Xiaoqing Cheng, Honghui Dong, Yong Qin)....Pages 21-31
A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore Powder Based on Machine Vision (Shanling Ji, Zhisheng Zhang, Zhijie Xia, Ying Zhu)....Pages 33-44
6D Pose Estimation of Texture-Less Object in RGB-D Images (Heng Zhao, Chungang Zhuang, Lei Jia, Han Ding)....Pages 45-58
Front Matter ....Pages 59-59
Improving Vision-Based Detection of Fruits in a Camouflaged Environment with Deep Neural Networks (Jinky G. Marcelo, Joel P. Ilao, Macario O. Cordel II)....Pages 61-69
Mechatronics for a LiDAR-Based Mobile Robotic Platform for Pasture Biomass Measurement (M. Sharifi, S. Sevier, H. Zhang, R. Wood, B. Jessep, S. Gebbie et al.)....Pages 71-80
Vision Guidance with a Smart-Phone (John Billingsley)....Pages 81-85
A High-Speed Camel Dung Collection Machine (Samuel N. Cubero, Mohammad Badi, Mohamed Al Ali, Mohammed Alshehhi)....Pages 87-103
Discussion of Soft Tissue Manipulation for the Harvesting of Ovine Offal (Qi Zhang, Weiliang Xu, Zhisheng Zhang, Martin Stommel, Alexander Verl)....Pages 105-115
Front Matter ....Pages 117-117
Fabrication and Characterization of 3D Printed Microfluidics (Swapna A. Jaywant, Muhammad Asif Ali Rehmani, Tanmay Nayak, Khalid Mehmood)....Pages 119-124
A Modified Bresenham Algorithm for Control System of FDM Three-Dimensional Printer (Ke Yu, Zhisheng Zhang, Zhiting Zhou, Min Dai)....Pages 125-139
Design and Experimental Study on the Self-Balancing Foot Device (Rui Peng, Liang Han)....Pages 141-151
Dynamic Characteristics Analysis and Optimization Design of Cross-Beam Assembly in 3D Printer (Weijie Chu, Limiao Gu, Xiaolong Liu, Fang Jia)....Pages 153-162
Design and Experimental Research of Automatic Tightening Method of Rubber Strip on the Side of Office Screen Panel (Ruonan Wang, Liang Han, Jinghui Peng, Rui Peng)....Pages 163-178
A Scene Feature Based Eye-in-Hand Calibration Method for Industrial Robot (Guoshu Xu, Yonghua Yan)....Pages 179-191
Vision-Based Trajectory Planning for a Five Degree of Freedom Assistive Feeding Robotic Arm Using Linear Segments with Parabolic Blend and Cycloid Functions (Priyam A. Parikh, Keyur D. Joshi, Reena Trivedi)....Pages 193-206
Structure Design and Closed-Loop Control of a Modular Soft-Rigid Pneumatic Lower Limb Exoskeleton (Jiangbei Wang, Yanqiong Fei)....Pages 207-214
Front Matter ....Pages 215-215
Real-Time, Dynamic Simulation of Deformable Linear Objects with Friction on a 2D Surface (Benjamin Maier, Marius Stach, Miriam Mehl)....Pages 217-231
A System for Capturing of Electro-Muscular Signals to Control a Prosthesis (Zeming Zhao, Bo Lv, Xinjun Sheng, Xiangyang Zhu)....Pages 233-243
Challenges in Robotic Soft Tissue Manipulation—Problem Identification Based on an Interdisciplinary Case Study of a Teleoperated Drawing Robot in Practice (M. Wnuk, F. Jaensch, D. A. Tomzik, Z. Chen, J. Terfurth, S. Kandasamy et al.)....Pages 245-262
Modeling of Lens Based on Dielectric Elastomers Coupling with Hydrogel Electrodes (Hui Zhang, Zhisheng Zhang)....Pages 263-268
Front Matter ....Pages 271-271
A General Monitoring Method for the State of Spandex Based on Fuzzy Evaluation and Its Application (Limiao Gu, Yan Wen, Yu Zhang, Weijie Chu, Yunde Shi, Fang Jia)....Pages 271-282
Study on the Type Identification of Cheese Yarn Based on Low-Resolution Pictures (Xiaolong Liu, Ran Hu, Yan Wen, Yu Zhang, Weijie Chu, Zhisheng Zhang et al.)....Pages 283-294
Research on High Feeding Speed System of L-Valve Rods Based on Two-in-One Device (Shiwei Cheng, Liang Han, Kai Yu, Rui Peng)....Pages 295-307
Back Matter ....Pages 309-309

Citation preview

John Billingsley Peter Brett Editors

Mechatronics and Machine Vision in Practice 4

Mechatronics and Machine Vision in Practice 4

John Billingsley Peter Brett •

Editors

Mechatronics and Machine Vision in Practice 4

123

Editors John Billingsley School of Mechanical and Electrical Engineering University of Southern Queensland Toowoomba, QLD, Australia

Peter Brett Agricultural Technologies and Robotics University of Southern Queensland Toowoomba, QLD, Australia

ISBN 978-3-030-43702-2 ISBN 978-3-030-43703-9 https://doi.org/10.1007/978-3-030-43703-9

(eBook)

© Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Introduction

These are selected and revised papers from the 26th Annual Conference on Mechatronics and Machine Vision in Practice, held in Toowoomba December 2–5, 2019. In the process of selecting the papers, many were rejected as lacking the essential ‘in practice’ element. The survivors have been grouped into those that deal predominantly with vision and optical sensing, those that relate to agricultural applications, more general robotics and devices, sensing methods and actuation, and finally industrial processes and products. The conference was held with the hospitality of USQ’s McGregor College and its Centre for Agricultural Engineering. The local team presented several keynotes not included here, which are likely to be published elsewhere later. There are strong contributions from a number of Chinese and New Zealand Universities, together with contributions from the Philippines, Emirates, and Germany. For more details on the topics, there are summaries in each of the part headings, covering a vast array of themes ranging from deep learning applied to vision analysis to a robotic mechanism used for collecting camel dung.

v

Contents

Vision and Optical Sensing The Design of Optical Sensing Needles for Tactile Sensing and Tissue Identification at Needle Tip . . . . . . . . . . . . . . . . . . . . . . . . . Zonglai Mo, Jun Li, Weiliang Xu, and Neil G. R. Broderick Object Detection on Train Bogies Using Structured Light Scanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangwen Yang, Yantao Sun, Xiaoqing Cheng, Honghui Dong, and Yong Qin A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore Powder Based on Machine Vision . . . . . . . . . . . . . . . . . . . . . . . . Shanling Ji, Zhisheng Zhang, Zhijie Xia, and Ying Zhu 6D Pose Estimation of Texture-Less Object in RGB-D Images . . . . . . . Heng Zhao, Chungang Zhuang, Lei Jia, and Han Ding

3

21

33 45

Agriculture Applications Improving Vision-Based Detection of Fruits in a Camouflaged Environment with Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . . Jinky G. Marcelo, Joel P. Ilao, and Macario O. Cordel II Mechatronics for a LiDAR-Based Mobile Robotic Platform for Pasture Biomass Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Sharifi, S. Sevier, H. Zhang, R. Wood, B. Jessep, S. Gebbie, K. Irie, M. Hagedorn, B. Barret, and K. Ghamkhar

61

71

Vision Guidance with a Smart-Phone . . . . . . . . . . . . . . . . . . . . . . . . . . . John Billingsley

81

A High-Speed Camel Dung Collection Machine . . . . . . . . . . . . . . . . . . . Samuel N. Cubero, Mohammad Badi, Mohamed Al Ali, and Mohammed Alshehhi

87

vii

viii

Contents

Discussion of Soft Tissue Manipulation for the Harvesting of Ovine Offal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Qi Zhang, Weiliang Xu, Zhisheng Zhang, Martin Stommel, and Alexander Verl Robotics and Devices Fabrication and Characterization of 3D Printed Microfluidics . . . . . . . . 119 Swapna A. Jaywant, Muhammad Asif Ali Rehmani, Tanmay Nayak, and Khalid Mehmood A Modified Bresenham Algorithm for Control System of FDM Three-Dimensional Printer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Ke Yu, Zhisheng Zhang, Zhiting Zhou, and Min Dai Design and Experimental Study on the Self-Balancing Foot Device . . . . 141 Rui Peng and Liang Han Dynamic Characteristics Analysis and Optimization Design of Cross-Beam Assembly in 3D Printer . . . . . . . . . . . . . . . . . . . . . . . . . 153 Weijie Chu, Limiao Gu, Xiaolong Liu, and Fang Jia Design and Experimental Research of Automatic Tightening Method of Rubber Strip on the Side of Office Screen Panel . . . . . . . . . . 163 Ruonan Wang, Liang Han, Jinghui Peng, and Rui Peng A Scene Feature Based Eye-in-Hand Calibration Method for Industrial Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Guoshu Xu and Yonghua Yan Vision-Based Trajectory Planning for a Five Degree of Freedom Assistive Feeding Robotic Arm Using Linear Segments with Parabolic Blend and Cycloid Functions . . . . . . . . . . . . . . . . . . . . . 193 Priyam A. Parikh, Keyur D. Joshi, and Reena Trivedi Structure Design and Closed-Loop Control of a Modular Soft-Rigid Pneumatic Lower Limb Exoskeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Jiangbei Wang and Yanqiong Fei Sensing Methods and Actuation Real-Time, Dynamic Simulation of Deformable Linear Objects with Friction on a 2D Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Benjamin Maier, Marius Stach, and Miriam Mehl

Contents

ix

A System for Capturing of Electro-Muscular Signals to Control a Prosthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Zeming Zhao, Bo Lv, Xinjun Sheng, and Xiangyang Zhu Challenges in Robotic Soft Tissue Manipulation—Problem Identification Based on an Interdisciplinary Case Study of a Teleoperated Drawing Robot in Practice . . . . . . . . . . . . . . . . . . . . . 245 M. Wnuk, F. Jaensch, D. A. Tomzik, Z. Chen, J. Terfurth, S. Kandasamy, J. Shahabi, A. Garrett, M. H. Mahmoudinezhad, A. Csiszar, W. L. Xu, O. Röhrle, and A. Verl Modeling of Lens Based on Dielectric Elastomers Coupling with Hydrogel Electrodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Hui Zhang and Zhisheng Zhang Industrial Processes and Products A General Monitoring Method for the State of Spandex Based on Fuzzy Evaluation and Its Application . . . . . . . . . . . . . . . . . . . 271 Limiao Gu, Yan Wen, Yu Zhang, Weijie Chu, Yunde Shi, and Fang Jia Study on the Type Identification of Cheese Yarn Based on Low-Resolution Pictures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Xiaolong Liu, Ran Hu, Yan Wen, Yu Zhang, Weijie Chu, Zhisheng Zhang, and Fang Jia Research on High Feeding Speed System of L-Valve Rods Based on Two-in-One Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Shiwei Cheng, Liang Han, Kai Yu, and Rui Peng Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Vision and Optical Sensing

The first chapter is more about sensing than vision, concerned with detecting force at a surgical needle-tip to identify the tissue layer that it has reached. Light is the chosen communication medium, being unaffected by the intense magnetic fields of a MRI environment. The next chapter concerns the build-up of detritus on a train bogy. Structured light is used to measure its volume. At a much smaller scale, vision is used to measure the breaking of spores that are used in Chinese medicine. These spores must be broken to act more efficiently. The fourth chapter in this part concerns the detection of the pose of an object that lacks texture features, for the purpose of grasping it. Although deep learning features in the method, there is a strong experimental basis.

The Design of Optical Sensing Needles for Tactile Sensing and Tissue Identification at Needle Tip Zonglai Mo, Jun Li, Weiliang Xu, and Neil G. R. Broderick

1 Introduction Needle insertion is a common surgery in a variety of procedures such as interventional radiology [1], neurosurgery [2], brachytherapy [3], regional anaesthesia [4], biopsies [5], drug delivery [6] and blood sampling [7]. It also represents one of the least invasive ways to access the internal organs of patients [8]. An example is epidural anaesthesia, a regional anaesthesia used for pain relief that can be performed at different locations along the spine depending on the surgery [9]. In England alone, about 280,000 epidural anaesthesia are performed yearly within the National Health Service [10]. In China, based on the 16 million newborns in 2017, about 8 million are potential users of epidural anesthesia, according to the c-section rate of more than 50% [11]. In the process of spinal epidural puncture anesthesia, the operator will inject anesthetics into the spinal epidural space between the spinal nerve and the yellow ligament through an injection needle (the width range is only about 2–7 mm, and the distance from the subcutaneous is about 20–90 mm). This operation requires the operator to determine the position of the needle tip immediately when the needle tip punctures the yellow ligament into the epidural space and stop the puncture. If Z. Mo (B) · J. Li School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing, China e-mail: [email protected] W. Xu Department of Mechanical Engineering, The University of Auckland, Auckland, New Zealand N. G. R. Broderick Department of Physics, The University of Auckland, Auckland, New Zealand Z. Mo · W. Xu · N. G. R. Broderick The Dodd-Walls Centre for Photonic and Quantum Technologies, Dunedin, New Zealand © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_1

3

4

Z. Mo et al.

the puncture continues, the spinal nerve will be easily punctured, resulting in shortterm or lifelong postoperative low back pain, and even paralysis in severe cases [12]. However, in the actual situation, the operator’s tactile perception of the tip is almost lost, and the judgment of the tip position mainly depends on the operator’s experience. Medical statistics show that the number of repuncture localization and various surgical side effects caused by the failure of puncture operation positioning can reach 13–47% [13, 14]. Therefore, how to capture tactile perception to accurately identify the position of needle tip has been an urgent problem in clinical practice. Force sensing is one of our main means of interacting with the environment. A force sensor can detect contact between itself and an object, and measures static or dynamic force magnitudes [15]. Some experimental studies [16, 17] have shown that, without force feedback, tissue trauma and unexpected damage to healthy organs increase during surgery. When force feedback is incorporated into teleoperated systems can reduce robotic force by 30–60%, peak forces by a factor of 2–6, operating time by 30% and error rates by 60% [18]. This shows the need for force sensors to be integrated into surgical instruments. However, the traditional electronic sensors are easily interfered by the electromagnetic imaging equipment used for surgery, and the electronic sensors are too large to be integrated into the needle, so the problem of sensor integration in the needle and tip has not been solved. In recent years, optical fiber sensor has been developed rapidly, which overcomes the shortcomings of the above methods. It has the characteristics of electromagnetic immunity and small size advantages, and it has unique advantages in the integrated technique of spinal puncture surgical needle. According to literature statistics, Fabry-Perot interference (FPI) and Fiber Bragg Grating (FBG) sensing technology are the main potential means of surgical needle integration. In the past years, only several researchers focused on tissue identification by needle-like probe. Liu et al. [19] designed a wheeled end-sensing probe with intensity modulated fiber optic force sensor, which was used to locate abnormal tissues in minimally invasive surgery. The experiment proved that the probe could effectively detect the location of tumor. Yip et al. [20] designed a three-axis end sensing probe (5.5 mm in diameter) based on a similar sensing structure, which was used for heart beat detection in vivo. The test showed that the detection error rate was less than 3%. Beekmans et al. [21] designed the terminal tactile feedback probe needle (5 mm in diameter) based on FPI optical fiber sensing technology combined with cantilever beam structure, and identified tissue types and boundaries by analyzing the detected soft tissue stiffness information. Carotenuto et al. [22] and others take the lead in the grating (FBG) sensors are integrated into the needle inside, used to detect needlepoint into epidural gap signal, but not its sensing structure temperature compensation measures, installation also affect the original function of epidural needles. Kumar et al. [23] designed the end sensing needle based on FBG to estimate the end force of the instrument in the process of software puncture, and the experiment proved that the sensing needle could detect the puncture signal of tissue boundary. In addition, Liu et al. [24] designed 3d end sensor needle for eye tissue sampling puncture surgery based on FPI principle or FBG principle to detect weak force feedback that cannot be perceived by human body during puncture. Beisenova et al. [25] integrated FBG

The Design of Optical Sensing Needles for Tactile Sensing …

5

distributed sensing on the outside of the needle body for the identification of epidural gap, which can detect the pressure signals between various parts of the needle and soft tissues, and estimate the types of soft tissues that have been punctured. It was verified by experiments with soft tissue composites of different kinds of animals. In this study, novel optical sensing solutions based on our previous designs [26– 28] are introduced and compared for needle-tip tactile sensing and bio-tissue identification, based on interference intensity modulated Fabry-Perot interference (FPI) principle. The design of the sensor and its optical circuit are detailed described. The system was verified under different working conditions, by conducting simulations, phantom tests, ex-vivo and in-vivo animal tests. Sensor properties and environmental influence factors on the sensor were investigated. The limitation of the designs and further improvements are also discussed in the paper.

2 Sensing Needle Designs 2.1 FPI Force Sensing Principle A typical FPI sensor (Fig. 1) has two cleaved single mode fibers embedded inside of a glass capillary by epoxy. There is a cavity in micrometer level between two fiber ends. An incident light is delivered into the fibers and four percent of the incident light is reflected by each cleaved fiber end. Consequently, the two beams of the reflected light are interfered with each other, resulting in an interfered light that can be detected by a photodiode sensor or a spectrometer. When an axial loading is applied, the cavity length varies, so does the intensity of the interfered light, which is described by,  I = I1 + I2 + 2 I1 I2 cos(φ)

(1)

where I is the intensity of the interfered light, I1 and I2 are the light intensities of the two reflected incident light beams, and is the intensity phase, which is the phase difference between the two reflected lights. The intensity phase can be calculated by,

Fig. 1 A typical FPI sensor

6

Z. Mo et al.

φ=

2π(L) λ

L = 2(d + d)

(2) (3)

where is the optical path-length difference between two reflected light beams, d and d are the initial cavity length and the change in cavity length, respectively and λ is the wavelength of the incident light.

2.2 Sensing Needle Designs To compensate for the influence of temperature variations, two identical FPI sensors were placed at the tips of surgical needles (an injection bevel-tip needle and an epidural needle), as shown in Fig. 2, with one sensor serving as a reference sensor subjected only to temperature variation and the other serving as a force sensor subjected to both temperature and axial loading at the tip of the needle. Figure 3a shows the schematic optical circuit of the temperature compensated FPI sensing system. A 1 mw hand-held laser with wavelength of 1550 nm is the source of the incident light. An optical isolator prevents reflected light returning to the source. The laser is split into two channels by a 50/50 splitter, which are transmitted to the two FPI sensors. Then the interfered light of the two FPI sensors are measured by two photodiode power sensors and the measurements are further processed by a computer. In this design, for the force sensor, the FPI cavity length change is induced by both force change and temperature change, while the reference sensor only has the temperature influence. Consequently, the temperature influence of the force sensor can be compensated for. d F = f 1 (F) + f 2 (T )

(4)

Fig. 2 Placement of the two FPI sensors for temperature compensation a Bevel tip injection needle. b Epidural needle

The Design of Optical Sensing Needles for Tactile Sensing …

7

a) Schematic optical circuit

b) The embedded sensor and overall system, (a) The sensing needle, (b) Handheld laser source, (c) Optical circuit box, (1) An FPI sensor, (2) The end faces of the sensor Fig. 3 Temperature-compensated FPI sensing system. a Schematic optical circuit. b The embedded sensor and overall system. a The sensing needle. b Handheld laser source. c Optical circuit box, (1) An FPI sensor, (2) The end faces of the sensor

d f = d F − kd R = f 1 (F) + f 2 (T ) − k f 3 (T )

(5)

where df is the effective cavity length change induced by force loadings, dR is the cavity length change of the reference sensor due to temperature variation, and k is the coefficient accounting for the difference in the temperature influence in the two sensors, which can be calibrated experimentally.

8

Z. Mo et al.

Fig. 4 Experimental tissues a Phantom skin. b Phantom fat. c Phantom muscle. d Phantom skinfat-muscle multiple layer tissue. e Phantom liver. f Pig tissue. g Mice

2.3 Tissue Samples Preparation Five kinds of phantom human tissues (SynDaverTM Labs, USA) were prepared for insertion experiments, i.e. phantom human skin, fat, muscle, liver, and multiple layer abdominal tissue (Fig. 4a–e). In addition, pig tissue and mice (Fig. 4f–g) were also used for experimental verifications.

2.4 System Performance After Temperature Compensation To evaluate the temperature dependence after the compensation, the needle was left in the chamber and heated to six different temperatures ranging from 23 to 45 °C with 6 °C intervals. In each temperature condition, the chamber temperature fluctuated up and down within 2 °C for fifteen minutes. The temperature compensation result (in terms of the intensity phase) is satisfactory, as shown in Fig. 5. To check the effectiveness of the temperature compensation involving axial loading, pulse forces were applied periodically at the needle tip during the temperature change of 23–37.5 °C, as shown in Fig. 6. The intensity phase of the force sensor is attributed to both the temperature and force, while the intensity phase of the reference sensor only to the temperature. A mean intensity phase error of 0.03 rad was found after temperature compensation. The intensity phase of the force sensor after compensation and the applied dynamic forces were recorded (Fig. 7), and then a linear relationship between them was obtained. The results show that the temperature compensated force sensor has a force measurement range of 0-8 N with a resolution of 0.3 N at the temperature of 23–37.5 °C.

The Design of Optical Sensing Needles for Tactile Sensing … 0.04

Intensity phase error after compensation (rad)

Fig. 5 Temperature dependence within 23–45 °C [27]

9

0.03

Mean compensation error

0.02 0.01 0 -0.01 -0.02

20

30

40

50

Fig. 6 Effective intensity phase after temperature compensation [26]

FPI intensity signal phase(rad)

Temperature (°C)

2.1 1.9 1.7 1.5 1.3 1.1 0.9 0.7 0.5 0.3 0.1 -0.1

Reference sensor signal phase Force sensor signal intensity phase Compensated force signal phase Compensation Error 37.5°C

23°C A pulse force

0

5

10

15

20

25

30

35

40

45

10 Force signal(N)

Fig. 7 FPI signal and applied force detected by dynamic force sensor [26]

Dynamic force sensor signal FPI sensor phase signal

8

0.6

6

0.4

4

0.2

2 0

0.8

0

50

100

150

FPI signal phase(uw)

Time(s)

0

Time(s)

3 Experimental Verifications and Needle Applications To characterize the needle’s force sensing capability, ex-vivo experiments were conducted using both phantom human tissues and bio tissues.

10

Z. Mo et al.

3.1 Temperature Change Versus the Time of Death To investigate the influence of temperature, the mouse body temperature was measured using a digital thermometer (DS18B20) with a precision of ±0.5 °C. Temperature change after death over time was recorded in Fig. 8. It shows that the body temperature dropped gradually within 35 min after death, from about 36 °C to near room temperature, yielding to a polynomial relationship. It also indicates that a mouse within one minute after death could be regarded the same as a live mouse in terms of body temperature and living organs. Figure 9 is the FPI interference light intensity signals of two skin-tumor-skin insertions and one abdomen insertion on a mouse at a time of two hours after death. The FPI reference sensor signal (the red curve in Fig. 9) shows that there was still a temperature influence that needs to be compensated for two hours after death. Although there might be only little temperature difference between mouse’s body and room temperature, the FPI signal induced by the influence is still large compared with the penetration force. The above results demonstrate that, for the mouse tissue penetration, the sensing needle needs temperature compensation at least within two hours after death of the mouse. Fig. 8 Temperature change post mortem [28]

Fig. 9 Insertions two hours after death [28]

The Design of Optical Sensing Needles for Tactile Sensing …

11

3.2 The Difference Between in-Vivo Insertion and Ex-Vivo Insertion In-vivo needle insertion is further proof of the temperature compensation effect of the sensing needle. The purpose of this section is to compare the difference between these two conditions, through conducting skin-tumor-skin insertion experiments. Four mice were sampled in this experiment. As shown in Fig. 10, the mouse was first anaesthetized in a box (Fig. 10a), and then moved to a tube with anaesthetic gas flow, putting the mouth and nose inside the tube for continuous anaesthesia during insertion experiments (Fig. 10b). The in-vivo insertion experiments then began, as shown in Fig. 10c. Figure 11 shows a typical FPI interference light intensity signal during the insertion, and its processed tip force signal after temperature compensation is shown in Fig. 12. The signal of one ex-vivo insertion on the skin-tumor-skin with 10 s time of death was added into Fig. 11 for comparison, shown as the dotted line. The result indicates that in-vivo insertions match well with the ex-vivo insertions on mice one minute from time of death. This experiment shows a database based on freshly-killed mice can be regarded equivalent to live mice, which can further be used for real-time in-vivo tissue identification during needle insertions.

Fig. 10 In-vivo mouse experiment. a Initial anesthesia. b Anesthesia setup during experiments. c Needle insertion on an anesthetized mouse

Fig. 11 In-vivo insertion FPI light intensity signal [28]

12

Z. Mo et al.

Fig. 12 In-vivo tip force signal after temperature compensation [28]

Fig. 13 Insertion tip force under different needle advancing rates [27]

3.3 Needle Advancing Rate The needle insertion force (hybrid tip/friction force) was believed relating to the needle advancing rate, both in phantom tissue [29] and biological tissue [30]. But studies on the influence of needle advancing rates on the needle tip force are scarce. Therefore, the relationship between tip force and needle advancing rate must be explicitly measured for the purpose of characterization. Three needle advancing rates were considered during experiments, i.e., 3.8 mm/s, 7.8 mm/s, and 14.5 mm/s, which are in line with that in routine surgeries. The experiment was carried out on both phantom human skin tissue and swine belly skin tissue. Based on the 10 tip force readings of each configuration, standard deviation was calculated, the results of which is shown in Fig. 13. The result indicates that manual insertion may achieve similar tip force at various constant advancing rates.

3.4 Automated and Manual Insertion To figure out the influence of insertion modes on needle tip force, automated and manual insertions were conducted on phantom human muscle, respectively, with an overall time of around 6 s. Needle advancing rate was set to 3.8 mm/s for automated

The Design of Optical Sensing Needles for Tactile Sensing …

13

Fig. 14 Automated and manual insertion on phantom muscle. a Four stages of one typical insertion. b Automated and manual insertion signals [27]

insertion, and the operator drove the needle as continuous and stable as possible in manual insertions. Five automated insertions and six manual insertions were performed on phantom muscle. The tip force signals of both situations are compared in Fig. 14. From the results, different insertion stages during the procedure can be clearly identified in both situations, indicated by a, b, c, and d in Fig. 14. However, some differences can be found when stages were changed. In the automated insertion, boundary displacement (from a to b in Fig. 14) took a time similar to that for the insertion out of the tissue (from c to d). While in the manual insertion, it highly depends on the needle advancing rate by the operator. This experiment shows that tissue identification in manual insertions is also possible, despite losing thickness measurement of interacted tissue.

14

Z. Mo et al.

Fig. 15 Tip force and hybrid tip/friction force during phantom muscle insertion [27]

3.5 Friction Force and Needle Displacement Figure 15 shows the force comparison between a reference force sensor and tip force sensing needle, from the insertion of a 10 mm thickness phantom human muscle tissue. The insertion procedure had four stages, denoted by a, b, c, and d, shown in Fig. 9. It shows that the sensing needle had similar values with ATI force sensor in the stage of boundary displacement (from a to b), until the layer was ruptured at around 6.7 N. However, after inserting entire tissue depth where denoted by d, friction force detected by ATI force sensor kept steady at around 12.5 N during continuous insertion, while the tip force from the sensing needle dropped to zero.

3.6 Phantom Tissue and Bio Tissues Needle insertions on different kinds of phantom human tissues and swine tissues were conducted. Four insertion rates were applied to the sensing needle, 3.8, 7.8, 11.3, and 14.5 mm/s. Each kind of tissue was inserted individually for 10 times under each rate. In total, 40 insertion events were executed in every kind of tissue sample. The results in Figs. 16 and 17 show that insertion forces of the identical tissue vary in a specific range at different needle advancing rates, but no general tendency is found. The results also show that some internal organs such as liver and kidney may have very similar insertion tip force at around 4–5 N. However, that has no influence on tissue identification as they lie at different body areas. To obtain the tip force database used for tissue identification in mouse, three mice were killed and dissected for collecting tip force data of various organs. Small internal organs, such as heart and lung, were inserted individually three times in each mouse, except skin, muscle, and tumor, with five insertions. In total, 9 insertions were conducted on each kind of internal organ and 15 insertions for skin, muscle, and tumor. Figure 18 shows the tip force database for different tissues and internal organs.

The Design of Optical Sensing Needles for Tactile Sensing …

15

Fig. 16 The tip force during insertions of different phantom tissues [27]

Fig. 17 The tip force during insertions of different swine tissues [27]

Fig. 18 Tip force per various murine mouse tissue during insertion [28]

3.7 Multiple Layer Tissue Insertion After having data field of tip forces of each types of tissue, the sensing needle has the potential to identify the tissue type during insertions. To assess its tissue identification function, insertion experiments on multiple layer tissues need to be done. Two groups of multiple layer tissue were prepared, i.e. the swine belly fat-muscle-liver tissue and the mouse tissues. Figure 19 is the tip force sensing during the insertion of the swine fat-muscle-liver tissue. During the insertions, the layer rupture information was the most wanted signal, which provides very useful information, such as tissue thickness

16

Z. Mo et al.

Fig. 19 Automated swine tissue insertion [27]

and average tip force. It shows that layer rupture information can be clearly captured by the tip force sensing needle. For the mouse insertions, three types of multiple layer insertion experiments were designed, the skin-leg muscle, the lateral direction of the abdomen, and from anus to head. Skin-leg muscle experiment was conducted after dissection, while the others were insertions first and then dissection was undertaken for confirmation, as shown in Fig. 20. Figure 21 shows the tip force results for two needle insertions of skin-leg muscle. The needle penetrated the skin first at around 1.2 and 1.9 N, respectively in the two insertions, as stage a shown in Fig. 16. It reached leg muscle and broke the muscle layer at around 3.3 N (Stage b). It shows that the sensing needle successfully captured the tissue layer breaking force during the needle insertion. Figure 22 is the insertion result of the abdomen in a lateral direction. It was found that the needle only penetrated the skin, leg muscle and another skin layer on the other side. Another experiment was performed for internal organs identification. The needle was inserted as stably as possible into the mouse body from the anus to head. Figure 23 gives a force-time history of one insertion. The result shows that there were several

Fig. 20 Multiple layers insertion experiments. a Skin-leg muscle insertion after dissection. b Dissection after insertion of the lateral direction in the abdomen. c Dissection after insertion from anus to head

The Design of Optical Sensing Needles for Tactile Sensing …

17

Fig. 21 Murine skin-leg muscle insertion [28]

Fig. 22 Internal abdomen insertion in a lateral direction [28]

Fig. 23 Insertion from anus to head, with possible tissue identification. a Colon. b Jejunum. c Liver. d Lung. e Heart [28]

organs experiencing penetration. According to needle displacement and mouse organ location, the possible organs are colon, jejunum, liver, lung, and heart. This was also confirmed via dissection. The experimental results show that the tissue layer breaking during insertion procedure is obvious and the main organs can be successfully identified by the tip force sensing needle. The tissue type can be confirmed according to tip force database

18

Z. Mo et al.

collected for various tissues, such as the data in Fig. 18, and with the help of the knowledge of anatomy. Consequently, a tissue layer sketch can be precisely drawn for clinicians during needle insertions.

4 Discussion This paper proposes optical sensing solutions for needle-tip applications of needle-tip tactile sensing and bio-tissue identification. The design of the sensor and its optical circuit are detailed described. The system is verified via simulations, phantom tests, ex-vivo and in-vivo animal tests. During the experiments, sensor properties and environmental influence factors on the sensor were investigated. The results show that the designed needles are capable of identifying tissue layers and types via analyzing force signals during the insertion process in real time mode in a temperature-changeable environment, with effective temperature compensation designs. Even though the FPI sensing needle design achieved some good results, there are still some design issues that need to be addressed. Firstly, the cantilever beam design inside the needle increases the overall size of the sensor, which cannot be integrated into the needle with the inner diameter less than 1 mm. In addition, the integrated method makes the injection needle lose its original function. Therefore, the sensor and its temperature compensation structure need to be improved, in terms of minimizing size and proper structure design. Secondly, the influence caused by the operator’s body movements has an important influence on the analysis of the terminal tactile signals. For example, without the feedback design of external factors, it is difficult to extract the effective tactile signals of the needle tip. Therefore, it is necessary to study the mechanism of influencing factors and design quantitative compensation methods.

References 1. Deipolyi, A. R., et al. (2017). Needlestick injuries in interventional radiology are common and underreported. Radiology, 285(3), 170103. 2. Petruska, A. J., Ruetz, F., et al. (2016). Magnetic needle guidance for neurosurgery: Initial design and proof of concept. In IEEE International Conference on Robotics and Automation (pp. 4392–4397). 3. Singh, M. K., Parameshwarappa, V., et al. (2016). Photoacoustic-guided focused ultrasound for accurate visualization of brachytherapy seeds with the photoacoustic needle. Journal of Biomedical Optics, 21(12), 120501. 4. Hadjerci, O., Hafiane, A., et al. (2016). Assistive system based on nerve detection and needle navigation in ultrasound images for regional anesthesia. Expert Systems with Applications, 61, 64–77. 5. Albanghali, M., et al. (2016). Construction of tissue microarrays from core needle biopsies–a systematic literature review. Histopathology, 68(3), 323–332.

The Design of Optical Sensing Needles for Tactile Sensing …

19

6. Cheung, K., & Das, D. B. (2016). Microneedles for drug delivery: trends and progress. Drug Delivery, 23(7), 2338–2354. 7. Nagata, K., Sawada, K., et al. (2017). Effects of repeated restraint and blood sampling with needle injection on blood cardiac troponins in rats, dogs, and cynomolgus monkeys. Comparative Clinical Pathology, 26(6), 1347–1354. 8. Barbé, L., et al. (2007). Needle insertions modeling: identifiability and limitations. Biomedical Signal Processing and Control, 2(3), 191–198. 9. Roux, P. -J. A., et al. (2016). 3D haptic rendering of tissues for epidural needle insertion using an electro-pneumatic 7 degrees of freedom device. In The 2016 IEEE International Conference on Intelligent Robots and Systems 2016. 10. Manoharan, V. (2011). Epidural needle insertion simulator: a device for training resident anaesthesiologists. Master thesis, Delft University of Technology. 11. Wang, L., & Yan, M. (2014). Research progress on labor analgesia. General nursing, 25, 2312–2314. 12. Tien, J. C., Lim, M. J., Leong, W. L., et al. (2016). Nine-year audit of post-dural puncture headache in a tertiary obstetric hospital in Singapore. International Journal of Obstetric Anesthesia, 28, 34–38. 13. Hermanides, J., Hollmann, M. W., Stevens, M. F., et al. (2012). Failed epidural: Causes and management. British Journal of Anaesthesia, 109(2), 144–154. 14. Tran, D., Hor, K. W., Kamani, A. A., et al. (2009). Instrumentation of the loss-of-resistance technique for epidural needle insertion. IEEE Transactions on Biomedical Engineering, 56(3), 820–827. 15. Dargahi, J., & Najarian, S. (2005). Advances in tactile sensors design/manufacturing and its impact on robotics applications–a review. Industrial Robot: An International Journal, 32(3), 268–281. 16. Travakoli, M., Patel, R. V., & Moallem, M. (2005). Robotic suturing forces in the presence of haptic feedback and sensory substitution. In Proceedings of 2005 IEEE Conference on CCA 2005, Control Applications. 17. Demi, B., Ortmaier, T., & Seibold, U. (2005). The touch and feel in minimally invasive surgery. In IEEE International Workshop on in Haptic Audio Visual Environments and their Applications. 18. Wagner, C. R., Stylopoulos, N. & Howe, R. D. (2002) The role of force feedback in surgery: Analysis of blunt dissection. In Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. Citeseer 19. Liu, H., Noonan, D. P., Challacombe, B. J., et al. (2010). Rolling mechanical imaging for tissue abnormality localization during minimally invasive surgery. IEEE Transactions on Biomedical Engineering, 57(2), 404–414. 20. Yip, M. C., Yuen, S. G., & Howe, R. D. (2010). A robust uniaxial force sensor for minimally invasive surgery. IEEE Transactions on Biomedical Engineering, 57(5), 1008–1011. 21. Beekmans, S. V., & Iannuzzi, D. (2016). Characterizing tissue stiffness at the tip of a rigid needle using an opto-mechanical force sensor. Biomedical Microdevices, 18(1), 15. 22. Carotenuto, B., Micco, A., Ricciardi, A., et al. (2017). Optical guidance systems for epidural space identification. IEEE Journal of Selected Topics in Quantum Electronics, 23(2), 371–379. 23. Kumar, S., Shrikanth, V., Amrutur, B., et al. (2016). Detecting stages of needle penetration into tissues through force estimation at needle tip using fiber Bragg grating sensors. Journal of Biomedical Optics, 21(12), 127009. 24. Liu, X., Iordachita, I. I., He, X., et al. (2012). Miniature fiber-optic force sensor based on low-coherence Fabry-Pérot interferometry for vitreoretinal microsurgery. Biomedical Optics Express, 3(5), 1062–1076. 25. He, X., Handa, J., Gehlbach, P., et al. (2014). A submillimetric 3-DOF force sensing instrument with integrated fiber Bragg grating for retinal microsurgery. IEEE Transactions on Biomedical Engineering, 61(2), 522–534. 26. Mo, Z., & Xu, W. (2016). Temperature-compensated optical fiber force sensing at the tip of a surgical needle. IEEE Sensors Journal, 16(24), 8936–8943.

20

Z. Mo et al.

27. Mo, Z., Xu, W., & Broderick, N. G. (2017). Capability characterization via ex-vivo experiments of a fiber optical tip force sensing needle for tissue identification. IEEE Sensors Journal, 18(3), 1195–1202. 28. Mo, Z., Mao, X., Hicks, K. O., & Xu, W. (2018). In-Vivo Tissue Identification on Mice Using a Fiber Optical Tip Force Sensing Needle. IEEE Sensors Journal, 18(15), 6352–6359. 29. Crouch, J., et al. (2005). A velocity-dependent model for needle insertion in soft tissue. In Medical Image Computing and Computer-Assisted Intervention–Miccai 2005 (pp. 624–632). 30. Mahvash, M., & Dupont, P. E. (2010). Mechanics of dynamic needle insertion into a biological material. IEEE Transactions on Biomedical Engineering, 57(4), 934–943.

Object Detection on Train Bogies Using Structured Light Scanning Tangwen Yang, Yantao Sun, Xiaoqing Cheng, Honghui Dong, and Yong Qin

1 Introduction By the end of 2016, China had built over 20,000 km high-speed railway, far more than the total mileage of the rest countries of the world, and this development attracts worldwide attention [1]. High-speed railway plays a critical role in the progress of China development. But railway accidents happen sometimes because of adverse weather, equipment failure, etc. Ensuring safe and efficient operation has become a primary issue of high-speed railway construction. With the rapid development of machine vision, three-dimensional (3D) measurement technology emerges in the field of object detection. Compared with twodimensional technology, 3D vision obtains the size, shape, volume and other information of an object, and meanwhile overcome the problems of lower image contrast and color interference. Structured light scanning technology is one of the non-contact 3D measurement methods. Based on the principle of geometrical triangulation, the depth information of a point on an object is calculated, and the 3D model of the object can be then obtained. Hence, the 3D technology can reconstruct and measure any threedimensional object. It has been widely used in many fields, such as three-dimensional digital modeling, industrial product design, animation, orthopedic medical, cultural relics protection, archaeological excavation, digital city, and the rail safety inspection [2–6].

T. Yang (B) · Y. Sun School of Computer and Information Technology, Institute of Information Sciences, Beijing Jiaotong University, Beijing 100044, China e-mail: [email protected] X. Cheng · H. Dong · Y. Qin The State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, Beijing 100044, China © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_2

21

22

T. Yang et al.

2 Related Work It is an important safeguard measure for railway safety operation to detect adhered objects on train bogie. Generally, this work is done by track walkers, which is tedious, costly and inefficient. With the development of computer and signal processing technology, new approaches are applied to the fault detection of train bogie. Based on EEMD denoising and manifold learning, Yu et al. [7] extracted the impulse components from the fault signals of a running bogie and identified the fault types. Xu et al. [8] used Hough transform to detect the tread profile in real time, which improves the extraction accuracy and efficiency of contour geometric parameters. Xie [9] proposed a feature extraction and fault diagnosis method based on Spark vibration data. By analyzing the vibration data of the train body, axle box and frame, the fault diagnosis of the train running part was effectively completed. He et al. [10] proposed a fault diagnosis method based on rough set (RS) and least squares support vector machine (LSSVM) for the fault diagnosis of high-speed train running gear rolling bearings. The method has higher accuracy and real-time performance, and provides a new method for fault diagnosis of high-speed train rolling bearings, and the multi-source information fusion can identify the fault of the running gear by assigning different weight information to the data of the sensor. The continuous improvement of data processing technology provides solutions to fault detection in the train bogie. Structured light technology has the characteristics of high precision, high speed and simple structure, and is widely used in various fields. More and more researchers have increasing interest in structural light measurement technology, and promoted scientific research results to industrial production and daily life. Zhang et al. [10] proposed a method based on monocular vision, which uses sequence images acquired by a passive sensor to reconstruct a long-distance scene, and quickly detects and locates obstacles. Liu et al. [11] proposed a method of using multiple lines structured light to determine whether high-speed railway fasteners are missing by extracting the feature quantity from images and deriving the characteristic parameters. The structured light scanning technology can be used to calculate the volume of an object by reconstructing the three-dimensional model of the object, which greatly improves detection accuracy and efficiency. Riccabona et al. [12] used ultrasonic scanning technology to reconstruct the object and the experimental accuracy could be acceptable. Zhang et al. [14] used multi-angle images to get the point cloud model and calculated the volume and surface area by incremental Delaunay triangulation. Zhang [15] proposed a shading method to recover the three-dimensional information of the measured object, and statistically sums the heights of all the pixels to obtain the pixel volume of the irregular object. Zhou et al. [16] established a linear model of the actual surface area and volume of object from the pixel surface area and volume, and used the linear model to predict the egg volume. Mao et al. [17] used binocular stereo vision technology to measure the volume of irregular pyramids. In general, the application of structured light technology in the measurement of irregular object volume provides a new way of measuring the volume of irregular objects and is of great help to improve the accuracy of three-dimensional measurement.

Object Detection on Train Bogies Using Structured Light Scanning

23

3 Methods and Principles 3.1 Line-Plane Model The linear structured light system consists of a linear laser and a camera. The relative positional relationship between the linear laser and the camera can convert the point coordinates in the two-dimensional image into three-dimensional coordinates in the camera coordinate system. The relative positional relationship between the laser and camera is determined by a line-plane model, where the line refers to the linear equation of the line connecting the point on the CCD plane and the optical center, and the plane refers to the plane equation of the sector projected by the laser. The line-plane model is established from the optical plane calibration. The line-plane model is shown in Fig. 1. The laser emits a beam of light onto an object and generates a distorted light stripe. P  is a pixel in image, corresponding to the point P on the distorted light stripe. The camera optical center Oc , and P  and P are on the same line. In the camera coordinate system, Oc is the origin, and pixel P  can be obtained by image processing. The linear equation connecting point Oc and point P  can be expressed as Yc − O y Z c − Oz X c − Ox = = Px − Ox Py − O y Pz − Oz

Fig. 1 The schematic diagram of line-plane model

(1)

24

T. Yang et al.

In the camera coordinate system, the equation of the structured light plane can be expressed as a X c + bYc + cZ c + d = 0

(2)

Using Eqs. (1) and (2) the coordinate of point P in the camera coordinate system can be calculated. Similarly, the coordinates of any point on the light stripe can be calculated, and the two-dimensional coordinates of the light stripe can be further converted into three-dimensional coordinates in the camera coordinate system.

3.2 Laser Triangulation Principle Laser triangulation is the basis of the three-dimensional measurement using linear structured light. To obtain information of object, the camera coordinate system should be transformed to the world coordinate system. The laser triangulation is used to transform the coordinate system of point clouds. The laser triangulation principle is shown in Fig. 2. First the position of the base plane is determined, and the height of the measured object refers to the height relative to the base plane. The laser is projected onto the surface of the object at point H . The actual height of the object measured is h. P N is the mapping of the actual height h on the CCD plane. The angles θ and α can be calculated using the cosine theorem and the known coordinates in the camera coordinate system. In terms of triangle similarity principle, the relation between P N and h is established as

Fig. 2 The triangulation principle

Object Detection on Train Bogies Using Structured Light Scanning

h=

O Q ∗ P N ∗ sin(θ ) Q P ∗ sin(θ ) + P N ∗ sin(α + θ )

25

(3)

where O Q is the distance from the focal point of the CCD optical axis and the laser optical axis to the lens center, Q P is the distance from the center of lens to the reference point, θ is the angle between the laser optical axis and the CCD optical axis, and α is the angle between CCD optical axis and CCD plane.

3.3 Light Stripe Center Extraction In the structured light scanning system, the width of the laser beam in image is about 10 pixels, depending on the laser’s projection distance and the surface characteristics of the object, as shown in Fig. 3. Light stripe center extraction can be divided into stripe extraction based on morphological characteristics and stripe extraction based on gray features [18, 19]. The method of light stripe center extraction based on morphological characteristics includes edge method, geometrical center method, threshold method, etc. Light stripe center extraction based on gray features includes gray centroid method and Hessian matrix method. The ideal structured light stripe is a Gaussian distribution, and the actual light stripe distribution is a Gaussian-like distribution shown in Fig. 4. The truncated Gaussian normal distribution based on curve fitting method is used here to fit the stripe and extract the center of the stripe. Here assuming X is a normal distribution, as x ∈ (a, b), the density of probability function X is expressed by f (x; μ, σ, a, b) =

1 φ( x−μ ) σ σ b−μ ( σ ) − ( a−μ ) σ

(4)

When x ∈ (−∞, a) ∪ (b, ∞), the probability density function f = 0. The mathematical expectation of truncated normal distribution is given below

Fig. 3 The laser light stripe width

26

T. Yang et al.

Fig. 4 a Standard normal distribution. b Actual gray-scale distribution

+∞ +∞ EX = x f (x)d x = (x − μ + μ) f (x)d x −∞

−∞

b =μ+ a

(x−μ)2 1  √ (x − μ)e− 2σ 2 d(x − μ) ( b−μ ) − ( a−μ ) 2π σ σ σ

= μ + σ 2 [ f (a) − f (b)]

(5)

If a and b are symmetric about μ, then f (a) = f (b), E X = μ. The center of light stripe is the symmetric center of truncated Gauss distribution.

3.4 ICP Algorithm The linear structured light system reconstructs a three-dimensional point cloud of object by collecting and analyzing the light stripe images. Because the threedimensional point cloud is scattered data, it is very difficult for a single point cloud to detect adhered substances on train bogie. While through reconstructing the point cloud twice, the difference of the two reconstructed point clouds can be then used to measure and detect the objects on the train bogie. To compute the difference of the two clouds, we have to register the point clouds first. The iterative closest point (ICP) algorithm is frequently used to achieve this goal. The ICP is an optimal registration algorithm based on the least square method. By calculating the corresponding points of a point cloud to be registered and a target point cloud. The optimal rotation parameters and translation parameters are found till the point clouds are converged. Here, the point clouds P and Q are obtained from the system. pi ∈ P, qi ∈ Q. The Euclidean distance between pi = (xi , yi , z i ) and  q j = x j , y j , z j can be expressed as

Object Detection on Train Bogies Using Structured Light Scanning

   d( pi , q j ) =  pi − q j  = (xi − x j )2 + (yi − y j )2 + (z i − z j )2

27

(6)

The registration of P and Q can be expressed as qi = Rp j + T

(7)

where R is the rotation matrix and the T is the translation matrix. The minimized objective function is expressed as N   q j − Rpi − T 2

E=

(8)

i=1, j=1

The optimal solution of R and T is obtained by singular value decomposition (SVD) algorithm.

4 Experimental Results 4.1 Experimental Setup The schematic diagram of the experimental system is shown in Fig. 5. It consists of two sets of photoelectric switches, a structured light scanning module and a miniaturized train bogie module. The photoelectric switches are placed on the left and right sides of the scanning module. The left switch starts a scanning, and the right one ends

Fig. 5 The schematic diagram of an experimental setup

28

T. Yang et al.

the scanning. Each switch has a laser emitter and a receiver. The signals from the switches are used to start or end a scanning. The structured light module is located at the rail track side, about 1 m away from the track. The images for the camera of the scanning module are recorded by a computer. The whole working process can be described below. As the train moves to the start point, the left switch triggers the structured light module to start scanning the train bogie data. As it reaches the stop point, the right switch will trigger the structured light module to end the scanning. The images are transmitted to data processing module for 3D reconstruction of the train bogie.

4.2 Three-Dimensional Reconstruction Using the pixels on the light stripes, the calibrated camera model and the triangulation principle, the depth of a point on the bogie can be computed with (3). When the depth information is obtained for all the pixels of the laser stripe, a 3D point cloud is generated. But there have a large number of background points in the reconstructed point cloud. These points are from the base plane, and the points behind the base plane are normalized to be the background points. They seriously decrease the speed of the point cloud calculation. A threshold method is used to remove the background points, as shown in Fig. 6. To improve the visualization of the cloud points, a greedy projection triangulation algorithm is used to project the three-dimensional points onto a plane through a normal line. The point cloud obtained by the projection is then triangulated, and the connection relationship of each point is calculated. The 3D reconstruction result is shown in Fig. 7.

Fig. 6 The background point removal

Object Detection on Train Bogies Using Structured Light Scanning

29

Fig. 7 The visualization result with greedy projection triangulation algorithm

4.3 Volume Measurement The volume measurement of the adhered substances on the train bogie is very important to eliminate the contigent risk of rail safety. Hence, a volume measurement approach is proposed here based on the numerical integration of a point cloud difference. The difference is calculated from two sets of point clouds, namely a scanning point cloud and an initial point cloud. The initial point cloud is obtained when the train bogie has no adhered substances. The numbers of the two sets of point clouds are different, and established at different time periods. Moreover, due to the delay to start the acquisition and the variation of the train running speed, the point clouds are offset. The two clouds are needed to set in the same coordinate system. Hence, the ICP algorithm introduced previously is used to registered them. To speed up the computation, the subsampled method is used to reduce the number of point cloud data, and the background points are removed as well. The registration results of two sets of point clouds are shown in Fig. 8. The left side is the point clouds to be

Fig. 8 The registration result of two sets of point clouds with ICP algorithm

30

T. Yang et al.

Fig. 9 The point cloud difference without and with filtering

registered, and the right side is the registered point clouds. Because of the discreteness characteristics of point cloud, there are many redundant noise points in the point cloud difference, as shown in Fig. 9a, and the noise points may decrease the accuracy of the volume computation of adhered substances. To remove the noise points, a threshold in the z axis is used to filter the redundant noise points. The point difference after filtering is shown in Fig. 9b. Once the redundant points are filtered, we can see that the adhered substance can be detected and figured out. Subsequently, the volume of differential point cloud is able to calculate by integration, too.

5 Conclusion An approach to detect adhered substances on train bogie is proposed using linear structured light scanning technology, and the volume of the substances is computed with numerical integration. The principle of structured light scanning is first introduced, and the extraction method of light stripe center is presented based on Gaussian distribution. To obtain the difference of a scanning point cloud and the initial background point cloud, the scanning point cloud is registered with the ICP algorithm to align in the same coordinate system of the background point cloud. A train bogie setup is used to validate the proposed algorithms, and experimental results show that the structured light scanning method can reconstruct the 3D model of the bogie, and the adhered substances on train bogie is able to be detected and the volume of the substances can be computed accurately with numerical integration.

Object Detection on Train Bogies Using Structured Light Scanning

31

References 1. Gao, W., & Zhang, X. (2016). High-rail current situation of China and future prospect research. Shanxi Architecture, 42(32), 172–173. 2. Zhang, L-b, Wang, P-j, Zhang, X., & Li, W-t. (2016). Research on 3-D structured light rail detection of high-speed railway and viewpoint optimization. Machinery Design & Manufacture, 4, 69–72. 3. Wang, J., Xu, Y-j, Wang, L., & Wang, P. (2014). The application research of machine vision in rail wear detection. Railway Standard Design, 9, 36–39. 4. Zhan, D., Yu, L., Xiao, J., et al. (2015). Multi-camera and structured-light vision system (MSVS) for dynamic high-accuracy 3D measurements of railway tunnels. Sensors, 15(4), 8664–8684. 5. Liu, Z., Sun, J., Wang, H., et al. (2011). Simple and fast rail wear measurement method based on structured light. Optics and Lasers in Engineering, 49(11), 1343–1351. 6. Ping, L. I., Wang, P. J., Chen, P., et al. (2018). Rail corrugation detection based on 3D structured light and wavelet analysis. Railway Standard Design, 62(4), 33–38. 7. Yu, P., Jin, W., & Qin, N. (2016). High-speed train running gear fault feature extraction based on EEMD denoising and manifold learning. Journal of the China Railway Society, 38(4), 16–21. 8. Xu, Z., & Chen, J. (2017). Tread profile of wheel detection method based on image processing and Hough transform. Electronic Measurement Technology, 6, 117–121. 9. Xie, J. (2015). Research on fault diagnosis of high-speed running gear based on deep learning under cloud platform. Southwest Jiaotong University. 10. He, D. Q., Chen, E. H., Li, X. M., et al. (2017). Research on fault diagnosis method of high-speed train running gear rolling bearing based on RS and LSSVM. Journal of Guangxi University, 42(2), 403–408. 11. Zhang, D.-z., Wang, Y.-t., Tian, J.-w., Wang, C.-q., Guo, Q. (2008). Efficient 3D reconstruction using monocular vision. Journal of Astronautics, 29(1), 295–300. 12. Liu, H., Qian, G., Zhang, H., Tao, W., Zhao, H., Wang, W., et al. (2011). High-speed railway fastener detection method based on structured-light. Measurement Technique, 5, 3–7. 13. Riccabona, M., et al. (1995). Distance and volume measurement using three-dimensional ultrasonography. Journal of Ultrasound in Medicine Official Journal of the American Institute of Ultrasound in Medicine, 14(12), 881–886. 14. Zhang, W., et al. (2016). A novel method for measuring the volume and surface area of egg. Journal of Food Engineering, 170, 160–169. 15. Zhang, N. (2015). Study on the volume measurement method of irregular object based on computer vision. Shaanxi University of Science and Technology. 16. Zhou, P., Zheng, W., Zhao, C., et al. (2008). Egg volume and surface area calculation based on machine vision. Computer and Computing Technologies in Agriculture II, 3, 1647–1653. 17. Mao, J., Lou, X., Weixian, L. I., et al. (2016). Binocular 3D volume measurement system based on line-structured light. Optical Technique, 42(1). 18. Wu, J. Y., Wang, P. J., et al. (2009). Method of linear structured light sub-pixel center position extracting based on gradient barycenter. Journal of Image & Graphics, 14(7), 1354–1360. 19. Gao, S., & Yang, K. (2011). Research on central position extraction of laser strip based on varied-boundary Gaussian fitting. Chinese Journal of Scientific Instrument, 32(5), 1132–1137.

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore Powder Based on Machine Vision Shanling Ji, Zhisheng Zhang, Zhijie Xia, and Ying Zhu

1 Introduction Ganoderma lucidum spore powder (GLSP)is an extremely tiny spore of Ganoderma lucidum that emerges from the pleats during the growth stage, and has all the genetically active substances of Ganoderma lucidum. Broken GLSP can be obtained by breaking the chitin shell of GLSP by bio-enzymatic method, physical method or chemical method. Some experiments have demonstrated that the medicinal effect of broken Ganoderma lucidum spores as a traditional Chinese medicine is better than that of unbroken Ganoderma lucidum spore [1]. Therefore, detecting the breaking rate of GLSP is an important link in the production process. Presently, microscopic detection and chemical fingerprinting are common methods for detecting the breaking rate of GLSP [2]. Although the two methods have high accuracy, they require more manual operational experience and the operation process is complicated. Chen et al. proposed the use of FTIR micro-spectroscopy to identify unbroken spores [3]. But this method was not applied to the calculation of the rate of cell wall breakage of Ganoderma lucidum spores. With the study of deep convolutional networks, people have made great progress in image recognition. However, directly using deep learning for image recognition places great demands on the amount of image data. And it is inevitable that data enhancement and other operations are required to obtain more trained images. Therefore, microscopic images of Ganoderma lucidum spore powder as shown in Fig. 1 are difficult to be directly recognized using deep learning with small quantities of images. This paper proposes a method based on machine vision to detect the breaking rate of Ganoderma lucidum spore powder, which is mainly to detect the number of intact spores from the microscopic image like Fig. 1 and then calculate the breaking rate. S. Ji · Z. Zhang (B) · Z. Xia · Y. Zhu School of Mechanical Engineering, Southeast University, Nanjing 211189, China e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_3

33

34

S. Ji et al.

Fig. 1 Microscopic image of broken Ganoderma spore powder

The detection step is first to filter the image bilaterally, and then extract each ROI region by using local threshold processing and connected domain extraction. For areas that are suspected of overlapping, segmentation is performed using distance conversion and Gaussian mixture clustering. Each of the divided sub-pictures is square, and then reset to the same large square. The features were extracted using AlexNet [4]. Finally, the support vector machine was used to classify and identify the same size square image, and the number of unbroken spores in the original image was counted to calculate the breaking rate. The details of the method based on machine vision for unbroken spore identification are given in Sect. 2. Section 3 examines the effectiveness of the local range threshold processing method and the breaking rate calculation platform proposed in this paper through experiments, and discusses the experimental results. Finally, the paper summarizes the full text and elaborates on the future work.

2 Methodology 2.1 Image Acquisition and Preprocessing The image acquisition system consists of an optical microscope, an electronic eyepiece, and a PC. The PC side collects a 500-fold magnified microscopic image with a resolution of 4076 × 3116. The on-site image acquisition equipment is shown in Fig. 2. Image bilateral filtering is a nonlinear filtering method [5]. The grayscale image is denoised by bilateral filtering, Gaussian filtering and mean filtering as shown in the Fig. 3. Compared to other filtering, bilateral filtering can well preserve the details of

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore …

35

Fig. 2 The on-site image acquisition equipment

Fig. 3 a Is grayscale image; b–d are respectively filtered by Gaussian filtering, mean filtering and bilateral filtering

the edges of the image and filter out the spatial noise. Therefore, the grayscale image of the GLSP microscopic image in this paper will be preprocessed using bilateral filtering.

2.2 Threshold Processing Based on Gray Level Difference of Image Local As can be seen from Fig. 3, the gray value changes of the background and foreground are sharp, and this feature can be well represented by the local range threshold processing. For each pixel point (xi , yi ), with r as the radius, the gray level variation S in the case of eight connections is obtained. If S is smaller than the threshold L,

36

S. Ji et al.

the pixel is set as the background (the gray value is 0). Otherwise, it is the foreground (the gray value is 1). The formula is expressed as (1) and (2). S = max(Sr (xi , yi )) − min(Sr (xi , yi ))  BW(xi , yi ) =

0, S < L 1, S ≥ L

(1)

(2)

Sr represents the gray value of all the pixels with the (xi , yi ) point as the core and radius r , and BW(xi , yi ) is the value of the binary image at the (xi , yi ) point. In order to make the image adaptive to get the appropriate L value, assign L as follow: L = 0.1 ∗ Sim

(3)

Sim is the range of gray values of the entire image. Using the local range threshold processing not only can better extract the foreground area, but also avoid the influence of uneven illumination. The comparison between the method used in this paper and other threshold processing methods is described in Sect. 3.1.

2.3 Split the Foreground and Extract the ROI Area In this paper, the ROI region is extracted first, and then the image classification method is used to identify the intact spores in the image. The image processed by the threshold in the previous step, after a simple morphological processing, can extract the connected domain well. The decision whether or not to further split is then determined by the area of the connected domain. The area calculated for each connected domain is the number of pixels. If the area is too large, it will be set to a suspected overlapping area and needs to be further divided. As shown in Fig. 4, the connected domain part identified by the binary map, for the non-overlapping part, directly settles its coordinate range, and uses a square to take a screenshot of the original image of the area. For overlapping areas, use the minimum box of the connected area to frame it and identify the area of the box in the original image, waiting for further segmentation. Various screenshots are shown in Fig. 5. In this paper, the method of segmenting the elliptical shape model touch unit by Winter et al. [6] is employed. Firstly, the distance conversion value is calculated for the binary map of the overlap region, and then the pixel is copied according to the distance conversion value of the pixel point, and finally the Gaussian mixture model is used for fitting and clustering. As shown in the Fig. 6, the area where the two spores overlap, the number of overlapping spores is obtained by calculating the area of the connected domain of the binary map, and then the cluster is fitted to the region, and finally the spores are separated as shown in the figure. For each of these areas, calculate the centroid, horizontal distance, and vertical distance of the coordinates,

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore …

37

Fig. 4 The left graph is a binary graph after the local extreme difference threshold processing, and the right graph is the connected domain extracted from the binary graph and marked in the grayscale graph

Fig. 5 The initially intercepted ROI region contains a single unbroken spore (a), a single broken spore (b), a small region of oil (c) and a suspected overlapping region (d) and (e)

Fig. 6 Segmentation process of two overlapping unbroken spores

and then use the two smaller values and the centroid as the side length and centroid of the subsequent screenshot. As shown in Fig. 7, there are a plurality of unbroken spore overlapping regions and suspected overlapping regions of broken spores. Since the calculated connected domain area exceeds the single spore area, the segmentation

38

S. Ji et al.

Fig. 7 Split overlapping area

is performed separately. Although the method of [6] was originally used to segment overlapping elliptical cells, it performed well in this paper.

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore …

ROI

Resize

227x227x3

AlexNet Feature

SVM

39

Marked in the original image

Fig. 8 Identifying unbroken spores

2.4 Classification of Intercepted ROI Areas After the previous steps, the spore image collected by the microscopy device has been divided into square images containing intact spore granules, broken spores and oil. Now we need to classify and identify these images and calculate the number of intact spores (Fig. 8). AlexNet [4] is a convolutional network proposed by Alex and Hinton in the ILSVRC2012 competition. It improves CNN and becomes the general structure of the CNN network. AlexNet is a deep convolutional network model with eight layers (5 layers of convolutional layers and 3 layers of fully connected layers) and the last layer for classification. This article uses 4096 data output from AlexNet Layer 7 as the extracted feature. Support vector machines show many unique advantages in solving small sample, nonlinear and high-dimensional pattern recognition, and can be applied to other machine learning problems such as function fitting. This paper uses AlexNet to extract the features of images and uses multiple types of support vector machines for classification.

2.5 Calculate the Breaking Rate For the same batch of GLSP, the same amount of spore powder was taken before and after processing, and an evenly distributed suspension was prepared using an equal amount of reagent. An equal amount of suspension was taken and images were taken with a 500-fold microscope, and the number of intact spores in the same field of view was separately counted. The number of intact spores sampled for the unprocessed spore powder was Na , and the number of intact spores after processing was Nb . The calculation formula of the breaking rate is (4). R =1−

Nb Na

(4)

40

S. Ji et al.

3 Experiment and Discussion 3.1 Comparison of Various Image Threshold Processing Methods Figure 9 showed the processing results of grayscale images processed by OTSU, maximum entropy, block OTSU, and local range threshold. It could be seen from the results that the proposed method performs better than OTSU, maximum entropy and block OTSU in the case of uneven illumination of the image. More importantly, the location of the foreground area extracted by the method of this paper was more accurate.

3.2 Test Results of the Breaking Rate Calculation Platform As shown in Fig. 10, the built-in breaking rate calculation platform could select functions such as segmenting the image, training the classifier or calculating the breaking rate. Table 1 showed the number of intact spores identified by the artificial identification and the platform of this paper, and the breaking rate in both cases was calculated using Eq. (4) and was shown in Table 2. According to the field conditions, a total of four batches were tested, and the accuracy of the wall breaking rate calculated by the platform of this paper was calculated according to the formula (5). R1 and R2 represented the calculated wall breaking rate and the wall breaking rate calculated by the platform, respectively. Accuracy = 1 −

|R1 − R2 | R1

(5)

The results of partial platform tests are given in Fig. 11. Combined with the chart, it can be concluded that the accuracy of the method given in this paper is at least 95%, which meets the requirements.

4 Conclusion and Future Work Based on machine vision, a platform was established for identifying intact spores from microscopic images and calculating the rate of broken wall. It can improve the detection efficiency of Ganoderma spore powder breaking rate and provided a new method for the detection of spore powder breaking and similar microscopic drugs. The method of segmentation recognition by extracting connected-domain images was established in this paper. Under the condition of collecting micro-image data, the average recognition rate can reach 95%. The proposed local range threshold

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore …

oral image

OTSU

max-entropy

block OTSU

proposed method

Fig. 9 Comparison of various image threshold processing methods

41

42

S. Ji et al.

Fig. 10 GLSP breaking rate calculation platform Table 1 Number of intact spores under different conditions Unprocessed GLSP

Processed GLSP

Batch

Artificial count

Platform count

Accuracy

1

20

20

1.0000

2

42

41

0.9762

3

19

20

0.9474

4

19

18

0.9474

1

4

4

1.0000

2

2

2

1.0000

3

2

2

1.0000

4

4

4

1.0000

Table 2 Platform breaking rate calculation accuracy

Batch

R1

R2

Accuracy

1

0.8000

0.8000

1.0000

2

0.9524

0.9512

0.9987

3

0.8947

0.9000

0.9941

4

0.7895

0.7778

0.9582

A Method for Detecting Breaking Rate of Ganoderma Lucidum Spore …

43

Fig. 11 Partial test results

processing method performed well in the case of uneven illumination. The method proposed in this paper provided a method to detect the wall breaking rate of Ganoderma lucidum spore powder in breaking machine. Acknowledgements This research work is supported by the National Natural Science Foundation of China (Grant Nos. 51775108).

References 1. Wang, W., Chen, G., Su, J., Lv, G., & Chen, S. (2017). Comparative study of tumor growth and VEGF expression in mice with Lewis lung cancer by Ganoderma lucidum spore powder and broken Ganoderma lucidum spore powder. Pharmacology and Clinics of Chinese Medicine, 33(02), 118–122. 2. Zhao, J. S., Wei, L., Yan, Y., et al. (2013). Research progress on the test method of the breaking rate of Ganoderma lucidum spore powder. Chinese Medicine Guide, 5, 431–434. 3. Chen, X., Liu, X., Sheng, D., Huang, D., Li, W., & Wang, X. (2012). Distinction of broken cellular wall Ganoderma lucidum spores and G. lucidum spores using FTIR microspectroscopy. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, 97, 667–672.

44

S. Ji et al.

4. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2, 1097–1105. 5. Chaudhury, K. N., Sage, D., & Unser, M. (2011). Fast $O(1)$ bilateral filtering using trigonometric range kernels. IEEE Transactions on Image Processing, 12, 3376. 6. Winter, M., Mankowski, W., Wait, E., et al. (2019). Separating Touching Cells Using Pixel Replicated Elliptical Shape Models. IEEE Transactions on Medical Imaging, 4, 883.

6D Pose Estimation of Texture-Less Object in RGB-D Images Heng Zhao, Chungang Zhuang, Lei Jia, and Han Ding

1 Introduction Texture-less objects are common in industrial environments and estimating their 6D poses (three in translation and three in rotation) has a wide range of applications in robotic tasks. For example, it is important for robot bin picking to recognize 6D pose of objects. However, 6D pose estimation still struggles to achieve fast and reliable results in real-world scenes. The real-world scenes are usually filled with objects of different shape and their appearances on images are easily affected by illumination, clutter and occlusions between objects. Traditionally, the problem of 6D pose estimation is tackled by matching local features like SIFT [1] and ORB [2] extracted from an RGB image to features in a 3D model of the object [3]. However, the stability of their features depends on the rich texture of the object. The lack of texture implies that the 6D object pose cannot be reliably recognized with these methods. In recent years, there are some works [4–6] which apply deep learning for 6D object pose estimation. These methods are not end-to-end or only estimate an approximate pose. They require further refinements to improve the accuracy. The classic refinement method is to handle it by using ICP algorithm [7], which linearly increases the running time.

H. Zhao · C. Zhuang (B) · L. Jia · H. Ding School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China e-mail: [email protected] H. Zhao e-mail: [email protected] L. Jia e-mail: [email protected] H. Ding e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_4

45

46

H. Zhao et al.

Fig. 1 Examples of real-world scene images (up) overlaid with colored 3D models at the estimated 6D poses (down)

In this paper, we propose an end-to-end deep neural network for 6D texture-less object pose estimation from RGB-D images. The network consists of two subnetworks. The pose estimation subnetwork takes data combined with RGB values and depth values at pixel level as input and predicts an approximated pose. An estimated point cloud is calculated by rendering an object based on the approximated pose. The estimated point clouds, together with the initial point clouds obtained from depth image, are fed into a pose refinement subnetwork to refine the pose. By iteratively re-rendering the object based on the improved pose, the two input point clouds to the refinement subnetwork become more and more similar, thereby enabling the subnetwork to generate more and more accurate pose estimates. Some examples of 6D pose estimation of texture-less objects are shown in Fig. 1. In order to reduce the cost of collecting datasets, a physically-simulated environment is constructed to generate dataset. The LINEMOD [8] and self-made industrial parts dataset are used to evaluate the proposed method. The experimental results show that the proposed method is competitive and outperforms the state-of-the-art methods on the LINEMOD dataset in some respects. Last, we also show its utility on the self-made industrial parts dataset. The main contributions of this work are: (1) an end-to-end approach for 6D textureless object pose estimation from RGB-D images. (2) a pose estimation subnetwork taking the data combined with RGB values and depth values at pixel level as input. (3) a pose refinement subnetwork taking point clouds as input. In the following sections, we discuss the related work in Sect. 2, describe our method in Sect. 3, present experimental results in Sect. 4. Conclusions are given in Sect. 5.

6D Pose Estimation of Texture-Less Object in RGB-D Images

47

2 Related Work RGB methods. Classical methods of 6D pose estimation using RGB images mostly rely on matching features extracted from an RGB image to features in a 3D model of the object [3]. The performance of these methods depends on the rich texture of the object. Recently, researchers focus on the texture-less objects based on the deep learning method. In PoseCNN [6], Xiang et al. proposed a network which estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. In SSD-6D [4], Kehl et al. extended the popular SSD [9] paradigm to cover the full 6D pose space. They decompose 3D rotation space into discrete viewpoints and in-plane rotation and treat the rotation estimation as a classification problem. In BB8 [5], Rad et al. applied to the detected objects a Convolutional Neural Network (CNN) trained to predict their 3D poses in the form of 2D projections of the corners of their 3D bounding boxes. In fact, the RGB image contains color and texture information of an object, while the depth image contains the geometry information. Due to the lack of geometry information, the performance of these methods using RGB image only is still not comparable to RGB-D based methods. RGB-D methods. With the advent of commodity depth cameras, researches focus on pose estimation using the RGB-D images [10–13]. The most traditional approaches are to use template matching [8, 14, 15]. LINEMOD [8] is the most notable work belonging to this category. The authors build the templates by rendering views of 3D models and embedding quantized color gradient and normal features. Rios et al. [14] proposed to learn the templates in a discriminative fashion and cascaded detections for higher accuracy and efficiency respectively. Recently, deep learning based RGB-D methods are used on object recognition and pose estimation. Kehl et al. [13] employed a convolutional auto-encoder for building regressed descriptors of locally-sampled RGB-D patches for 6D vote casting. Li et al. [16] presented a framework for accurately inferring 6D object pose from single or multiple views by integrate three new capabilities into a deep CNN: an inference scheme, the fusion of class priors and the fusion of class priors. Wang et al. [17] presented DenseFusion, a generic framework for estimating 6D object pose from RGB-D images. The core of their approach is to embed and fuse RGB values and point clouds at per-pixel level. The method most relevant to the proposed method is DeepIM [18], in which the network of only RGB image as input is able to iteratively refine the pose by matching the rendered image against the observed image. In order to eliminate the dependence on color and texture, our pose refinement subnetwork refines pose by matching rendered point clouds instead of RGB image. Finally, we show its utility on the LINEMOD and self-made industrial parts dataset.

48

H. Zhao et al.

3 Methodology 3.1 Architecture Overview The goal of 6D object pose estimation is to obtain the rigid transformation from the object coordinate system to the camera coordinate system. Given an RGB-D image, we design the network to directly output a relative transformation that consists of a 3D rotation R and a 3D translation T . Since we estimate the 6D pose of the objects from camera images, the poses are defined with respect to the camera coordinate system. Figure 2 illustrates the framework of 6D texture-less object pose estimation. We first use a segmentation network to generate object instance masks. The masks of RGB-D image are then passed individually through the pose estimation subnetwork, which outputs a 6D approximated pose for each object. We assume that the 3D model of the object is available. An estimated point cloud is calculated by rendering the 3D model of the object based on the approximated pose. The estimated point clouds, together with the initial point clouds obtained from depth image, are fed into a pose refinement subnetwork to refine the pose.

3.2 Instance Segmentation Instance segmentation focuses on detecting the bounding box location and object category as well as predicting the predefined category for each pixel in the bounding

Fig. 2 An overview of our 6D pose estimation framework

6D Pose Estimation of Texture-Less Object in RGB-D Images

49

box from an RGB image. The focus of this work is to develop a pose estimation network. Thus we use an existing instance segmentation architecture YOLACT [19]. The network takes in RGB images and outputs segmentation labels, which are converted into bounding box and binary instance masks with associated object classes and fed into the subsequent pose estimation network.

3.3 Pose Estimation Subnetwork The goal of pose estimation subnetwork is to predict an approximated pose. The   pose is represented by its position p = (x, y, z) and orientation q = q0 , q1 , q2 , q3 , which are translations and rotations relative to the camera coordinate frame. The subnetwork takes RGB and depth images patch cropped by the bounding box as input. The first step is to extract correct information from the color and depth channels. Feature Extraction. Inspired by the DenseFusion [17], separately to generate color and geometric features from RGB image and 3D point clouds. We extract color and geometric features from RGB and depth images. In order to make color and depth present a similar format, normalization processing is carried out. The subnetwork follows a simple CNN architecture consisting of a ResNet-18 [20] feature extractor followed by a multilayer perceptron. Feature Fusion. After feature extraction, per-pixel feature maps from RGB and depth images separately are generated. In order to minimize the effects of occlusion and segmentation noise, the feature maps are sampled at per-pixel level according to binary instance mask generated by instance segmentation stage. Next, we need to combine the features. Figure 3 illustrates the structure of feature fusion network. The network takes two sampled feature maps with channel dimension 32 generated by the feature extraction stage as inputs. The multilayer perceptron is used to regress position and quaternion values. Inspired by the PointNet [21], local and global information aggregation is carried out. The local color and depth features are concatenated in the layers of dimension 32, 64 and 128. After computing the global feature, we feed it back to per-pixel local features by concatenating the global feature with each of the local features. Then we extract new per-pixel features based on the combined the features. Now, the per-pixel feature is aware of both the local and global information. The per-pixel features are fed into a final network to predict the 6D object pose. Loss Function. The loss function is defined as the distance between the target point clouds and the point clouds transformed by the predicted the ground   pose.Given    truth pose Ttar = [R|t ] and the estimated pose set Tˆ = pˆ i  pˆ i = Rˆ i ˆti , the loss is computed as: Le =

N

 1



(Rxi + t) − Ri xi + ti

N i=0



(1)

50

H. Zhao et al.

Fig. 3 The architecture of pose estimation subnetwork

where N denotes the total number of per-pixel features and xi is 3D point on the object model at the corresponding pixel location. Pose Clustering. We obtain the 6D object pose for each per-pixel features from above section. Now, we need to choose a best pose as the estimated final pose and feed it into the subsequent pose refinement network. Inspired by the PPF [22]. First, a new cluster is created with the highest vote pose hypothesis. Then, similar poses are grouped together. If a pose is obviously different from the existing clusters, a new cluster will be created. The score of a cluster is the sum of the number of all contained poses. Finally, the pose with highest score will be return.

3.4 Pose Refinement Subnetwork So far we have obtained an approximated pose T 0 using pose estimation subnetwork. The goal of pose refinement network is to predict more accurate 6D pose based on T 0. Rendering Point Clouds. Inspired by the DeepIM [18], in which the authors render 3D model of the object to obtain RGB image. However, this method requires that the color and texture of 3D model is as same as possible to real-world scenes. We assume that the 3D model of the object and camera-intrinsic parameters are available.

6D Pose Estimation of Texture-Less Object in RGB-D Images

51

In order to eliminate dependence on color and texture, an estimated 3D point cloud is obtained by rendering the 3D model of the object under 6D pose T 0 . Network Structure. Figure 4 illustrates the structure of the subnetwork. The subnetwork takes the estimated point clouds and the initial point clouds obtained from depth image as input and predicts T between the current approximated pose and the target pose. Inspired by the PointNet [21].The T has to be invariant if the pose of the two point clouds is changed together. Therefore an affine transformation matrix is predicted by a mini-network and directly apply this transformation to the coordinates of the two input point clouds. The mini-network is composed by basic modules of point independent feature extraction, average pooling and fully connected layers. Unlike pixel arrays in images or voxel arrays in volumetric grids, point cloud is a set of points without specific order. Therefore the average pooling layer as a symmetric function to aggregate information from the two point clouds. Finally, several linear layers is used to predict t and R between the current approximated pose and the target pose. Deserved to be mentioned, the pose refinement subnetwork predicts a pose T for all point cloud instead of poses for every points. This procedure can be applied iteratively and generate potentially finer pose estimation each iteration. Loss Function. The loss function is define as the distance the target   point clouds ˆ and the current estimated point clouds. The estimated pose T = Rˆ tˆ is obtained from the concatenation of the per-iteration estimations:             Tˆ =  Rˆ M−1 tˆM−1 ·  Rˆ M−2 tˆM−2 · · ·  Rˆ 0 tˆ0 · Rˆ 0 tˆ0

Fig. 4 The architecture of pose refinement subnetwork

(2)

52

H. Zhao et al.

   where M is iterations, Rˆ 0 tˆ0 is the result of pose clustering in the part of pose estimation subnetwork. Given the ground truth pose Ttar = [R|t ], the loss is computed as: Lr =

N 

1

ˆ i + tˆ

(Rxi + t) − Rx

N i=0

(3)

where N denotes the total number of per-pixel features and xi is 3D point on the object model at the corresponding pixel location.

4 Experiments In this section, the experiments are discussed as below. We test our approach on two datasets, the LINEMOD dataset and self-made industrial parts dataset. The LINEMOD dataset is a widely-used dataset that allows to compare with a broader range of the existing methods. In order to prove the effectiveness of our method and reduce the cost of collecting datasets in real-world scenes, a physically-simulated environment is constructed to generate industrial parts dataset. The networks are trained on this physically-simulated dataset and are tested on the images of real-world scenes. The proposed networks are implemented by PyTorch deep learning library. The test images for industrial parts are captured from Intel RealSense SR300 with size 480 × 640. The networks are trained on a NVIDIA GeForce GTX 1080Ti. Each mini-batch has one image. The learning rate is set to 0.001 for first 5 k iterations. After 5 k iterations, the learning rate is set to 0.0001. The self-made industrial parts dataset for training and validation is set to 9:1.

4.1 Datasets LINEMOD [8] has become a de facto standard benchmark for 6D pose estimation of texture-less objects in cluttered scenes. It contains 13 texture-less objects with discriminative color, shape and size. Each object is associated with a train and test image set showing one annotated object instance with significant clutter but only mild occlusion. A full 3D model representing the object is also provided. We compare our method with existing methods on this dataset. Self-made Industrial Parts Dataset is generated using physically-simulated engine in order to reduce the cost of constructing dataset in real application. The two industrial parts are shown in Fig. 5. The physically-simulated environment is constructed on the basis of the Blender [23] python API. The objects in the workspace are randomly pushed into the simulation environment. By using mesh based collision

6D Pose Estimation of Texture-Less Object in RGB-D Images

53

Fig. 5 a The 3D models of two industrial parts. b The real-world image of two industrial parts. c Generated RGB image using the physically-simulated environment. d Generated depth image correspond with the RGB image

detection technique, the simulated pose distribution of objects is similar to the realworld scene. The various rendered images are obtained when the camera and light are variably configured in the physically-simulated environment. The physicallysimulated dataset is used for training networks only. We also showcase its utility on the images of real-world scenes using the trained networks.

4.2 Evaluation Metrics The average 3D distance of model points (referred to as ADD metric) [8] is used for performance evaluation. The metric computes the average distance between the 3D model points transformed using the estimated pose and the ground truth pose. For symmetric objects, we use the closest point distance in computing the average distance. An estimated pose is correct if the average distance is within 10% of the 3D model diameter. For most objects, this is approximately a 20 mm threshold but for smaller objects, the threshold drops to about 10 mm.

4.3 Evaluation on LINEMOD Dataset The comparison between our method and other methods is shown in Table 1. The bold font means the highest recognition rate for each sequence. It is worth mentioning that these methods is based on deep learning. As for time efficiency, Wadim et al. [4] and Mahdi et al. [5] require a further pose refinement step for improved accuracy. The classic refinement method is to handle it by using ICP algorithm [7], which linearly increases the running time. Our method needs around 100 ms to complete the whole pipeline of 6D pose estimation for objects on this dataset. It suggests that the proposed method is characterized by low computation cost. We render the original image with 3D model at the estimated 6D pose. The visualization results are shown in Fig. 6.

54

H. Zhao et al.

Table 1 Comparison with other methods on the LINEMOD dataset Wadim et al. [4] (%)

Mahdi et al. [5] (%)

Ape

76.3

96.6

Wang et al. [17] (%)

B.Vise

97.1

90.1

93.2

96.1

Camera

92.2

86.0

94.4

97.1

Can

93.1

91.2

93.1

90.0

Cat

89.3

98.8

96.5

94.0

Driller

97.8

80.9

87.0

93.0

Duck

80.0

92.2

92.3

96.2

Egg.B

93.6

91.0

99.8

100.0

Glue

76.3

92.3

100.0

100.0

Hole.P

71.6

95.3

92.1

90.5

Iron

98.2

84.8

97.0

97.9

Lamp

93.0

75.8

95.3

95.2

Phone

92.4

85.3

92.8

96.2

Average

88.5

89.3

94.3

95.1

92.3

Ours (%) 89.4

Fig. 6 Proposed pose estimation method on LINEMOD dataset. The first and third column: original images. The second and fourth column: images overlaid with colored 3D models at the estimated 6D poses

6D Pose Estimation of Texture-Less Object in RGB-D Images

55

4.4 Evaluation on Self-made Industrial Parts Dataset In order to show the utility of the proposed approach, the experimental results on industrial object are also presented. The industrial parts dataset is generated using physically-simulated engine. There are about 1 k images for the two objects. Each object features in about 3 k instances. After the dataset generation and network training, the networks on the dataset and real-world scene images are simultaneously evaluated. In Table 2, we show the performance of our network after training and evaluating on the self-made industrial parts dataset. There are about 700 images for each object. The metric computes the average distance between the 3D model points transformed using the estimated pose and the ground truth pose. The threshold is set by 10% of the 3D model diameter. The visualization evaluation results on the self-made industrial parts dataset are shown in Fig. 7. The evaluation results on real-world scene images are shown in Fig. 8. We overlie the original images with colored 3D models at the estimated 6D poses. For better visibility, the background is kept in gray. The first column shows several original test images. The second column shows the results of pose estimation subnetwork. The third column shows the results of pose refinement subnetwork. The last column shows the comparison results. The red and green lines represent the silhouettes of the pose without and with pose refinement, respectively. The evaluation results on industrial parts dataset show that the trained network is effective. Table 2 Evaluation results of industrial object dataset

Diameter (m)

Threshold (m)

Evaluation (%)

obj_01

0.1121

0.0112

98.1

obj_02

0.1373

0.0137

98.5

Fig. 7 a Histogram showing errors of our network when the network is trained with and without pose refinement subnetwork. b and c The visualization evaluation results of obj_01 and obj_02 for each images

56

H. Zhao et al.

Fig. 8 Proposed pose estimation method for industrial application. Original real-world scene images, pose estimation subnetwork results, pose refinement subnetwork results and comparison results are presented in each column

5 Conclusion In this paper, an efficient method is proposed for 6D pose estimation of texture-less objects with RGB-D images. In order to eliminate dependence on texture, RGBD images are fed into a pose refinement subnetwork to predict an approximated pose. The pose refinement subnetwork of 3D point clouds as input is used to iteratively refine the approximated pose by matching the rendered point cloud against the observed point cloud. Two experiments using the LINEMOD dataset and self-made industrial parts dataset are conducted to verify the effectiveness of the proposed method for pose estimation. The experimental results show the effectiveness of our method on the LINEMOD dataset and its utility on self-made industrial parts dataset. The proposed method performs best for five out of thirteen objects, and has highest score for 95.1% average recognition rate on the LINEMOD dataset. In addition, the method achieves 98.3% average recognition rate on self-made industrial parts dataset. This mean that the proposed method of estimating 6D pose of texture-less object can be used in industry, such as bin picking and warehouse automation.

6D Pose Estimation of Texture-Less Object in RGB-D Images

57

Acknowledgements The insightful comments of the reviewers are cordially appreciated. This research work is supported in part by National Natural Science Foundation of China under grant No. 51775344.

References 1. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110. 2. Rublee, E., Rabaud, V., Konolige, K., et al. (2011). ORB: An efficient alternative to SIFT or SURF. In IEEE International Conference on Computer Vision (pp. 2564–2571). 3. Rothganger, F., Lazebnik, S., Schmid, C., et al. (2006). 3d object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints. International Journal of Computer Vision, 66(3), 231–259. 4. Kehl, W., Manhardt, F., Tombari, F., et al. (2017). SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In IEEE International Conference on Computer Vision (pp. 1521–1529). 5. Rad, M., & Lepetit, V. (2017). BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In IEEE International Conference on Computer Vision (pp. 3828–3836). 6. Xiang, Y., Schmidt, T., & Narayanan, V., et al. (2017). Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199. 7. Besl, P. J., & McKay, N. D. (1992). Method for registration of 3-D shapes. Sensor Fusion IV: Control Paradigms and Data Structures., 1611, 586–606. 8. Hinterstoisser, S., Lepetit, V., Ilic, S., et al. (2012). Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian Conference on Computer Vision (pp. 548–562). 9. Liu, W., Anguelov, D., Erhan, D., et al. (2016). Ssd: Single shot multibox detector. In European Conference on Computer Vision (pp. 21–37). 10. Brachmann, E., Krull, A., Michel, F., et al. (2014). Learning 6d object pose estimation using 3d object coordinates. In European Conference on Computer Vision (pp. 536–551). 11. Choi, C., & Christensen, H. I. (2012). 3D textureless object detection and tracking: An edgebased approach. In International Conference on Intelligent Robots and Systems (pp. 3877– 3884). 12. Choi, C., & Christensen, H. I. (2016). RGB-D object pose estimation in unstructured environments. Robotics and Autonomous Systems, 75, 595–613. 13. Kehl, W., Milletari, F., Tombari, F., et al. (2016). Deep learning of local RGB-D patches for 3D object detection and 6D pose estimation. In European Conference on Computer Vision (pp. 205–220). 14. Rios-Cabrera, R., & Tuytelaars, T. (2013). Discriminatively trained templates for 3d object detection: A real time scalable approach. In IEEE International Conference on Computer Vision (pp. 2048–2055). 15. Tejani, A., Tang, D., Kouskouridas, R., et al. (2014). Latent-class hough forests for 3D object detection and pose estimation. In European Conference on Computer Vision (pp. 462–477). 16. Li, C., Bai, J., & Hager, G. D. (2018). A unified framework for multi-view multi-class object pose estimation. In European Conference on Computer Vision (pp. 254–269). 17. Wang, C., Xu, D., Zhu, Y., et al. (2019). Densefusion: 6d object pose estimation by iterative dense fusion. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 3343– 3352). 18. Li, Y., Wang, G., Ji, X., et al. (2018). Deepim: Deep iterative matching for 6d pose estimation. In European Conference on Computer Vision (pp. 683–698).

58

H. Zhao et al.

19. Daniel, B., Zhou, C., Xiao, F., et al. (2019). Yolact: Real-time instance segmentation. arXiv preprint arXiv:1904.02689. 20. He, K., Zhang, X., Ren, S., et al. (2016). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). 21. Qi, C. R., Su, H., Mo, K., et al. (2017). Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 652– 660). 22. Drost, B., Ulrich, M., Navab, N., et al. (2010). Model globally, match locally: Efficient and robust 3D object recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 998–1005). 23. Blender. [online]. Available: https://www.blender.org/.

Agriculture Applications

When using machine vision to count fruit in situ, a major problem is the occlusion of the fruit by other fruit and foliage. The first chapter in this part attacks the problem with convolutional neural nets. In great detail, the second chapter gives the design and construction of a robot for assessing pasture biomass. LIDAR data is combined with location by satellite in order to map the field. The third chapter shows the practicality of implementing a visual steering within a simple Android smartphone. An HTML page is linked from which the reader can download all the code that is needed to demonstrate the effectiveness. The fourth chapter describes a robot for the collection in an arid environment of a valuable resource—camel dung. Far from being a source of amusement, the machine shows itself to be of great practical value. An important component of the harvesting of lamb meat is the offal. Special manipulators are needed for the soft tissue of the ovine intestines.

Improving Vision-Based Detection of Fruits in a Camouflaged Environment with Deep Neural Networks Jinky G. Marcelo, Joel P. Ilao, and Macario O. Cordel II

1 Introduction Detecting objects from a similarly colored environment has been an open problem in computer vision. Camouflage or color similarity is one of the major issues where the object gets occluded or merged when they are of similar color with the environment resulting in a difficult object detection [1]. One of the applications in agriculture where camouflage problem exists is at detecting fruits or vegetables in a camouflaged environment for yield estimation, e.g. the foreground contains green fruits or vegetables and the background contains green foliage. Harvest time is critical particularly for vegetables and fruits; thus, there is a need to automatically detect and localize fruits in images for yield estimation for proper planning in labor, market and transport arrangement [2, 3]. Recent advances (e.g. [2, 3]) in computer vision has led to obtaining fruit detection and counting from images; however, this area still faces challenges because of illumination changes, scale variation, occlusion and situations of camouflage. Camouflage situations refer to the blending of the fruits to its foliage, stems and other objects of the environment which in turn makes the fruit detection a difficult task. Initial efforts on fruit detection and counting using deep neural networks were presented in [2–6, 9]. Keresztes et al. [4] achieved an R2 correlation of 0.96 for 45 J. G. Marcelo (B) · J. P. Ilao · M. O. Cordel II De La Salle University, Manila, Philippines e-mail: [email protected] J. P. Ilao e-mail: [email protected] M. O. Cordel II e-mail: [email protected] J. G. Marcelo Central Mindanao University, Maramag, Bukidnon, Philippines © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_5

61

62

J. G. Marcelo et al.

samples of grapes and 0.85 for 150 samples of apples between the manual and automatic counting. Chen et al. [2] proposed fruit counting based on a combination of two convolutional neural networks which achieved an accuracy of 0.96 on 71 images with 7,200 oranges and 0.91 on 21 images with 1,749 apples. Rahnemoonfar and Sheppard [3] attained a 0.91 accuracy on 100 real images for automatic yield estimation using synthetic training data. Fourie et al. [5] implemented a fruit detection and localizer algorithm on 21 apple images with 442 objects resulting in 0.98 accuracy. Stein et al. [6] used a pre-trained detector and Lidar to efficiently detect, track, count and localize every piece of fruit with an error of 0.014 in a total of 522 trees with 71,609 mangoes. Sa et al. [9] developed a real-time fruit detector that can perform up to a 0.83 F1 score with a field farm dataset comprising of at most 170 samples. All the mentioned prior works have promising results; however, these systems were trained to detect objects (e.g. oranges and tomatoes) with high color difference from the leaves. This allows them to train their network with few samples. By increasing the number of samples for the training dataset, deep learning models can achieve better performance and generalization. Though general object detection frameworks have brought remarkable breakthroughs in detecting different types of objects [7, 8], the current object detection algorithms fail in specific application scenarios like fruit detection due to occlusion and color similarity between the objects and the environment as illustrated in Fig. 1, last column. As shown in the last column of Fig. 1, the model pre-trained on MS COCO dataset [10] fails to detect any bell pepper or chili pepper. On the contrary, as illustrated in Fig. 1, third column, our work performs object detection and counting on fruits with heavy occlusion and high color similarity with its environment by

Fig. 1 a Input image: input images with high occlusion and color similarity of the objects with the environment (bell pepper or chili pepper). b Ground truth: each image contains groundtruth rectangular bounding box around each object which identifies the xmin, ymin, xmax, ymax for the object. c Ours: by re-purposing the region-based convolutional network to bell pepper and chili pepper images, our proposed system performs well in detecting heavily-occluded and camouflaged objects in a similarly-colored foreground and background. d Faster R-CNN: the result predicted by a generic object detection system

Improving Vision-Based Detection of Fruits in a Camouflaged …

63

increasing the number of images and re-purposing a region-based convolutional neural network. The contributions of this paper are: (1) we built two datasets (Bell Pepper and Chili Pepper) for sweet pepper detection. The sample images were taken from the sweet pepper field, capturing the natural settings of the object. The datasets are composed of 7700 images with 29,915 chili peppers and 3312 images with 14,548 bell peppers; (2) we exhaustively tested various region-based convolutional neural networks (Faster R-CNN Inception, Faster R-CNN Resnet50, Faster RCNN Resnet101, R-FCN Resnet101) for a very challenging task of detecting heavily occluded objects and highly-similar color of objects with its environment. To our knowledge, this is also the first attempt that a region-based convolutional neural network is used in recognizing and localizing camouflaged objects for agricultural applications.

2 Proposed System Figure 2 shows the overall block diagram of the proposed system. The network takes an image and outputs a set of objects with rectangular bounding boxes and the probabilities associated with it.

2.1 Image Datasets Sweet pepper was chosen as the sample dataset since it is considered as a high-value crop in the Philippines. Two varieties of peppers (green bell pepper and green chili pepper) were used, that were high in occlusion and have high color similarity with the environment. The samples for the green bell pepper dataset were collected during the

Fig. 2 Block diagram of our proposed system. The proposed system leverages the strength of region-based convolutional neural network for fruit detection in a camouflaged environment

64

J. G. Marcelo et al.

day with no artificial lighting in a greenhouse in Impasugong, Bukidnon, Philippines. A total of 552 images were taken for green bell pepper dataset. The samples for the green chili pepper dataset were collected under an uncontrolled variability in illumination since the climatic condition of the farm location was relatively cold and humid in a cultivated farm in Lantapan, Bukidnon, Philippines. A total of 2200 images were taken for green chili pepper dataset. The images of both datasets were acquired of size 3008 × 2000 in jpeg format using a Nikon D3200 24.2 MP digital SLR camera.

2.2 Data Augmentation and Annotation To improve the performance and generalization of deep neural networks, data augmentation was applied to the existing datasets. Data augmentation was implemented by applying horizontal flipping, rotations, shears, cropping and translation to existing datasets. These techniques ensure that the model can work under multiple angles and different orientations. The augmented images were added as additional samples to the training and test sets. After data augmentation, there was a total of 3312 images from 552 naturally captured images of bell pepper and 7700 images from 2200 naturally captured images of chili pepper. The regions of interest for ground truth annotations were drawn and extracted using a labeler software [11]. As illustrated in the second column of Fig. 1, a rectangular box was drawn around each object in each image and these generated bounding boxes were exported into xml files in Pascal Voc format which stores the coordinates of the bounding box of regions of interest.

2.3 Implementation Details The deep learning models were implemented using Tensorflow [14] by leveraging on transfer learning of deep vision systems. To fully explore the capability of CNNs in detecting and localizing fruit objects, four pre-trained models from MS COCO dataset [10] were fine-tuned and evaluated for bell pepper and chili pepper datasets. These pre-trained models include Faster R-CNN Inception-v2 [7, 12], Faster R-CNN Resnet50 [7, 13], Faster R-CNN Resnet101 [7, 13] and R-FCN Resnet101 [8, 13]. The pre-trained model was downloaded from Tensorflow Object Detection API [14], which is an open-source framework built on top of Tensorflow and trained on the Microsoft COCO [10] dataset. Table 1 shows a summary of the training and testing datasets. We fine-tuned the modified object detector networks [7, 8] using our respective datasets for green chili and green bell pepper, with momentum equal to 0.9 and an initial learning rate of 0.0003. The learning rate decreases by a factor of 3 × 10−5 every 9 × 105 iterations. The learning rate was further reduced to 3 × 10−6 at 1.2 × 106 iterations. All

Improving Vision-Based Detection of Fruits in a Camouflaged …

65

Table 1 Summary of training and testing datasets. The annotated bell pepper dataset and chili pepper dataset were randomly split into two datasets: 70% for training and 30% for validation Dataset

Bell peppers images objects

Chili peppers images objects

Training (70%)

2318

5390

Validation (30%) Total images and objects

10,141

20,361

994

4407

2310

9554

3312

14,548

7700

29,915

the models were trained with a momentum optimizer. We trained our system using NVIDIA GeForce RTX2070. A network was trained separately for each dataset with a batch size of 1. The network was trained for 6000 epochs and 20,000 epochs for bell pepper and chili pepper dataset, respectively.

2.4 Evaluation Metrics Object classification per category was evaluated using average precision (AP). As shown in Eq. (1), the AP score is defined as the mean precision at the set of 11 equally spaced recall values. AP =

1 × (A Pr (0) + A Pr (0.1) + · · · + A Pr (1.0)) 11

(1)

In order to evaluate the model on the task of object localization, we determined how well the model predicted the location of the object. As shown in Eq. (2), the localization task was evaluated based on the thresholds of Intersection over Union (IoU). A threshold of 0.5 was set which means that if the IoU exceeds the threshold, then the detection is marked as correct detection. The model with the highest average precision at 0.5 IoU was selected as the model for detecting and localizing camouflaged fruits for inference to validation data. I oU =

Ar ea o f Overlap Ar ea o f U nion

(2)

3 Results Table 2 and 3 present the result of the detection performance of four different models and their corresponding training time to bell pepper dataset and chili pepper dataset, respectively. Out of the four fine-tuned models, Faster R-CNN Resnet101 exhibited the best performance for the bell pepper dataset, yielding an AP of 0.966. For the

66 Table 2 Comparison of the results obtained by our proposed system employing different region-based CNNs on the bell pepper dataset. Faster R-CNN Resnet101 (Bold text) outperforms other models with [email protected] = 0.966

Table 3 Comparison of the results obtained by our proposed system employing different region-based CNNs on the chili pepper dataset. Faster R-CNN Resnet101 (Bold text) outperforms other models with [email protected] = 0.922

J. G. Marcelo et al. Method

Basenet

Training time

Average precision

R-FCN

Resnet101

34 min

0.963

Faster R-CNN

Resnet101

33 min

0.966

Faster R-CNN

Resnet50

23 min

0.964

Faster R-CNN

Inception

21 min

0.960

Method

Basenet

Training time

Average precision

R-FCN

Resnet101

1 h 42 min

0.904

Faster R-CNN

Resnet101

1 h 35 min

0.922

Faster R-CNN

Resnet50

1 h 5 min

0.917

Faster R-CNN

Inception

47 min

0.903

chili pepper dataset, Faster R-CNN Resnet101 also outperformed other models with an average precision of 0.922. The features of Faster R-CNN Resnet101, a very deep network, were sufficient in the transfer learning for the detection of bell peppers and chili peppers. It is evident that the region proposal network contributed to higher accuracy and efficiency. From this result, it can be concluded that the model performs well in predicting the occurrence and position of the fruits in an image amidst high levels of occlusion and even those highly-similar in color between the fruits and the background. After finishing the training, the model trained with the highest average precision was selected as the best model and exported to a single file for inference. The inference system’s performance was measured using the validation datasets for bell peppers and chili peppers. Figure 3 presents the sample result of inference of our proposed system

Fig. 3 Inference system on the highly-similarly colored environment. Our proposed system (3rd column) substantially performed better than the generic region-based object detector (4th column) on detecting bell peppers and chili peppers in a camouflaged environment

Improving Vision-Based Detection of Fruits in a Camouflaged …

67

Fig. 4 Visualization of feature maps to bell pepper and chili pepper input. a Input images (bell pepper and chili pepper); b visualization image of feature maps of the first convolutional layer; c visualization image of feature maps of the last convolutional layer

and a pre-trained Faster R-CNN object detection system on images with high color similarity and heavy occlusion. Despite the high degree of color similarity between the fruits and the foliage, our proposed method can detect the fruits efficiently and correctly. Also, the system correctly recognized and localized the fruits even those fruits which are almost hidden due to heavy occlusion. Figure 4 shows the visualization of feature maps after applying the filters at the first and last convolutional layer in the Resnet101 model for bell pepper and chili pepper input, respectively. It can be observed that the result of applying filters in the first convolutional layer retains most of the input image features. This means that there are many activations on the edges and textures within the image. But as the network goes deeper into the model, the feature maps become more sparse and visually less interpretable. This implies that the filters abstract the features from the image into more general concepts and convert it to the required output classification domain.

4 Conclusion and Future Work We presented a system that automatically detects and localizes fruits from images captured from the natural settings of the fruits. By increasing the number of images and leveraging on the four pre-trained networks, the evaluation results show that the fine-tuned model on Faster R-CNN Resnet101 performed the best among all the models in detecting heavily-occluded and camouflaged fruits. It yielded an average precision of 0.92 for chili pepper and 0.96 for bell pepper. The inference shows that the fine-tuned model on Faster R-CNN detected very well to heavy-occluded and

68

J. G. Marcelo et al.

similarly-colored foreground and background bell pepper and chili pepper images. This indicates that the trained fruit detection and counting model can be integrated into applications for precision agriculture such as automated fruit harvesting, yield estimation, and plant phenotyping. One direction of future work is to integrate the trained fruit detector to an unmanned ground vehicle. Moreover, it can also be extended to detect other parts of a plant such as leaves, flowers, and stems which may be used for plant phenotyping and plant pathology. The proposed system can still be improved by extending its functions to more camouflaged images in agriculture and other domains. Acknowledgements We would like to thank the farm owners and farmers for helping us in data collection. The first author acknowledges the Commission on Higher Education, in collaboration with De La Salle University and Central Mindanao University for funding the scholarship grant.

References 1. Ratthi, K., Iyshwarya. V. S., Yogameena, N. B, Menaka, K. (2017) Foreground segmentation using motion vector for camouflaged surveillance scenario. In International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) (pp. 172–176). 2. Chen, S. W., Shivakumar, S. S., Dcunha, S., Das, J., Okon, E., Qu, C., et al. (2017). Counting apples and oranges with deep learning: A data-driven approach. IEEE Robotics and Automation Letters 2(2). 3. Rahnemoonfar, M., & Sheppard, C. (2017). Deep count: Fruit counting based on deep simulated learning. Sensors, 17(4), 905. 4. Keresztes, B., Abdelghafour, F., & Randriamanga, D. (2018) Real-time fruit detection using deep neural networks. In Proceedings of the 14th International Conference on Precision Agriculture. 5. Fourie, J., Hsiao, J., & Werner, A. (2017). Crop yield estimation using deep learning. In 7th Asian-Australasian Conference on Precision Agriculture. 6. Stein, M., Bargoti, S., & Underwood, J. (2016). Image based mango fruit detection. Sensors: Localisation and Yield Estimation Using Multiple View Geometry. 7. Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (pp. 91–99). 8. Dai, J., Li, Y., He, K., & Sun, J. (2016). R-FCN: Object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems (pp. 379–387). 9. Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., & McCool, C. (2016). Deepfruits: A fruit detection system using deep neural networks. Sensors, 16(8), 1222. 10. Lin T. et al. (2014) Microsoft COCO: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele & T. Tuytelaars (Eds.), Computer Vision ECCV 2014. Lecture Notes in Computer Science, vol: 8693. Cham: Springer. 11. Tzutalin (2015). LabelImg. Git code. https://github.com/tzutalin/labelImg. 12. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2818–2826).

Improving Vision-Based Detection of Fruits in a Camouflaged …

69

13. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770– 778). 14. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., … & Ghemawat, S. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.

Mechatronics for a LiDAR-Based Mobile Robotic Platform for Pasture Biomass Measurement M. Sharifi, S. Sevier, H. Zhang, R. Wood, B. Jessep, S. Gebbie, K. Irie, M. Hagedorn, B. Barret, and K. Ghamkhar

1 Introduction With the world’s growing population and the significance of sustainable agriculture, agricultural productivity, and food supply, digital and precision agriculture is at the top of the priority list for speeding up the efficiency of agricultural productivity. In New Zealand, pastoral sector is the largest export contributor, where dairy exports generate $14b and meat export generates $5b per annum [1]. To evaluate agricultural productivity, it is important to accurately and consistently measure yield in pasture plants. This trait can vary depending on different factors such as soil, environment, fertilization, cultivation time, etc. Pasture yield can be more specifically measured using fresh weight (FW) and dry weight (DW), which increases the efficiency of methods such as genomic selection, thus predicting forage productivity with minimal cost [2]. The conventional method used to estimate dry matter yield (DMY) in plots of forage grasses involves destructive mechanical harvesting of a known area, drying the harvested material, weighing the dry material and then using the data to calculate the dry weight per area harvested. Fresh weight can be estimated using visual scoring by an experienced plant breeder. These conventional methods limit plant breeders to effectively (in terms of both cost and accuracy) assess DMY and FW in large scale capacities, as well as precluded grazing which is critical for selecting and evaluating the pasture plants. Furthermore, the ability to measure pasture growth rate at high M. Sharifi (B) · S. Sevier · H. Zhang · R. Wood · B. Jessep · S. Gebbie Development Engineering, Lincoln Research Centre, AgResearch, Lincoln, New Zealand e-mail: [email protected] B. Barret · K. Ghamkhar Forage Science, Grasslands Research Centre, AgResearch, Palmerston North, New Zealand K. Irie · M. Hagedorn Red Fern Solutions Ltd, Christchurch, New Zealand © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_6

71

72

M. Sharifi et al.

spatio-temporal resolution is indeed constrained by current technologies in pasture plant breeding, despite its economic value and impact in the selection process [3, 4]. Plant phenomics or the use of sensors and digital technologies for measuring traits in plants has enabled efficient and reliable phenotyping of many plant traits. Mobile, in-field robotic platforms equipped with sensors and computational power can assist plant breeders and other researchers to conduct the required high-throughput plant phenotyping. [5–7]. Light detection and ranging (LiDAR) sensing technology, measures distance to a target by emitting the target with frequent impulsive laser signals and simultaneously capturing those reflected laser signals. Unlike traditional cameras, LiDAR scanners directly capture distance and distribution data [8]. LiDAR is a preferred tool for characterisation of plant biophysical [9, 10] and detailed physical characteristics in the field [11, 12]. Ground-based LiDAR systems have shown promising results in the estimation of biomass in many types of crops as well as pasture [13]. With the successful application of ground-based mobile robotics in agriculture to enhance precision and productivity, our objective in this paper is that an improved LiDAR-based mobile robotic platform based on an early prototype could offer increased efficiency in forage yield measurement. Here we present the mechatronic design and development of a LiDAR-based mobile robotic platform for rapid and accurate measurement of pasture biomass. The mechatronic process including system design specification, mechanical design and 3D CAD modelling, electrical and control system architecture, alongside control software architecture is presented. This is followed by a brief review of field experimental results and the overall performance of the system.

2 Methodology In this section, the integrated mechatronic design and development process of the LiDAR-based robotic platform has been presented.

2.1 Design Overview and System Specification Following a consultation meeting with the end-users, the specifications of the robotic platform were identified as below: • Be a lightweight and robust platform, all electrically powered, for the purpose of carrying LiDAR to measure pasture biomass of grass plots. • Be equipped with a high accuracy real-time kinematic (RTK) Global Navigation Satellite System (GNSS) to map measured pasture biomass of grass plots.

Mechatronics for a LiDAR-Based Mobile Robotic Platform …

73

• Be able to be operated and controlled remotely through a wireless controller, monitored through a graphical user interface which is handheld by a nearby operator. • Be transportable on a trailer from its base (workshop) to the field, with maximum incline of 1 in 3 for loading/unloading and maximum incline of 1 in 20 during pasture biomass scanning operation. • Be suitable for environmental conditions including weatherproofing and all types of agricultural terrain. • Be easy to clean using gentle water pressure followed by drying and spraying with disinfectant. The main chassis was designed to have no suspension (rigid frame), and 600 mm minimum ground clearance to minimise disturbance of pasture plots. The wheel centres width across the platform were fixed at 1400 mm, and wheel centres from front to back wheels were 1250 mm. To accurately measure pasture biomass, accurate ground height measurement (from ground to LiDAR) was needed, thus an additional measurement mechanism was included. This mechanism included two ‘floating’ wheels to detect, and record LiDAR to ground variations, these were placed on each side of the chassis. The LiDAR unit was positioned in between the front and rear wheels at a height of 1500 mm and was facing down to the grass plots vertically. In this position it is able to scan grass plots up to 1400 mm wide. To provide a simple, reliable, and manoeuvrable drive for precision operation in the pasture plots the wheeled robotic platform was designed to encompass a differential steering drive system. This was provided by the two rear wheels while the two front wheels are castering. Each of the driving wheels was equipped with a geared high torque brushless DC (BLDC) motor, electrical braking system for greater safety, and encoder feedbacks for closed loop speed control. Inflated tyres were utilized for all wheels to provide a softer ride while minimising height variations of LiDAR relative to the ground. All the electrical and electronic components were graded with a minimum of IP54 protection class (protected against dust that could interfere with operation of the equipment and protected from splashing water from all directions), but ideally IP65 (protection against dust and protection from jets of water from all directions). The platform included a stand-alone industrial embedded control system to control the operation modes and drive, this was interfaced with a central high-level computing system to carry out the LiDAR measurement and had a graphical user interface. Three operation modes were considered including: high torque low speed (up to1 km/h) for loading/unloading from the trailer, fast speed mode (up to 6 km/h) for any other nonscanning movements, and scan-mode with a constant speed of 3 km/h. The ground height measurement system was designed to be engaged during scanning mode, and disengaged during the other two modes.

74

M. Sharifi et al.

2.2 3D CAD Mechanical Design and Manufacturing The robotic platform was designed using Autodesk Inventor Professional 2019 running on Windows 10. Initially, a concept design was modelled in CAD, which enabled further refinement of the specification and design. Once final design was complete, CAD data was leveraged to enable efficient manufacture using modern methods, including CNC cutting, folding and machining. 3D printing was also used for rapid prototyping of small components. This design included all the parts that needed to be manufactured and parts supplied off the shelf. 3D CAD modelling enabled agile and improved design and also expedited rapid prototyping and manufacture of mechanical parts. It also enabled further computer aided engineering (CAE) and analysis of the mechanical parts and driving joints in terms of structural and motion analysis. Figure 1 shows an overview of the 3D CAD model of the robotic platform including the height measurement mechanism. As mentioned earlier, the 3D CAD model enabled rapid prototyping and manufacturing of the robotic platform through computer-aided manufacturing (CAM). In the design process, it was also aimed to design the robotic platform by considering factors such as modularity and manufacturability. The robot chassis was manufactured through laser cutting and CNC machining. Figure 2 depicts the manufactured robot chassisand as an example.

2.3 Electrical, Control, and Sensing Systems Architecture The electrical and control system architecture on this robotic platform was comprised of a low-level control system interfaced with a high-level computing processor. The low-level control system encompassed an embedded brushless DC (BLDC) motor speed controller and is responsible for controlling platform operation modes, receiving speed commands and speed feedbacks while controlling and providing Pulse-Width-Modulation (PWM) power to both BLDC drive motors in a closed-loop PID control system. The high-level computing process system encompassed a touch screen industrial computer, that is interfaced to the embedded speed controller, RTKGNSS receiver, and LiDAR. This high-level processing system runs software with a graphical user interface. This software is responsible for collecting and processing the LiDAR, RTK-GNSS geo-location information, and speed feedback. The algorithms embedded in this software generate both pasture biomass estimation and a scanning map. Figure 3 presents the electrical and control system architecture including both low level and high-level systems and attached sensors.

Mechatronics for a LiDAR-Based Mobile Robotic Platform …

75

Fig. 1 3D CAD model of the robotic platform with all componentry: (1) chassis, (2) rear driving wheels, (3) front castering wheels, (4) ground height measurement system, (5) control system and electronics, (6) RTK-GNSS receiver, (7) LiDAR, (8) Battery, (9) Height detection reflector plate

2.4 Software System Architecture The software system is composed of both low and high-level processing systems. The low-level processing system, the embedded speed controller, accommodates the required software for the robotic platform motion and operation modes control. A kinematic based model responsible for differential driving system, receives the input linear and angular velocity commands, calculates the required power for each of the driving motors, and controls the platform with the desired speed through inverse kinematics. The controller also provides different operation modes (slow, fast, and scan) with set speeds through user input. In the scan mode, the height measurement system is engaged with the ground (through an active linear actuator), while it is disengaged in both slow and fast operation mode.

76

Fig. 2 Manufactured chassis of the robotic platform

Fig. 3 Electrical, control, and sensing system architecture of the robotic platform

M. Sharifi et al.

Mechatronics for a LiDAR-Based Mobile Robotic Platform …

77

The high-level software comprises real-time data streaming and recording from the LiDAR, RTK-GNSS receiver, and motor-drive software, along with in-field data analysis to calculate biomass (post-capture).

3 Experimental Results and Discussions Several in-lab and in-field experiments were conducted to evaluate the performance of the platform in different operation modes and conditions. When in trailer loading and unloading operation mode (slow mode) the platform could reliably drive a ramp with the expected incline angle of 1 in 3 (~18°). This was evaluated through several ramp tests inside Engineering Development Laboratory (EDL), AgResearch Ltd. The experimental results for this operation mode showed that the drawn current by each of the driving BLDC motors were within the expected and safe limits. Figure 4 shows the ramp test setup and the plotted electrical currents drawn by the driving motors and from the two batteries during the experiment. In-field experiments were carried out for both fast and scanning operation modes to test the low-level control system and driving performance of the platform. In the scanning mode, the platform is expected to scan the pasture plots on a direct path with minimal steering requirement. Thus, the steering rate was limited to a maximum of 10 degrees to protect the ground height measurement system from unwanted damage

Fig. 4 The ramp experiment using the slow mode control: a the platform and the test ramp setup, b the drawn electrical current from driving motors and batteries

78

M. Sharifi et al.

Fig. 5 Fully developed LiDAR based robotic platform during the in-field experiments

due to sharp turns (or spot turns). Figure 5 shows the completed platform during the in-field experiments. In-field experiments were also carried out in scan-mode to evaluate and validate the performance of the pasture biomass measurement system through LiDAR data and the high-level software processing system. The developed algorithms within the high-level software system, automatically segment the scanned LiDAR data from a sequence of raw measures into volumetric estimates for multiple rows of pasture plots. Ground height can vary substantially between the scans. Thus, the ground height is also calculated at each sequence of the LiDAR scan through measuring the vertical variations of the reflector plate of the ground height measurement mechanism. Through further processing, pasture biomass is estimated using volumetric data from the segments. Figure 6 shows example LiDAR 3D data of a single row of perennial ryegrass achieved from the high-level processing system. From the obtained LiDAR volumetric data, pasture fresh weight (FW) and dry weight (DW) were estimated. Finally, the percentage dry matter yield (%DMY) was calculated from the two indicators as the main pasture yield indictor.

Mechatronics for a LiDAR-Based Mobile Robotic Platform …

79

Fig. 6 Example of LiDAR volumetric data (Top-down view) of a single row of perennial ryegrass pasture [13]

4 Conclusion A LiDAR-based mobile robotic platform for non-destructive and rapid measurement of pasture biomass was developed. An integrated mechatronic design and development process was incorporated to develop the mobile robotic platform including mechanical, electrical, electronic, sensors, and software system. A low-level software system was responsible for motion control and driving the robotic platform, while a high level software system was responsible for carrying out the integration of LiDAR measurement, GNSS-RTK receiver data, and the odometry from the low level control system. The integrated data from different sources were processed in real-time to generate LiDAR 3D volumetric data. The LiDAR volumetric data was then further processed to estimate pasture FW, DW, and DMY. Early results show that this integrated approach provides a precise, non-destructive, and cost-effective way for real-time in-filed measurement of pasture yield with highly anticipated scientific and commercial benefits. Acknowledgements We acknowledge technical support from staff at PGG Wrightson’s Seeds and New Zealand Agriseeds. Technical staff at AgResearch: Craig Anderson, Angus Heslop, Anthony Hilditch, Peter Moran, and Jana Schmidt are highly appreciated for their input in designing the machine. The research to develop the LiDAR, electronics, and mechanics of the system was funded by Pastoral Genomics, a joint venture co-funded by DairyNZ, Beef+Lamb New Zealand, Dairy Australia, AgResearch Ltd, New Zealand Agriseeds Ltd, Grasslands Innovation Ltd, and the Ministry of Business, Innovation and Employment (New Zealand).

80

M. Sharifi et al.

References 1. Ministry for Primary Industry, New Zealand. (2019). https://www.mpi.govt.nz/exporting. 2. Pollock, C. J., Abberton, M. T., & Humphreys, M. O. (2005). Grass and forage improvement: Temperate forages. Grassland: a Global Resource 57–68. 3. Cayley, J. W. D., & Hannah, M. C. (1995). Response to phosphorus fertilizer compared under grazing and mowing. Australian Journal of Agricultural Research, 46(8), 1601–1619. 4. McNaughton, S. J., Milchunas, D. G., & Frank, D. A. (1996). How can net primary productivity be measured in grazing ecosystems? Ecology, 77(3), 974–977. 5. Lingfeng, D., et al. (2011). A novel machine-vision-based facility for the automatic evaluation of yield-related traits in rice. Plant Methods, 7(1), 44. 6. Wang, L., et al. (2014). Estimation of leaf biochemical content using a novel hyperspectral full-waveform LiDAR system. Remote Sensing Letters, 5(8), 693–702. 7. Fernando, S., et al. (2008). Active sensor reflectance measurements of corn nitrogen status and yield potential. Agronomy Journal, 100(3), 571–579. 8. Molebny, V., Kamerman, G., & Ove, S. (2010). Laser radar: from early history to new trends. In Electro-Optical Remote Sensing, Photonic Technologies, and Applications IV. (vol. 7835). International Society for Optics and Photonics. 9. Holmgren, J., Nilsson, M., & Olsson, H. (2003). Estimation of tree height and stem volume on plots using airborne laser scanning. Forest Science, 49(3), 419–428. 10. Næsset, E. (2002). Predicting forest stand characteristics with airborne scanning laser using a practical two-stage procedure and field data. Remote Sensing of Environment, 80(1), 88–99. 11. Harding, D. J., et al. (2001). Laser altimeter canopy height profiles: Methods and validation for closed-canopy, broadleaf forests. Remote Sensing of Environment, 76(3), 283–297. 12. Lovell, J. L., et al. (2003). Using airborne and ground-based ranging LiDAR to measure canopy structure in Australian forests. Canadian Journal of Remote Sensing, 29(5), 607–622. 13. Ghamkhar, K., et al. (2018). Using LIDAR for forage yield measurement of perennial ryegrass (Lolium perenne L.) field plots. In Breeding grasses and protein crops in the era of genomics. Cham: Springer, pp. 203–208.

Vision Guidance with a Smart-Phone John Billingsley

1 Introduction A paper describing the development of a vision-guidance strategy for farm vehicles has been cited over a hundred times, including six citations within the last two years. This simple strategy has now been encapsulated in the form of JavaScript code suitable for a smart-phone or tablet and is demonstrated in action. A URL is given from which the source can be downloaded for further research and development. The vision processing that exploits the camera of a smartphone can be complemented by the inbuilt sensors of GPS and three-axes of acceleration. These give all that is required for low-cost guidance in a variety of situations. With the addition of simple Bluetooth communication to a steering module, this means that precision automatic guidance along a row crop can now be applied to the smallest of mobile devices at a budget cost. A goal of agricultural robotics has been the development of autonomy for farming vehicles. With autonomy comes a relaxation of the requirement that the machine must be large to optimize the effort of a human driver. It is suggested that on safety grounds small autonomous vehicles would be more acceptable than large ones. The software outlined here is still in an early stage of development, but the implementation is already such that it can be demonstrated in the form of a simple HTML web page, supported by JavaScript code. This was transplanted almost untouched from the C++ of the original application. The working software can be found at the web location http://www.essdyn. com/rowfit.htm [1] from which the source can be saved for further research and development.

J. Billingsley (B) University of Southern Queensland, Toowoomba QLD4350, Australia e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_7

81

82

J. Billingsley

2 Background and History The original 1990s project used a camera that was part of a camcorder, webcams were not yet common. The computer used was a PC in a tower case, since there were no tablets or laptops. The display was a bulky monochrome CRT. After an industry sponsored launch that was a commercial failure, the project was relaunched some years later Recent developments in HTML5 mean that video data can be accessed from the camera with ease, allowing software written in JavaScript to be run by common browsers in the form of a web page. It was not difficult to take the original C++ code and knit it into a JavaScript environment. This runs at an adequate speed to analyse the video frames at the full frame rate.

3 Evolution of the Central Algorithm The first vision guidance project really commenced before the founding of USQ’s National Centre for Engineering in Agriculture, where the later work was based. A Masters student, Murray Schoenfisch, gave a presentation that concerned the use of buried cable for sensing a steering error. It was clear from his slides that rows of crop could readily be discerned, so the emphasis turned to machine vision [2]. The first step was to record some video footage from a car that was driven somewhat erratically along the rows. The video footage was captured to a PC using a primitive home-grown frame-grabber. This was only capable of capturing a binary image at relatively low resolution. The later paper describing the strategy [3] receives citations to this day. Recent examples such as Zhang et al. [4] and García-Santillánet al. [5] involve substantial computation such as clustering algorithms to achieve their objective. In contrast the early strategy bypassed many of the conventional ‘tools’ of image filtering. In consequence it has led to algorithms that can readily be migrated to other technologies. A feature common to many image analysis techniques is the attribution of pixels to membership of a set. In this case, the sets are “plant” or “soil” and when accessed, the pixels were tagged in this conventional way. In the early days, discrimination was by means of a brightness threshold and then, when colour capture became possible, by inspection of the chrominance component of a YUV stream. An innovation was in automatic threshold control. A ‘farmer’s guess’ was entered to signify the proportion of the image that should be “plant”, based on the state of the crop. When lighting conditions changed, the discrimination threshold was automatically varied until the proportion of “plant” pixels in the portholes of interest reached that level.

Vision Guidance with a Smart-Phone

83

Fig. 1 An early example of ‘keyholes’ fitted to the rows

Having divided the pixels into two sets, a ‘toolbox’ technique would have been to apply a filter that identified pixels on the boundary between the regions, seeking to define the shape and location of a crop row. In emerging or weedy conditions, no such connected or compact regions might exist. Instead, the strategy treated “plant” pixels as simple data points, through which regression lines could be drawn to identify any formation into rows. The parameters defining these lines then determine the lateral displacement and heading of the vehicle. The heading relative to the rows is given by movement of the vanishing point, while lateral displacement is given by the deviation of the mean slope from the vertical. The calculation also included a quality figure to indicate the degree of discernment and reliability. To fit such lines, it was necessary to ensure that the processed pixels were members of a single row at a time. ‘Keyholes’ were selected that should each contain only one row. When the parameters of the fitted line were extracted, they defined corrections to the keyhole position and slope for use in the following image frame, Thus the keyholes could be moved rapidly from frame to frame to track the rows, while the hardware could then take its time to steer the vehicle and bring the set of keyholes back to their datum position (Fig. 1). With values for displacement and heading, a control strategy was able to demand a heading change proportional to the lateral error. This was limited in magnitude. The difference between observed and demanded heading then generated a steering demand, also limited in magnitude.

4 Methodology: Bringing the Implementation up to Date With the advent of HTML5 and Canvas, the programming task has become much easier. The original rowfit.cpp code can be simplified and dropped straight into a

84

J. Billingsley

JavaScript file rowfit.js. The same code calculates the parameters and quality of the regression fit. The video stream is now accessed by invoking [6]. navigator.mediaDevices.getUserMedia(constraints) This first asks the user for permission to access the camera, then presents the frames as an array data that is a property of a variable that has been equated to readFrame(). This data differs from the format of DirectX data, in that there are four bytes per pixel, not three. The fourth byte controls opacity, allowing an underlying image to show through if the opacity is not set at 255. Also, the colour bytes are in the reverse order. The picbit(x, y) function is still used, returning 0 or 1. It is now calculated as the difference between the green and red components of the pixel when compared against a threshold. This threshold is controlled just as before to maintain the proportion of marked pixels to be the ratio entered during setup. As before, the quality is assessed in terms of the ‘moment of inertia’ of the plant pixels about the fitted line. If the keyhole bridges a neighbouring row, that moment will be large and the quality will be low. If the quality for both rows is persistently of an unacceptable standard, the vanishing point and slope both decay to their datum values. The Canvas environment gives a further advantage. A mouse or finger drag can be used to set up the datum parameters. • The vanishing point and horizon can be dragged to match the image on the screen by dragging above the top of the stripes. • By dragging horizontally below the bottom of the stripes, the datum tilt can be set to compensate for any lateral offset of the camera. • By dragging in the lower half of the stripes, the angular separation of the keyholes can be set to match the rows in the image. • In the upper half of the stripes, the width of the keyholes can be adjusted. To gain access to the ‘promise’ of the ‘mediaStream’, the web page must either be stored on the user’s own machine or be at an address that is reached by a secure ‘https’ call. A demonstration page can be found at http://www.essdyn.com/rowfit. htm. Since this page is not secure, you must first save it as a ‘web page complete’ and then open that file with your browser. In the process, you will be able to see all the finer points and shortcomings of the software by examining the source.

5 Further Work Although this code demonstrates the ability to generate steering signals, the Bluetooth communication protocol must be devised, including all necessary safety features. The present GPS signals available to low cost devices are not yet of a quality that will allow precision steering, though they should suffice for headland turns. With the

Vision Guidance with a Smart-Phone

85

escalation of constellations of satellites, this will be remedied in a very short time, allowing vision and GPS to work in mutually-supporting partnership. Tracking the intersection of the fitted lines would allow the horizon to be tracked in undulating ground. By tracking their angular split, precise altitude could be measured of a drone being flown for crop inspection or selective spraying. This is a technology that is easily accessible and should stimulate a host of further research projects.

6 Conclusions The release of the algorithm in this form has many interesting implications. It makes vision guidance accessible to anybody who wishes to exploit it, though it is sincerely hoped that they will attribute its origin. Some tractor manufacturers are known for the difficulty of interfacing a new sensor with their system. They use encryption to preserve exclusivity. Could the attraction of simple ‘apps’ such as this open up the way for farmers to use their own ingenuity to upgrade their machinery? In the consumer field, the concept of computer as entertainer has spun off a dazzling array of products and services. In this flood of products and technologies there are many that can be exploited for solving ‘real’ engineering problems, such as those that have so long confronted the farmer. As these new opportunities are seen to emerge, new problems are brought to light that will require new strategies for their solution.

References 1. Billingsley, J. (2019). Rowfit demonstration of row tracking, seen at. http://www.essdyn.com/ rowfit.htm. 2. Billingsley, J., & Schoenfisch, M. (1995). Vision and Mechatronics Applications at the NCEA. In Fourth IARP workshop on Robotics in Agriculture and the Food Industry, Toulouse. 3. Billingsley, J., & Schoenfisch, M. (1997) The successful development of a vision guidance system for agriculture. In Computers and Electronics in Agriculture (journal). Amsterdam Netherlands: Elsevier, pp. 147–163. 4. Zhang, X., et al. (2018). Automated robust crop-row detection in maize fields based on position clustering algorithm and shortest path method. In Computers and Electronics in Agriculture vol. 154, pp. 165–175. 5. García-Santillán, I., Guerrero, J. M., Montalvo, M., et al. (2018). Curved and straight crop row detection by accumulation of green pixels from images in maize fields. Precision Agriculture, 19(1), 18–41. 6. MDN (many contributors). (2019). MediaDevices.getUserMedia() In Mozilla Develpers Network, viewed at. https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUse rMedia.

A High-Speed Camel Dung Collection Machine Samuel N. Cubero, Mohammad Badi, Mohamed Al Ali, and Mohammed Alshehhi

1 Introduction Animal dung (manure or droppings) from horses, bovines (cattle) and camels are rich in bacteria and nutrients. They are manually collected and used to enrich soils and fertilize farm crops, plants and gardens. Large beaches are often strewn with small bottles, empty cans, and other forms of litter that need to be collected regularly and disposed of to maintain public safety and cleanliness. In the UAE (United Arab Emirates, a small oil-rich country beside Saudi Arabia that boasts the world’s tallest skyscraper in Dubai), temperatures often reach as high as 50 °C during the summer. At present, collecting camel dung, horse manure and litter on a large scale is considered to be exhausting and tedious work, and is usually done manually, using shovels, wheelbarrows and small hand tools. This paper briefly describes the ‘prior art’, or existing machines designed for collecting animal dung and small litter, including their advantages and disadvantages. A practical ‘mechatronic’ design project is described, leading to the creation of a working prototype for a machine that shows good potential for use as a litter and dung collector on sandy ground and beaches. It was developed at Khalifa University, Abu Dhabi, during mid-2018 to mid-2019, at the Department of Mechanical Engineering. It is a manually-steered self-propelled vehicle that— with a few modifications and improvements—could become a highly reliable dung collector which can be successfully mass-produced and marketed worldwide in the very near future. Modeling and engineering analysis for this machine’s front-wheeldrive motor and the rake conveyor motor are also described in detail. Several performance problems were experienced and their potential solutions and remedies are briefly discussed.

S. N. Cubero (B) · M. Badi · M. A. Ali · M. Alshehhi Department of Mechanical Engineering, Khalifa University, PO Box 2533, Abu Dhabi, UAE e-mail: [email protected]; [email protected] URL: https://www.samcubero.com © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_8

87

88

S. N. Cubero et al.

Fig. 1 Typical a camel dung, b horse dung, and c cow or bull dung

1.1 Different Types of Animal Dung The manure (or dung) usually produced by horses and camels is quite similar in size, shape and texture, and is typically shaped in the form of round balls. In contrast, the manure produced by cows contains more moisture and typically appears formless, similar to a slurry or a sticky wet mud. The dung shown in Fig. 1a, b are typical for camels and horses, respectively, and they appear to be much dryer and possess lower density (or mass per unit volume) compared to the cow dung shown in Fig. 1c. It is important to note that teeth on a rake would not be very effective at collecting wet cow dung, but a rake can collect horse and camel dung balls that are larger than the gaps between the rake’s teeth. A moving scraper, a fan blade or a solid paddle (with no holes or gaps) would be able to collect all types of dung. Unfortunately, such a collection mechanism would also collect unwanted sand which may take up a lot of room in the collection bin or storage area for the dung. Cow dung collection requires a mechanism that can handle the sticky slurry-like dung while avoiding the collection of too much sand and dirt. In this project, our priority was to focus mainly on collecting camel and horse dung at high speed (or at least, faster than an average human worker).

1.2 Prior Art in High-Speed Litter Collection Machines This section briefly examines the features, pros and cons of some patented designs and commercially available litter cleaners—machines that could potentially collect horse and camel dung effectively, especially off sandy or desert ground. The different kinds of litter collection machine designs investigated fall into the following 4 categories: 1. 2. 3. 4.

Rotating blade or screw collection machines Rake conveyor machines Vibrating filter machines Vacuum-type litter collection machines.

A High-Speed Camel Dung Collection Machine

1.2.1

89

Rotating Blade or Screw Collection Machines

Hansson [1] patented a ‘Manure removing machine’ in 1978 that uses a rotating feeder screw to scrape and transfer manure off the ground. The mass flowrate of manure that can be removed by this design would only be very small due to the small contact area between the screw blades and the ground. The circular or cylindrical shape of the feeder screw means that it can only touch the ground at one outside edge, and the axis of rotation points in the forwards direction of driving for the vehicle, therefore, only a small area of manure can be removed off the ground in one pass of the vehicle. This design could be improved if one or two feeder screws have their axis (or axes) of rotation(s) oriented orthogonal to the forwards direction of driving for the vehicle to increase contact area of the feed-screw with the ground. Unfortunately, this design does not seem like it would have a high manure collection rate. The feedscrews would also be very expensive to manufacture due to their complex geometries, and the entire machine requires a large truck or vehicle to tow it. Vinyard [2] patented a ‘Rear-mounted manure gathering machine’ in 1994. This device uses rotating blades (resembling several paddle wheels, stacked one on top of each other) to scrape and lift (or fling) manure off the ground to higher levels of rotating fan blades, which transports the manure to an elevated tank at the rear of the machine. Unfortunately, this design would also pick up a lot of sand and dirt, along with the manure. Sand and fine particulates can be sifted out and prevented from entering a collection bin or container using appropriate filters. The entire machine is towed behind a tractor. The video shown in [3] shows how dry and wet cow manure (or cow patties) can be collected effectively using a conventional ‘ride-on’ mower for cutting grass. However, the video shows that some grass is also collected. If this ‘rotary mower’ type dung collection method is used on a beach or in a sandy desert, the rotating blades will collect a large amount of sand, along with any animal dung or litter, thus wasting much space in the collection bin. Although very simple and useful for collecting cow dung on grassy pastures, this method needs some way to filter out or separate the sand from the dung. A video demonstrating a hand-powered ‘cow dung collecting machine’ is shown at [4]. This hand-powered cow manure collecting machine seems to experience mechanism seizure or jamming if there is too much dung at the bottom of the ramp, because one of the rotating blades on the chain conveyor must compress the dung beneath it at the front of the ramp, requiring very high torque. This problem could be avoided if the conveyor wheel (sprocket) radius was much larger, or the radial length of the scraper blades was made much longer (to avoid compressing too much manure). The solid blade scraper design seems to be effective at collecting moist cow dung which cannot be easily collected by teeth on a rake (because wet dung can easily pass through the gaps between rake teeth). Just like other scraping blade designs, some sand and soil would also be collected along with the manure, and these need to be filtered out.

90

S. N. Cubero et al.

Fig. 2 The Barber ‘Surf Rake’ litter collecting machine. Source www.hbarber.com

1.2.2

Rake Conveyor Machines

The ‘Barber Surf Rake’ is perhaps the most popular type of design for collecting small pieces of litter on sandy beaches because rakes tend to act like filters or sieves and do not collect much sand. Each rake only collects large pieces of litter that cannot fit between the gaps of the rake teeth. A demonstration video [5] shows this machine being towed behind a large tractor, collecting litter very reliably at a fairly high mass flowrate. The ‘Surf rake’ uses a ‘chain conveyor’ to move rakes up a steep incline, to transport litter from the ground up to the top of the collection bin behind the conveyor. This machine can also automatically raise and rotate its large collection bin, so it can empty all its contents into a larger collection bucket or garbage bin (or dumpster), and finally return its collection bin to its original position behind the conveyor. This action is similar to how contemporary garbage trucks use hydraulicpowered manipulators to empty large dumpsters and return them to their starting positions (Fig. 2). A similar version of the ‘Surf Rake’ machine, called the ‘Litter picker’ [6], is widely used for collecting small litter in grassy parks. The Barber ‘Turf Rake’ machine is also very similar to the ‘Surf Rake’ but is used for picking up stones. Demonstration videos of these commercially available machines can be found at www.hbarber.com.

1.2.3

Vibrating Filter Machines or Sieves

The Barber ‘SAND MAN’ sand sifter machine [7] operates on a very different concept, unlike the previous rake conveyor-type machines. It first scoops up any litter and sand from the ground using rotating buckets. These buckets elevate the sand and the litter (or all debris) and places them on a vibrating sifting screen, positioned higher above the ground, to sieve the debris and retain objects that cannot pass through the screen (or mesh, which functions as a filter). The debris gradually moves towards a collection bin, due to the cyclic backward movements of the screen, while any sand

A High-Speed Camel Dung Collection Machine

91

and small pieces of dirt falls through the screen to the ground. Theoretically, this vibrating filter design should work well for fairly solid, dry, ball-shaped manure, and warrants future investigation and testing. However, it is very likely that wet or very moist dung will cause problems by blocking the screen (mesh) holes. Also, some of the holes in the filtering screen may occasionally become blocked by tight fitting stones or objects.

1.2.4

Vacuum-Type Litter Collection Machines

Large volumes of litter can be removed from the ground using powerful vacuum suction cleaners mounted on large trucks and steerable vehicles, like the machines shown in [8] and [9]. A handheld, hand-steered, self-propelled and human-guided vacuum cleaning machine is shown in [10], and is effective for small to medium litter collection jobs. While very effective for collecting litter on hard ground, like sealed street surfaces and grassy areas, it would easily collect a great deal of sand and even water, which needs to be removed or prevented from entering the collection bin. The overall effectiveness of this type of collection method for collecting animal dung may be examined in a future project.

1.3 Selection of the Animal Dung Collection Method Several companies manufacture and sell high-speed litter-collection machines, such as BarberTM , HermanesTM and WidontecTM . These machines usually require a large tractor or truck to move them around, and they are quite expensive. For example, the Barber ‘Beach Rake’ model litter collection machine costs approximately $55,000 USD (not including the towing tractor), and is used by many different cleaning companies worldwide to perform large scale litter collection on public beaches. This project, however, is targeted more towards meeting the needs and budgets of local camel and horse ranchers who cannot justify the purchase of very expensive machines and tractors, or who do not wish to hire costly laborers to manually collect animal manure and keep their stables and ranches clean on a regular basis. Such a machine needs to be easy to transport, reasonably affordable (below $10,000) and easy to operate by one person. Therefore, after considering the previous collection methods, this team decided to focus on developing a self-propelled camel and horse dung collection machine based on the BarberTM ‘Surf Rake’ or ‘Litter Picker’ rake conveyor design (but without the collection bin lifting and tipping feature), since it appears to perform reliably on sandy ground.

92

S. N. Cubero et al.

2 Drive Motor Modeling and Selection The main goal of this section is to calculate the required power needed from a driving motor to rotate the two front wheels, assuming ‘worst-case loading’ conditions. The total mass of the entire machine (approximately) is set to mtotal = 100 kg (including a full payload). The maximum mass of the vehicle is 60 kg (approx.) and the maximum payload is 40 kg. Treat the polar mass moments of inertia for all 4 wheels as negligible (Fig. 3). Assume maximum slope of hill or sand dune to climb is: θ = 30° (worst case driving). Force to overcome static gravity on a slope of θ = 30°: Fg = mtotal g . sin θ

(1)

Static gravity force is Fg = 100 × 9.81 × sin (30°) = 490.5 N Approximate drag force of rake teeth pushing through sand under vehicle: Fdrag = 250 N. Drive force needed to (barely) climb a 30° slope: Fdrive = Fg + Fdrag = 740.5 N.

(2)

On flat level ground, maximum linear acceleration: a = (Fdrive /mtotal ) = 7.4 m/s2 .

(3)

Choose maximum linear (forward) velocity: v = 0.2 m/s. Wheel radius: r = 0.2 m (front wheel diameter: d = 0.4 m) Wheel speed: ω = v/r = 0.2/0.2 = 1 rad/s Fig. 3 Modelling the vehicle as a single mass on a constant slope hill

(4)

A High-Speed Camel Dung Collection Machine

93

(= 1 rad/s × 1 rev/(2π rad) × 60 s/min = 9.55 RPM, maximum front wheel speed) Theoretical Power needed for drive motor: Pth = Fdrive v = 740.5 × 0.2 = 148.1 W

(5)

Assume 85% motor and gear efficiency, required motor power: Preq = (Pth /0.85) = 174 W. Torque needed: T = Fdrive r = 740.5 × 0.2 = 148.1 Nm.

(6)

This is the minimum output torque needed for the worm-drive gearbox driving the front two wheels, at a speed of 9.55 RPM. Therefore, we need to select a motor with a continuous power rating >174 W. TRAMECTM (Italy) manufactures and supplies a suitable motor and gearbox unit that has a rated continuous power rating Pmotor = 180 W or 0.18 kW. The front wheels and axle rotate at the same speed as the output shaft of the worm gearbox of this motor (1:1 ratio). A 3 phase AC motor was selected to avoid the wear and maintenance issues associated with brushed DC motors. Its speed is set by a Panasonic VF200 speed controller. (Alternatively, a BLDC motor can be used.)

3 Conveyor Design, Motor Modelling and Selection The manure collecting mechanism must perform several critical functions in a predictable and reliable manner, namely: • • • • • •

Collect as much manure as possible off the ground (faster than a human worker) Remove or exclude sand or dirt (e.g. some kind of filtration process) Transport or elevate the manure to the top of a collection bin or container Deal with shock, or adapt to immovable obstacles (e.g. strong stones, roots) Prevent human injury or accidents Allow for easy maintenance work (replacement of worn parts) and repair work.

McGuire [11] describes the designs of many different kinds of conveyor systems, including ‘Tabletop chain conveyors’. For this dung collection application, the table is inclined at a steep 58° (degrees) to the horizontal, to keep the base of the vehicle (or distance between the front and back wheels) fairly short and compact so the vehicle can make ‘tight turns’, or have a small turning circle. Fayed and Skocir [12] describe how plain chain conveyors are suitable for constant speed operations and only require lubrication and minor maintenance. This may be true for a clean factory environment, however, if a great deal of dirt or sand builds up on the chains and sprocket wheels—as would be expected on a farm, beach or a sandy desert—the chain conveyor could eventually encounter much more friction, require

94

S. N. Cubero et al.

more motor torque, and eventually fail to rotate. Therefore, to keep the conveyor chains and sprocket wheels as clean as possible, and to prevent human injury, clear plastic covers (made of transparent acrylic sheet) were designed and built to cover the entire conveyor area, including all power transmission elements of the machine. This is a very important requirement to satisfy the ‘safety regulations’ or ‘safety standards’ for most countries. The following discussion describes how to calculate the approximate required power needed for the driving motor to rotate a ‘chain conveyor’ that moves animal dung from the ground to the collection bucket, while also rotating all elements of the conveyor system (i.e. both conveyor chains, all rakes and all rotating conveyor shafts and sprockets/wheels). We will also calculate the ‘effective’ or ‘reflected inertia’ of the entire conveyor system and (maximum possible) payload of the dung at the ‘drive shaft’ (at position A, as shown in Fig. 4). The existing design of the conveyor system for the first prototype is shown in Fig. 4 above. The ‘straight’ rake teeth are aligned in a row, spaced approximately 20 mm apart. This spacing distance between each tooth can be reduced to collect smaller sized dung pellets if necessary, however, this will result in greater drag force in the sand and a heavier rake, hence, more power would be required from the conveyor motor. The rake teeth are made of round steel (similar to thick nails, but approximately 2 mm in diameter), each welded onto an UA (Unequal Angle) or L-shaped rake bar which is bolted to a conveyor chain at each of its ends. The ‘rake bar’ is approximately 980 mm wide. The 2 conveyor chains are similar to bicycle chains but larger, and wrap around the sprocket wheels at positions A, B, C and D. The left-side and right-side chains both travel at the same speed, driven by 2 sprocket wheels located on opposite ends of a long shaft, located at position A. Animal dung is quite lightweight (and density varies based on moisture content), so for a fully-loaded conveyor, assume that the ‘worst case’ total mass of the animal dung being carried up the straight 58° slope is mpayload = 5 kg (assuming all ascending

Fig. 4 Conveyor design for collecting animal dung (Each sprocket wheel radius is r = 60 mm)

A High-Speed Camel Dung Collection Machine

95

rakes are full). The masses or inertias of the 21 ‘L-shaped’ UA sections (‘Unequal Angle’ solid steel bars) that are used for the rake bars that hold all the teeth, is significant, because each rake bar, 2 connection bolts and 4 nuts, and all straight teeth weighs approximately 1.4 kg. Two different types of teeth are shown in Fig. 5. For the first prototype, all the rake bars were made with ‘straight teeth’. In fact, the largest load on the chain conveyor system will be caused by the ‘drag force’ of the teeth being pulled through the sand, beneath the machine. This drag force was assumed to be 250 N (Fdrag ). Drag force through the sand can be experimentally determined by measuring the force on a load cell or a ‘spring balance’ (that measures tension force) in series or inline with a pulling rope connected to the front of the prototype vehicle (shown in Fig. 9), as the vehicle is being pulled across level sandy ground. Fdrag is proportional to the number of teeth (or number of rakes) dragging through the sand underneath the vehicle. Dragging more than one row of teeth through the sand may not even be necessary, although it increases the chances of collecting manure that happens to pass through the first rake. The total torque needed for the driving motor to rotate the shaft at position A, and to rotate the entire conveyor system can be calculated from this dynamics equation: Ttotal = Tdynamic + Tconstant

(7)

where Tdynamic is the torque needed to accelerate the entire system from rest up to the top speed, in a given time t, and Tconstant is the sum of all persistent or constant torques that need to be overcome, such as torque due to gravity loading of the payload (under worst case loading conditions) Tgravity , drag torque Tdrag created by the total force of all the rake teeth between B and C being pulled through the sand, and the sum of all other continuous friction torques, Tfriction . The torque due to gravity loading of the rake bars will be negligible because the conveyor system is almost perfectly

Fig. 5 ‘Straight teeth’ and ‘Bent teeth’ rake designs for the conveyor

96

S. N. Cubero et al.

‘balanced’, having the same number of rakes on the ‘ascending’ side (going up the 58° ramp, between positions B and A) as on the ‘descending’ side (between positions A and C). For simplicity, let us assume Tfriction is negligible and focus on calculating the Tgravity and Tdrag torques as ‘seen’ at the drive shaft at position A (which will be rotating at the same speed as the output shaft of the gear box connected to the driving motor that drives the entire chain conveyor system). Each conveyor wheel (or sprocket) shown in Fig. 4 has a radius r = 60 mm. Tconstant = Tgravity + Tdrag Tgravity = mpayload g . sin(58◦ ) . r = 5 × 9.81 × sin(58◦ ) × 0.06 = 2.5 Nm Tdrag = Fdrag . r = 250 × 0.06 = 15 Nm

(8) (9) (10)

Tconstant = Tgravity + Tdrag = 2.5 + 15 = 17.5 Nm Assuming 21 rakes, each weighing 1.4 kg, and spaced approximately 144 mm apart, Tdynamic = Jr e f α Jr e f =

13  mi r 2 i

i=1

2

+ 21m rake rw2

(11)

(12)

where J ref is the ‘Reflected inertia’ or ‘equivalent inertia’ of the entire conveyor system (including all rotating components) as ‘seen’ by the shaft at A, where the output of the motor gearbox equals the speed of the shaft at A. Note that ‘i’ is the index number for all solid rotating bodies rotating at each position in Fig. 4 (namely, positions A, B, C and D), mi is the mass of the cylindrical shaped body (which could be a sprocket wheel, or a 1 m long shaft), and r i is the radius of that round solid. The mass of each rake is mrake = 1.4 kg, and its radial distance to the axis of rotation is rw = 0.06 m. This equation above is only approximate since it does not consider the ‘centers of mass’ for each tooth on every rake, because mrake is treated like a point mass in-line with the chain. Also, the inertias of the two long chains (on the left and right-hand sides) are considered negligible in this calculation. Also note that all rotating masses on the conveyor system are rotating at the same angular velocity and angular acceleration because there are no gear reductions (no changes in torque nor rotational speed) from one shaft to another. All rotating sprockets and shafts are rotating at the same speed as the output shaft of the gearbox unit connected to the drive motor at A. For more details about the derivation of the general equation for ‘reflected inertia’, also known as ‘equivalent inertia’, refer to Klafter et al. [13], ‘Robotics Engineering—An Integrated Approach (Table 1).’

A High-Speed Camel Dung Collection Machine

97

Table 1 Mass moments of inertia for all rotating sprockets and shafts of the conveyor (approx) I. No.

1

2

3

4

5

6

7

8

9

10

11

12

13

mi mass

0.9 kg

0.9

0.9

2.6

0.9

0.9

2.5

0.9

0.9

2.5

0.9

0.9

2.5

ri radius

0.06 m

0.06 0.06 0.01 0.06 0.06 0.01

0.06 0.06 0.01

0.06 0.06 0.01

mi r2i /2 × 1.62 kg.m2 1.62 1.62 0.13 1.62 1.62 0.125 1.62 1.62 0.125 1.62 1.62 0.125 10−3

J ref = 15.1 kg.m2 (approximately) for the entire conveyor, as ‘seen’ at the shaft at A. α is the constant angular acceleration of the conveyor system to accelerate from rest up to maximum speed within a given time t. Assume that the maximum speed ωmax = 0.5 rev/s = π rad/s = 3.14 rad/s must be reached within a maximum time t = 3 s, then the maximum constant angular acceleration of the conveyor α = ωmax /t = 1.05 rad/s2 . It is important to check that the speed of the rake teeth relative to the ground is faster than the maximum forward velocity of the vehicle, or the rake rotation may be too slow to pick up the animal dung, causing the dung to ‘pile up’ or build up in front of the vehicle. The length of the ‘straight’ tooth is 60 mm, so when this is added to the sprocket wheel radius of 60 mm, the distance from the axis of rotation to the tooth tip is approximately 120 mm, or rtooth = 0.12 m. Tangential velocity of the tooth tip is v = r ω, so the tooth will be traveling at v = 0.12 × 3.14 = 0.377 m/s, which is still higher than the vehicle’s top speed of 0.2 m/s (almost double). Therefore, the operating speed of 0.5 rev/sec is suitable as the top operating speed for the conveyor system. Assuming the entire front face of the conveyor is fully loaded, with 5 kg of wet manure (i.e. all front rakes are fully loaded), we can find the fastest possible manure collection rate. • • • •

Distance travelled along front face of conveyor from B to A is d = 1.15 m Maximum linear conveyor speed v = r ωmax = 0.06 π rad/s = 0.19 m/s Speed v = Distance/Time = d/t, so t = d/v = 1.15/0.19 = 6.05 s Mass flow = Mass/Time = 5 kg/6.05 s = 0.826 kg/sec or 49.6 kg/min.

The total dynamic torque needed to accelerate the entire conveyor from rest to top speed can now be calculated as: Tdynamic = Jr e f α = 15.1 × 1.05 = 15.9 Nm. Total required motor torque is Ttotal = Tdynamic + Tconstant = 15.9 + 17.5 = 33.4 Nm. Theoretical motor power required to drive the conveyor is Pth = Ttotal ωmax = 33.4 × 3.14 = 104.9 W

(13)

98

S. N. Cubero et al.

This is the minimum torque needed from the motor and gearbox driving the shaft at A. Assume 85% motor and gear efficiency, required motor power: Preq = (Pth /0.85) = 123.4 W, therefore, the motor for driving the conveyor system can be a lot smaller than the vehicle driving motor. In order to avoid piling up manure, which would increase drag force on the vehicle, the conveyor operating speed should always be higher than 1.67 rad/s when the vehicle is going at the maximum forward speed of 0.2 m/s.

4 Steering Mechanism Kershaw and Van Gelder [14] describe different kinds of steering systems used for steered vehicles and contemporary automobiles. To keep the design as simple as possible, the entire vehicle is manually steered in a manner that is similar to how a boat is steered with a rudder. Figure 6 shows a ‘Top view’ diagram of the steering shaft (at the center) which moves connecting rods that rotate the rear wheels. To turn the vehicle right, the operator rotates the steering shaft clockwise using a handle, so the two (unpowered) rear wheels point left, and vice versa for turning the vehicle left. This is known as a ‘parallelogram steering mechanism’ and is only recommended for low-speed applications. High vehicle speeds would cause significant tyre (or wheel) scrubbing, or lateral wheel dragging, because the rear wheels need to point in a direction that is tangent to its appropriate ‘turning circle’ which is centered on the point of rotation (as seen in a ‘top view’). To avoid wheel scrub (which can cause significant tyre wear and high ground friction), it is better to use an ‘Ackermann’ steering

Fig. 6 Manual steering mechanism at rear of the vehicle

A High-Speed Camel Dung Collection Machine

99

Fig. 7 Photo of steering mechanism from under the vehicle

mechanism [15, 16] which points both wheels tangent to their own turning circles, which happen to be concentric, centered on the same point of rotation. However, because this vehicle is designed for use mainly on sandy ground, like beaches or in the desert, the ‘parallelogram’ steering mechanism was considered adequate for this environment (Fig. 7).

5 Field Test Results 5.1 Real-World Test Runs The first prototype (based on the design in Fig. 4) had a dismal success rate of about 10% at collecting animal dung off sandy ground. i.e. For every 10 pieces of dung that the vehicle drove over, usually one or no pieces were transferred to the conveyor. Horse and camel dung tended to move forward and roll off the ‘straight teeth’ on each rake. One of the big problems encountered was the problem of new or moist dung sticking to the rake teeth (which had to be manually removed by hand, with a cloth or paper towels). Pieces of dung that stick to a rake can easily fall off or return to the ground when they are dragged under the vehicle. This problem can be partially solved by adding a ‘cleaning rake’ to act as a filter or a ‘comb’ for the conveyor rakes, to push off any sticky dung, allowing it to roll into the collection bin, as shown in Fig. 8. For example, the ‘cleaning rake’ may be oriented perpendicular to line AB at Position A, with its teeth almost tangent to the wheel at A, at the very end of the conveyor, as shown in Fig. 4, so that its teeth can slide through the gaps of the moving teeth on the conveyor, and can direct any dung to roll down into the collection bin.

100

S. N. Cubero et al.

Fig. 8 Cleaning rake can force animal dung into the collection bin

As shown in Fig. 4, the ‘Stationary curved ramp’ was added to help catch pieces of dung and stop them from falling off the straight teeth of the rake. Without this curved ramp, the dung pieces kept falling or rolling forward off the teeth. However, because the lowest part of the curved ramp had to be as low as possible to the ground to prevent pieces of dung falling forward off the teeth, the curved ramp ended up blocking and piling up dung pieces in front of the vehicle, in a similar manner to a snow plough pushing snow forward, except sand is piled up. The curved ramp was an impediment or obstacle for the pieces of dung, and was not a good solution to the problem of dung pieces falling off each rake. Numerous field tests proved that the ‘straight tooth’ rake and ‘curved ramp’ idea was unsuccessful and unreliable for collecting dung pieces and transferring them to the conveyor in a reliable manner. A feasible solution to this problem will be discussed in the next section.

5.2 Possible Improvements to the Rake Design A feasible solution to stop dung pieces from falling off the teeth was proposed by the first author. Instead of using forward-moving straight teeth that tend to flick any dung forwards, bent teeth can be used on all rakes to scoop up dung and stop any pieces from falling off the rake near position B (See Figs. 5 and 10) so that a ‘curved ramp’ is no longer necessary. As seen in Fig. 10, the bent teeth on each rake can act like a filtering ‘scoop’ and can hold the dung pieces in place before they begin ascending up the steep ramp to the collection bin. A more reliable, simpler, lower-cost design for the animal dung collection machine is shown in Fig. 10c. The ‘rotating rake’ shown in this diagram consists of 4 straight-tooth rakes attached to a rotating shaft. It serves to push out any dung that may cling to the bent teeth on the moving conveyor rakes, in a similar manner to the ‘cleaning rake’ shown in Fig. 8. This design offers many advantages over the design of the existing conveyor shown in Fig. 4, namely: • It requires far fewer shafts and sprockets (i.e. simpler design, less material needed) • It uses far fewer rakes because the conveyor chains are much shorter (hence, much less cost and setup time needed during manufacture)

A High-Speed Camel Dung Collection Machine

101

• Much fewer components means far less weight and less manufacturing cost • It does not create as much friction or drag force with the ground (because not many teeth are in contact with the ground—i.e. Only one rake is dragging through the sand, unlike the many rakes seen in Fig. 4, therefore, Tdrag or Fdrag would be much lower, so a much smaller conveyor motor may be used. • There is no need for any curved ramp (which doesn’t help with collection anyway!) • Less space is needed for the conveyor, so the collection bin can be much larger. Unfortunately, this potentially better solution – using ‘bent teeth’ as shown in Figs. 10a, b—was too late to implement by the time the project was completed, so this is left as future work. At the back of the machine (see Fig. 9) is the steering handle which rotates the steering shaft for turning the back wheels. In the side view, the lower motor and worm-drive gearbox is shown connected to the front axle using a chain, for driving the front two wheels. The upper motor also has a worm-drive gearbox for rotating the conveyor system, driven by the top shaft. In the back view, Fig. 9b, near the left wheel, is a 12 V DC car battery which is connected to a 12 V DC to 240 V AC power converter. This powers the 2 AC motor speed controllers shown in the bottom right corner. There are 2 ‘On-Off’ switches for activating or de-activating each AC motor in the upper right-hand corner of Fig. 9b. A video showing the KU prototype in action is available for viewing at [17]. The total cost of all parts and materials to build the prototype in Fig. 9 comes to approximately $3000 USD or about 11,000 AED. Using flexible rubber or plastic material for the ‘bent teeth’ could improve each rake’s effectiveness at picking up small pieces of litter on hard, rocky or rough ground surfaces. Alternatively, if each bent tooth, or each rake was ‘spring-loaded’ to allow backward flexing, the conveyor would perform much better on hard ground or on rough surfaces, because the tips of the rake teeth would be able to follow the ground’s contours or rub against the ground, thus preventing high resistance collision forces

Fig. 9 a Side view and b back view of the animal dung collector prototype built at KU

102

S. N. Cubero et al.

and large gaps under the rake (which could miss some pieces of dung). These kinds of improvements will help to significantly reduce or prevent shock loads and very high drag forces caused by rigid rake teeth getting caught on hard immovable objects, or grass roots, while improving the effectiveness of manure collection. Therefore, if the improvements mentioned earlier are implemented, and the parts costs can be reduced by about half, this design could become a commercially feasible product if it is mass produced on a large scale.

6 Conclusions This paper described the design, engineering analysis, performance results, and observed problems and their potential solutions for a camel and horse dung collection machine. This paper described the current ‘state of the art’ in litter and dung collecting machines, motor modelling and selection for vehicle locomotion, steering mechanics and motor modelling and selection for the chain conveyor mechanism. Animal dung collection was examined and analyzed briefly. At present, the current prototype is unable to reliably collect round clumps of camel dung off the ground, mainly because of inherent problems with the ‘straight teeth’ rake design. With a few improvements to the rake design, and some modifications—i.e. implementing flexible or spring-loaded ‘bent teeth’—this machine could become very reliable at collecting camel and horse dung. A future version of this machine that uses the conveyor design shown in Fig. 10c could be built to keep manufacturing costs as low as possible.

Fig. 10 a and b A ‘Bent tooth’ design; c A simplified lower-cost conveyor design

A High-Speed Camel Dung Collection Machine

103

References 1. Hansson, B. O. (1978, Oct 25). Vehicle carried manure removing machine. U.S. Patent 4,289,439. 2. Owen Vinyard (1994, Mar 29). Rear-mounted manure gathering machine and method of handling manure. U.S. Patent 5,297,745. 3. OldManStino. Best Way to Collect Cow Manure. (July 1, 2018). Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/598HG4M0kuQ. 4. Vibhute, R. (2017 Sep 2). Cow Dung collecting machine. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/9SkNVRLZoMI. 5. Barber, H. & Sons. (2012, Dec 21). USA. Hurricane Sandy Beach Cleanup with Beach Cleaning Machine. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/NyyRmrdI_-M. 6. Barber, H. & Sons (2016, July 1). USA. Litter Picker Machine for Event Cleanup. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/REY7eP8ewyY. 7. Barber, H., & Sons. (2011, Oct 27). USA. SAND MAN: Barber’s Walk-Behind Beach Cleaner. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/6J96qxqIQn0. 8. Awesome inventions. (2018, Mar 12). Meet ‘Hermanes’—The giant street cleaning vacuum. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/RABRJgNuuUw. 9. Widontec. (Oct 20 2014). Widontec MC3 self-propelled trike vacuum cleaner. Accessed Sep 11 2018. (Online Video). Available: https://youtu.be/vczNjC5atpU. 10. Chau, H. (May 14 2017). E-Vacuum Machine/ Litter picker. Accessed 11 Sep 2018. (Online Video). Available: https://youtu.be/MfmW8PRlqvY. 11. McGuire, P. M. (2009). Tabletop Chain Conveyor. In Conveyors: Application, Selection, and Integration (Systems Innovation Book Series) 1st ed., USA: CRC Press, pp. 17–23. 12. Fayed, M. E., & Skocir, T. S. (1996). Chain-Type Conveyors. In Mechanical Conveyors: Selection and Operation, USA: CRC Press, pp. 303–304. 13. Klafter, R. D., Chmielewski, T. A., Negin, M. (1989). Mechanical systems: Components, dynamics, and modeling. In Robotic Engineering: an integrated approach. Englewood Cliffs, USA: Prentice-Hall, pp. 119–124. 14. Kershaw, J., & VanGelder, K. (2017). Steering systems. In Automotive Steering and Suspension (Master Automotive Technician) Kindle Edition, USA: Jones & Bartlett Learning, pp. 365–366. 15. Wilson, C. E., & Sadler, J. P. (2003). Mechanisms for specific applications. In Kinematics and Dynamics of Machinery 3rd ed., India: Pearson India Education Services, pp. 75–77. 16. Norris, W. (1906). Steering. In Modern Steam Road Wagons (Ed.), London (pp. 63–67). UK: Longmans Green & Co. 17. Alali, M. (April 30, 2019). Animal Dung Collecting Machine. Accessed 23 Oct 2018. (Online Video). Available: https://youtu.be/vrpnm_3tm8g.

Discussion of Soft Tissue Manipulation for the Harvesting of Ovine Offal Qi Zhang, Weiliang Xu, Zhisheng Zhang, Martin Stommel, and Alexander Verl

1 Introduction Soft tissues exist in a wide range of fields, including industrial, domestic, medical and food applications, and their robotic manipulations have attracted a lot of attention in recent years. There are many challenges in soft tissue manipulation, which includes: (a) 3D soft tissue modelling requires very high computational cost; (b) the shape of soft tissue changes significantly during manipulation; (c) the interaction between soft tissue and manipulator is very complicated; (d) strategies and techniques of rigid objects manipulation cannot be applied directly to soft tissue. The applications of soft tissue manipulation mainly include medical assistance, surgical suturing, meat cutting and food harvesting. Ovine offal harvesting is one of the significant studies in soft tissue manipulation because ovine offal is a major export co-products of meat processing. In New Zealand, ovine offal made up a third of the total edible offal, with 22,184 tones and worth 63 Q. Zhang · W. Xu (B) Department of Mechanical Engineering, The University of Auckland, Auckland, New Zealand e-mail: [email protected] Q. Zhang e-mail: [email protected] Z. Zhang School of Mechanical Engineering, Southeast University, Nanjing, China e-mail: [email protected] M. Stommel Department of Electrical and Electronic Engineering, Auckland University of Technology, Auckland, New Zealand e-mail: [email protected] A. Verl Institute of Control Engineering (ISW), Stuttgart University, Stuttgart, Germany e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_9

105

106

Q. Zhang et al.

million dollars [1]. At present, ovine offal is harvested manually. The market volume is limited due to the low efficiency and high cost of manual labour. Plants generally halted offal collection, especially the lower-value products, when the labour is scarce. This phenomenon also happens in Australia, with a yield of only 63–91% even for the higher-value organs such as heart, liver and kidney [2]. Existing robotic systems for organ sorting mainly are used for organ harvesting of small animals such as poultry [3–5]; they cannot be used for large animals such as sheep due to their size and mechanical properties. The automatic processing of large animal carcass is after the removal of the organ package as a whole [6]. There is currently no automatic system for harvesting lamb organs. Therefore, it is necessary to develop a robotic system that can automatically and efficiently sort internal sheep organs and study its manipulation. In this paper, the requirements for robotic sorting of ovine offal are analyzed in Sect. 2. Section 3 introduces the current practices in the animal slaughtering and processing industry. Section 4 proposes a robotic system for ovine offal sorting and summarizes the problems to be solved. The potential solutions for the above problems are discussed in Sect. 5. Section 6 gives the conclusion and future work of ovine offal harvesting.

2 Requirements Analysis The aim of this research is to use a robotic system to manipulate and separate the internal organs of sheep (Fig. 1). We use “organ package” to represent the internal sheep organs that need to be manipulated. It contains mainly heart, lungs, liver, stomach (i.e. rumen, abomasum, omasum and reticulum), spleen, gallbladder and intestines with the folded structure, connected by blood vessels and connection tissues. We assume that the organ package has been put on the table manually or automatically. The robotic system first needs a machine vision to capture the configuration of the organ package. The removal device is then needed to cut and remove individual organs. The sorting device is also required to order the configuration of the organ package. This is because the organ packages are generally unordered and have

Organ package

Heart

Lung

Liver Spleen

Fig. 1 Schematic diagram of the research objective

Discussion of Soft Tissue Manipulation for the Harvesting …

107

a spatial configuration (i.e. the organs may overlap/occlusion), making it impossible for machine vision to recognize organs directly. The manipulation devices (including sorting device and removal device) should satisfy the following requirements: (a) The diameter of the table is about 600– 1000 mm, depending on the size of the organ package; (b) The payload of the devices is about 5–20 kg, depending on the weight of the organ package; (c) The requirement of position accuracy is generally about ±3 mm; (d) The devices can be washable and withstand high temperature and high-pressure cleaning; (e) The components of the device such as grease, lubricant, tools and fixture materials should be food grade. At the same time, the sorting of ovine offal should satisfy some special needs of the food industry. Soft tissue manipulation requires high hygiene requirements in food industries because of their susceptibility to bacterial contamination. In addition, quality control is another important requirement for this industry. This because soft tissues are relatively fragile and are easily damaged when they come into contact with the manipulator. The unique environmental conditions of soft tissue manipulation should not be neglected. A manipulation environment with the appropriate temperature, humidity and pressure are required. The manipulation tasks in ovine offal sorting include adjusting the configuration of the organ package, grasping the single organ, cutting off the connection tissue and removing the single organ. The purpose of adjusting the configuration of the organ package is to expose the overlapping organs for identification and to adjust the recognizable organs to a suitable position for grasping and removal.

3 Current Practices In the animal slaughtering and processing industry, most of the processing, such as carcass-splitting, deboning and packaging, can be performed by robotic systems, although the system has some shortcomings. Applications of robotic systems in the animal slaughter and processing industry are shown in Fig. 2. Figure 2a shows carcass-splitting automation using a 6-DoF robot arm, the manipulation task cutting off the connection tissue in ovine offal sorting can be achieved using the similar robotic system. Figure 2b shows a robotic chicken deboning system. Figure 2c shows a robotic system for auto-shacking of poultry, the manipulation task grasping single organs and removing single organs in ovine offal sorting can be achieved using the similar robotic system.

4 Proposed System and Its Operation The proposed system consists of three parts, (A) machine vision, (B) sorting device and (C) dual-arm robot (Fig. 3). The machine vision is used to identify the types and positions of organs and guide the action of the robotic system. The sorting device is

108

Q. Zhang et al.

Fig. 2 Current practices in the animal slaughtering and processing industry [6]

Fig. 3 The concept of a robotic system for ovine offal harvesting. (A) machine vision, (B) sorting device, (C) dual-arm robot: (C1 ) a vacuum gripper and (C2 ) a specific cutting device

used to order and manipulate the organ package, adjusting its pose to expose single organs. It can be a rotation table, a peristaltic table [7–10], a shaking platform [7, 8], a robotic arm, or a soft machine table [11]. The dual-arm robot is used to cut and remove single organs. One end effector of the dual-arm robot is (C1) a vacuum gripper with many small suckers, and the other is (C2) a specific cutting device for soft tissue. The removal process involves lifting the identified single organ first with the vacuum gripper and then cutting the connection tissue of this single organ with

Discussion of Soft Tissue Manipulation for the Harvesting …

109

Fig. 4 The operating process of a robotic system for ovine offal harvesting

cutting device, and finally placing this separated organ to the designated position with the vacuum gripper again. In this paper, we propose a partial approach to separating the organ package. We mainly focus on four organs, the heart, lungs, liver and spleen. Figure 4 shows the process of the robotic system harvesting sheep internal organs. The operation begins with the placement of the unordered package on the table. At the same time, the camera begins to capture images of the organ package and then judges the configuration of the organ package. If there is a separable organ (i.e. can be recognized and separated), the organ will be removed by the dual-arm robot. If there is a recognizable organ (i.e. can be recognized but not separated), the organ package will be manipulated by the sorting devices until this organ can be separated. If there is no organ can be recognized, select an organ with distinct features (i.e. the most recognizable feature) and manipulate the organ package with the sorting devices until this organ becomes separable. The next step is to repeat the above judging process until all the organs are separated. The final step is to clean up the table and start a new process. This step can also be executed if the organs cannot be detected after a long unsuccessful manipulation. There are many problems that need to be solved in ovine offal sorting. 1. Mechanics modelling of the organ package. It is difficult to build the organ package model because of the complexity of its components. The organ packages are generally unordered, with the organ having an irregular surface shape, special microstructure (e.g. the heart has a unique laminar structure, and the lungs contain alveoli) and physical loading properties (e.g. stress and strain). The connection tissues are located in different parts of different organs and have different physical loading properties. 2. The contact between the organ and the manipulator. The softness and stickiness of the organs complicate the contact between the organ and the manipulator. The surface of the organ is distributed or covered with fatty tissue, mucus, and membranes, which affect the manipulation and grasping of the organ. 3. The estimation of the organ package deformation. Different organs have different behaviors under the external force. They interact with each other during manipulation by connecting tissues. Some organs like the intestines and stomach are

110

Q. Zhang et al.

Fig. 5 Single organ manipulation

filled with intestinal or digestive fluids and partially digested food debris. It is difficult to estimate the organ deformation required for manipulation control with high accuracy. 4. The control of soft tissue manipulation. There is an indirect simultaneous localization problem [12] in the control of soft tissue manipulation, which is multiple points called control points on an object should be manipulated to their desired points simultaneously (Fig. 5). The control points are difficult to manipulate directly. Therefore, the other points called manipulation points on the object surface are needed to realize the manipulation. The manipulation points are the contact points between the manipulator and the object, and the control points are the points on the object that needs to be controlled.

5 Discussion 5.1 Mechanics Modelling of the Organ Package There are three modelling methods available for soft tissue modelling: finite element method (FEM), mass-spring-damper (MSD) method, and reproducing kernel particle method (RKPM). FEM is a numerical method to solve engineering and mathematical physics problems. It is prone to error when modelling complex objects. Moreover, its solution accuracy is low when analyzing large deformed objects. It is difficult to meet the requirements of high precision and high efficiency at the same time. MSD method describes the object by a set of points with mass, connected by springs or dampers or their combination. The method can well describe the viscoelastic behavior of the object and has higher computational efficiency than FEM. However, the MSD method has several disadvantages. The parameters of these models are difficult to estimate. The MSD model has systematic errors and

Discussion of Soft Tissue Manipulation for the Harvesting …

111

localization of the deformation for a complex network. It is difficult to handle large deformations of objects using the MSD method. RKPM is a mesh-free method using a correction function in an integral transformation to impose reproducing conditions. This method can accurately model extremely large deformation without the mesh distortion problem because its computation does not require an explicit mesh. Adaptive modelling can be easily accomplished by changing particle definitions for the desired refinement without re-meshing. Compared with FEM, the non-uniformity of the RKPM node spacing does not cause the irregular shape of the mesh. Consequently, it has higher solution accuracy under large deformation. RKPM has a higher efficiency than FEM for handling large material deformation due to its smooth shape function. At present, RKPM is mainly used to build the model of 2D objects, and its application in 3D objects can be studied in the future. In this proposed system, FEM will be used to build the mechanics model of a single organ, MSD method will be used to establish the mechanic model of connection tissue, and the combination of FEM and MSD method will be used to establish the mechanics model of organ package.

5.2 The Contact Between the Organ and the Manipulator Different handling devices used to manipulate soft tissues has different contact types and contact forces, which can result in different deformation mechanics of soft tissues. For example, the finger has a point contact and creates a point force when it contacts with the soft tissue. The table has a surface contact and creates surface forces. Multi-finger manipulator is the most commonly used manipulator in soft tissue manipulation. We can extend the fingertip contact theory [13] to soft tissue manipulation. Soft-finger contact refers to any external forces and pure torques that can be exerted at the contact point as long as its direction inside or on the friction cone. The soft-finger contact can be used as a contact between the rigid finger and the soft tissue. The contact part changes during the manipulation. For instance, the contact between the finger and the object changes from a point to a set of points (Fig. 6), some of which stick to the tip and some of which slide on the tip [14]. Therefore, it is necessary to build the contact model between the organ and the manipulator. The purpose of this model is to compute the manipulative force and velocity applied in soft tissue manipulation. In the proposed system, MSD method will be used to build the contact model of fabricated organs. The concept of the model is shown in Fig. 7, which the unknown MSD model is fixed at one end and connected to the force applied by the manipulator at the other end.

112

Q. Zhang et al.

Fig. 6 Object a before and b after two-finger grasp [14]

Fig. 7 The concept of the contact model

5.3 The Estimation of the Organ Package Deformation Soft tissue satisfies the energy balance during the manipulation process [15], expressed by the equation W p = Wk 

 δu i, j τi j d − x

x

W f − Wd − Wb = Wk   δu i f i d − δu i h i d = δu i ρ u¨ i d h

x i

(5)

x

which is the potential energy W p consumed during organ manipulation is equal to the kinetic energy Wk produced. The potential energy contains the work generated by the external force W f and the stored energy in its deformation Wd when it has no boundary constraint. The work generated by the boundary constraint Wb is added to

Discussion of Soft Tissue Manipulation for the Harvesting …

113

the potential energy when the organ is constrained. The estimation of organ deformation can be achieved by minimizing its potential energy, which is computed from the static equilibrium [16] equation ∂ W p /∂u = 0

(6)

where u is the displacement of the point on the organ. In the proposed system, we extend this theory to organ package manipulation to obtain the transfer of energy between organs and connection tissues. When the same external force is applied to the organ package and an individual organ respectively, the transfer energy between organs and connection tissues can be obtained by the organ0 organ0 and stored energy Wd in the deformation of an individual kinetic energy Wk organ organ and stored energy Wd in the deformation organ minus the kinetic energy Wk of organ package. This is expressed by trans f er

Wd

organ0

= Wk

organ0

+ Wd

organ

− Wk

organ

− Wd

(7)

5.4 The Control of Soft Tissue Manipulation There are generally two kinds of control methods for soft tissue manipulation: modelbased control method and model-free control method. For the model-based control method, the controller design is based on the error between the desired deformation and the feedback and the model error between the theoretical deformation calculated by the soft tissue mathematical model and the feedback. For the model-free control method, the controller design is based on the error between the expected deformation and the feedback deformation and the estimated deformation Jacobian matrix obtained by the estimator. The most common feedback in soft tissue manipulation is the deformation of control points, called the deformation feature. The description of the deformation feature is very important for the model-free control method because the control is realized based on these deformation features. The types of deformation feature include points, distances, angles, curvatures, contours and surfaces. In the proposed system, we prefer to use the model-based control method, because it is difficult to recognize the deformation feature of organ package during manipulation. The sensing system includes machine vision for detecting shapes or selecting manipulation points to guide the action of the robot and force/tactile sensors for detecting shapes or contact conditions to supplement contact information between the robot and the object. The sensing system is used to verify the models of soft tissue manipulation in model-based control method and to detect the deformation features in model-free control method. The visual sensor is the major detection equipment in this paper, and the tasks of it are model verification, parameters identification, shape estimation and tissue/organs classification. Existing image processing methods can apply in the proposed system.

114

Q. Zhang et al.

6 Conclusion There is currently no automatic system available for ovine offal sorting. This paper analyzed the requirements of robotic sorting system of ovine offal from the aspects of system composition, equipment requirements and food industry requirements. Current practices of the animal slaughtering and processing industry were described. We then introduced a robotic system including machine vision, sorting device and dual-arm robot and introduced its manipulation process. The problems to be solved in ovine offal sorting were summarized, which includes mechanics modelling of the organ package, the contact between the organ and the manipulator, the estimation of the organ package deformation and the control of soft tissue manipulation. The potential solutions to these problems were discussed by reviewing the existing methods and theories of soft tissue manipulation. We plan to use the combination of FEM and MSD method to build the mechanics model of the organ package. MSD method will be used to build the contact model between the organ and the manipulator. The energy balance theory and its extension will be used to estimate the deformation of the organ package. Model-based control method will be used to implement the manipulation of organ package efficiently. Our future work is to complete the first step of the ovine offal sorting, which is manipulating the connecting organs from the initial configuration to the desired configuration. Acknowledgements The research reported in this paper is supported by the Royal Society of New Zealand. The first author acknowledges the provision of a doctoral scholarship from the China Scholarship Council (CSC).

References 1. MIA. Annual Report 2018. (2018). Meat Industry Association MIA (Trade Association representing New Zealand meat processors, exporters and marketers). 2. Meat Technology Update. (2008). CSIRO Food and Nutritional Sciences: Meat Industry Services. 3. Jansen, T. C., & Spijker, R. (2011). Method and apparatus for mechanically processing an organ or organs taken out from slaughtered poultry. U.S. Patent No. 20110244773 A1. 4. Jansen, T. C., & Spijker, R. (2012). Method and apparatus for mechanically processing an organ or organs taken out from slaughtered poultry. U.S. Patent No. 8303383B2. 5. Jansen, T. C., & Spijker, R. (2015). Method and apparatus for mechanically processing an organ or organs taken out from slaughtered poultry. U.S. Patent No. 9004987B2. 6. Choi, S., Zhang, G., Fuhlbrigge, T., Watson, T., & Tallian, R. (2013). Applications and requirements of industrial robots in meat processing. In International Conference on Automation Science and Engineering (CASE) (Vol. 2, pp. 1107–1112). 7. Stommel, M., Xu, W. L., Lim, P. P. K., & Kadmiry, B. (2014). Robotic sorting of ovine offal: Discussion of a soft peristaltic approach. Soft Robot, 1(4), 246–254. 8. Stommel, M., Xu, W. L., Lim, P. P. K., & Kadmiry, B. (2015). Soft peristaltic actuation for the harvesting of ovine offal. In Robot Intelligence Technology and Applications 3 (Advances in Intelligent Systems and Computing) (Vol. 345, pp. 605–615).

Discussion of Soft Tissue Manipulation for the Harvesting …

115

9. Stommel, M., & Xu, W. L. (2016). Optimal, efficient sequential control of a soft-bodied, peristaltic sorting table. IEEE Transactions on Automation Science and Engineering, 13(2), 858–867. 10. Stommel, M., & Xu, W. L. (2016). Learnability of the moving surface profiles of a soft robotic sorting table. IEEE Transactions on Automation Science and Engineering, 13(4), 1581–1587. 11. Deng, Z., Stommel, M., & Xu, W. L. (2016). A novel soft machine table for manipulation of delicate objects inspired by caterpillar locomotion. IEEE/ASME Transactions on Mechatronics, 21(3), 1702–1710. 12. Henrich, D., & Wörn, H. (2000). Robot manipulation of deformable objects. London: Springer. 13. Nguyen, V. D. (1988). Constructing force-closure grasps. The International Journal of Robotics Research, 7, 3–16. 14. Guo, F., Lin, H., & Jia, Y. B. (2013). Squeeze grasping of deformable planar objects with segment contacts and stick/slip transitions. In International Conference on Robotics and Automation (ICRA) (pp. 3736–3741). 15. Chen, J. S., Pan, C., Wu, C. T., & Liu, W. K. (1996). Reproducing kernel particle methods for large deformation analysis of nonlinear structures. Computer Methods in Applied Mechanics and Engineering, 139(1–4), 195–227. 16. Navarro-Alarcon, D., & Liu, Y. H. (2018). Fourier-based shape servoing: a new feedback method to actively deform soft objects into desired 2-D image contours. IEEE Transactions on Robotics, 34(1), 272–279.

Robotics and Devices

The part starts with the first of several chapters that concern 3D printing. Here, the focus is on the fabrication of structures to control very small flows of liquid. The second chapter in this part concerns the control of the 3D printer itself. Its movement must be decoded form the files that describe the object being fabricated. A more prosaic problem is addressed in the third chapter, that of removing the wobble from a four-legged table by automating one or more of the legs. The fourth chapter returns to the topic of the 3D printer. In this case, the stability of the cross-beam is of particular importance because the printer is very large. The fifth chapter in the part is more practical than theoretical, dealing with the automatic tightening of the rubber strips linking sections of an office partition. Next is the chapter that deals with the calibration of a robot. By mounting a camera in the end effector, vision of a set of fixed targets can be used to determine the pose of the hand. A feeding robot can be used in a domestic or patient care situation, where vision locates the mouth to be fed and the dynamics of the movement must be carefully controlled to avoid spilling. The final chapter in this part concerns the design, control and experimental results of a lower-limb exoskeleton that combines soft and rigid links.

Fabrication and Characterization of 3D Printed Microfluidics Swapna A. Jaywant, Muhammad Asif Ali Rehmani, Tanmay Nayak, and Khalid Mehmood

1 Introduction Microfluidics is an integral part of lab-on-chip (LOC) and a micro total analysis system (µTAS) and sometimes also referred by these names. The field of microfluidics has proven high potential in many applications ranging from environmental assays to clinical analyses. This includes various point-of-care diagnostic tools, therapeutic devices, and water quality monitoring techniques and so on [5, 6, 8, 10, 12, 14, 15]. Several techniques are available today for manufacturing of microfluidic channels such as injection moulding, softlithography and paper microfluidics [1]. Among many methods, softlithography technique using polydimethylsiloxane (PDMS) micro-moulding is a highly popular method [2, 7]. Microfluidics fabrication using PDMS can be easily prototyped with simple procedures [4]. However, this multi-step process requires special equipment, and in many cases access to a cleanroom. Furthermore, it generally manufactures the final product at the second step (casting). The process is manual and cannot be fully automated [2]. Due to advancements in the modern additive manufacturing methods, 3DP has been shown as a promising platform for the fabrication of microfluidic devices. 3D printers convert a computer-aided design (CAD) into a physical 3D object by depositing the desired material in a layer by layer fashion [11]. The main advantages of 3DP are automated fabrication process, cost-effectiveness, higher printing resolution, etc. Additionally, these machines make the process simpler and lower the size of the required infrastructure and can be used as desktop printers [3].

S. A. Jaywant (B) · M. A. A. Rehmani · T. Nayak · K. Mehmood Department of Mechanical and Electrical Engineering, SF&AT, Massey University, Auckland 0632, New Zealand e-mail: [email protected] K. Mehmood e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_10

119

120

S. A. Jaywant et al.

In microfluidics, the major benefit of using 3DP is the elimination of the need for a mould to cast the final shape/product. This allows a significant reduction in the material cost, creates the possibility of mass manufacturing, and saves significant development time. Many common 3DP techniques like FDM, stereolithography (SLA), and polyjet printing have been compared using open channel micro-mixers by several researchers [9, 13]. Mixing in the micro-mixers is primarily dependent upon diffusion of two different flows. Hence, effective mixing is only possible with a slower flow rate and longer channel length [13]. However, investigation of internal features of microfluidic devices, the effect of flow rate and the channel size are also important parameters to explore the possibilities of leakage through the microfluidic devices. In this paper, we explore the possibility of using FDM and SLS technology for internal features of the microfluidic devices. These technologies have been compared in terms of their ability to fabricate microchannels. The comparison is based on the minimum possible channel size, fluid flowrate, and leakage in the microchannel body.

2 Materials and Method 2.1 Instrumentation The FDM-based 3D-printer used was a Tiertime UP 02, equipped with a 0.2 mm nozzle and the printer was controlled through UPStudio software. Stereolithography file (stl format) of 3D parts was constructed using parametric 3D modeling software—Autodesk Inventor Professional 2020. An orange spool of polylactic acid (PLA) filament having a dimeter of 1.70 mm used to deposit the layers of modeled microfluidic channels. The SLS-based 3D-printer used was DTM Sinterstation 2500 Plus. The 3D-objects were printed with a nylon powder known as Precimid 1170. Pico Plus Syringe Pump from HARVARD APPARATUS was used for injecting the water at different flow rates. Pressure measurements were performed with the help of PX3 Series heavy duty Honeywell pressure transducer having pressure measurement range of 667 psi with total error band (TEB) of ±1% full scale span (FSS).

2.2 Fabrication of Microchannel The microchannels were fabricated with different diameter sizes. Each microchannel consisted of an inlet port and the main channel as shown in Fig. 1. The input connector was connected at the inlet port. The inner diameter of the inlet port was kept constant at 0.5 mm. However, the main channel was fabricated with various diameter sizes (0.25, 0.3, 0.35, 0.4, 0.45, and 0.5 mm).

Fabrication and Characterization of 3D Printed Microfluidics

121

Fig. 1 a CAD image of Microchannel. b Cross-sectional view of microchannel. c Pictorial view showing the internal measurements of a microchannel

Table 1 3D printers and working parameters

3D printer Layer thickness Nozzle Material used (mm) diameter (mm) FDM

0.1

0.2

PLA

SLS

0.1

NA

Precimid 1170

Two different processes were explored during the fabrication of these microfluidic channels: FDM and SLS. Pressure measurements were performed on these channels at different flow rates. In the SLS printer, the inside power of the laser was set at 18 watts and outline power was kept at 9 watts. The slicer fill spacing was kept at 0.15 mm. The FDM printer was optimised with nozzle temperature at 207 °C and platform temperature at 68 °C to achieve the best results during the printing process. Other process parameters used during manufacturing are summarized in Table 1. 3D printed channels were cleaned, and the syringe was connected to each input port of the microchannels to provide the inlet connection.

2.3 Experimental Setup The experimental set up is explained in Fig. 2a. Initially, all the printed channels were cleaned with compressed air jets. Water at different flow rates (varying from 0 to 40 µL/Min. in steps of 5) was injected at the input port with the help of the syringe pump. A disposable, 10 mL syringe was actuated on the syringe pump. A pressure

122

S. A. Jaywant et al.

Fig. 2 a Experimental set up. b Microchannel printed with FDM method

sensor was placed in-between the syringe pump and the input port of the channel for corresponding pressure measurement. The pressure was obtained as voltage value which, in turn, was converted into psi value using the data-sheet. The change in flowrate at the input port resulted in a change in pressure at the input port and leakage was observed in the microchannel.

3 Results and Discussion The SLS technology uses nylon powder for laser printing, due to which the microchannels with a diameter of less than 1.5 mm created using this method were filled up with the nylon powder resulting in blocked microchannels. However, microchannels with a diameter of 1.5 mm and above created using this method did not have any blockages. Whereas the microchannels created using the FDM method having diameters ranging from 0.25 to 0.5 mm did not have any blockages (Fig. 2b). As a result of these observations, the SLS method was dropped and the FDM method was continued for further experiments. The water was passed through the FDM method based microchannels to study the effect of various flow rates on the leakage. The graph (Fig. 3) explains the relationships among the applied flow rate, the pressure generated at the input port, and the diameter size. As depicted in the graph there is a proportional relationship between the flow rate and the pressure. Additionally, at a certain flow rate, there is a linear increase in the pressure for a decrease

Fabrication and Characterization of 3D Printed Microfluidics

123

Fig. 3 Change in flow rate versus developed pressure at the input port

in the diameter size. In the case of microchannels with a diameter of 0.4, 0.45, and 0.5 mm, no leakage is observed. However, in the microchannel with a diameter of 0.35 mm, no leakage has been observed until pressure 0.2 psi. Above this pressure, the body of the microchannel started to leak from the top and bottom side near the input port. It has been observed that the microchannels with a diameter of 0.3 mm and less started to leak even at a minimum flow rate.

4 Conclusion In this paper, we demonstrated the fabrication of internal features of the microfluidic device below 0.5 mm by optimizing the printing parameters of PLA material. Moreover, the experimental results showed the correlation of internal feature (microchannel diameter) with the pressure the microfluidic device can handle without

124

S. A. Jaywant et al.

any leakages. Further treatment of microchannel with void filling material or chemical can also enhance the overall fluidic pressure in the device. Comparison between the SLS and FDM cannot be presented due to the inability of SLS for printing internal features of microchannel since the cleaning of the internal features at this scale was not successful after numerous attempts. The printed SLS internal features were clear with channel sizes exceeding more than 1.25 mm which does not fall in the category of a microfluidic device. Acknowledgements This work was supported by Massey University Research Funds (MURF) provided by the College of Sciences.

References 1. Amin, R., Knowlton, S., Hart, A., Yenilmez, B., Ghaderinezhad, F., Katebi-far, S., et al. (2016). 3d-printed microfluidic devices. Biofabrication, 8(2), 022001. 2. Bhattacharjee, N., Urrios, A., Kang, S., & Folch, A. (2016). The upcoming 3d-printing revolution in microfluidics. Lab on a Chip, 16(10), 1720–1742. 3. Bressan, L. P., Robles-Najar, J., Adamo, C. B., Quero, R. F., Costa, B. M., de Jesus, D. P., et al. (2019). 3d-printed microfluidic device for the synthesis of silver and gold nanoparticles. Microchemical Journal, 146, 1083–1089. 4. Chen, C., Mehl, B. T., Munshi, A. S., Townsend, A. D., Spence, D. M., & Martin, R. S. (2016). 3d-printed microfluidic devices: fabrication, advantages and limitations: A mini review. Analytical Methods, 8(31), 6005–6012. 5. Kou, S., Cheng, D., Sun, F., & Hsing, I. M. (2016). Microfluidics and microbial engineering. Lab on a Chip, 16(3), 432–446. 6. Lafleur, J. P., Joensson, A., Senkbeil, S., & Kutter, J. P. (2016). Recent advances in lab-on-a-chip for biosensing applications. Biosensors & Bioelectronics, 76, 213–233. 7. Lee, K. G., Park, K. J., Seok, S., Shin, S., Park, J. Y., Heo, Y. S., et al. (2014). 3d printed modules for integrated microfluidic devices. RSC Advances, 4(62), 32876–32880. 8. Liao, Z., Wang, J., Zhang, P., Zhang, Y., Miao, Y., Gao, S., et al. (2018). Recent advances in microfluidic chip integrated electronic biosensors for multiplexed detection. Biosensors and Bioelectronics. 9. Macdonald, N. P., Cabot, J. M., Smejkal, P., Guijt, R. M., Paull, B., & Breadmore, M. C. (2017). Comparing microfluidic performance of three-dimensional (3d) printing platforms. Analytical Chemistry, 89(7), 3858–3866. 10. Samiei, E., Tabrizian, M., & Hoorfar, M. (2016). A review of digital microfluidics as portable platforms for lab-on a-chip applications. Lab on a Chip, 16(13), 2376–2396. 11. Waheed, S., Cabot, J. M., Macdonald, N. P., Lewis, T., Guijt, R. M., Paull, B., et al. (2016). 3d printed microfluidic devices: enablers and barriers. Lab on a Chip, 16(11), 1993–2013. 12. Whitesides, G. M. (2006). The origins and the future of microfluidics. Nature, 442(7101), 368. 13. Yi-Qiang, F., Hong-Liang, W., Ke-Xin, G., Jing-Ji, L., Dong-Ping, C., & Zhang, Y. J. (2018). Applications of modular microfluidics technology. Chinese Journal of Analytical Chemistry, 46(12), 1863–1871. 14. Zeraatkar, M., Filippini, D., & Percoco, G. (2019). On the impact of the fabrication method on the performance of 3d printed mixers. Micromachines, 10(5), 298. 15. Zhang, J., Yan, S., Yuan, D., Alici, G., Nguyen, N. T., Warkiani, M. E., et al. (2016). Fundamentals and applications of inertial microfluidics: A review. Lab on a Chip, 16(1), 10–34.

A Modified Bresenham Algorithm for Control System of FDM Three-Dimensional Printer Ke Yu, Zhisheng Zhang, Zhiting Zhou, and Min Dai

1 Introduction The stepper motor is a device that is controlled in an open-loop method with many advantages, such as fast response speed, accurate movement and strong antiinterference ability. With the aid of a stepper driver, a stepper motor could complete a small angle move in a single step precisely. In the case of the 3D printing, stepper motors are usually used to move the extrusion nozzle of the printer to a particular location. After a 3D model is sliced on the host computer, a series of G-codes are generated. According to the feeding speed and target coordinates specified by the G-codes, a group of stepper motors cooperate to complete the point-to-point motion. The linear motion of the nozzle is made up from motors on three axes by interpolation algorithm, through mechanical structures such as transmission belts and pulleys. At present, three-axis interpolation algorithms which are commonly used in 3D printing are DDA algorithm [1] and Bresenham algorithm. The Bresenham algorithm was firstly proposed by Bresenham [2] in 1965 and Angel [3] gave a method to speed it up by breaking the line into segments for computer graphic systems, in 1991. Sun [4] also proposed a parallel Bresenham algorithm to interpolate a line symmetrically. Dai [5] applied the Bresenham algorithm on a control system for stepper motors. This paper introduces a modified Bresenham algorithm. The improved algorithm is more efficient than the conventional one and suitable to be applied on the control system of the FDM 3D printer.

K. Yu · Z. Zhang (B) · Z. Zhou · M. Dai School of Mechanical Engineering, Southeast University, Nanjing 211189, China e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_11

125

126

K. Yu et al.

2 Bresenham Line Generation Algorithm As a common algorithm to draw straight lines in computer graphics, Bresenham algorithm is usually utilized in computer numerical systems. Linear rasterization refers to a methodology that simulates a line with a series of pixel points. The algorithm constructs a set of vertical grid lines through the center of each row of pixels. According to the generation order of a line from the starting point to the ending point, the algorithm calculates and generates the intersection point of the target line and vertical grid lines. Then the linear interpolation is performed. (x0 , y0 ) and (x1 , y1 ) are used to represent the starting and ending coordinates of the target line to be interpolated. The slope of the trajectory line should be k=

dy y1 − y0 = dx x1 − x0

(1)

If the slope k < 1, for every pixel increased in the x direction, whether a pixel is increased in the y direction is determined according to the error term epsilon, as shown in Fig. 1. Suppose the line starts at the origin [i.e. (0, 0)], with ε0 = 0, and x is increased by one step further then, i.e., x++. In this case, εi+1 = εi + k. If εi+1 >0.5, then the value of y should be increased by one step further, i.e., y++, and εi+1 is substituted by εi+1 − 1 for the next calculation. If the εi+1 ≤ 0.5, there is no step advanced in the y direction, with the εi+1 is kept as the original value for the next calculation. This operation is repeated in a loop until the trajectory reaches the final coordinate in the x direction. xi+1 = xi + 1  yi+1 =

Fig. 1 Diagram of the decision process of Bresenham algorithm

εi + k − 0.5 < 0 yi , yi + 1, εi + k − 0.5 ≥ 0

(2)

(3)

A Modified Bresenham Algorithm for Control System …

 εi+1 =

εi + k − 0.5 < 0 εi + k, εi + k − 1, εi + k − 0.5 ≥ 0

127

(4)

There are a number of floating-point calculations in this loop, which affects the efficiency of the process. In order to avoid this kind of floating-point arithmetic, following transformation is applied on the formulas. Let dy = y1 − y0 , dx = x1 − x0 , Ei = 2εi · dx, and the formulas then become: 

E i + 2dy − dx < 0 yi , yi + 1, E i + 2dy − dx ≥ 0

(5)

E i + 2dy − dx < 0 E i + 2dy, E i + 2dy − 2dx, E i + 2dy − dx ≥ 0

(6)

yi+1 =  E i+1 =

Through this transformation, integer data are used only on the calculation to make the decision whether a step should be advanced in the y direction, avoiding the floating-point arithmetic and division calculations (Fig. 2).

3 Modification on Bresenham Algorithm There is still room for improvement on the computational efficiency of the Bresenham algorithm. In the existing decision process, it is necessary to determine for each step to feed. When the slope of the interpolated line is in a small amount, there are several steps fed in the x direction with no feeding steps in the y direction. In this case, if it is possible to calculate the number of feeding steps on the same row of x axis, there is no need to make a decision on whether or not to feed in the y direction after each step is advanced in the x direction, thereby saving time and improving efficiency. In other words, it is possible that a set of interpolation points on the same row could be predicted at a time. Before the introduction of the modified method, a slight transformation takes place on the Formulas (5), (6), which were introduced in Sect. 2. Let ei = Ei + 2dy − dx, and the formulas become: 

ei < 0 yi , yi + 1, ei ≥ 0

(7)

ei < 0 ei + 2dy, + 2dy − 2dx, ei ≥ 0 ei

(8)

yi+1 =  ei+1 =

This can make the formulas more simple for the subsequent work. On this basis, straight interpolated lines with the slope in the interval [0, 1] are discussed here. Any straight lines whose slopes are not within this interval can be adjusted by simply exchanging x and y.

128

K. Yu et al.

Fig. 2 The decision process of conventional Bresenham algorithm (ei = Ei + 2dy − dx)

Start

Input starting and end points; Calculate dx and dy.

e=2dy-dx

e>0 Y

N

x++ ; y++

x++

e=e+2(dy-dx)

e=e+2dy

step_completed++ N

step_completed>dx Y

End

For ease of understanding, a custom definition “feeding stair” is introduced. When the slope of the straight line is in the interval [0, 1]. Since the projection of the straight line in the x direction is longer than which in the y direction, according to the conventional Bresenham method, one step is certainly fed in the x direction after each decision formula is calculated, and whether a step should be fed in the y direction will be based on the result of decision. Under this circumstance, there will be a case where no steps are fed on the y direction and a certain number of steps are fed on the x direction. In this case, the steps fed in the x direction are defined as a “feeding stair”, and the width of this stair is the number of interpolated points in the x direction (i.e. the number of steps fed on the x direction increased by 1). Each time a step in the y direction is advanced, the whole trajectory is defined to have risen by one stair.

A Modified Bresenham Algorithm for Control System …

129

Fig. 3 A feeding stair with the width m

Here a hypothesis is introduced: Among the several feeding stairs obtained by the Bresenham method in a straight line, except the first stair and the last stair, the width m of all other stairs satisfies F ≤m ≤ F +1

(9)

where F stands for the downward rounding value of dx/dy. Proof Suppose there are m interpolated points on a certain stair, the decision variables corresponding to these points are named by e1 , e2 , …, em, and the decision variable corresponding to the last interpolated point on the previous stair is named by e0 . Furthermore, the decision variable corresponding to the last but one interpolation point on the previous stair is named by e−1 . (See Fig. 3.) If there is more than one interpolated points on the previous feeding stair, the following formulas could be derived directly: e0 = e−1 + 2dy

(10)

em−1 = e0 + 2(m − 1)dy − 2dx

(11)

em = em−1 + 2dy

(12)

where: e−1 ≤ 0; e0 > 0; em−1 ≤ 0; em > 0. Substitute (10), (11) into (12): em = em−1 + 2dy = e−1 + 2(m + 1)dy − 2dx

(13)

130

K. Yu et al.

Because em > 0, e−1 + 2(m + 1)dy − 2dx > 0; and because e-1 ≤ 0, 2(m + 1)dy − 2dx > 0, and here it could be derived that, m > dx/dy − 1. According to the previous definition, m ≥ F. Because em-1 ≤ 0, e0 + 2(m − 1)dy − 2dx ≤ 0; and because e0 > 0, 2(m − 1)dy − 2dx ≤ 0, and here it could be derived that, m ≤ dx/dy + 1. According to the previous definition, m ≤ F + 1. Therefore, F ≤ m ≤ F + 1. From this conclusion, the width of every other stairs must be F or F + 1, except the first and the last stairs. If the decision variable of the last interpolated point of a certain stair is e0 , the number of feeding steps of the next stair can be predicted by determining the sign of the decision variable (set to ef ) of the feeding step F (i.e. the Fth step) on this feeding stair. As we calculate that ef = e0 + 2Fdy − 2dx, if ef < 0, the width of stair is F + 1, and ef + 2dy is the value of the decision variable of the last point of the next stair. Otherwise, the width of next stair is F, and the decision variable of the last point of the stair is ef . Through this conclusion, the number of steps to be fed on the same stair can be calculated by one time of decision, and the efficiency of the algorithm is improved (Fig. 4).

4 The Software Implementation of the Modified Algorithm on the FDM 3D Printer The work flow of the FDM 3D printing is briefly introduced in Fig. 5. After the 3D model is drawn in the modeling software on the computer, it needs to be converted into a STL format model and sliced in the slicing software. The model is finally transformed into a series of G-codes which could be easily realized by the control system of the printer. The subsequent work is similar to the processing of G-code in GRBL firmware. Each G-code is parsed into a structure in C language, named by “block”. In this structure, parameters which are required to run a G-code are calculated, and the implementation process of the nozzle path is planned. Then, using the parameters inside the structure, the pins of the MCU can be controlled to emit a series of pulses in timer interrupts to control the movement of stepper motors. In a timer interrupt, Bresenham method could be applied to combine the multi-axis motion planning, with the acceleration and deceleration process of stepper motors. According to the previous introduction, the algorithm needs to be applied with the consideration of the specific environment of a 3D printer. If the stepper motor starts directly at the feeding speed which is preset by the G-code, out-of-step phenomenon may occur. The acceleration and deceleration process is generally introduced during the movement to avoid this problem. Taking trapezoidal acceleration and deceleration process [6] as an example, since the total number of the stepper motor’s feeding steps in the x direction (Suppose dx > dy, otherwise the role of x and y should be exchanged)

A Modified Bresenham Algorithm for Control System …

131

Start

Input starting and endIng points(x0,y0),(x1,y1); Calculate dx and dy.

e=2dy-dx

e>0

N

Y

x++ ; y++

x++

step_completed++

step_completed++

e=e+2dy

ef=e+2Fdy-2dx>0 Y

step_completed++

N

x=x+F ; y++

N

x=x+F+1 ; y++

step_completed>dx step_completed+=F

step_completed+=(F+1)

e=ef

e=ef +2dy

Y

y f1 N, so the drive wheel is designed as a V-shaped groove to increase the driving friction. The design of the driven wheel uses a stainless steel toothed wheel as the driven wheel. Because the toothed drive wheel has the characteristic of increasing friction, it can grab the rubber strip well. The three-dimensional diagram of the friction wheel feeding feed mechanism is shown in Fig. 4.

Fig. 4 The three-dimensional diagram of the friction wheel feeding feed mechanism. (1) Synchronous motor, (2) primary reducer, (3) drive wheel, (4) driven wheel

Design and Experimental Research of Automatic Tightening …

167

2.3 Rubber Strip Pressing System Design The rubber strip pressing system is hoisted on the two-dimensional screw module, and moves around the screen panel under the driving of the screw module. The screen pressing system is the core part of the whole system. The three-dimensional design of the strip pressing system is shown in Fig. 5.

3 Multi-motor Synchronous Control System In most industrial production, the traditional PID controller is mainly used to improve the response rate of the controller. Traditional PID controller is simple to operate and easy to realize the required motion, but it is difficult to tune the PID parameters. If the speed of the motor can not be well controlled, the soft and elastic rubber strip may be broken during movement. The smooth embedding of rubber strips requires synchronous and cooperative motion of eight motors. For such a non-linear, strong

Fig. 5 The three-dimensional design of the strip rubber pressing system. (1) Rubber strip release mechanism, (2) rubber strip buffer mechanism, (3) friction wheel feed mechanism, (4) cutting mechanism, (5) pressure roller mechanism, (6) dragging mechanism

168

R. Wang et al.

coupling control object, the traditional PID controller is not enough to achieve the purpose of motion. [3, 4] Because of the strong non-linear approximation ability and self-learning ability of the neural network, people in recent years have formed the neural network PID control by combining the BP neural network algorithm and the traditional PID. The control can automatically adjust the parameters of the controller and solve the difficult problem of setting the parameters of the PID. In the process of in-depth study, it is found that the traditional BP neural network algorithm has many advantages, but it has the disadvantages of slow convergence speed and easy to fall into local minimum. In this study, by introducing inertia term, momentum term and improving learning rate strategy, the PID controller based on improved BP neural network is redesigned to improve the performance.

3.1 Modeling of Permanent Magnet Synchronous Motor Permanent magnet synchronous motor (PMSM) has the characteristics of high energy density, light weight, small volume, easy control and high precision. So PMSM is used in this study [5]. The PMSM is shown in Fig. 6. Stator voltage formula of permanent magnet synchronous motor in three-phase static coordinate system: Fig. 6 The model of PMSM

Design and Experimental Research of Automatic Tightening …

⎧ dψA ⎪ uA = Rs iA + ⎪ ⎪ ⎪ dt ⎪ ⎨ dψB uB = Rs iB + ⎪ dt ⎪ ⎪ ⎪ ⎪ dψC ⎩ uC = Rs iC + dt The flux formula is shown as follows: ⎧ ⎨ ψA = LA iA + LAB iB + LAC iC + ψf cosθr  ψ = LBA iA + LB iB + LBC iC + ψf cosθr − 2π 3  ⎩ B ψC = LCA iA + LCB iB + LC iC + ψf cos θr − 2π 3

169

(4)

(5)

The torque formula is shown as follows: te = P0 ψf × is

(6)

In the formula, uA ,uB ,uC —phase voltage of three-phase winding A, B and C; iA , iB , iC —line current of three-phase winding A, B and C; ψA , ψB , ψC —full flux linkage of three-phase winding A, B and C; LA , LB , LC —A, B, C three-phase stator winding self-inductance; LAB , LBA —mutual inductance between phase A and phase B stator winding; LAC , LCA —mutual inductance between A phase and C phase stator winding; LBC , LCB —mutual inductance between B-phase and C-phase stator winding; RS —stator resistance; ψf —rotor permanent magnet flux linkage; θr —the angle between axis A and axis D of DQ shafting. From the above voltage formula and flux formula, it can be seen that the voltage and flux linkage in ABC three-phase static coordinate system are relatively complex and vary with the relative position between stator and rotor. In order to facilitate the analysis and research, the model is further simplified from three-phase stationary coordinate system to two-phase rotating coordinate system. The simplified model is shown in the Fig. 7. According to the principle of magneto motive force equivalence, the static ABC coordinate system is transformed into the rotating dq coordinate system, and the transformation relationship is obtained as follows: ⎛ ⎞

    iA   4π 2 cos θM cos θM − 2π cos θ − id 3  3  ⎝i ⎠ M  = B 4π − sin θ iq − 3 − sin θM − sin θM − 2π M 3 3 iC

(7)

The conversion from static ABC coordinate system to rotating dq coordinate system is equivalent to the equivalent of permanent magnet synchronous motor to DC motor. The voltage formula of permanent magnet synchronous motor under DP shafting at this time:

170

R. Wang et al.

Fig. 7 The simplified model of PMSM

ud = Rs id +

dψd − ω r ψq dt

(8)

uq = Rs iq +

dψq + ω r ψd dt

(9)

The electromagnetic torque formula is shown as follows:    1 te = P0 ψf is sin β + Ld − Lq i2s sin 2β 2

(10)

In the formula, ud , uq —stator voltage dq axis component; id , iq —stator current dq axis component; Ld , Lq —stator inductance component on dq axis; ψd , ψq —stator flux component on dq axis;te —electromagnetic Torque of Motor; ωr —rotor speed; is —stator current; β—angle between d-axis and stator magneto motive potential.

3.2 Optimized BP Neural Network Algorithms As shown in Fig. 8, BP neural network is a kind of multi-layer feedforward neural network with hidden layer in its structure [3, 6]. Gradient descent is used to adjust the connection weights of each neuron in the network, which can minimize the error of the network. In the learning stage of training network, forward calculation formula is shown as follows:

Design and Experimental Research of Automatic Tightening …

171

Fig. 8 Structure of BP neural network

⎧ M  ⎪ ⎪ ⎪ ⎪ neti = ωi j o j ⎪ ⎪ ⎪ ⎪ j=1 ⎪ ⎪ ⎪ ⎪ ⎪ oi = f (neti ) ⎪ ⎪ ⎨

e x − e−x f (x) = tanh(x) = ⎪ e x + e−x ⎪ ⎪ ⎪ q ⎪ ⎪  ⎪ ⎪ ⎪ netk = ωki oi ⎪ ⎪ ⎪ ⎪ i=1 ⎪ ⎪ ⎩ ok = f (netk )

(11)

In the formula, neti —input of the i-th neuron in the hidden Layer; Oi —output of the i-th neuron; netk —input of the k-th neuron in the lower layer of oi ; Ok —output of the k-th neuron. After calculating the output value of the system, the calculated output value is compared with the expected output value. If there is a deviation between the calculated output value and the expected output value, the deviation needs to be fed back to the system to correct the connection weights between layers so that the calculated output of the neural network is consistent with the expected output. The weight correction formula is as follows:

172

R. Wang et al.

⎧ L ⎪ 1 ⎪ ⎪ E = (d pk − o pk )2 ⎪ P ⎪ ⎪ 2 ⎪ k=1 ⎪ ⎪ ⎪ ⎪ N L ⎪ ⎪ 1  ⎪ ⎪ E = (d pk − o pk )2 ⎪ ⎪ ⎪ 2 p ⎪ p=1 k=1 ⎪ ⎪ ⎪ ⎪ ⎪ ∂Ep ⎪ ⎪ ωki = −η ⎪ ⎪ ∂ωki ⎪ ⎪ ⎪ ⎪ ∂ E ∂ E ⎪ p p ∂netk ⎨ = ∂ωki ∂netk ∂ωki  q  ⎪ ⎪ ⎪ ∂netk  ⎪ ∂ ⎪ ⎪ = ωki oi ⎪ ⎪ ∂ωki ∂ωki i=1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ∂ E p ∂ok ⎪ ⎪ δk = − ⎪ ⎪ ⎪ ∂ok ∂netk ⎪ ⎪ ⎪ ⎪ ∂Ep ⎪ ⎪ = −(dk − ok ) ⎪ ⎪ ∂ok ⎪ ⎪ ⎪ ⎪ ∂ok ⎪ ⎪ ⎩ = f k (netk ) ∂netk

(12)

Formula (13) can be obtained from Formula (12): 

δk = Ok (1 − Ok )(dk − Ok ) ωki = ηOk (1 − Ok )(dk − Ok )Oi

(13)

The traditional BP neural network uses the stochastic gradient descent algorithm to make the weights better. The convergence speed of the traditional BP is slow, and it is easy to fall into the local minimum. In this study, the gradient momentum term optimization algorithm is introduced to improve the convergence speed. The weight correction formula after introducing the gradient momentum term is as follows: ⎧ ∂Ep ⎪ Vdw = (1 − β) ⎪ ⎪ ⎪ ∂ωki ⎪ ⎪  ⎪ ⎪ ⎪ ∂Ep 2 ⎪ ⎪ S = (1 − β ) dw 2 ⎪ ⎪ ∂ωki ⎪ ⎪ ⎪ ⎨ V dw c Vdw = 1 − β1t ⎪ ⎪ ⎪ ⎪ ⎪ Sdw ⎪ c ⎪ ⎪ ⎪ Sdw = 1 − β t ⎪ ⎪ 2 ⎪ ⎪ ⎪ V c ⎪ ⎪ ⎩ ωki = −η  c dw Sdw + ε

(14)

Design and Experimental Research of Automatic Tightening …

173

In the formula, Vdw —gradient momentum accumulated in iteration of loss function; Sdw —gradient momentum accumulated during iteration of loss function; β1 = 0.9, β2 = 0.999, ε = 10−8 .

3.3 Design of PID Controller Based on BP Neural Network Digital PID controller can be divided into position PID control and incremental PID control. The position PID control output sampling time data is related to the output data of the previous time. It needs to accumulate the deviation, and the calculation is huge. So the position PID control is not easy to apply to this system. BP neural network algorithm is easy to implement, and it can also approximate arbitrary nonlinear model. Using the self-learning ability of the neural network, we can find the best proportion, integral and differential parameters in the operation of the system. The PID controller of BP neural network consists of two parts: traditional PID controller and neural network algorithm. The output of the neuron in the output layer of the neural network corresponds to the PID parameters respectively, so that the PID parameters can be adjusted by the self-learning of the network. The PID structure of BP neural network is shown in the Fig. 9. The expression of incremental PID is as follows [7]: u(k) = u(k − 1) + k p (e(k) − e(k − 1)) + k I e(k) + k D (e(k) − 2e(k − 1) + e(k − 2))

(15)

In the formula, k—sampling number; u(k)—k-time system output value; e(k), e(k − 1)—deviation between input and output of system at K and (k-1) time; Kp — proportional constant; KI —integral constant; KD —differential constant.

Fig. 9 The PID structure of BP neural network

174

R. Wang et al.

In this study, a three-layer BP neural network is used, and its structure is shown in Fig. 10. The calculation formula is as follows: ⎧ (1) o0 = e(k) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ o1(1) = e(k − 1) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ o2(1) = e(k − 2) ⎪ ⎪ ⎪ ⎪ 3 ⎪  ⎪ ⎪ (2) (1) ⎪ net (k) = ωi(2) ⎪ j o j (k) ⎪ i ⎪ ⎪ j=0 ⎪ ⎪ ⎪ ⎪ ⎨ o(2) (k) = f (net (2) (k)) i i (16) 4 ⎪  ⎪ ⎪ (3) (3) (2) ⎪ ⎪ netl (k) = ωli oi (k) ⎪ ⎪ ⎪ l=0 ⎪ ⎪ ⎪ ⎪ ⎪ ol(3) (k) = g(netl(3) (k)) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ o0(3) (k) = k p ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ o1(3) (k) = ki ⎪ ⎪ ⎪ ⎩ (3) o2 (k) = k D In the formula, f(.) = tanh(x); g(.) = 1/2(1 + tanh(x)). The mean square error function is as follows: E=

1 (r (k + 1) − y(k + 1))2 2

Fig. 10 Three-layer BP neural network

(17)

Design and Experimental Research of Automatic Tightening …

175

Fig. 11 The simulation results

3.4 Simulation Experiment Analysis Although the system requires eight synchronous motors to work together, the control algorithm for each motor is the same. Therefore, this simulation experiment built a traditional PID control system and a BP neural network PID control system in the Matlab/Simulink environment. The simulation of BP neural network PID is realized by writing S function [8]. The simulation results are shown in the Fig. 11. It can be seen from the figure that the control effect of the optimized BP neural network PID control algorithm is obviously better than the traditional PID control algorithm.

4 Tightening Experiment In order to verify the rationality of the design, a complete prototype was built and preliminary experiments were carried out. The experimental prototype is shown in the Fig. 12. Through the cooperation of the speed parameters of the rubber strip release motor and the rubber strip friction feed motor, the rotary motor of the indenter device and the running motor of the indenter device, the best effect of the rubber strip embedded in the screen plate was debugged. The experimental sample obtained is shown in Fig. 13. Table 1 is the experimental data of total rubber strip feed under multi-motor

176

R. Wang et al.

Fig. 12 The experimental prototype. (1) Lead screw module, (2) screen board, (3) flexible fixed platform for screen panel, (4) frame, (5) supporting device, (6) indenter mechanism

Fig. 13 The experimental sample

synchronous drive. The experimental data of the length of the rubber strip winding is shown in the Fig. 14. The width of the groove on the side of the bare leaky screen was 3 mm. When the clamping experiment was carried out, the screen was wrapped around tablecloth with thickness of 0.5 mm, and the width of the groove became 2 mm. Rubber strip diameter is 3 mm, So the deformation of rubber strip in the process of embedding was related to the quality of rubber strip embedding. One of the important factors affecting the deformation of rubber strip was whether the motors were synchronized and coordinated. By analyzing the experimental data, it can be concluded that the total winding length of the tape in two minutes was nearly linear with the speed, which provided a guarantee for the smooth embedding of the tape. The preliminary experimental results showed that the rubber strip can be embedded automatically by the method proposed in this study. The width of the groove on the side of the covered screen plate was smaller than the diameter of the rubber strip, which caused the hoop

Design and Experimental Research of Automatic Tightening …

177

Table 1 Tightening experimental data Length of Speed of tape winding tape release (mm/2 min) motor (mm/s)

Speed of front friction wheel motor (mm/s)

Speed of rear Speed of friction drag motor wheel motor (mm/s) (mm/s)

Speed of rotating motor (mm/s)

Speed of XY directional motor (mm/s)

726

6

8

8

8

6

10

945

8

10

9.6

9.4

7

12

1338

12

14

13.8

12

9

16

1675

14

16

15.8

14

11

18

1917

16

18

17.6

16

13

21

2275

19

22

21.6

19

17

24

2773

23

26

25.4

23

21

28

3365

28

31

30.6

27

25

33

3847

32

35

34.2

32

30

37

4214

35

38

37.6

35

33

40

Fig. 14 The experimental data

178

R. Wang et al.

force was insufficient and the rubber strip was easy to fall off. It is believed that the effect of stirrups will be better in the future by optimizing the experimental conditions. Better clamping effect can be obtained by optimizing experimental conditions in the future.

5 Conclusion In this paper, a method of automatic tightening of rubber strip on the side of office screen panel is described. Through mathematical modelling static analysis, the rationality of the design was verified. In addition, the task of multi-motor synchronous control used in the automatic tightening method of rubber strip on the side of office screen panel was analyzed in this paper. By comparing the PID and neuron PID control algorithms, the simulation results of MATLAB showed that neuron PID control had strong robustness. In order to verify the rationality of the method, an automatic tightening experiment device was set up and relevant experiments were carried out. The preliminary experimental results showed that the design of automatic tightening method for rubber strip on the side of office screen panel was reasonable and feasible. This method provided a design basis for the practical production device of automatic tightening of rubber strip on the side of office screen panel.

References 1. Yong, J. (2015). A preliminary study on the relationship between modern corporate culture and modern office environment. Interior Design, 25–27. 2. Wang, Y. (2015). Research and design of FDM rapid prototyping feed system. Huazhong University of Science and Technology. 3. Cui, B., Li, Y., & Duan, Y. (2011). Spanning slab system based on neural network PID control. Journal of Shenyang University of Technology, 33(20), 188–192. 4. Hu, J., & Wang, W. (1999). Research on neural network training methods with addition al items. Computing Technology and Automation, 18(2), 16–19. 5. Wang, C., Xia, J., & Sun, Y. (2013). Modern motor control technology. Beijing: Mechanical Industry Press. 6. Xie, W. (2017). Research on multi-motor synchronous control based on BP neural network PID algorithm. Shenyang University of Technology. 7. Tao, Y., Yin, Y., & Ge, L. (1999). New PID control and its application. Beijing: Mechanical Industry Publishing. 8. Lin, R., & Qiu, G. (2001). Shenmu PID control simulink imitation based on S function true model. China Instrumentation, 6, 4–5.

A Scene Feature Based Eye-in-Hand Calibration Method for Industrial Robot Guoshu Xu and Yonghua Yan

1 Introduction Eye-in-hand calibration is a basic task before the robot works. When the RGB-D camera is mounted on the end-effector of the robot, the transformation matrix from the robot end-effector coordinate system to the camera coordinate system is obtained by Eye-in-hand calibration. After calibration, visual grabbing work, three-dimensional scene reconstruction work and so on can be carried out. Eye-in-hand calibration is usually divided into two steps: solving for camera intrinsic parameters and solving for hand-eye transformation matrix. Firstly, intrinsic parameters of camera like focal length and principal points need to be obtained. At present, the commonly used method is Zhang’s calibration method [1], which uses the camera to take multiple checkerboard pictures at different positions to solve the camera intrinsic parameters. In this method, checkerboard is a man-made marker. The advantage is that it can obtain high calibration accuracy, however this method is an artificially involved and time-consuming work. Besides, high quality checkerboard plays a key role in the accuracy of calibration result. For the industrial occasion where optical parameters (focal length, principle points) are often changed, it is necessary to re-calibrate at each time, thus Zhang’s method can not be easily implemented. There are some other calibration methods based on the constraints of essential matrix [2]. The principle is constructing cost function according to the essential matrix having two equal non-zero singular values. Stochastic optimization method like dynamic mountain climbing method [3] is used to find parameters that can minimize the cost function and the optimal parameters are the camera intrinsic parameters. However, stochastic optimization algorithm can not strictly prove its convergence, so the obtained optimal parameters are often local optimal values rather than global optimal values. G. Xu · Y. Yan (B) School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_15

179

180

G. Xu and Y. Yan

Fig. 1 The model of hand-eye calibration for robots

Once getting the camera intrinsic parameters, we can further get the hand-eye transformation matrix. Tsai’s method [4] is usually used to solve the hand-eye calibration model for robots (Fig. 1). The core of eye-in-hand calibration is to solve the matrix equation like AX = XB. A represents the relative transformation between two camera poses, B represents the relative transformation between two end effector poses, X is the hand-eye transformation matrix which need to be solved. N groups of camera relative pose transformation Ai,i+1 and end effector relative pose transformation Bi,i+1 (i = 0…N − 1) are obtained by N times pose transformation of robot. Assuming that matrix Hi represents the end effector coordinate system relative to the robot base coordinate − . Since Hi is directly obtained system in the ind transformation, so Bi,i+1 = Hi Hi+1 from the robot, Bi,i+1 can be obtained easily. As for Ai,i+1 , we can use chessboard as a reference object and calculate the camera pose transformation by obtaining the camera pose matrix relative to the chessboard. It is seen that man-made markers like chessboard play an important role in traditional hand-eye calibration methods, and these marker-based methods are manual and time-costing. However, labor-intensive and time-consuming work are intolerable in industrial application. What is more, auto parameters adjustment and online camera calibration is a trend in the future. In order to overcome the shortcomings of traditional hand-eye calibration methods, an automatic eye-in-hand calibration method based on scene feature is proposed. Firstly, the camera intrinsic parameters are solved by using ORB feature extraction and Bundle Adjustment to deal with small motion image sequence. Then, ORB feature and PnP are used to solve the pose transformation between multi-view images, BA is also used to optimize the result. Finally, the eye-in-hand transformation matrix is obtained by solving the corresponding relations between the pose transformations of multi-view images and the pose transformations of the robot. Experiments show that our method is simple and feasible with high accuracy. The robot can re-calibrate quickly for many times with no rely on markers. It is believed that our method has good prospects in industrial applications.

A Scene Feature Based Eye-in-Hand Calibration Method …

181

Fig. 2 Camera pinhole model

2 Methodology 2.1 Camera Imaging Model Camera imaging process is a mapping between the 3D world and a 2D image. Pinhole camera model is the simplest camera model. The model is shown in Fig. 2. The imaging process can be described by formula 1. ⎞⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ Xc Xc u fx 0 cx w⎝ v ⎠ = ⎝ 0 fy cy ⎠⎝ Yc ⎠ = K⎝ Yc ⎠ 0 0 1 Zc Zc 1

(1)

T  In formula 1, assuming a point P in the world space, Xc Yc Zc is the coordinate  T of point P in the camera coordinate system and u v is the coordinate of point P in the pixel coordinate system. The matrix K is called the camera intrinsic calibration matrix. From formula 1, we can get the transformation relationship from camera coordinate system to pixel coordinate system (formula 2). 

u = fx · v = fy ·

Xc Zc Yc Zc

+ cx + cy

(2)

2.2 Camera Intrinsic Parameters The first step of hand-eye calibration is to solve the camera intrinsic matrix K. Bundle Adjustment (BA) [5] is a non-linear least squares optimization model. BA can be used to optimize camera parameters and pose estimation by minimizing the re-projection error between images. Ha [6] uses BA to estimate the camera parameters from a

182

G. Xu and Y. Yan

Fig. 3 Small motion image sequence used in BA

small motion image sequence. In his method, Harris corner detector is used in the reference image to extract the local features and KLT algorithm is used to find the corresponding features in other images. Experiments show that Ha’s method is useful for outdoor scenes but less effective in indoor scenes. Considering that most industrial robots work in indoor environment, we present a method for solving camera intrinsic matrix by using ORB feature and Bundle Adjustment. The principle is shown in Fig. 3. In Fig. 3, O0−xyz is the camera coordinate system of 0-th frame, Oi−xyz is the camera coordinate system of i-th frame. (Ri , Ti ) is the homogeneous transformation matrix from O0−xyz to Oi−xyz . Xj is a feature point, Xoj 0 is Xj coordinate in O0−xyz , Xoj i is Xj coordinate in Oi−xyz . u0j (in red) is the ORB feature point corresponding to Xj in 0-th frame, uij (in red) is the ORB feature point corresponding to Xj in i-th frame. uij (in blue) is the reprojection point corresponding to u0j in i-th frame. The 0-th frame is thought as reference frame,

ORB feature point in the the j-th reference frame is u0j and its coordinate is xu0j , yu0j . The depth value of u0j is Zj , which can be directly gotten from corresponding depth image. According to the camera imaging model (formula 1), we can get the coordinate of Xj in the camera coordinate system O0−xyz (formula 3). ⎡ xu0j −cx ⎢ Xoj 0 = ⎣

fx yu0j −cy fy

· Zj

⎤

⎥ · Zj ⎦

(3)

Zj Function T maps the point coordinate from O0−xyz to Oi−xyz , so the coordinate of Xj in Oi−xyz can be described by formula 4.

Xoj i = T Xoj 0 = R(ri ) · Xoj 0 + ti

(4)

A Scene Feature Based Eye-in-Hand Calibration Method …

183

T  In formula 4, ri = ri,x ri,y ri,z ∈ R3 represents the rotation vector, ti =  T ∈ R3 represents the translation vector. The transformation from ti,x ti,y ti,z rotation vector to rotation matrix can be achieved by Rodrigues formula (formula 5). R(ri ) = I · cos θ + (1 − cos θ)ri riT + sin θ · ri

(5)

In formula 5, θ = ri ,  is the conversion symbols from vector to antisymmetric matrix. According to the formula derived by Yu and Gallup [7], when the rotation angle is very small, Rodrigues formula can be approximated to formula 6. ⎤ 1 −ri,z ri,y R(ri ) = ⎣ ri,z 1 −ri,x ⎦ −ri,y ri,x 1 ⎡

(6)

According to formula 2, the projection from Xoj i = XXoij , YXoij , ZXoij to uij can be described by formula 7. ⎡

XXoi

⎤

⎢ fx ZXoij + cx ⎥ ⎥ uij = ⎢ ⎦ ⎣ YXoij fy Z oi + cy j

(7)

Xj

The reprojection from u0j to uij is a combination of formulas 3, 4, 5, 6, and 7. For a small motion image sequence, the error function W is the sum of reprojection errors for all the feature points u0j (j = 1…n) in the reference frame. RP((p |K, R, T, Z)) is an abstract symbol for reprojection, so the error function W can be described by formula 8.  2  2  1  1  ρij · ei,j  = ρij · uij − RP u0j |K, ri , ti , zj  2 i=1 j=1 2 i=1 j=1 i=n j=m

W=

i=n j=m

(8)

In formula 8, n is the image numbers in the small motion image sequence, m is the numbers of extracted ORB features point. If feature point Xj has projection in i-th frame, then ρij = 1; else ρij = 0. K is the camera intrinsic calibration matrix. ri , ti are the rotation vector and translation vector from the coordinate system of reference frame to the coordinate system of i-th frame. zj is the depth value of the ORB feature point u0j . ORB feature [8] consists of key point and descriptor. The key point is Oriented FAST corner point which is scale invariant and rotation invariant. The descriptor is a binary descriptor named BRIEF. ORB feature points are extracted in every frame and use bidirectional Brute-Force [9] to match the corresponding feature points. Figure 4 shows the matching of ORB feature points between two similar images. At the same time, threshold is set to preserve matching point pairs with high matching accuracy.

184

G. Xu and Y. Yan

Fig. 4 Matching of ORB feature points in two images

BA is used to solve the variables K, ri , ti , zj by minimizing the cost function W. Since it is a small motion image sequence, the initial value of ri , ti can be set as zero vector. Camera intrinsic calibration matrix contains four unknown parameters fx , fy , cx , cy . The initial value of fx and fy is half of sum of image width and height. The initial value of cx is half of the image width, the initial value of cy is half of the image height. The initial value of zj can get from corresponding depth image. After setting all initial values, we utilize Ceres [10] to do the BA optimization. The detailed working flow is shown in Table 1. Table 1 Flow of solving camera intrinsic parameters

A Scene Feature Based Eye-in-Hand Calibration Method …

185

2.3 Hand-Eye Transformation Matrix After getting the camera intrinsic calibration matrix, the hand-eye transformation matrix can be calculated by Tsai’s method. The key problem is how to get the pose transformations between multi-images? We have discussed traditional methods which rely on marker like chessboard in Chap. 1. Here, we take the idea from camera location in ORB-SLAM [11, 12] and put forward a method to solve pose transformations between multi-images with ORB and PnP, followed by BA optimization. Although ORB feature and BA are still used like the way we use in calculating camera intrinsic calibration matrix, the clear difference is that the image sequence used here is from obvious camera moving rather than small motion. As a result, PnP is used to solve the pose transformations between two adjacent frames. PnP [13] is a method for estimating camera pose by N sets of 3D space points and their corresponding 2D projection points. The pose transformations between multi-view images is shown in Fig. 5. Firstly, ORB feature points are extracted in two adjacent color frames and bidirectional Brute-Force is used to find 2D-2D matching pairs. We further find the 3D-2D matching pairs with the depth image and camera intrinsic calibration matrix K. Considering the measurement error of camera sensor, the depth images are preprocessed before been used to find the 3D-2D matching pairs. Preprocessing contains the following three aspects: 1. Fill the hole in the depth image and replace it with the mean of neighborhood pixels. 2. Apply Gaussian filter to the depth image to reduce noise. 3. Considering that error of measured depth value is large when it is too close or too far away, so we set an interval from 0.5 to 5 m. And it is thought that these points between 0.5 and 5 meters as “good points”, only “good points” are used in 3D-2D matching pairs. Fig. 5 Pose transformations between multi-view images. R12 , T12 represent the pose transformation from 1-th frame to 2-th frame

186

G. Xu and Y. Yan

Fig. 6 Solve pose transformation between adjacent images

We utilize PnP to solve pose transformation Ai,i+1 between two adjacent images, and set the Ai,i+1 as initial value, then BA is used to optimize Ai,i+1 . The detailed process flow is shown in Fig. 6. Finally, N groups of camera pose transformation matrix Ai,i+1 (i = 0…N − 1) can be obtained. End effector pose transformation matrix Bi,i+1 (i = 0…N − 1) can be gotten directly from robot. Substitute the Ai,i+1 and Bi,i+1 into AX = XB, and we can get the hand-eye transformation matrix X by Tsai’s method.

3 Experiments and Results 3.1 Experimental Setup The experiment system includes a PC, a JAKA Zu7 six-axis industrial robot and an Intel RealSense D435 RGB-D camera. Experiment platform is shown in Fig. 7. The software environment is Ubuntu16.04 LTS with ROS, OpenCV and PCL. The parameters of camera are shown in Table 2.

A Scene Feature Based Eye-in-Hand Calibration Method …

187

Fig. 7 Experiment platform

Table 2 Camera parameters

Intel RealSense D435 camera Sensor

Stereo IR + RGB

Color image resolution (max)

1920 * 1080

Depth image resolution (max)

1280 * 720

Maximum frame radio fps

90

Work distance (m)

0.105–10

Interface

Type-C, USB3.0

3.2 Results and Analysis 1. Camera Intrinsic Parameters We get color image sequence of small motion with 30 frames and take the first frame as reference frame. The depth image of reference frame is also captured to initialize depth value of each feature point. Since it is a small motion, there is little difference between frames. The resolution of each frame is 640 * 480. The image sequence is shown in Fig. 8. The experiment results are shown in Table 3. Results of our method are the average result of 10 independent repeats. At the same time, Zhang’s method results are thought as standard values. From the Table 3, the relative error of focal length (fx , fy ) is 0.5%, the relative error of principle point (cx , cy ) is less than 1%. In other calibration method like Jiang’s method [14], the relative error of focal length is 0.5% and the relative error of principle point is 1.5%. So, it can be concluded that our calibration method has a good precision. Meanwhile, our method needs no man-made marker, thus it is a better choice in application.

188

G. Xu and Y. Yan

Fig. 8 The small motion image sequence. a Color image of reference frame. b Depth image of reference frame. c Color image of last frame. d Depth image of last frame

Table 3 Camera intrinsic parameters fx

fy

cx

cy

Zhang’s method

614.18

614.17

322.25

245.24

Our method

617.24

617.24

324.14

247.52

0.5

0.5

0.6

0.9

Relative error (%)

2. Hand-Eye Transformation Matrix We get color image sequence of obvious motion with 50 frames and its corresponding depth image sequence. The resolution of each frame is 640 * 480. The image sequence is shown in Fig. 9. The result is shown in Table 4. 3. Object Grasping Experiment Since the accuracy of hand-eye transformation matrix is not easy to be evaluated quantitatively, an object grasping experiment is designed to evaluate indirectly. If the grasping error is within the acceptable range, it can be declared that the accuracy of hand-eye transformation matrix is acceptable because the grasping error includes the hand-eye matrix error, vision detection error, robot motion error and so on. In our experiment, the grasping object is a box as shown in Fig. 10. The object pose in camera coordinate system can be calculated from the color image, depth image and camera intrinsic matrix K, then the pose transformation

A Scene Feature Based Eye-in-Hand Calibration Method …

189

Fig. 9 The obvious motion image sequence. a The 1-th color frame. b The 2-th color frame. c The 3-th color frame. d The 4-th color frame Table 4 Hand-eye transformation matrix

Matrix

Trans vector Euler angles

⎛

⎞ 0.9987 0.0353 −0.0361 −35.163 ⎜ ⎟ ⎜ −0.0351 0.9994 0.0050 −107.188 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ 0.0362 −0.0037 0.9993 26.185 ⎠ 0 0 0 1

−35.163 −107.188 26.185 mm

x : −0.287 y : −2.068 z : −2.026 ◦

Fig. 10 The grasping object. a Color image. b Depth image

190

G. Xu and Y. Yan

Fig. 11 The grasping scene

from camera coordinate system to end-effector coordinate system can be solved by the hand-eye transforming matrix. The transformation from end-effector coordinate system to robot base coordinate system is known, the object pose in robot base coordinate system can be finally solved. Before grasping, it is vital to set the tool coordinate system. The grasping scene is shown in Fig. 11. It is found that the sucker is about in the center of the box and the grasping result is satisfied. According to the foregoing analysis, we think the hand-eye transformation matrix is of good accuracy and can be used in practice.

4 Conclusion In the paper, we put forward a scene-feature based eye-in-hand calibration method. In first step, ORB feature extraction and Bundle Adjustment are used to deal with small motion image sequence to get camera intrinsic parameters. In second step, ORB feature extraction and PnP are used to deal with obvious motion image sequence, followed by BA optimization. Finally, we get hand-eye transformation matrix. We compare our experiments results with existing methods results, and find that our method is applicable and effective. At the same time, since our method requires no man-made marker, it is time-saving and can be conveniently used for multicalibration in practice.

A Scene Feature Based Eye-in-Hand Calibration Method …

191

References 1. Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. 2. Mendonca, Paulo, R. S., & Cipolla, R. (1999). A simple technique for self-calibration. In IEEE Computer Society Conference on Computer Vision & Pattern Recognition. IEEE (pp. 500–505). 3. Anthony, W., & Gerhard, R. (2004). Estimating intrinsic camera parameters from the fundamental matrix using an evolutionary approach. Eurasip Journal on Advances in Signal Processing, 2004(8), 1–12. 4. Tsai, R. Y., & Lenz, R. K. (2002). A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Transactions on Robotics and Automation, 5(3), 345–358. 5. Triggs B. (1999). Bundle adjustment—A modern synthesis. In Proceedings of the International Workshop on Vision Algorithms: Theory and Practice. Springer (pp. 298–372). 6. Ha, H., Im, S., Park, J., et al. (2016). High-quality depth from uncalibrated small motion clip. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (pp. 5413–5421). 7. Yu, F., & Gallup, D. (2014). 3D reconstruction from accidental motion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (pp. 3986–3993). 8. Rublee, E., Rabaud, V., Konolige, K., et al. (2012). ORB: An efficient alternative to SIFT or SURF. In 2011 International Conference on Computer Vision. IEEE (pp. 2564–2571). 9. Jakubovi´c, A., & Velagi´c, J. (2018). Image feature matching and object detection using Bruteforce matchers. In 2018 International Symposium ELMAR (pp. 83–86). 10. Agarwal, S., Mierle, K., et al. (2014). Ceres solver. http://ceres-solver.org. 11. Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015). ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5), 1147–1163. 12. Mur-Artal, R., Tardos, J. D. (2017). ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 1–8. 13. Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 6(6), 381–395. 14. Jiang, Z., & Wu, W. (2010). An essential matrix-based camera self-calibration method. Journal of Image and Graphics, 15(4), 565–569.

Vision-Based Trajectory Planning for a Five Degree of Freedom Assistive Feeding Robotic Arm Using Linear Segments with Parabolic Blend and Cycloid Functions Priyam A. Parikh, Keyur D. Joshi, and Reena Trivedi

1 Introduction and Background This paper mainly focuses on the trajectory planning of a serial manipulator using joint space scheme and face reorganization algorithm. Trajectory planning of a multidegree of freedom robot can be done using Cartesian scheme as well as joint space scheme [1]. Since Cartesian scheme mainly deals with end-effectors’ position, orientation and their time derivatives, it is not recommended to design trajectory in Cartesian plane. Researchers do not use Cartesian space for trajectory planning, because the inverse of the motion transfer matrix or Jacobian matrix does not exist [2]. The robotic arm is designed to feed the physically challenged people, who are suffering from Parkinson, neurological disorder, paralyses. End-effector of the robot is a spoon, having single degree of freedom. Therefore, it is important to control joint acceleration and angular velocity as the arm carries food. Many trajectory planning methods are available in joint space scheme; e.g. cubic polynomial, fifth order polynomial, sixth order polynomial and many more. It is quite obvious that the higher the order of polynomial, smoother will be the trajectory. However trajectory planning using lower order polynomial or linear polynomial is also possible, but it gives discontinuities in joint rates, which in turn causes more vibrations. Moreover, actuator must provide almost an infinite acceleration to achieve instantaneous velocity in the case of straight line or first order polynomial [3]. In order to counter this issue, straight line trajectory is divided in three parts, where first part and last part are generated P. A. Parikh (B) · R. Trivedi Nirma University, Ahmedabad, India e-mail: [email protected] R. Trivedi e-mail: [email protected] K. D. Joshi Ahmedabad University, Ahmedabad, India e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_16

193

194

P. A. Parikh et al.

using second order polynomial and the middle part is a straight line. The meaning of the parabolic blend is blending of parabola with a straight line. Researchers have concluded that parabolic blend gives continuity in velocity but there is no continuity in the acceleration [4]. A discontinuity in acceleration and velocity is not desired for such a robotic arm which carries food. On the other hand, higher order polynomials give smoother trajectory, but provide acceleration peaks in the middle of the trajectory. To resolve that problem one can also achieve zero velocity and acceleration at the start point and end point of the trajectory without resorting to a higher order polynomial [5]. This can be achieved by designing trajectory using cycloid functions. However sometimes full cycloid function also provides peaks of acceleration in middle of the trajectory, but the magnitude of the peaks are lesser compared to higher order polynomials [6]. To overcome above mentioned problems, a trajectory is designed using semi cycloid function, which gave smoother trajectory, cut down the peaks of acceleration and provided zero acceleration at the ends. However it provided nonzero value of velocity at the end of the trajectory. Figure 1a shows the 3D model of the robot, whereas hardware setup of the robot with nomenclature is shown in Fig. 1b, c respectively. All the hardware details of the robot are shown in Table 1. This paper is divided in three parts; first part discusses about the hardware setup, problem description and literature survey. In the second part of the paper, the kinematics and inverse kinematics are discussed. The trajectory planning is shown in the last part of the paper. In addition with that comparison of trajectory generated by LSPB (Linear segment with parabolic blend), cycloid and semi cycloid function are shown along with angular velocity and angular acceleration.

Fig. 1 Details of feeding robotic arm a 3D CAD design, b actual model, c nomenclature, and d D-H matrix frame assignment

Vision-Based Trajectory Planning for a Five Degree of Freedom …

195

Table 1 Hardware details #

Details of feeder robotic arm used Parameter

Description

1

Degree of freedom

Five

2

Type of robotic arm

T-R-R-R-R

3

Joint actuators (total five)

Servo motor: metal gears, stall torque 15 kg/cm

4

Working speed

0.13 s/60° at 7.2 V (no load)

5

Working voltage

4.8–7.2 V

6

Controller board used

Arduino Mega

7

Battery

LI-Po 7.2 V

1.1 Literature Review Researchers have worked on trajectory planning to make it smoother and to get continuity in acceleration as well as velocity. Reham [7] worked on trajectory tracking control for robot manipulator using fractional order fuzzy PID controller, using which they achieved lesser steady state error. However they applied this algorithm to 5th order polynomial. Zhang et al. [8] designed trajectory for 6DOF robot manipulator using genetic algorithm to reduce robot operation time. Many researchers tried to redevelop inverse kinematics algorithms, some of them tried minimize the number of solutions, but the issue of singularity was always there. Wang et al. [9] analyzed singularity for 7R 6DOF painting robot to prevent robot joint from folded back situation. Valente et al. [10] developed jerk-bounded trajectory for industrial robots which was based on sine jerk motion profile. Furthermore Zhao et al. [11] gave a comparison between cubic polynomial and quintic polynomial for a 6DOF robotic arm. Gasparetto and Zanotto [12] deigned trajectory using cubic B spline and quintic B-spline method for 6DOF industrial robots. Li et al. [13] gave an approach for smoother trajectory planning for high speed pick and place parallel robots using quintic B-splines. Gallant and Gosselin [14] extended the capability of a robotic manipulator using optimizing robot trajectory with the help of SQP algorithm. Kucuk [15] designed trajectory using optimal trajectory generation (OTGA) for serial and parallel manipulators.

1.2 Problem Description and Methodology The robot trajectory is divided in three parts; the first part of the trajectory deals with deceleration, in which the end-effector of the robot grabs the food. In the second part robot carries the food and tries to reach at the destination, which is an acceleration mode. In the last part of the trajectory, end-effector decelerates near the destination point. It becomes important to control over the acceleration and velocity.

196

P. A. Parikh et al.

Initially the trajectory was designed using sixth order polynomial. But this method was not able to perform in the middle part of the trajectory as it was giving positive and negative peaks of acceleration in the middle stage. In the next iteration, trajectory was designed using combination of parabolic blend and first order polynomial. This method provided continuity in the velocity, but failed to provide continuity in acceleration. It also provides zero acceleration at the middle of the trajectory, which is also not suitable. Furthermore, trajectory was designed using fully cycloid function. It satisfied zero velocity and acceleration conditions at the end-points of the trajectory, but provided multiple peaks of acceleration and deceleration, which is not desired. At last, the trajectory was designed semi cycloid function, which made trajectory smoother, protected middle part of the trajectory from zero acceleration, cut down the acceleration peaks at the middle and protected actuator from providing instantaneous velocity and infinite acceleration. The webcam attached to the robot recognizes the face of the user and locates mouth. Based on mouth position, it finds the position vector in XYZ plane. As shown in Fig. 1(d, the robot works in XZ plane (as per DH matrix), hence the position vector in X direction remains same for all patients. Camera finds only the position vector in Z direction (height). The values of position or target vector are fed in inverse kinematics algorithms, which calculate joint angles. Based on these joint angles, trajectories are planned in joint space.

2 Trajectory Planning 2.1 Kinematic Analysis As discussed earlier, it becomes essential to perform inverse kinematics before getting into the joint space trajectory planning. Inverse kinematic is done using Peter-Corke toolbox in MATLAB. Robotic arm’s home position before applying any kinematics assignments is shown in Fig. 2. For the particular home position, initial angles along

Fig. 2 Simulated initial (Home) and final position of the feeding robotic arm

Vision-Based Trajectory Planning for a Five Degree of Freedom …

197

Table 2 Joint parameters forward and inverse kinematics where JA stands for joint angle, JD stands for joint distance, TA stands Twsting angle and LL stands for link length Joint #

Joint parameters forward kinematics

Inverse Kinematics

JA initial (°)

JD (mm)

TA (°)

LL (mm)

JA initial (°)

JA final (°)

1

180

50

−90

0

−21

0

2

90

0

0

140

119

61

3

0

0

0

120

0

−43

4

0

0

0

100

0

−7

5

−90

0

0

80

−87

−10

with DH parameters are shown in Table 2. For the initial and final position of the robot, corresponding transformation matrices of end-effector with respect to base are shown in Eqs. 4 and 5 in Chap. 15 respectively. It should be noted that final position varies with the height of patient. The initial and final angles found from inverse kinematics are shown in Table 2. Equations for Joint 1 and Joint 4 are not taken in consideration as the difference between their boundary conditions is approximately zero. ⎡

⎤ −1 0 0 −80 ⎢ 0 0 −1 0 ⎥ ⎥ T05 = ⎢ ⎣ 0 −1 0 310 ⎦ 0 0 0 1 ⎡ ⎤ 0.98 0.17 0 380 ⎢ 0 0 −1 0 ⎥ ⎥ T05 = ⎢ ⎣ −0.17 0.98 0 −110 ⎦ 0 0 0 1

(1)

(2)

2.2 Locating the Target The webcam from INTEX is mounted on the upper base of the robotic arm. Camera captures an image only for one time per user to get information of user seating in the chair. This image is processed to locate the height of the user’s mouth. The face was recognized by using Viola Johns algorithm [16]. The height of mouth from top of a face on average was found to be around 80% of the total height of the face from the top. This height of the mouth was set as target for the feeding robotic arm. This is required only one time for one user. Small variation in the mouth location (e.g. around ±0.5 cm) is not a concern, as mouth position can be adjusted by the user.

198

P. A. Parikh et al.

2.3 Parabolic Blend with Linear Path Trajectory planning using parabolic blend and linear path is very common in industries. As shown in Fig. 3, this trajectory is divided in three parts; (i) constant acceleration or ramp velocity (ii) zero acceleration and (iii) constant deceleration. The total trajectory time is T = 6 s. A generalized plot of joint angles versus time is shown in Fig. 4 to identify the conventions used in this paper. Time taken by robot to reach from θi to θA is tf1 = 1 s. Similarly, it takes tf3 = 1 s to reach from θB to θf. . Robot takes tf2 = 4 s to reach from θA to θB . All the joint angles with their intermediate points are shown in Table 3 with angular acceleration and velocity. For the first part of the trajectory, a second order polynomial with its two time derivatives is shown in Eqs. 4 to 5 in Chap. 15 respectively. All the initial and final conditions are shown in Eqs. 6, 7 in Chap. 15, and Eq. 1 respectively. Angular displacement and velocity at intermediate points A and B, can be found using Eqs. 3– 5. Angular displacement for joint 2, 3 and 5 are shown jointly in Eq. 6. Angular velocity in the linear path can be found using Eq. 6 (Table 4). θ(t1 ) = a + bt1 + ct21

(3)

θ(t˙1 ) = b + 2ct1

(4)

θ(t¨1 ) = 2c

(5)

Fig. 3 Robotic feeding arm forward pass end position according to position of the user’s mouth

Vision-Based Trajectory Planning for a Five Degree of Freedom …

199

Fig. 4 Trajectory segmentation with convention used in this paper

Table 3 Joint angles with Intermediate points Joint

Joint angles, angular velocity and angular acceleration θ¨ (°/s2 ) Angular velocity in the linear path (°/s) θi θA θB θf

θ˙ A (°/s)

1

0

0

0

0

0

0

0

2

−85

−75

33

45

20

27

20

3

−75

−65

−52

−42

20

3.25

20

4

37

0

0

37

0

0

0

5

124

114

−28

−38

20

−35.5

20

Table 4 Values of cooficients for the parabolic blend and linear parth a b c

Parabolic blend first half

Middle Part

Parabolic blend last half

θi θ˙ i 

θA

θf θ˙ B +

θA −θi t2f1

θB −θA tf2

−

θ˙ i tf1

NA

2(θB −θf ) T−tf1 −tf2

(θf −θB ) (T−tf1 −tf2 )2 θ˙ B T−tf1 −tf2

−



t1 = 0 s θi = a, 2 θA = a + btf1 + ctf1 , t1 = tf1

θ˙ = b, t1 = 0 s θ(t˙1 ) = ˙ i θA = b + 2ctf1 , t1 = tf1

θ(t1 ) =

(6)

(7)

200

P. A. Parikh et al.



θ¨ i = 2c, t1 = 0 s θ¨ A = 2c, t1 = tf1 ⎡ ⎤⎡ ⎤ ⎡ ⎤ 1 0 0 a θi ⎣ 1 tf1 t2 ⎦⎣ b ⎦ = ⎣ θA ⎦ f1 θ¨ i 0 1 0 c

θ(t¨1 ) =

(8)

(9)

¨ 2f1 θA = θi + 0.5(θ)t

(10)

θB = θf − θA + θi

(11)

¨ f1 θ˙ A = θ¨ i + θt

(12)

Angular velocity in the linear path =

θB − θA T − 2tf1

⎧ ⎨ −85 + 10t21 , for joint 2 θ(t1 ) = −75 + 10t21 , for joint 3 ⎩ 124 − 10t21 , for joint 5

(13)

(14)

Similarly, angular displacement, angular velocity and angular acceleration for the last part are given in Eqs. 7 to 9 respectively. Initial and final conditions for the last part of the trajectory are given in Eqs. 10 to 13 respectively. Angular displacement for joint 2, 3 and 5 are shown jointly in Eq. 13.

θ(t3 ) =

θ(t3 ) = a + b(T − t3 ) + c(T − t3 )2

(15)

θ(t˙3 ) = −b + 2c(t3 − T)

(16)

θ(t¨3 ) = 2c

(17)

θB = a + b(T − tf1 − tf2 ) + c(T − tf1 − tf2 )2 , t3 = tf1 + tf2 = 5 s (18) t3 = T θf = a,

θ˙ = −b + 2c(tf1 + tf2 − T), t3 = tf1 + tf2 s θ(t˙3 ) = ˙ B (19) θf = −b, t3 = T

θ¨ = 2c, t3 = tf1 + tf2 s ¨ θ(t3 ) = ¨ B (20) θf = 2c, t3 = T

Vision-Based Trajectory Planning for a Five Degree of Freedom …

⎤⎡ ⎤ ⎡ ⎤ 1 T − tf1 − tf2 (T − tf1 − tf2 )2 a θB ⎣1 ⎦⎣ b ⎦ = ⎣ θF ⎦ 0 0 θ˙ B 0 −1 2(tf1 + tf2 − T) c ⎧ ⎨ −38 + 10(6 − t3 )2 , for joint 2 θ(t1 ) = −42 − 10(6 − t3 )2 , for joint 3 ⎩ 45 − 12(6 − t3 )2 , for joint 5

201

⎡

(21)

(22)

Middle path of the trajectory, which is a linear displacement, is shown along with its first and second time derivatives in Eqs. 15–17. Its initial and final conditions are shown in Eqs. 18 and 19 respectively. Angular displacement for joint 2, 3 and 5 are shown jointly in Eq. 22. All the coefficients are calculated using Eqs. 2, 13 and 21. θ(t2 ) = a + b(t2 − tf1 )

(23)

θ(t˙2 ) = b

(24)

θ(t¨2 ) = 0

(25)



t2 = tf1 s θA = a, θB = a + btf2 , t2 = tf1 + tf2

θ˙ A , t2 = tf1 s θ(t˙2 ) = b, t2 = tf2

0, t2 = tf1 s θ(t¨2 ) = 0, t2 = tf2      1 0 a θ = A θB 1 tf2 b ⎧ ⎨ −75 + 27(t2 − 1), for joint 2 θ(t2 ) = −65 + 3.25(t2 − 1), for joint 3 ⎩ 114 − 35.5(t2 − 1), for joint 5 θ(t2 ) =

(26)

(27)

(28)

(29)

(30)

2.4 Trajectory Design Using Cycloid Function This sub-section discusses about trajectory design using cycloid functions. A cycloid is defined as the curve traced by a point on a circle as it rolls on a straight line without slipping. It is a geometric entity used by gears and cams [17]. Angular displacements

202

P. A. Parikh et al.

along with its first and second time derivatives are shown in Eqs. 23 to 25 respectively. Angular displacements for all the joints are shown in Eq. 27.   T 2πt θf − θi t− sin θ(t) = θi + T 2π T   ˙ = θf − θi 1 − cos 2πt θ(t) T T   ¨ = θf − θi 2π sin 2πt θ(t) T T T   ⎧ 6 2πt sin t − ⎨ −85 + 21.67 2π 6  , for joint 2  6 πt θ(t2 ) = −75 + 5.5 t − 2π sin 6 , for joint 3  ⎩ 6 sin πt , for joint 5 124 − 27 t − 2π 6

(31) (32) (33)

(34)

2.5 Trajectory Design Using Semi Cycloid Function Angular displacements along with its first and second time derivatives are shown in Eqs. 28 to 30 respectively. Angular displacements for all the joints are shown in Eq. 31.   T πt θf − θi t − sin θ(t) = θi + T π T   ˙ = θf − θi 1 − cos πt . θ(t) T T   θf − θi π πt ¨ θ(t) = sin T T T   ⎧ 6 πt sin  , for joint 2 ⎨ −85 + 21.67 π 6  t − θ(t¨2 ) = −75 + 5.5 t − π6 sin πt , for joint 3 6  ⎩ , for joint 5 124 − 27 t − π6 sin πt 6

(35) (36) (37)

(38)

3 Results A trajectory (angular displacement v/s time) generated and compared for joint 5 using LSPB, cycloid and semi cycloid is shown in Fig. 5. Comparison of angular velocity and angular acceleration is shown in Figs. 6 and 7 respectively.

Vision-Based Trajectory Planning for a Five Degree of Freedom …

203

Fig. 5 Angular displacement versus time using all three trajectory methods for Joint 5

Fig. 6 Angular velocity versus time using all three trajectory methods for Joint 5

As shown in Fig. 5, Semi cycloid trajectory is smoother than cycloid and LSPB as is does not contain any linear part as well as it does not produce any peaks of displacement. As shown in Fig. 6, LSPB generates trapezoidal velocity response to protect actuator from providing infinite acceleration at the beginning. The cycloid function provides zero velocity at the beginning and at the end, whereas semi cycloid trajectory is not able to provide zero velocity at the end point. As shown in Fig. 7,

204

P. A. Parikh et al.

Fig. 7 Angular acceleration versus time using all three trajectory methods Joint 5

LSPB is not able to provide zero acceleration at the end of the trajectory, which is not desired. Cycloid and semi function are able to achieve zero acceleration at the ends.

4 Conclusion and Future Scope The trajectory generated by LSPB gives continuity in velocity, but there is no continuity of acceleration. LSPB fails to protect servo actuator from providing instantaneous velocity in the middle of the trajectory. It also fails to protect actuator from zero acceleration in the middle part. Therefore, constant velocity and zero acceleration forces actuator to draw more current instantaneously, which can cause winding damage along with unwanted vibrations. Due to above mentioned reasons, trajectory generated by LSPB may not be able to help robotic arm in order to protect the food in the middle of the trajectory. The trajectory generated with cycloid function satisfies zero acceleration and velocity at the beginning and end of a trajectory. However it provides peaks of acceleration and deceleration, which is a concern for a robot. Furthermore, a trajectory generated with cycloid function is not smoother due to its sinusoidal behavior. The trajectory generated using semi cycloid function removes zero acceleration and constant velocity in the middle of the trajectory, which helps servo actuator from drawing lesser current from the battery. Moreover semi cycloid function removes acceleration peaks form the trajectory and gives smoother trajectory as compared to LSPB and full cycloid function.

Vision-Based Trajectory Planning for a Five Degree of Freedom …

205

In future, we are planning to achieve continuity in acceleration and velocity by designing trajectory using ANN, Genetic algorithm and fuzzy PID controller. We plan to use an advanced open source cameras with inbuilt processors such as PIXY cam, ESP32 CAM, NOIR 16MP with raspberry pi to add more features and options for the user.

References 1. Chiaverini, S., Siciliano, B., & Egeland, O. (1994). Review of the damped least-squares inverse kinematics with experiments on an industrial robot manipulator. IEEE Transactions on Control Systems Technology, 2(2), 123–134. https://doi.org/10.1109/87.294335. 2. Cheah, C. C., Kawamura, S., & Arimoto, S. (1999). Feedback control for robotic manipulator with an uncertain Jacobian matrix. Journal of Robotic Systems, 16(2), 119–134. https://doi. org/10.1002/(sici)1097-4563(199902)16:2%3c119:aid-rob5%3e3.0.co;2-j. 3. Guan, Y., Yokoi, K., Stasse, O., & Kheddar, A. (2005). On robotic trajectory planning using polynomial interpolations. In 2005 IEEE International Conference on Robotics and Biomimetics-ROBIO. https://doi.org/10.1109/robio.2005.246411. 4. Rossi, C., & Savino, S. (2013). Robot trajectory planning by assigning positions and tangential velocities. Robotics and Computer Integrated Manufacturing, 29(1), 139–156. https://doi.org/ 10.1016/j.rcim.2012.04.003. 5. Macfarlane, S., & Croft, E. A. (2003). Jerk-bounded manipulator trajectory planning: Design for real-time applications. IEEE Transactions on Robotics and Automation, 19(1), 42–52. https://doi.org/10.1109/tra.2002.807548. 6. Seraji, H., Long, M. K., & Lee, T. S. (1993). Motion control of 7-DOF arms: The configuration control approach. IEEE Transactions on Robotics and Automation, 9(2), 125–139. https://doi. org/10.1109/70.238277. 7. Mohammed, R. H., Bendary, F, & Elserafi, K. (2016). Trajectory tracking control for robot manipulator using fractional order-fuzzy-pid controller. International Journal of Computer Applications, 134(15), 22–29. https://doi.org/10.5120/ijca2016908155. 8. Zhang, J., Meng, Q., Feng, X., & Shen, H. (2018). A 6-DOF robot-time optimal trajectory planning based on an improved genetic algorithm. Robotics and Biomimetics, 5(1). https://doi. org/10.1186/s40638-018-0085-7. 9. Wang, X., Zhang, D., Zhao, C., Zhang, H., & Yan, H. (2018). Singularity analysis and treatment for a 7R 6-DOF painting robot with non-spherical wrist. Mechanism and Machine Theory, 126, 92–107. https://doi.org/10.1016/j.mechmachtheory.2018.03.0. 10. Valente, A., Baraldo, S., & Carpanzano, E. (2017). Smooth trajectory generation for industrial robots performing high precision assembly processes. CIRP Annals, 66(1), 17–20. https://doi. org/10.1016/j.cirp.2017.04.105. 11. Zhao, X., Wang, M., Liu, N., & Tang, Y. (2017). Trajectory planning for 6-DOF robotic arm based on Quintic polynomial. In Proceedings of the 2017 2nd International Conference on Control, Automation and Artificial Intelligence (CAAI 2017). https://doi.org/10.2991/caai-17. 2017.23. 12. Gasparetto, A., & Zanotto, V. (2010). Optimal trajectory planning for industrial robots. Advances in Engineering Software, 41(4), 548–556. https://doi.org/10.1016/j.advengsoft.2009. 11.001. 13. Li, Y., Huang, T., & Chetwynd, D. G. (2018). An approach for smooth trajectory planning of high-speed pick-and-place parallel robots using quintic B-splines. Mechanism and Machine Theory, 126, 479–490. https://doi.org/10.1016/j.mechmachtheory.2018.04.0. 14. Gallant, A., & Gosselin, C. (2018). Extending the capabilities of robotic manipulators using trajectory optimization. Mechanism and Machine Theory, 121, 502–514. https://doi.org/10. 1016/j.mechmachtheory.2017.09.

206

P. A. Parikh et al.

15. Kucuk, S. (2017). Optimal trajectory generation algorithm for serial and parallel manipulators. Robotics and Computer-Integrated Manufacturing, 48, 219–232. https://doi.org/10.1016/ j.rcim.2017.04.006. 16. Viola, P., & Jones, M. J. (2001). Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1 (pp. 511–518). 17. Saha, S. K. (2014). Introduction to robotics 2e. Mcgrawhill education. ISBN 93-329-02801.

Structure Design and Closed-Loop Control of a Modular Soft-Rigid Pneumatic Lower Limb Exoskeleton Jiangbei Wang and Yanqiong Fei

1 Introduction The exoskeletons have been developed from academic to commercial applications [1, 2] in recent decades. The conventional rigid exoskeletons are constituted by rigid links and joints actuated by electric or hydraulic motors [3]. They have the advantages of high force output, good controllability and greater accesses to mature fabrication and integration technologies. However, the rigid exoskeletons are confronted with challenges such as poor compliance with the human body, difficult alignment with the biological joints [4] and large weight/inertia applied on the biological extremities, degrading the wearing comfort, safety and metabolic efficiency [5]. The soft exoskeletons that adopt soft actuators, soft sensors and soft structural materials such as cables [6] and fabrics [7] provide new approaches. The soft exoskeletons have no explicit mechanical joints or take the soft actuators as the flexible joints [8]. Two mainstream configurations of the soft exoskeletons are the Tendon-Driven Exoskeletons (TDEs) [6] and the Bending-Driven Exoskeletons (BDEs) [9], differentiated by their soft actuators. The TDEs have no explicit mechanical joints and can be made lightweight, lowprofile and highly flexible, indicating good wearability and portability. However, the high tension forces (up to 80 N [6]) required to actuate the biological joints, induce large compression forces to the joints and shear forces to the skin of extremities, which may reduce the wearing comfort. Reasearch supported by the National Natural Science Foundation of China under Grant No. 51475300 and 51875335, Joint fund of the Ministry of Education under Grant No. 18GFAZZ07-171, Institute of Medical Robotics of Shanghai Jiao Tong University under Grant No. IMR2019QY01. J. Wang · Y. Fei (B) Research Institute of Robotics, Shanghai Jiao Tong University, Shanghai 200240, China e-mail: [email protected] Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_17

207

208

J. Wang and Y. Fei

The BDEs utilize the soft bending actuators as their active soft hinges to provide torques for rotation of the biological joints [9]. The torque is exerted as several couples of transverse forces on the wearer’s extremities, producing no net pulling or pushing forces on the biological joint and only normal instead of shearing pressures on the biological limbs, which overcomes the aforementioned disadvantage of the TDEs. From the ergonomics, the exoskeletons with a variable-stiffness or heterogenous structure [10] exhibit better conformity with the human body than those with a homogeneous structure [11]. Therefore, it is necessary to design the BDEs into a soft-rigid structure, i.e. with soft hinges and rigid links. The soft hinges can supply compliant actuation for the wearer’s joints, and the rigid links can transfer the actuation forces from the exoskeleton to the human body effectively. The rigid links also leave spaces for integrating sensors and circuits for data acquisition and wireless communication, enabling the feedback control without the tethering of electric wires. In the paper, we propose the novel modular soft-rigid BDEs actuated by the CPAMs, incorporate the sensing circuits into it, and further achieve the closed-loop control.

2 Structure Design In anatomy, the lower limb of human body mainly includes three joints (hip, knee and ankle) and three skeletal regions (thigh, crus and foot), in which the hip joint links the whole lower limb to the waist [12]. The proposed soft-rigid exoskeleton is designed to assist the flexion and extension of the hip, knee and ankle joints through three customized bidirectional CPAMs which are connected by four 3D-printed rigid parts corresponding to the waist, thigh, crus and foot respectively (Fig. 1a). Sensors and transmission circuits are embedded into each rigid part to monitor the wearer’s motion and publish the sensed signals wirelessly (Fig. 1b). According to the locations on the human body, the exoskeleton is divided into four modules, i.e. the waist-hip, thigh-knee, crus-ankle and foot modules (Fig. 1c). Each module consists of a CPAM (sealed by the proximal and distal rigid caps) and a rigid case (with the upper and lower parts 3D-printed individually and then assembled by the screws and nuts for containing the sensing unit) except the foot module which only has the rigid case. Two different modules can be joined by the complementary convex and concave surfaces at their two ends. The main body of the bidirectional CPAM is in shape of elliptical cylinder (for the hip and knee joints) or torus (for the ankle joint), which is made of the longitudinally elastic fabric (polyester & latex) and the reinforcement inextensible fabric (cotton & linen) on the neutral plane (Fig. 1b). To ensure the air tightness, two elastomeric inner bladders (ELASTOSIL® M 4601 A/B) are inserted into the two cavities respectively, resulting in two closed air chambers. The proximal and distal rigid caps are plugged into the two ends of the cylinder or torus and then tied with the fabrics by the hose clamps, resulting in the bidirectional CPAM. For each CPAM, pressurization of the

Structure Design and Closed-Loop Control …

209

Fig. 1 Structure design of the modular soft-rigid pneumatic lower limb exoskeleton. a The external electro-pneumatic control system. b The structure of single module. c Four modules of the exoskeleton ➀ waist-hip module, ➁ thigh-knee module, ➂ crus-ankle module, ➃ foot module

chamber on one side leads to the bending deformation towards another side, and thus the bidirectional bending can be achieved. The left side chamber’s pressure denotes positive pressure while the right one denotes negative pressure. The sensing unit of each module (Fig. 1b) mainly includes two pressure sensors for monitoring the inflating pressure of the two bladders respectively, an Inertia Measurement Unit (IMU) for detecting the inclination angle of each skeletal segment, and two flex sensors stacked face-to-face and sandwiched between the two inextensible fabric

210

J. Wang and Y. Fei

layers for measuring the bidirectional bending angle of the bidirectional CPAM. The sensed signals are collected by the Arduino Pro Mini and transmitted by the Bluetooth module wirelessly. For reducing the exoskeleton weight, the electro-pneumatic control system is not integrated with the exoskeleton (Fig. 1a). The only tethering between the exoskeleton and the control system is the detachable pneumatic pipes. On the control board (Arduino Mega 2560), four Bluetooth packages are incorporated to receive the signals transmitted from the four sensing units of the exoskeleton individually. According to the received signals and the programmed control algorithm, the control board can automatically adjust the inflating pressures of the soft hinges of the exoskeletons via the Electro-Pneumatic Regulators (EPRs). The whole electronic system is powered by a linear DC source and the pressure is supplied by an air compressor.

3 Closed-Loop Control and Experiment Able to detect the inflating pressures and bending angles of the exoskeleton hinges, we can further implement the closed-loop control for the exoskeleton. The overall control scheme (Fig. 2a) consists of the position controllers (i.e. the external closed loop), the pressure servos (i.e. the internal closed loop), the CPAMs and the soft-rigid exoskeleton mechanism. This closed-loop control scheme makes the two processes (i.e. the pressure regulation and the actuator’s bending) decoupled and thus allows their parameters to be tuned individually. The pressure servo (Fig. 2b) is to control the Electro-Pneumatic Regulators (EPRs) to supply a target inflating pressure (Pri ) for the air chambers of the CPAM. First, the target pressures are decomposed into the reference pressures for the left and right chambers of the CPAM, i.e. PLri and PRri . Then the errors (PLi and PRi ) between the reference and actual pressures are calculated and amplified by the proportion   and f R0i ). Signs of the component (K PPi ), resulting in the impulse frequencies ( f L0i frequencies represent the rotation directions of the stepper motors of the EPRs, i.e. d Li and d Ri . To ensure the stepper motors work normally, the frequencies are saturated by the maximum and minimum frequencies (f min = 32 Hz and f max = 8000 Hz), resulting in f Li and f Ri . For safety, a pressure limit switch with the algorithm Eq. (1) is used to limit the inflating pressures of the chambers.  δLi,Ri =

0, ifPLi,Ri > Pmax and dLi,Ri = 1    1, if PLi,Ri > Pmax and dLi,Ri = −1 or PLi,Ri < Pmax

(1)

The proportional gain (K PPi ) is tuned to make the pressure servo response fast as well as avoid the oscillation. The final tuned gains for the three exoskeleton hinges are K PP1 = 55 Hz kPa−1 (hip), K PP2 = 90 Hz kPa−1 (knee) and K PP3 = 110 Hz kPa−1 (ankle).

Structure Design and Closed-Loop Control …

211

Fig. 2 Control scheme of the exoskeleton. a The overall control scheme. b The pressure servo. c The position controller. i = 1, 2, 3 represents the hip, knee and ankle joints respectively

The position controller (Fig. 2c) estimates the error between the target and actual bending angles (i.e. θ i ) of the exoskeleton hinges and then generates the reference pressure (Pri ) through a Proportion-Integral (PI) component. The reference pressure is then applied to the CPAM through the pressure servo and thus actuate the exoskeleton hinge to bend according to the target signals. The proportional gain (K Pθ i ) is to improve the response speed while the integral gain (K Iθ i ) is to eliminate the static error of the bending angles. For the position control, the gains are tuned by the Good Gain method [13], resulting in K Pθ 1 = 43 kPa rad−1 , K Iθ 1 = 80 kPa rad−1 s−1 (hip), K Pθ 2 = 70 kPa rad−1 , K Iθ 2 = 130 kPa rad−1 s−1 (knee) and K Pθ 1 = 86 kPa rad−1 , K Iθ 1 = 158 kPa rad−1 s−1 (ankle). The pressure servos and position controllers are applied to control the exoskeleton to track a standard gait cycle [14]. Figure 3a shows that the proposed exoskeleton can achieve the motion of all phases in the gait cycle. It should be noted that gait cycle time is elongated into 40 s in the experiment due to limit of the exoskeleton’s response speed.

212

J. Wang and Y. Fei

Fig. 3 The exoskeleton tracking the joint angles of a gait cycle. a The final states of different phases. b–d The joint angles. ➀ Initial contact. ➀~➁ Loading response. ➁~➂ Mid stance. ➂~➃ Terminal stance. ➃~➄ Pre swing. ➄~➅ Initial swing. ➅~➆ Mid swing. ➆~➇ Terminal swing

According to Fig. 3b–d, the bending angles of the exoskeleton hinges can be controlled effectively. In the stable stage (10–120 s), the maximum tracking errors of the hip, knee and ankle joints are 5.9°, 7.4° and 1.7° respectively. The tracking errors majorly occur when the reference pressures (Pri ) cross zero. The inflating pressures of the left (right) chambers of the exoskeleton hinges should be zero when the reference pressures are negative (positive) in theory, i.e. PLi = 0 if Pri < 0, and PRi = 0 if Pri > 0. But in practice, due to the offset of the pressure sensors and the residual of the regulators, it is impossible to measure or control the zero pressure. Therefore, to ensure the controllability of the inflating pressure, we set a lower boundary for it, i.e. Pmin = 0.01 MPa. It is why the inflating pressures (PLi and PRi ) cannot track the reference pressures (Pri ) in the range from −0.01 to 0.01 MPa, which is the main cause of the position tracking errors. Figure 3b–d also indicates that the static response of the exoskeleton exhibits an overall delay of 0.64 s (hip), 0.58 s (knee) and 0.44 s (ankle) relative to the reference signals. In addition, the transition response (5–10 s) at start of the tracking shows a natural oscillation frequency of 2.5 Hz (or period of 0.4 s), which constricts the respond speed of the exoskeleton and causes the aforementioned time delays. We can reduce the effect of the time delays on the tracking accuracy by elongating the gait cycle (to 40 s for instance), as shown in Fig. 4, but at cost of low operation speed. In general, the proposed exoskeleton shows good controllability despite the slow response speed. The proposed exoskeleton is able to assist the basic movement of the lower limb at low speed.

Structure Design and Closed-Loop Control …

213

Fig. 4 Gait tracking of the exoskeleton with cycle time of a 2 s, b 5 s, c 20 s, d 40 s

4 Conclusion In this work, a novel modular soft-rigid pneumatic exoskeleton for lower limb is presented. It consists of three soft-rigid modules, one foot module and an external electro-pneumatic control system. In each soft-rigid module, onboard sensing units are integrated with the exoskeleton for detecting the inflating pressures and bending angles of the pneumatically actuated soft hinges, and also transmitting the signals to external subscribers wirelessly. The exoskeleton is controlled by an external electropneumatic control system which receives the signals from the sensing units and then supplies the appropriate inflating pressures for the exoskeleton accordingly. The gait tracking experiments show that the soft-rigid structure allows the exoskeleton to conform well with human body and its controllability is verified.

214

J. Wang and Y. Fei

References 1. Eschweiler, J. M. P., Gerlach-Hahn, K., Jansen-Troy, A., & Leonhardt, S. (2014). A survey on robotic devices for upper limb rehabilitation. Journal of NeuroEngineering and Rehabilitation. 2. Rupal, B., Rafique, S., Singla, A., Singla, E., Isaksson, M., & Virk, G. (2017). Lowerlimb exoskeletons: Research trends and regulatory guidelines in medical and non-medical applications. International Journal of Advanced Robotic Systems, 14, 1–27. 3. Redlarski, G., Blecharz, K., D˛abkowski, M., Pałkowski, A., Tojza, P. M. (2012). Comparative analysis of exoskeletal actuators. Pomiary Automatyka Robotyka. 4. Schiele, A. (2009). Ergonomics of exoskeletons: Objective performance metrics. In World Haptics 2009 Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 103–108). 5. Browning, R. C., Modica, J. R., Kram, R., & Goswami, A. (2007). The effects of adding mass to the legs on the energetics and biomechanics of walking. Medicine and Science in Sports and Exercise, 39(3), 515–525. 6. Asbeck, A. T., De Rossi, S. M. M., Holt, K. G., & Walsh, C. J. (2015). A biologically inspired soft exosuit for walking assistance. The International Journal of Robotics Research (IJRR), 34(6), 744–762. 7. Yap, H. K., et al. (2017). A fully fabric-based bidirectional soft robotic glove for assistance and rehabilitation of hand impaired patients. IEEE Robotics and Automation Letters, 2(3), 1383–1390. 8. Koh, T. H., Cheng, N., Yap, H. K., & Yeow, C. H. (2017). Design of a soft robotic elbow sleeve with passive and intent-controlled actuation. Frontiers in Neuroscience. 9. Hassanin A. F., Steve, D., & Samia N. M. (2017). A novel, soft, bending actuator for use in power assist and rehabilitation exoskeletons. In IEEE International Conference on Intelligent Robots and Systems. 10. Yap, H. K., Lim, J. H., Nasrallah, F., Goh, J. C. H., & Yeow, R. C. H. (2015) A soft exoskeleton for hand assistive and rehabilitation application using pneumatic actuators with variable stiffness. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 4967–4972). 11. Polygerinos P., et al. (2013). Towards a soft pneumatic glove for hand rehabilitation. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1512–1517). 12. OpenStax. (2013). Anatomy and Physiology, 1st ed. Houston: OpenStax. 13. Haugen, F. (2012). The good gain method for simple experimental tuning of PI controllers. Modeling, Identification and Control. 14. Charalambous, C. P. (2014). Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait. In Classic Papers in Orthopaedics.

Sensing Methods and Actuation

In this part, the first chapter concerns prediction of the way that flexible objects will move when manipulated. The effects of friction are simulated, and then, experiments are performed to validate the results. For the advanced control of an actuated prosthesis, it is necessary to derive signals from the remaining muscles. This chapter describes the development, construction and evaluation of a highly portable sensor. Next is another chapter dealing practically with the interaction between a robot and soft tissue. The human eye alters its focal length by muscular deformation of the tissue of the lens. In this chapter, a hydrogel is used to obtain the same objective by the application of an electrical potential.

Real-Time, Dynamic Simulation of Deformable Linear Objects with Friction on a 2D Surface Benjamin Maier, Marius Stach, and Miriam Mehl

1 Introduction Stationary robotic arms can be used to help in the assembly process of various devices, e.g., for populating electronic circuit boards, assembling pumps or casings of appliances. Many of those fields of application involve handling flexible, linear objects such as electrical and optical wires, strings and cords, hoses and flexible tubes. Often, these objects are manipulated on a workspace with a flat surface, as this simplifies sensing and motion planning. In order to control their handling, control algorithms can benefit from models of the linear object, e.g., as is done in model predictive control. A key requirement is that the computation of the model is realtime, i.e., ideally having a lower runtime than the duration of the simulated process. Much research has dealt with modeling and simulation of deformable linear objects, which usually are well approximated by one-dimensional objects. A method of fitting a model to observation data by minimizing an energy function is presented in [1]. The authors of [2] establish a framework based on differential geometry for modeling deformable linear objects. They describe flexure, torsion and extension and extend their static model to account for dynamics and contact in [3] and [4]. However, one end of the object is always fixed. The authors in [5] use a similar approach but represents the linear object using straight segments to reduce computational costs. Another model based on differential geometry is presented in [6], discrete cosine transform is applied to reduce the number of parameters and computational costs, however, it only considers a static scenario. Nonlinear Finite Element Methods have also been used to model the deformation of one-dimensional objects [7, 8], but they have the disadvantage of increasing computational complexity for higher numbers of degrees of freedom.

B. Maier · M. Stach (B) · M. Mehl Institute for Parallel and Distributed Systems, University of Stuttgart, Stuttgart, Germany e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_18

217

218

B. Maier et al.

Fig. 1 The left end of a cable is moved and rotated, friction forces occur only at the parts of the cable, where it touches the surface of the table, as can be seen from the shadow

We base our work on [3] and extend their dynamic model of an inextensible, deformable linear object in a two-dimensional space by the possibility to prescribe an arbitrary motion and angle of one end of the object. Furthermore we account for friction between the object and the surface. Depending on the scenario and the rigidity of the flexible object, often only some portion of the object has contact with the surface, whereas the rest hovers slightly above and experiences no friction, as in Fig. 1. Thus, our model allows to specify friction forces for portions of the object. In order to run our simulation in real-time, we develop an efficient, parallel simulation program. We give details on our considerations and optimizations, demonstrate the real-time capabilities and make the source code freely available. By comparison with experiments we validate the modeling approach. The remainder of this paper is structured as follows: In Sect. 2 the model is derived, in Sect. 3 we address the efficiency of the program and in Sect. 4 the modelling approach is validated by comparing the simulation to experiments.

2 Modeling In this section, we model the dynamic behavior of an inextensible, elastic, onedimensional object in a 2D geometric setting. In Sects. 2.1 and 2.2, we introduce our notation which follows the work of [2]. We extend this formulation to incorporate the prescribed position and angle at the end of the object in Sect. 2.3 and friction between parts of the object and the 2D ground surface in Sect. 2.4.

2.1 Representation of the Linear Object The linear object of length L is defined as the set of points P(s) where the linear coordinate, s ∈ [0, L], is the distance along the object of point P(s) from the start ˜ y˜ ), point, P0 = P(0). This start point is assumed to have the prescribed position, (x, as depicted in Fig. 2. Let θ (s, t) be the angle at time t between the x-axis and the object   at coordinate s. The Cartesian coordinates of P(s) at time t, x(s, ˆ t), yˆ (s, t) ∈ R2 , can now be formulated as

Real-Time, Dynamic Simulation …

219

Fig. 2 Coordinates of the start point P0 and a point P(s) on the object

 P(s) =

x(s, ˆ t) yˆ (s, t)



 =

x(t) ˜ y˜ (t)



 +

x(s, t) y(s, t)



 =

x(t) ˜ y˜ (t)



s

+∫ 0



 cos θ (u, t) du. sin θ (u, t) (1)

Derivation with respect to time yields the velocity, 

˙ˆ t) x(s, y˙ˆ (s, t)



 =

x(t) ˜˙ y˙˜ (t)



 +

x(s, ˙ t) y˙ (s, t)



 =

x(t) ˜˙ y˙˜ (t)



  − sin θ (u, t) ˙ θ(u, t)du. +∫ cos θ (u, t) 0 (2) s

To spatially discretize the formulation,  we subdivide the interval of the coordinate  s into n equal-sized intervals si−1 , si , i = 1 . . . n, of size h = L/n. We use linear hat functions     s − si  ,0 (3) Ni (s) = max 1 − h    as nodal basis with Ni s j = δi j to discretize the angle θ (s, t): θ (s, t) =

n

θi (t)Ni (s)

i=0

Here, the coefficients, θi , are the degrees of freedom. The vector n = (θ0 , . . . , θn ) fully specifies the discretized state of the object, the θi values therefore serve as generalized coordinates. This ansatz approximates the shape of the object as a sequence of circular arcs. Note that, while the presented formulation leads to the same mathematical objects as

220

B. Maier et al.

in [2], we use a node based approach instead of the authors’ element based indexing, as this simplifies the equations.

2.2 Equations of Motion To formulate the dynamics, we use the Lagrangian form of the equations of motion, d ∂L ∂L − = 0∀i = 0, . . . , n ∂θi dt ∂ θ˙i

(4)

where the Lagrange function L = T − U is the difference between kinetic energy T and potential energy U . The kinetic energy T depends on the density ρ given as mass per cross section area. For now, we assume P0 = (0, 0) and, thus, xˆ = x, yˆ = y. Then, T is given by L 1 T = ∫ ρ(x˙ 2 + y˙ 2 )ds. 0 2

(5)

Inserting (2) and (5) into (4) yields the kinetic energy contribution to the Lagrange equation of motion, ¨ n + Y ˙n ˙ n − M Z The entries m i,k , Z r,k and Yr,i of the matrices M, Z and Y are defined as L

m i,k = ∫ ρ(Si Sk + Ci Ck )ds,

(6)

0

1 ∂m i,k ∂m r,i = ( − )θ˙i , 2 i=0 ∂θr ∂θk n

Z r,k

1 ∂m r,i θ˙k , 2 k=0 ∂θk n

Yr,i = − where the abbreviations s

s

0

0

Si (s, t) = ∫ sin θ (u, t)Ni (u)du, Ci (s, t) = ∫ cos θ (u, t)Ni (u)du

(7)

are used. The potential energy U is assumed to result from flexural bending of the object. It is given by

Real-Time, Dynamic Simulation …

221

U=

1   K n , 2 n

with the tridiagonal stiffness matrix, ⎡

1 −1

0

⎤

⎥ ⎢ ⎥ ⎢ −1 2 . . . ⎥ ⎢ R f lex ⎢ ⎥ . . . .. .. .. K = ⎥, ⎢ ⎥ h ⎢ ⎥ ⎢ .. ⎣ . 2 −1 ⎦ 0 −1 1 and flexural rigidity R f lex . Combining the contribution of U with the term for T , we get the equations of motion, ¨ n + Y ˙ n − K n = 0. ˙ n − M Z

(8)

2.3 Prescribed Position and Angle of the Start Point In this section, we extend the formulation to account for the prescribed, time varying displacement (x(t), ˜ y˜ (t)) of the start point P0 as formulated in Eq. (1). Furthermore, the start angle of the object at P0 is also set to a prescribed transient value. Consequently, the new kinetic energy Tˆ has contributions from the prescribed movement that is superimposed on the internal movement of the object. Following definitions (1) and (5), we get  2  2  L 1 ds Tˆ = ∫ ρ x˙ + x˙˜ + y˙ + y˙˜ 0 2  L 1  L 1   = ∫ ρ x˙ 2 + y˙ 2 ds + ∫ ρ 2 x˙ x˙˜ + 2 y˙ y˙˜ + x˙˜ 2 + y˙˜ 2 ds = T + T˜ 0 2 0 2 with the additional contribution T˜ := Tˆ − T . When calculating the derivatives of T˜ with respect to θ, θ˙ and t, as needed for the Lagrange equations of motion (4), one gets values  L  wi = ∫ ρ Ci y¨˜ − Si x¨˜ ds.

(9)

0

These terms have to be added to the left hand side of the governing equation, Eq. (8).

222

B. Maier et al.

To prescribe the angle at the start point P0 , the respective degree of freedom θ0 is simply fixed to the prescribed value, θ0 := θˆ , and removed from the vector of unknowns in the solution process.

2.4 Friction on the Underlying Surface Next, we add a formulation for the friction to our model. Friction forces occur between the object and the surface on which it slides when being manipulated. To be able to specify friction for any portion of the object, we define n + 1 discrete friction forces Rsi acting at equidistant positions, si = i · h for i = 0, . . . , n. Every force is proportional to the gravitational force Fsi at its point of action and to the respective sliding friction coefficient μi . In order to specify no friction at some intervals of the object, the corresponding values of μi can be set to zero. The direction of the friction forces is always opposite to the current direction of motion of the points of action. Thus, the discrete friction forces can be formulated as Rsi = −μi Fsi

v(si , t) , i = 0, . . . , n |v(si , t)|

 T ˙ˆ i , t), y˙ˆ (si , t) . In general, dissipative forces can with the velocities v(si , t) = x(s be considered by adding generalized forces Ri to the Lagrange equations of motion: d ∂L ∂L − + Ri = 0∀i = 0, . . . , n. ∂θi dt ∂ θ˙i For the friction, Ri is calculated as Ri =

n j=0

Rs j ·

n ∂ xs j = θ˙ j Vi, j + u i ∂θi j=0

(10)

with the symbols Vi, j

  n −μk Fsk Si (sk )S j (sk ) + Ci (sk )C j (sk )   =  v(sk ) 

(11)

k=0

and   ˙ ˙˜s Ci (sk ) n −μ F k sk − x˜ sk Si (sk ) + y k   . ui =  v(sk )  k=0 Finally, the governing equation that follows from Eqs. (8)–(12) is given by

(12)

Real-Time, Dynamic Simulation …

223

˙ n − M ¨ n + Y ˙ n − K n − w + V  ˙ n + u = 0, Z

(13)

where the entries Vi, j , u i and wi of matrix V and vectors u and w are given by Eqs. (11), (12) and (9), respectively. Note that this is a nonlinear equation in the vector of coordinates n because all vectors and matrices also depend on n . ˙ n , we transform Eq. (13) into a system of first-order, nonlinear By defining ωn :=  differential equations: ˙ n = ωn ,  M ω˙ n = Z ωn + Y ωn − K n − w + V ωn + u.

(14)

An explicit time stepping scheme can be used to solve this system of differential equations in the variables (n , ωn ) in time. Consequently, a linear system with system matrix M has to be solved in every time step.

3 Efficient Implementation In the following, we illustrate how the presented model can be solved efficiently to enable real-time computation. In Sects. 3.1–3.3, we present algorithmic, numerical and methodological considerations that reduce the runtime of a simulation program compared to naïve approaches. We give details on our reference implementation, which we make available as an open-source project.1 For every efficiency consideration, we quantify the effect in our implementation.

3.1 Algorithmic Considerations A simulation program has to compute the vectors u and w and the matrices M, Z , Y, K and V of Eq. (14) in every timestep. We consider the following term, which appears twice in the computation of Z and once in the computation of Y :   L ∂m i,k ∂ Si ∂ Sk ∂Ci ∂Ck = ∫ρ Sk + Si + Ck + Ci ds. ∂θr ∂θr ∂θr ∂θr ∂θr 0

(15)

To save computation time, in every timestep we first compute the values of L

∫ρ 0

L ∂C i ∂ Si Sk ds, and ∫ ρ Ck ds, ∂θr ∂θr 0

1 https://github.com/maierbn/dynamic_linear_object.

(16)

224

B. Maier et al.

for all required combinations of the indices i, k and r, each of them occurring twice in Eq. (15). Thus we can get every value of ∂m i,k /∂θr just by adding four of these terms. In our implementation, we measured a reduction in the total simulation runtime by 77%. A second opportunity to precompute values comes with the symmetry properties of the involved matrices. It can be easily verified that the matrices M, Y, K and V as well as the expressions in Eq. (16) are symmetric whereas matrix Z is skewsymmetric. Computing the off-diagonal entries once and reusing them leads to an additional drop in computation time by 32% in our implementation.

3.2 Numerical Considerations Several of the computed quantities are defined as integrals over a part or the whole length of the linear object, e.g., Si , Ci and wi . For their computation, a numerical quadrature scheme has to be chosen. Such schemes typically approximate the integral value by a weighted sum of the integrand evaluated at specified sampling points. For example, the composite Simpson’s rule is such a scheme. It uses equally spaced sampling points and is known to integrate polynomials of up to order 3 exactly. Also adaptive schemes exist which subdivide the integration interval recursively until a required error tolerance is met. The adaptive Archimedes quadrature is such a scheme. It approximates the integrals by the surface areas of trapezoids. We determined by numerical experiment that when using the same composite Simpson’s rule for every quadrature that occurs during computation, at least 14 sampling points per rule are required, for the whole simulation to remain numerically stable. When using the adaptive Archimedes quadrature instead, in our case we have to set the error tolerance parameter to 10−5 . It turns out that as a result we get only 2–3 function evaluations per quadrature on average, which reduces the total computation time by 44%. An explanation for this effect is that the occurring integrals are diverse, some have small integration intervals where the integrand is smooth while others involve more complex integrands. There exist nested integrals, e.g., to compute m i,k in Eq. (6), we integrate over an expression containing Si , which again consists of an integral, as can be seen in Eq. (7). To implement the quadrature efficiently, it is  crucial to only integrate over the support of the integrands. E.g., in the interval si−1 , si , only the ansatz functions Ni−1 and Ni have nonzero values. For the solution of the differential equations, Eq. (14), we use a forth order RungeKutta scheme. Using such a high order scheme allows to achieve stable results with larger time steps. Compared to using the first order forward Euler integration scheme, we could increase time steps such that the total computation time was reduced by 75%.

Real-Time, Dynamic Simulation …

225

3.3 Technical Considerations If a real-time implementation is favored, a suitable low-level programming language should be selected. By transferring our initial MATLAB implementation to C++ code, we reduce the runtime by 96%. The computations of the entries of the matrices in Eq. (14) are independent of each other and thus can be done concurrently. We parallelize our code to run on multiple cores of shared-memory computers by employing OpenMP.2 To assess the efficiency of our code, we use the Gprof tool to profile simulation runs. This can reveal bottlenecks resulting from a non-optimal implementation, which can be fixed in consequence. Thus, among other optimizations, we removed avoidable ‘if’ branches in the implementation of the ansatz function given by Eq. (3) and reduced the number of arithmetic operations in the adaptive quadrature implementation. The two mentioned functions account for 23 and 52% of the total runtime. In total, improvements lead to runtime reductions of factors of approximately 2–3.

4 Experiments In this section, four simulation experiments are carried out using the developed model. In the first two, we demonstrate the effects of friction and bending, in the last two, we compare the results to real experiments. Finally, we present runtime and scalability results.

4.1 Experiments #1 and #2—Friction and Bending In experiment #1, the parameters R f lex , ρ and L are set to 1. Initially, the onedimensional object is positioned horizontally with its start point on the right side. The start point is then constantly accelerated in y-direction with yˆ = t 2 for a time span of two seconds. The angle at the start point is not prescribed. Two runs are performed where the friction coefficient, μ, is set for the whole object to 0 or 1, respectively, corresponding to no friction and constant friction. The time step width is set to dt = 10−2 s and n = 3 elements are used. The results of these two runs are depicted in the left and right image in Fig. 3. The trajectories of the prescribed start point and of the opposite end of the object are shown by the solid blue and dashed grey lines. The colorful lines depict the states of the linear object in intervals of 0.2 s. It can be seen that the friction prevents the object from swinging to the right side. Instead, the object gets dragged along a path that closely follows the prescribed trajectory. 2 www.openmp.org/.

226

B. Maier et al.

Fig. 3 Experiment #1—accelerated movement in y-direction from bottom to top left: no friction, right: with friction

In the two runs of the next experiment, #2, the flexural rigidity, R f lex , is set to the two different values 1 and 5, while the remaining parameters are set to ρ = 1, L = 1 and μ = 0.3. The initial position of the object is horizontal with the start point on the left side. The start point is moved to the right side for one second, following the prescribed trajectory xˆ = 6t 2 + 3t. Additionally, the angle of the object at the start point describes a prescribed counterclockwise rotation of θˆ = (π/2)t. Like in experiment #1, n = 3 elements are used with a time step width of dt = 10−2 s. Figure 4 shows the results of both runs, the depicted states of the object are in the interval of 0.1 s. It can be seen that the object bends to the right at first, then to left as it is pushed over the surface. In the case with lower flexural rigidity, inertia causes the object to strongly bend such that the free end finally points downwards whereas in the case of a more rigid object the direction stays approximately the same. These two examples have shown that bending and friction play an important role to the dynamics of a one-dimensional object in the described setting. The presented modelling approach is capable of describing these effects and their influence can be studied in simulations by varying material parameters.

4.2 Experiments #3 and #4—Validation Next, we compare our model to experiments carried out with the help of a robotic arm. In experiment #3, a cable is placed horizontally on a wooden table with the right end being gripped by a Franka Emika Panda robot. The gripper is turned counterclockwise by 180° for ten seconds with a constant angular velocity, as shown in Fig. 5. The length

Real-Time, Dynamic Simulation …

227

Fig. 4 Experiment #2—accelerated movement in x-direction and counterclockwise rotation of the start angle with different flexural rigidities. top: Rflex = 1, bottom: Rflex = 5

Fig. 5 Experiment #3—manipulation of a cable

from the gripper and the density of the object are L = 40 cm and ρ = 17.8 g/m. Flexural rigidity and friction coefficient are estimated as R f lex = 9 · 10−4 Nm2 and μ = 0.3. The same action is simulated with n = 3 elements and a time step width of dt = 5 · 10−3 s. Experiment #4 is conducted using a flexible tube. Again, the gripper holds one end of the object and performs a defined movement. The movement in this scenario is a translation by the vector (40 − 10 cm) and a rotation by 135°. The movement

228

B. Maier et al.

lasts four seconds and has a smooth velocity profile of zero velocity, acceleration and jerk at both t = 0 s and t = 4 s. The experiment is visualized in Fig. 6. Again, a simulation of the scenario is run with n = 3 elements, for this case with time step width dt = 10−2 s. The used parameters are L = 80 cm, ρ = 150 g/m, R f lex = 5 · 10−2 Nm2 and μ = 0.6. The simulation results of experiments #3 and #4 are depicted in Fig. 7. In the left image, the state of the simulated object in experiment #3 is shown for intervals of 0.5 s. By visually comparing them to the photos in Fig. 5, we find a good match. In both experiment and simulation, the object initially starts to bend with the free end

Fig. 6 Experiment #4—manipulation of a flexible tube

Real-Time, Dynamic Simulation …

229

Fig. 7 Simulation results for experiment #3 (left) and #4 (right). The colorful lines depict states of the object in intervals of 0.5 s

remaining at a similar location until approximately t = 1 s, before the whole object rotates and maintains a rigid shape. Similarly, the simulation results of experiment #4 compare well to the photos in Fig. 6. It can be seen that the final position matches well. Additionally, the simulation results show that, in the beginning, the left end of the tube moves slightly upwards. This behavior was also observed during the experiment. Further comparisons between experiments and simulations are given in Fig. 8. The movements of experiments #3 and #4 are captured from above and overlaid with respective simulation results. For the left image, the same velocity profile as for experiment #4 has been used for the rotation. Again, a good match is found. The comparisons between experiments and simulations show that the model can accurately compute real behavior of deformable flexible objects, such as cables and flexible tubes. The choice of using three elements in the discretization means that the simulated object is comprised of three circular arcs. The visual comparison with the experiment justifies this low number for the presented scenarios.

Fig. 8 Overlay of simulation results (blue dotted lines) over pictures of experiment #3 with accelerated start and end of the movement (left) and experiment #4 (right)

230

B. Maier et al.

Table 1 Runtimes of the simulation Experiment

Duration in reality (s)

Runtime 4 threads

2 threads

1 thread

Parallel efficiency (%)

Real-time factor

#1

2

661 ms

785 ms

1.26 s

48

3.0

#2

1

356 ms

394 ms

610 ms

43

2.8

#3

5

4.10 s

5.36 s

8.61 s

53

1.2

#4

4

1.44 s

1.66 s

2.56 s

44

2.8

4.3 Evaluation of Runtime Table 1 compares the durations D of the presented experiments and the runtimes of the simulations. We executed each simulation on a computer with Intel® Xeon® E3-1585 v5 quad-core processor with base frequency of 3.5 GHz and 8 MB cache. We measured runtimes T1 , T2 and T4 using 1, 2 and 4 threads. With runtime we mean the total duration of the simulation program, including input and output and without any visualization step. The parallel efficiency for the run with 4 threads, E p = T1 /(4 · T4 ), is listed, as well as the real-time factor D/T4 . A value higher than 1 means that the simulation took less computational time than the process in the real experiment. It can be seen that for each experiment the runtime decreases, if more threads are used, as the computational work is distributed to more cores. Because only the computation of the matrix and vector entries in Eq. (14) is parallelized, the parallel efficiency for 4 threads is at a low value of around 50%. The absolute value of the runtime for a simulation depends mainly on the used time step width, which, in turn, has to be small enough for the simulation to remain stable. This depends on the scenario, number of elements and material parameters. In the simulations carried out in the previous sections, the time step widths could be chosen such that a computation time below real-time was possible. This is shown by the real-time factors in the last column of Table 1.

5 Conclusion This paper developed a dynamic model of an inextensible, one-dimensional object in a 2D geometric setting, based on differential geometry and using the Lagrangian formulation of the equations of motion. The model accounts for a prescribed position and rotation of one end point and models linear friction between parts of the object and the underlying table. The model was implemented as open source code and shown to run in real-time. Algorithmic, numerical and technical considerations of an efficient implementation were presented. Simulations were carried out to show the behavior of the model. Experiments with a robot arm were conducted and reproduced

Real-Time, Dynamic Simulation …

231

by the simulation. Finally, the simulation runtime was evaluated and shown to be up to a factor of 3 below the duration of the process in the experiment. Future work can use this model for control or combine it with a simulation in a 3D space over the 2D workspace, where also gravity plays a role. Acknowledgements The authors acknowledge the International Research Training Group on Soft Tissue Robotics “Simulation-Driven Concepts and Design for Control and Automation for Robotic Devices Interacting with Soft Tissues” funded by Deutsche Forschungsgemeinschaft as GRK 2198/1, as well as the support from the Institute for Control Engineering of Machine Tools and Manufacturing Units at the University of Stuttgart.

References 1. Javdani, S., Tandon, S., Tang, J., O’Brien, J. F., & Abbeel, P. (2011). Modeling and perception of deformable one-dimensional objects. In 2011 IEEE International Conference on Robotics and Automation (pp. 1607–1614). Shanghai. 2. Wakamatsu, H., & Hirai, S. (2004). Static Modeling of linear object deformation based on differential geometry. The International Journal of Robotics Research, 23(3), 293–311. 3. Wakamatsu, H., Takahashi, K., & Hirai, S. (2005). Dynamic modeling of linear object deformation based on differential geometry coordinates. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation (pp. 1028–1033). Barcelona, Spain. 4. Wakamatsu, H., Yamasaki, T., Tsumaya, A., Arai, E., & Hirai, S. (2006). Dynamic modeling of linear object deformation considering contact with obstacles. In 2006 9th International Conference on Control, Automation, Robotics and Vision (pp. 1–6). Singapore. 5. Huang, J., Di, P., Fukuda, T., & Matsuno, T. (2008). Dynamic modeling and simulation of manipulating deformable linear objects. In 2008 IEEE International Conference on Mechatronics and Automation (pp. 858–863) Takamatsu. 6. Luo, C., Mo, Z., Zhou, Y., & Kuang, M. (2017) Static modeling and simulation of linear object based on differential geometry and discrete cosine transform. In 2017 IEEE International Conference on Mechatronics and Automation (ICMA) (pp. 1342–1347). Takamatsu. 7. Simo, J. C., & Vu-Quoc, L. (1988). On the dynamics in space of rods undergoing large motion: a geometrically exact approach. Computer Methods in Applied Mechanics and Engineering, 66(2), 125–161. 8. Boyer, F., & Primault, D. (2005). Finite Element of nonlinear cable: applications to robotics. Far East Journal of Applied Mathematics, 19(1), 1–34.

A System for Capturing of Electro-Muscular Signals to Control a Prosthesis Zeming Zhao, Bo Lv, Xinjun Sheng, and Xiangyang Zhu

1 Introduction In recent years, more and more attention has been paid to the amputated patients. However, it is not good enough to focus on treating their illness only. It is also crucial to improve their quality of life in the future and help them return to their normal lives [1, 2]. Installing a prosthesis is an ideal solution to the problems mentioned above. The prosthesis can be traced to Ancient Rome, which was used as decorations merely for many years in that era. In recent years, with the development of driver and electronic technology, controllable prosthetic hands have emerged. With the increasing functions and degree of freedoms (DOFs) of the prosthetic hands, it is getting harder and harder to control it properly. Electromyography (EMG) signal whose essence is the superposition of the action potential of the muscle tissue cells occurring during voluntary contraction is considered to be an excellent carrier for controlling the multiDOF prosthetic hands. It is intuitive for people to use it to control the prosthesis to a desired goal. Because the amputees need little onerous and suffering anti-intuitive training. EMG signal can be divided into two parts, surface EMG (sEMG) signal and invasive EMG (iEMG) signal. Compared with iEMG, sEMG is more popular because of its non-invasive acquisition, safety, and convenience [3]. In this paper, all the work focuses on sEMG only. In 1967, During [4] first used sEMG collected from a pair of antagonistic muscles to control a prosthesis successfully. Since then, a lot of work has been done on the inverse decomposition and application of the sEMG signal. Before 2000, much Z. Zhao · B. Lv · X. Sheng (B) · X. Zhu State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, 800 Dongchuan Road, Minhang, Shanghai, China e-mail: [email protected] URL: http://bbl.sjtu.edu.cn Z. Zhao e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_19

233

234

Z. Zhao et al.

work has focused on exacting relevant features from the sEMG signal to reduce the data dimension and then classify them by using pattern recognition algorithms such as LDA or SVM [5, 6]. In the past 20 years, people began to use sEMG signal to complete much more complicate tasks such as synchronous proportional control [6] for some specific joint angles and decomposition of MUAPt [7]. Previous work like During did, though brilliant, requires only a few channels of sEMG data because it can be done with relatively little information of muscles. However, as much information as possible has to be collected from the muscles we interested to complete the complicate tasks mentioned above. Under this circumstance, the high-density sEMG (HD-sEMG) acquisition equipment which can easily meet the extremely high requirement is a substitute for the traditional equipment. It supports deploying tens or even hundreds of electrodes on the upper arm at the same time, which means we can collect even hundreds of channels of sEMG data simultaneously from the target area. Because it has a promising prospect, many companies have introduced their HDsEMG acquisition system. DELSYS INC. launched NeuroMap System, Bagnoli System and Tiber [8]. TMSi launched Refa and SAGA [9]. OT Bioelectronica developed a series of products such as Quattrocento, EMG-USB2+, Sessantaquattro, Quattro, and Due [10]. Among all these systems, Tiber, Quattro, Due, and SAGA are portable devices. Referring to the technical indicators like number of acquisition channels, sampling frequency, A/D Resolution, these portable devices are not comparable with the desktop devices. Most of the commercial portable HD-sEMG acquisition devices have only a sampling frequency of 1 kHz at most. Since the energy of sEMG signal is mainly distributed in the frequency range of 20–450 Hz, the device only with a sampling frequency of 1000 Hz can only meet the minimum requirements of the sEMG acquisition equipment [11, 12], which is not suitable for high-quality sEMG acquisition. But on the other hand, desktop devices are not portable. This limits the location and the condition of experiment that makes harder to recording sEMG signal. Herein, we develop a new portable sEMG recording system with high performance. In this paper, the acquisition system will be introduced first. After that, the results of the test and related experiment will be presented briefly.

2 System Description As we mentioned above, it is essential to develop a portable HD-sEMG acquisition system of high performance. The system we developed will be introduced briefly in this chapter. And the block diagram of the system is shown in Fig. 1. The specific parameters of the system are listed in Table 1. The whole system consists of two parts: the acquisition device and the adaptable upper computer software. The acquisition device is expandable so that the number of acquisition channels can be adjusted from 64 to 256. It can satisfy the demand

A System for Capturing of Electro-Muscular …

235

Fig. 1 Block diagram of system

Table 1 Comparsions of specification parameters Tiber

Quattrocento

Proposed System 64, 128, 192, 256

Channels

64, 128

96, 192, 288, 284

A/D resolution

16 bits

16 bits

12, 16, 20 bits

Dynamic range

11 mVpp

33 mVpp

21 mVpp

Noise baseline

f , v > f , the real image is upside-down

Modeling of Lens Based on Dielectric Elastomers Coupling …

265

Table 2 Different thicknesses d corresponding to focal lengths f Varies

Value (cm)

R1

2

2

2

2

2

2

R2

2

2

2

2

2

2

d

0.5

1

2

3

4

5

f

2.2

2.3

2.5

2.8

3.1

3.5

u > f , v > f , the real image is upside-down

The lens system Smile logo The Camera 60 cm

5 cm Focus distance Fig. 2 Demonstration of optical tuning using the DEA

Table 2, R1 and R2 equal 2 cm. When d increases, and the focal length has a little change. High transparency allows DEAs to transmit electrical signals without impeding optical signals. Figure 2 shows the setup and tunable focus on the objects. The focus distance is from 5 to 60 cm by theoretically. A greater lens deformation could reach to larger scale focus. We adopt the model of ideal dielectric elastomers, and the stretchable materials are taken as incompressible. The energy of the active region is attributed to the stretching of the dielectrics [6]. The areal strain is defined as σ areal = (A − Ap )/Ap × 100%, where A is the area of the active region covered by the hydrogel for various voltages, and Ap is the area of this region after the pre-stretch. In Fig. 3, the equations exist: μJm (λ2 −λ−4 ) Jm −2λ2 −λ−4 +3

λ2 −λ2

− σp = ελHΦ2 , and σ = λ2 p × 100%. p Figure 3 represents the predicted voltage-strain curve with the hydrogel of thickness t = 0.5 mm. The lens will be stretched radially to two times its original radius, λp = 2. The hydrogel also contributes to the total volume of the active region. The parameters used in our simulation are μ = 10 kPa and J lim = 125. The relative permittivity of DE is set to be E = 4.159 × 10−11 F/m. The model accounts for homogeneous deformation of the active region. The strain-voltage curve of the actuator using the hydrogel electrodes is similar to the one with carbon grease [7]. 4

2

Fig. 3 Areal strain as a function of voltage for soft-actuated lens using hydrogel electrodes

H. Zhang and Z. Zhang

Areal Strain, %

266

λp=2

Voltage (kV)

3 Discussion Large actuated strain is attainable under the condition of large prestretched dielectrics and thin hydrogels. The rate of the hydrogel evaporation can be reduced by encapsulation in the condition of cyclic loading. In Fig. 4, the hydrogel and the VHB are subject to tensile load and unload. The displacement is described as d = (λ − λp ) × R. The parameters are set as Ref. [8]. The shear modulus μα = 16 kPa and μβ = 60 kPa, J lim = 150 and η = 0.2. The curvature of the lens system is R = 2 cm. The thickness of the lens membrane is 1 mm. The pre-stretch λp of the dielectrics is imposed to be 1.5. According to the Gent model, the Helmholtz free energy density of the lens is given by the sum of stretching energy density with viscoelasticity as [9] (a)

(b)

Fig. 4 a The simulation results of the soft lens subjected to three cycles of voltage pattern. b Displacement as a function of voltage for the three cycles

Modeling of Lens Based on Dielectric Elastomers Coupling …

267

    μβ Jβ 2λ2 + λ−4 − 3 2λ2 ξ −2 + λ−4 ξ 4 − 3 μα Jα log 1 − log 1 − W (λ, ξ ) = − − , 2 Jα 2 Jβ

(1) W , then, There is the relation given by λS + εE 2 = λ ∂∂λ

 2   2μα 4λ − 4λ−5 μα × 20λ−6 + 4  2 −4 2 −  2 −4  −6 2 2λ +λJα −3 − 2 Jα × 4λ +2λ − 2 Jα   2  2μβ 4λξ −2 − 4ξ 4 λ−5 μβ × 4ξ −2 + 20ξ 4 λ−6 + −  2 −2 4 −4   2ξ −2 4 −4 2 λ −3 4λ +2ξ λ −6 − 2 2 2λ ξ +ξ J × − 2 β Jβ

∂S = ∂λ

Jβ

−

3εΦ λ , H2 2 2







β



(2)

 μβ × 8λξ −3 + 16ξ 3 λ−5 2μβ × 4λ2 ξ −3 − 4ξ 3 λ−4 × 4λξ −2 − 4ξ 4 λ−5 ∂S = , −  2 −2 2 4 λ−4 −6 2(2λ2 ξ −2 +ξ 4 λ−4 −3) ∂ξ −2 Jβ 4λ ξ +2ξ − 2 Jβ J 

(3)

A significant hysteresis is observed during cyclic loading and it becomes repeatable in Fig. 4. The peak of the displacement is shifted from the peak of the voltage signal and occurs after it. To account for these viscoelastic phenomena, a constitutive model is developed by employing several dissipative non-equilibrium mechanisms as Eq. (4).  dξ dt dλ dt

⎡ ⎢ =⎣

− η2 ×

λ2 ξ −3 −ξ 3 λ−4

2ξ −2 λ2 +ξ 4 λ−4 −3 −1 Jβ ∂ S dΦ ∂ S dξ − − ∂Φ dt ∂ξ dt ∂S ∂λ

⎤ ⎥ ⎦

(4)

4 Conclusion In conclusion, we demonstrated the use of DEA with hydrogel electrodes to make optical lens system. The transparent electrode features are for both considerable stretch and high conductivity. The internal pressure inside the lens enclosure is changed to cause the shape change. Small deformation of the electro-active membrane will cause large deformation in the liquid lens enclosure. When the thickness d of a lens keeps a constant, the focal length increases as the curvature of the soft–actuated lens becomes larger. When d increases, the focal length has a little change.

268

H. Zhang and Z. Zhang

The strain-voltage curve of the actuator using the hydrogel electrodes is similar to the one with carbon grease. Large actuating strain is attainable under the condition of large prestretched dielectric and thin hydrogel. The rate of evaporation of the hydrogel can be reduced by encapsulation in the condition of cyclic loading. Future work will do more experiments to verify the theoretical analysis and further reduce the actuating voltage for the lens system.

References 1. Keong, G. K., La, T. G., Shiau, L. L., et al. (2014). Challenges of using dielectric elastomer actuators to tune liquid lens. In: Electroactive Polymer Actuators and Devices (EAPAD) 2014 International Society for Optics and Photonics. 2. Shian, S., Diebold, R. M., & Clarke, D. R. (2013). Tunable lenses using transparent dielectric elastomer actuators. Optics Express, 21, 8669–8676. 3. Choi, D. S., Jeong, J., Shin, E. J., et al. (2017). Focus-tunable double convex lens based on non-ionic electroactive gel. Optics Express, 25, 20133. 4. Chen, B., Bai, Y., Xiang, F., et al. (2014). Stretchable and transparent hydrogels as soft conductors for dielectric elastomer actuators. Journal of Polymer Science Part B: Polymer Physics, 52, 1055–1060. 5. Zhang, H., Dai, M., & Zhang, Z. S. (2019). The analysis of transparent dielectric elastomer actuators for lens. Optik, 178, 841–845. 6. Sun, J. Y., Zhao, X., Illeperuma, W. R., et al. (2012). Highly stretchable and tough hydrogels. Nature, 489, 133–136. 7. Brochu, P., & Qibing, P. (2010). Advances in dielectric elastomers for actuators and artificial muscles. Macromolecular Rapid Communications, 31, 10–36. 8. Zhang, H., Dai, M., & Zhang, Z. S. (2019). Application of viscoelasticity to nonlinear analyses of circular and spherical dielectric elastomers. AIP Advances, 9, 045010-1—045010-5. 9. Gu, G. Y., Gupta, U., Zhu, J., et al. (2017). Modeling of viscoelastic electromechanical behavior in a soft dielectric elastomer actuator. IEEE Transactions on Robotics, 1–8.

Industrial Processes and Products

The part starts with a practical process for detecting breakage of the yarn in the spinning of Spandex. The next chapter has nothing to do with the dairy product, but concerns identification by machine vision of the spool on which the yarn is wound. The end of this cylinder is coded with the information that describes the colour of yarn. The final chapter relates to the assembly-line feeding of parts for the construction of spray equipment. The components in question are of an awkward shape and call for an ingenious design to attain the required speed.

A General Monitoring Method for the State of Spandex Based on Fuzzy Evaluation and Its Application Limiao Gu, Yan Wen, Yu Zhang, Weijie Chu, Yunde Shi, and Fang Jia

1 Introduction The demand of core spun yarn is rising in the modern textile industry. As the important raw material of core spun yarn, the spandex is prone to break due to the improper pre-draft times and the wear of the godet rolls resulting that the defects of core spun yarn increase. Timely detection and treatment of broken spandex are the important procedures to ensure the quality of core spun yarn. At present, the manual inspection adopted by most spinning enterprises not only fails to find the problem in time, but also increases the labor intensity. A myriad of scholars and enterprises have studied different monitoring ways of spandex and core spun yarn in the spinning process. Gorbunov et al. [1] proposed an automated control system based on machine vision to identify fabrics parameters to find defects. Kim et al. [2] proposed a method to monitor yarn wrapping quality to identify hollow yarns. The QUANTUM3 system of Uster company was able to monitor the defects in the production of core spun yarn, and the defective segments are separated manually. Ling-yun et al. [3] developed a sensor to identify the core spun yarn breakage based on the tunnel magnetoresistance effect. Wang Senxiao et al. [4] proposed a dual-coil differential sensor and detection device based on the principle of electromagnetic induction to judge the state of yarn. Catarino et al. [5] developed a sensor to measure the input tension of yarn, and the state of the core yarn was judged by analyzing the tension. Most researches pay attention to the direct detection of the state of core spun yarn at present, but the research schemes on the monitoring of spandex have been rarely reported. Meanwhile, the existing detection algorithms are complex and require high L. Gu · Y. Wen · Y. Zhang · W. Chu · Y. Shi · F. Jia (B) Department of Mechanical Engineering, Southeast University, 211189 Nanjing, China e-mail: [email protected] L. Gu e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_22

271

272

L. Gu et al.

hardware cost [6, 7]. Aiming at the above problems, this paper proposes a general monitoring method for the state of spandex based on fuzzy evaluation to real-time monitor and timely solve the problems of spandex.

2 A Low-Cost Monitoring System for the State of Spandex The core spun yarn is produced by twisting the roving in a certain direction with spandex and wrapping on the textile bobbin. Figure 1 shows the spinning process of core spun yarn, including roving 1, spandex 2, the guide hook 3, the godet rolls 4, twisting end of spinning frame 5, core spun yarn 6 and the textile bobbin 7. As shown in Fig. 2, the godet roll is composed of N40 NdFeB magnet 1, white painted wheel 2 and threaded shaft 3. The spandex is located in the upper part of the spinning frame about 50 cm away from the godet roll [8, 9]. Nevertheless, the spandex is liable to shake in a certain range because of its small diameter and long drawing distance. Therefore, to directly monitor the state of the spandex is very challenging for hardware [10]. The godet roll is driven to rotate because of the spandex under normal conditions. If the spandex breaks, the godet roll will stop running due to friction. Hence, this system judges whether the spandex breaks by detecting the rotation of the godet roll, so as to lessen the hardware requirement and complexity of the algorithms. Fig. 1 The spinning process of core spun yarn

Fig. 2 The composition of the godet roll

A General Monitoring Method for the State of Spandex …

273

Fig. 3 The system composition and behavior

The monitoring system consists of sensors, the signal preprocessing module, the threshold comparison module, the pulse detection module, the MCU and actuators as shown in Fig. 3. The upper waveforms represent the output signal of modules and the action of the system under normal rotation of the godet rolls. When the godet rolls are abnormal, the output signal and the behavior of the system are shown below. The communication and power supply of the system are realized with chain structure which the circuit boards are connected one by one to reduce the length of the wires and cost.

3 The Design of Monitoring and Communication Algorithms 3.1 An Intermittent Monitoring Algorithm Based on Fuzzy Evaluation The monitoring algorithm for spandex needs to meet the high real-time requirements and the computation efficiency. This algorithm is based on fuzzy evaluation and intermittent inquiry to balance efficiency and time with strong robustness. As shown in Fig. 4, the hierarchical structure model in view of the spinning technology is established, which includes rational speed, historical detection data and other positions’ status. Since the output signal generated by the pulse detection module lasts for 0.4 s after receiving the single pulse, the low rotational speed caused by the wear of the godet roll will generate pulse signals at the pulse detection module. Therefore, the rotational speed layer includes the previous speed and the current speed. The historical monitoring data layer includes spandex breakage, the wear of the godet roll and the damage of the godet roll. The evaluation set of the state of the godet roll is V, which includes spandex breakage V 1, the wear of the godet rolls V 2 and the damage of the godet rolls V 3. According to the hardware design of the monitoring system and the opinions of

274

L. Gu et al.

Fig. 4 The hierarchical structure model for the state evaluation of the godet roll

the technological engineers, the membership degree of the evaluation object to the evaluation set is determined from every single factor. The evaluation matrix for the ration speed is S = [S1 S2]T , and the evaluation matrix for historical detection data is H = [H1 H2 H3]T . The elements of each matrix are the membership degree of a certain evaluation object in a certain evaluation factor. Finally, the initial fuzzy relation matrix B = [S H P]T is introduced. The monitoring algorithm will query 8 godet rolls at an interval of 1 s and revise the fuzzy relation matrix. The weight matrix A is constructed by the analytic hierarchy process based on the scale theory and processed according to the formula 1. The factor weight of the solution layer is determined as W. Similarly, the factor weights of the criteria layers can be obtained as W 1, W 2 and W 3  n ⎧  ⎪ ⎪ ai j = ai j ak j (i, j = 1, 2, 3) ⎪ ⎪ ⎪ k=1 ⎪ ⎨ n  Wi = ai j (i = 1, 2) . (1) ⎪ k=1 ⎪ ⎪ n ⎪  ⎪ ⎪ W i (i = 1, 2) ⎩ Wi = W i i=1

The ak j means the elements of matrix A. On this basis, the criterion layer is judged:

A General Monitoring Method for the State of Spandex …

⎧  ⎪ ⎪ S = W1 ◦ S ⎪ ⎨ H = W2 ◦ H . ⎪ P = W 3 ◦ P ⎪  ⎪ ⎩ R = S H  P  T

275

(2)

The final comprehensive evaluation results are obtained, and the membership degree Re of matrix E is calculated according to the principle of maximum membership degree:

E=W◦R Re = max {ei }(ei ∈ E)

(3)

1≤i≤n

The actual state of the godet roll is distinguished to judge the state of spandex. In addition, the permissible detection marks are introduced into the algorithm and the marks of unproductive positions are closed. If it is judged as spandex breakage, the actuator will be triggered and the historical monitoring data will be updated. Before an actuator executes, the detection marks of other circuit boards are temporarily turned off by the close command to avoid multiple actuators working at the same time. Another command will be sent to enable other boards to continue detecting after the actuator has done. If it is judged as the wear of the godet roll, the system will provide early warning but not cut roving. If the godet roll is identified as damaged, the actuator will run and give an alarm. The flow of the intermittent monitoring algorithm is shown in Fig. 5. The godet rolls may be accidentally touched by the operators during the process of reconnecting the spandex, which leads to the misjudgment that the spandex breaks again. By analyzing the joint process, it is known that the rotation duration due to the unintentional touching does not exceed 2 s. Hence, the electromagnet working mark is added to the monitoring algorithm. As shown in Fig. 6, after the electromagnet executed, the permissible working mark is temporarily closed. In this situation, the actuator will not be triggered even if the operators mistakenly touch the godet roll. After the joint is completed, the godet roll begins to rotate again. If the rotation lasts longer than 6 s, the working mark will be turned on. At this moment, the actuator will be triggered if the spandex breaks.

3.2 A Process Identification Algorithm The speed of the godet rolls will have an obvious change with the process alternation. The process identification algorithm is based on the change rate of speed to identify the current process by variance evaluation. According to the technology and monitoring system, the following conclusions can be drawn:

276

L. Gu et al.

Fig. 5 Flow chart of intermittent general monitoring algorithm

• The time of all godet rolls will not exceed 3 s from rotating to stopping in the doffing process. • The speed of the godet rolls is low in the initial spinning stage and the pulse detection module will generate low-frequency pulses. • The speed of the godet rolls begins to rise such that the pulse frequency of the pulse detection module will increase in middle spinning stage. • During the high-speed spinning stage, the pulse detection module keeps high level. Based on these points, if all 8 godet rolls are in the stall identified by the monitoring algorithm, the current process is judged as doffing and the actuator will not work. If it does not belong to the doffing, the average speed of all position will be calculated. The change rate of speed is obtained with the average speed of the previous moment. The process array A is constituted of the rational speed change rate in a few seconds. It can be concluded that the current process has changed when the array data change dramatically. The reference arrays of each process are generated according to the slope of pre-measured speed of the initial spinning, middle spinning, high-speed spinning and doffing processes. For example, the rational speed of doffing shows a downward trend and the absolute value of slope will increase gradually. Formula 4 is used to calculate the square error between the array A and reference arrays.

A General Monitoring Method for the State of Spandex …

277

Fig. 6 Flow chart of mistake recognition algorithm

 n 1 2 s = ∗ (ai − xi ) n 1 2

(4)

The ai is the element of array A and the xi is the element of reference arrays of each process. The square error is the similarity between arrays so that the current process is corresponding to the process reference array with the minimum square error.

3.3 An Adaptive Communication Algorithm The monitoring system installed on both sides of the spinning frame have opposite communication directions. This communication algorithm not only suits the direction of data transmission, but also improves the efficiency and reliability through appropriate communication protocol and FSM (Field Signature Method). The data will be stored in certain array corresponding to the communication direction, and then are processed based on FSM function and communication protocol. The improved FSM function includes sending direction, array name and length. The sending directions are left and right instead of usart1 and usart2. According to the direction information in the type frame, the meaning of left and right corresponding to usart1 and usart2 can be changed in real time, and the parameter does not need

278

L. Gu et al.

Table 1 Communication protocol framework No.

Type

Number of bytes

Instructions

1

Synchronization header

1

0×7E

2

Type frame

1

Upper 5 bits represent the command type Third bit is the direction of communication Lower 2 bits mean the number of length type

3

Length frame

0, 1, 2

Setting delay time and electromagnet command have no length frame Setting working state and cutting command have 1 byte IAP command is 2 bytes

4

Content frame

N

N is the number of content frame

5

BCC check frame

1

The XOR value is calculated by first four content

to be modified. Hence, the direction requirement can be easily satisfied. The framework of the communication protocol is shown as Table 1, including synchronization header, type frame, length frame, content frame and BCC check frame.

4 Experimental Verification 4.1 Experimental Platform and Conditions The experiment of given monitoring method for spandex includes 264 monitoring circuit boards based on the chain structure and four spinning frames. The experimental environment is shown in Fig. 7 including roving 1 and sensors 3. The spinning frame 2 and the right frame are equipped with monitoring system and monitored by them. The spinning frame 4 and the left frame are monitored with manual inspection by 4 operators. When the number of work positions is large enough, the probability of the spandex breakage in this experiment tends to be about 5–7%. Hence, it can be assumed that the number of the spandex breakage of four spinning frames is the same [11, 12]. The experimental conditions are as follows: • Spandex specifications include 15D, 20D, 30D, 40D and 70D, and the pre-draw time is set as 4. • The count of core spun yarn is set as 60, the front roller speed is set as 120 r/min and the diameter of the front roller is 25 mm. • Every spinning frame spins at 512 positions and each specification spandex is spun 2 round.

A General Monitoring Method for the State of Spandex …

279

Fig. 7 Experimental environment

4.2 System Running Test The output signal can be obtained from the threshold comparison module of the monitoring system as shown in Fig. 8. The output signals were smooth and stable and conformed to the actual speed. It can be seen that the strong interference of the spinning workshop can be resisted by system, and a reliable basis is provided for the monitoring algorithm. The comparison of the spandex breakage and normal spinning is shown in Fig. 9. The spandex of middle positions was broken. The roving 1 is stopped feeding by the white actuators 3 and the LED 2 in the senor end gave an alarm. Meanwhile, the monitoring data were transmitted to the PC according to the communication algorithm and the abnormalities were displayed as shown in Fig. 10. The actuators 3 were not triggered when the godet rolls were touched in reconnecting process. The spandex of right positions was in the normal state so that the actuators did not work Fig. 8 The output waveform of comparison module

280

L. Gu et al.

Fig. 9 Comparison of the spandex breakage and normal positions

Fig. 10 The experiment of PC communication on field

and LED 4 was in the extinguished state. When the doffing process was carried, all the godet rolls stopped running within 2 s. The doffing process was distinguished by the process identification algorithm and the actuators were not triggered.

4.3 Analysis of Experimental Results The experiment shows that the proposed monitoring method has higher accuracy and is suitable for various specifications of spandex. As shown in Table 2, the monitoring accuracy of most of the larger diameter spandex is over 98%. The monitoring accuracy is equal to the correct number divided by the positions’ number of spandex breakage and misjudgments. In the case of low speed of the front roller, the godet rolls are not driven by the smaller spandex, which is prone to have misjudgments in initial stage. With the accumulation of monitoring data, the misjudgments gradually decrease and the accuracy will be over 98%. However, the accuracy of the manual inspection in 1 round is far less than 80%.

A General Monitoring Method for the State of Spandex …

281

Table 2 Experimental results No.

The specifications of spandex

Monitoring accuracy (%)

Average discovery time (min)

Average roving length saved (m)

1

15D

97.7

1.8

16.65

2

20D

98.2

1.5

13.82

2

30D

99.2

1.2

10.99

4

40D

99.7

1

9.11

5

70D

99.7

1

9.11

Compared with manual inspection and processing, the monitoring method stops the roving feeding in time to reduce the waste of raw materials, and the roving length saved is shown in Table 2. The roving length used is determined by formula 5: L = s ∗ (π ∗ d) ∗ t

(5)

The L is the used roving length, and the s is the speed of roller. The d represents the diameter of roller and t means the discovery time. If the spandex breaks, it can be recognized by the monitoring system in 2 s so that the roving length used is about 0.314 m. However, the average discovery time of operators is shown in Table 2. The spandex of smaller diameter is prone to break, and the treatment of spandex will affect the inspection of other positions. Based on this, the average discovery time will increase gradually. The operators are responsible for a large number of spinning frames instead of two in reality. Therefore, the average discovery time and the number of roving saved will be more.

5 Conclusion In view of the shortcomings of existing detection methods for spandex, this paper proposes a low-cost monitoring system, which makes the design of hardware and algorithms more easily. To realize real-time monitoring of spandex and process identification, an intermittent monitoring algorithm based on fuzzy evaluation and a process identification algorithm are proposed with high robustness. Furthermore, the adaptive communication algorithm suits the change of direction of data transmission and realizes real-time display of the situation on host computer. Finally, the experiments show that the accuracy of the monitoring method for spandex is more than 98% and the raw materials are saved more than 9.11 m per station, which is better than existing methods and applicable to various spandex. Acknowledgements The authors greatly acknowledge the grant of Daiyin Group (www.daiyin. com) which supported this research and professors who provided suggestions.

282

L. Gu et al.

References 1. Gorbunov, V., Bobkov, V., Htet, N. W., & Ionov, E. (2018). Automated control system of fabrics parameters that uses computer vision. In 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), Moscow (pp. 1728–1730). 2. Kim, H. J., Kim, J. S., Lim, J. H., & Huh, Y. (2009, November). Detection of wrapping defects by a machine vision and its application to evaluate the wrapping quality of the ring core spun yarn. Textile Research Journal, 79(17), 1616–1624. 3. Ling-yun, X., Dong-fang, Z., & Qing-guang, C. (2017). Design of yarn break detection based on tunnel magnetoresistance effect. In 2017 16th International Conference on Optical Communications and Networks (ICOCN), Wuzhen (pp. 1–3). 4. Wang, S., Wang, J., Shang, L., & Wang, T. (2016). Double coil electromagnetic induction differential ring ingot break detection system. In 2016 National Metallurgical Automation Information Network Conference (pp.180–184). 5. Catarino, A., Rocha, A., & Monteiro, J. (2003). Low cost sensor for the measurement of yarn input tension on knitting machines. In 2003 IEEE International Symposium on Industrial Electronics (Cat. No. 03TH8692), Rio de Janeiro, Brazil (Vol. 2, pp. 891–896). 6. Pinto, J. G., Monteiro, J., Vasconcelos, R., & Soares, F. O. (2002). A new system for direct measurement of yarn mass with 1 mm accuracy. In 2002 IEEE International Conference on Industrial Technology, 2002. IEEE ICIT ‘02, Bankok, Thailand (Vol. 2, pp. 1158–1163). 7. Shuai, W., Chongqi, M., & Hanming, L. (2010). Yarn quality tracking system based-on RFID. In 2010 International Conference on Computer and Information Application, Tianjin (pp. 103– 105). 8. Wang, W., & Liu, J. (2018). Spinning breakage detection based on optimized hough transform. Journal of Textile, 39(04), 36–41. 9. Carvalho, V., Monteiro, J., Vasconcelos, R. M., & Soares, F. O. (2004). Yarn mass analysis with 1 mm capacitive sensors. In 2004 IEEE International Symposium on Industrial Electronics, Ajaccio, France (pp. 633–638). 10. Jun, L. D. (2007). The research of broken filaments detection device on viscose filament yarn. In 2007 International Conference on Computational Intelligence and Security Workshops (CISW 2007), Heilongjiang (pp. 910–913). 11. Roy, S., Sengupta, A., Maity, R., Sengupta, S. (2013). Yarn parameterization based on image processing. In 2013 IEEE International Conference on Signal Processing, Computing and Control (ISPCC), Solan (pp. 1–6). 12. Shanghai textile holding company. (2006). Cotton textile manual (3rd Edn). China Textile Press.

Study on the Type Identification of Cheese Yarn Based on Low-Resolution Pictures Xiaolong Liu, Ran Hu, Yan Wen, Yu Zhang, Weijie Chu, Zhisheng Zhang, and Fang Jia

1 Introduction With the development of Industry 4.0, the traditional manufacturing factory’s direction has gradually transformed into intelligent factory. As an important pillar industry, the spinning industry urgently needs to upgrade production methods. With the support of the artificial intelligence technology, the use of machine vision and neural networks for automated operation is the future direction of the spinning industry. In view of the machine vision-based sorting method, many scholars have proposed different algorithms for identification. Wang [1] proposed an improved self-organizing feature mapping network algorithm for image color feature extraction. P. Sundara Vadivel proposed an efficient image retrieval system based on color histograms [2], edges and texture features. After identifying images with color similarity, only texture and edge-based search of the identified images is required. This process greatly reduces the time required for the retrieval process by avoiding the fusion process. Surina Borjigin proposed a color image segmentation algorithm based on multi-level Tsallis-Havrda-Charvát entropy [3]. Chen Qian proposed a main color extraction algorithm for color feature extraction of clothing images to judge the main color of the clothing color according to the H, S channels of the color histogram [4]. This method can identify the main color of the clothes, but needs further optimization in the identification of flower colors. Chen Huiyuan and Liu Zeyu designed a cascaded convolutional neural network detection framework for the rapid detection of large-scale remote sensing image ship targets, which improved the speed of ship detection in remote sensing images, but the accuracy was low [5]. Li Wenyu proposed an intelligent detection algorithm for color fabric defects based on energy X. Liu · R. Hu · Y. Wen · Y. Zhang · W. Chu · Z. Zhang · F. Jia (B) Department of Mechanical Engineering, Southeast University, 211189 Nanjing, China e-mail: [email protected] X. Liu e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_23

283

284

X. Liu et al.

local binary mode. By analyzing the energy feature images of the color fabrics [6], they found that the defects are usually irregular, non-uniform local bright areas, while the background pattern of the color fabric presents a regular, uniform brighter area in the energy feature image, this method can detect color defects and tissue structure defects, and the detection accuracy is 94.9%, which is higher than the accuracy of 89.4% using BP neural network. Machine vision is an indispensable part of the process of intelligent and automated development of manufacturing. The application of machine vision is of great significance for the classification and identification of cheese yarn. It is of great significance to realize the classification and identification of the package yarn by machine vision. The traditional machine vision-based processing method has a large amount of computation, a slow processing speed, and high-performance requirements for hardware devices. This paper combines manual data processing and machine vision technology, locate and extract the digital information from the label, then Processing data by BP neural network. This paper proposes a new method for identifying the type of package yarn [7].

2 Location of Labels Based on Hough Algorithm 2.1 Imagine Preprocessing Analysis of the image in the gray space can avoid the interference of the color of the image on the shape feature. Therefore, after acquiring the image, the grayscale processing is first performed to convert it into a grayscale image. The formula for grayscale conversion is [8]: G(x, y) = 0.299 × r (x, y) + 0.587 × g(x, y) + 0.114 × b(x, y)

(1)

Among these: G(x, y) is the gray value obtained by transforming each feature point on the image. r(x, y), g(x, y), b(x, y) represent the color of the pixel with coordinate (x, y) in the original image. After grading the picture, use the mean filtering to remove the noise in the picture, then use the histogram to equalize the image. The gray information of the comparative set is stretched to the entire gray range. This method increases the contrast of the image and makes the image look sharper (Fig. 1). In this experiment, the gray value of the yarn and the label is very different to the gray value of the hole in the middle of the picture. By binarizing the image, the hole in the middle of the image can be converted to black and the other parts to white.

Study on the Type Identification of Cheese Yarn …

285

Fig. 1 a Original image; b grayscale processing; c mean filtering; d histogram equalization; e binarization

2.2 Hough Algorithm Based on Circular Symmetry The label to be searched for in this paper is circular. Hough transform is a commonly used circular search method, but it has a large amount of calculation and takes up a large storage space, which is not suitable for the real-time operation in this paper. Therefore, this paper proposes a random Hough transform based on far symmetry in combination with the actual situation [9–11]. The distance and angle of the bobbin relative to the camera vary over a small range, so the size of the label and the angle of view are almost the same in the captured image. On this basis, the position of the label can be determined on the contour image according to the symmetry of the circle, as follows: First, use a line L1 with a line width of two pixels to pass through the image. The line will intersect the circle at two points P1 (x1, y1) and P2 (x2, y2). The center of the circle must be at the vertical line “M” of the connection of these two points. Since the image itself may have interference information, this paper uses the line set L = {L1, L2, L3, …, L10} to intersect with the picture, and the distance between each line is the same, as shown in Fig. 2. In the process of performing the above steps, use a box containing 2 * 2 pixels to slide along the line Ln. When all the four pixels contained in the box are black, record the coordinates of the point Pn (x n , yn ) as shown in Fig. 3. Continue to slide the  box forward. When white appears in the square, record the coordinates Pn xn , yn of the point. By the coordinates of Pn and Pn , according to Formula 2, the set C x = {C 1 , C 2 , …, C 10 } of abscissas of the center of the circle can be obtained: C xn =

 Pxn + Pxn 2

(2)

286

X. Liu et al.

Fig. 2 Lines cut circular diagram

Fig. 3 Locating the coordinates of the intersection

Then get the average of the horizontal axis of the circle: 10 Cx =

k=1

10

C xk

(3)

If the coordinates of the center of the set C x = {C 1 , C 2 , …, C 10 } differ from the average by more than 10% of the radius of the label, then recalculate the average of the abscissa of the center without that point. Then, calculate the ordinate C y of the center of the circle in the same way and get the coordinates C(x, y) of the center of the circle. The method has small calculation amount and small storage space, and is suitable for applications with high real-time requirements.

Study on the Type Identification of Cheese Yarn …

287

2.3 Label Locating Algorithm Flow The height of the camera remains the same, so the size and shape of the label in the picture is fixed, so the radius r of the label can be obtained by statistics. After the label is located by the manner described above, the pixel points within the range are extracted in the original image and saved in a new image to obtain a label image of the bobbin. The overall algorithm flow is as follows: Step 1: Acquire image information and convert the image to gray space; Step 2: Use mean filtering to remove image noise and use histogram equalization to improve picture contrast; Step 3: Binarize the image; Step 4: Locating the center of the label using a Hough transform algorithm based on the symmetry of the circle; Step 5: Obtain an image within the range of the label in the original image.

3 Extraction of Neural Network Input Information In order to improve production efficiency and ensure the correct rate of industrial production, the color of the labels in the factory is printed according to the discrimination habits of the human eye. Currently, in all color spaces, the HSV color space is expressed in the closest way to the color-resolving habits of the human eye, and can well express the color information perceived by the human eye, and is less affected by the illumination. The H channel represents the hue and the S channel represents the saturation. Therefore, the color channel selected in the label feature extraction is the H channel and the S channel of the HSV color space [12]. (r, g, b) represents a pixel in the RGB color space, the value of “r g b” is a number between 0 and 1, and max is the maximum of the three [13], and min is the minimum of the three, then the conversion of RGB space to HSV space The formula is: ⎧ ⎪ 0◦ , if max = min ⎪ ⎪ g−b ⎪ ◦ ◦ ⎪ ⎪ ⎨ 60 × max−min + 0 , if max = r, g ≥ b g−b ◦ ◦ h = 60 × max−min + 360 , if max = r, g < b ⎪ g−b ⎪ ⎪ 60◦ × max−min + 120◦ , if max = g ⎪ ⎪ ⎪ g−b ⎩ 60◦ × ◦ max−min + 240 , if max = b 0, if max = 0 s = max−min min max = 1 − max , otherwise v = max

(3)

(4) (5)

288

X. Liu et al.

Fig. 4 Histogram information

Extract the color histogram of the H channel and the S channel, and quantize the value of the H channel from 0 to 360 to 0 to 10. The quantization interval used is 36, and the value of the S channel ranges from 0 to 50, and the quantization value is 0 to 5. Obtain the quantized color histogram, quantize and normalize it, and finally calculate the color ratio. The obtained label histogram information is shown in Fig. 4. Algorithm flow: Step 1: When the sensor is triggered, the camera takes a picture of the package yarn; Step 2: Use the method in Part II to get the image of the label; Step 3: Preprocessing the image by using medium filtering and bilateral filtering; Step 4: Convert the image from the RGB color space to the HSV color space; Step 5: Quantize the global color histogram of the H channel and the S channel to obtain a quantized color histogram. Step 6: Normalize the quantized color histogram, and the ordinate value in the figure can represent the proportion of each color.

4 Processing Method Based on Deep Neural Networks 4.1 Process of Algorithm Deep neural network (DNN) is a neural network with multiple hidden layers (two or more layers). The theory has proved that single-layer neural networks cannot solve the linear indivisible problems, considering that the data of this project is complicated, DNN was selected to demand the requirement. After the feature information of the picture is extracted by the color histogram, the feature information will processed by the neural networks. In this paper, the sample information, the detected cheese yarn information and the flag information are input into the input layer of the neural networks, training the neural networks to obtain the weights and then import the weights to the industrial equipment. Compared with the

Study on the Type Identification of Cheese Yarn …

289

convolutional neural networks, this method can reduce calculation and shorten the operation time [14–16]. The process of the cheese yarn classification neural networks mainly includes data preparation, network establishment, network training and classification effects evaluation. The training data was obtained by the simulation experiment platform, the neural network needs to determine the depth, the number of nodes in each layer and the activation function, to optimize network performance, the neural network adjusts the weights constantly during the training process. Finally, the optimal algorithm was obtained after the classification effect evaluation.

4.2 Preparation of Training Data The training samples are the data that includes the classification flag, and the weights of the neural network can be adjusted by training to improve the classification accuracy. In this experiment, samples are obtained under the simulation experiment platform. Each group of samples consists of three parts, which are the characteristic information of the to-be-detected cheese yarn, the characteristic information of the reference cheese yarn, and the classification flag. The characteristic information of the to-be-detected cheese yarn and the characteristic information of the reference cheese yarn are determined by the H and S values of the image obtained under the simulation experiment platform in the HSV color space. In the HSV color space, the values of H, S, V are 0–180, 0–255, 0–255 respectively. The range of H was divided into 10 intervals to obtain the number of pixels in each interval, the number of pixels constitutes a 1 × 10 matrix. Similarly, the value range of V is divided into five intervals, and a 1 × 5 matrix is obtained. According to the above, a 1 × 30 matrix composed of the characteristic information of the to-be-detected cheese yarn and the characteristic information of the reference cheese yarn is obtained. If the to-be-detected cheese yarn is the same type as the reference cheese yarn, the value of classification flag is 1, otherwise, the value of classification flag is 0. In conclusion, each training sample is a 1 × 31 matrix.

4.3 Establishment of Neural Networks Considering that the data of this project is relatively complicated and the calculation performance of the industrial field is limited, this experiment chooses the double hidden layer neural network, to reduce the amount of calculation, the input layer and the hidden layer 1 are not set to be fully connected. The structure of neural networks is shown in Fig. 5. H, S, H0 , S0 are training samples, hidden layer 1 was set 6 nodes, where a11 , a12 , a13 are obtained by H, S, a14 , a15 , a16 are obtained by H0 , S0 , weights ω(11) , ω(12) are 15 × 3 matrix:

290

X. Liu et al.

Fig. 5 The structure of the neural networks (11) (11) (11) ⎤ ω12 ω13 ω11 (11) (11) (11) ⎥ ω

⎢  ⎢ 21 ω22 ω23 ⎥

H1 · · · H10 S1 · · · S5 ⎢ . ⎥ = a11 a12 a13 . . .. .. ⎦ ⎣ .. (11) (11) (11) ω15,1 ω15,2 ω15,3 ⎡ (12) (12) (12) ⎤ ω11 ω12 ω13 ⎢ ω(12) ω(12) ω(12) ⎥

(0)   22 23 ⎥ ⎢ 21 (0) (0) ⎥ = a14 a15 a16 H1 · · · H10 S1 · · · S5(0) ⎢ . . . .. .. ⎦ ⎣ ..

⎡

(6)

(7)

(12) (12) (12) ω15,2 ω15,3 ω15,1

Hidden layer 2 was set 2 nodes, connected with hidden layer 1 fully, the weight ω(2) is a 6 × 2 matrix, ω(3) is a 2 × 1 matrix: (2) (2) ⎤ ω12 ω11 (2) (2) ⎥ (3)  ω ⎢ ⎢ 21 ω22 ⎥ ω1 · · · a16 ⎢ . .. ⎥ (3) = [y] ⎣ .. . ⎦ ω2 (2) (2) ω61 ω62

⎡



a11

(8)

According to the Y = ∅(y), y outputs Y between 0 and 1. Y represents the probability that the to-be-tested cheese yarn and the sample cheese yarn are the same type, 1 − Y represents the probability that cheese yarns are the different type. The combination of loss function and activation function has great influence on training success rate and classification accuracy of neural network. The activation function is the nonlinear distortion force in the neural network structure, the loss function is the comparison between the neural network output value and the real value,

Study on the Type Identification of Cheese Yarn …

291

reflect the degree of data fit. The 2-classification neural network model usually selects the Sigmoid function as activation function. To eliminate the gradient disappearance, the cross entropy was selected as loss function. The Sigmoid function, the Sigmoid derivative function, and the cross entropy function are as follows: ∅(x) =

1 1 + e−x

(9)

˙ ∅(x) = ∅(x)[1 − ∅(x)]

(10)

L(y, G(x)) = −[y ln G[x] + (1 − y) ln(1 − G(x))]

(11)

Finally, the neural network was built by the Tensorflow framework.

5 Algorithm Verification The data acquisition and extraction methods proposed in this paper were tested, photographing the label position of the cheese yarn in the laboratory environment, then the information of the pictures was extracted and training the neural network.

5.1 Label Location and Data Collection The shapes of label for the data collection include solid color, star shape, square shape, triangle shape, and diagonal shape, the colors include black, green, purple, red, and blue. The pictures of different labels are as shown in Fig. 6. A total of 1000 labeled images were processed in the experiment, the labels that center deviation distance less than 5% of the radius (rough measurement) was accurate. According to the results, the accuracy of label location was 95.4%. The results are shown in Fig. 7. After locating the coordinates of the label, removing the background of the images according to the coordinates of the label, the H and S channels of the remaining part of the image in the HSV color space are extracted to obtain the quantized color histogram.

5.2 Verification of Neural Network The experimental platform is a black box structure with OpenMV as the core, the camera is OV7725 and the processor is STM32H743VI, which is controlled by

292

X. Liu et al.

Fig. 6 The labels of cheese yarn

Python. The training data was collected on the experimental platform. A total of 36,524 samples of 30 types of cheese yarn were collected. After the post-processing and histograms, 26,350 available samples were obtained. The available samples were divided to 2 groups, with 20,000 training samples and 6350 test samples. The neural network was built based on Python 3.6 and Tensorflow 1.12, training on the PC. The correct rate was detection by the test samples during the training process. The training was completed while the correct rate above 99.8% stably. Obtain weight matrixes ω(11) , ω(12) , ω(2) , ω(3) after the training, import the weight matrixes into OpenMV, a large number of experiments were performed on the experimental platform to detect the correct rate. In summary, the overall correct rate is 99.76%, which beyond the requirements of the 99.5% in industrial field.

6 Conclusion In this papers, a method of label location based on symmetry of circle is proposed, which reduces the computational complexity and can locate the label in a shorter time. On the basis of locating labels, color information of labels is picked up, and color histograms of H and S channels are extracted for quantification and normalization.

Study on the Type Identification of Cheese Yarn …

293

Fig. 7 Accuracy of labels location

References 1. Wang, Y. (2016). An improved image compression algorithm based on self-organizing feature mapping. Radio Engineering, 36(12), 18–20. 2. Sundara Vadivel, P., Yuvaraj, D., & Navaneetha Krishnan, S. (2018). An efficient CBIR system based on color histogram, edge, and texture features. Concurrency and Computation Practice and Experience, 57(6), 738–748. 3. Borjigin, S., & Sahoo, P. K. (2019). Color image segmentation based on multi-level Tsallis– Havrda–Charvát entropy and 2D histogram using PSO algorithms. Pattern Recognition, 92, 107–118. 4. Lv, M., Gao, T., & Zhang, N. (2019). Main color extraction algorithm and its application to clothing image retrieval. Journal of South China Normal University, 51, 111–119. 5. Chen, H., Liu, Z., & Guo, W. (2019). Fast detection of ship targets for large scale remote sensing image based on a cascade convolutional neural network. Journal of Radars, 8(3), 413–424. 6. Li, W. (2014). Research on automatic detection for yarn-dyed fabric defect based on machine vision and image processing. Doctor, Donghua University. 7. Cheng, Y. Y., Li, H. Y., & Zhang, Y. F. (2011). A new method of denoising mixed noise using Limited Grayscale Pulsed Couple Neural Network. In Cross Strait Quad-Regional Radio Science and Wireless Technology Conference (pp. 1410–1413). 8. Boerner, H. (1989). Feature extraction by grayscale morphological operations—A comparison to DOG filters. In International Workshop on Industrial Applications of Machine Intelligence and Vision.

294

X. Liu et al.

9. Shi, D. C., Zhang, B., & Wang, N. (2014). Fast circle detection based on improved randomized Hough transform. In 7th International Symposium on Advanced Optical Manufacturing and Testing Technologies. 10. Djekoune, O. (2013). A new modified Hough transform method for circle detection. In 5th International Joint Conference on Computational Intelligence. 11. Jiang, L., Tang, P., Zhu, Y., Jiang, J., & Zhang, Y. (2013). Multi-circle detection algorithm based on symmetry property. International Journal of Applied Mathematics and Statistics, 47(17), 337–345. 12. Marcu, G., & Abe, S. (1996). New HSL and HSV color spaces and applications. In Imaging sciences and display technologies. 13. Morshidi, M. A., Marhaban, M. H., & Jantan, A. (2008). Color segmentation using multi layer neural network and the HSV color space. In International Conference on Computer and Communication Engineering. 14. Zhang, M., Ding, X. Q., & Li, X. (2013). Neural network based color recognition for bobbin sorting machine. Telkomnika-Indonesian Journal of Electrical Engineering, 11(7), 3728–3735. 15. Liu, Y. X. (2010). BP neural network classification method under Linex loss function and its application to face recognition. Journal of Jilin University (Science Edition), 48(3), 411–413. 16. Uchimura, S., Hamamoto, Y., & Tomita, S. (1995). On the effect of the nonlinearity of the sigmoid function in artificial neural network classifiers. In International Conference on Neural Networks Proceedings.

Research on High Feeding Speed System of L-Valve Rods Based on Two-in-One Device Shiwei Cheng, Liang Han, Kai Yu, and Rui Peng

1 Introduction With the introduction of Industry 4.0 and China Manufacturing 2025, industrial automation has become more and more prominent in the manufacturing industry. At present, many enterprises in China are shifting from labor-intensive enterprises to high-tech enterprises with high degree of automation [1]. The key part of the production line is to arrange the workpieces according to a specific attitude to complete the subsequent processing of the workpiece. The speed of loading and unloading directly affects the production efficiency of the entire assembly line. In this paper, an automatic feeding system is designed for the LVR, which is a key component of spray equipment. The system is required to order the disorderly LVR in the correct direction and accurately place it in the specified position, and improve the efficiency to meet the feed rate requirements of the line. The process flow chart of the LVR automatic feeding system device designed in this paper is shown in Fig. 1.

2 LVR Feeding Device Overall Design 2.1 Structural Design Requirements After analyzing LVR orientation requirements and the installation space of the system device, the following technical objectives are formulated: (a) Orient the L valve, move the big endian down and the little endian forward to the next station. The physical drawing of LVR and the positioning attitude are shown in Fig. 2. (b) The feeding speed of LVR automatic feeding device should be greater than 180 pieces/min to meet S. Cheng · L. Han (B) · K. Yu · R. Peng School of Mechanical Engineering, Southeast University, Nanjing 211189, China e-mail: [email protected] © Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9_24

295

296

S. Cheng et al.

Fig. 1 The process flow chart of the LVR automatic feeding system device

Fig. 2 The physical drawing of LVR and the positioning attitude

the speed requirement. (c) Ensure that the loading system can send the workpiece to the next station in a stable, correct and uniform speed. At the same time, it should be ensured that the workpiece will not be damaged or even broken during the loading process.

2.2 Overall Design Scheme According to the shape of LVR and the requirement of feeding speed, the electromagnetic bowl vibratory feeder is used to arrange and orient the workpiece [2]. At present, for the LVR workpieces, the output speed of a single bowl vibratory feeder

Research on High Feeding Speed System of L-Valve Rods …

297

Fig. 3 The position layout of LVR automatic feeding system device

is 100 pieces/min, which does not meet the loading speed requirement of the device. Therefore, the two bowl vibratory feeders are used to simultaneously sort and orient the workpieces, and the sorted and oriented workpieces are respectively output from the two curved fixed sections and the vertical moving sections, and finally the workpieces are output in turn by the common vertical fixed section, so as to realize the feeding mode of the two channels in one. Since the number of workpieces needs to be identified and the two vertical moving sections are moved, sensor detection devices and pneumatic devices are also required in the design. Figure 3 shows the position layout and schematic diagram of LVR automatic feeding mechanism in the whole workpiece processing device.

2.3 The Design of Workpiece Channel According to the overall design of the system, the whole feeding device is divided into vibration feeding device, workpiece channel, positioning device, sensor detection and pneumatic device. The workpiece channel is composed of two left and right curved fixed sections, two vertical moving sections and one vertical fixed section. First,

298

S. Cheng et al.

Fig. 4 The partial enlarged view of LVR automatic feeding system device

the two vibratory feeders sort the disorderly LVR, and then transport the directional workpieces to the left and right curved fixed sections and the vertical moving sections. When the sensor detects that the workpiece in channel is full, the left cylinder pushes the left moving section to the right, so that the discharge port of the moving section is aligned with the feeding port of the vertical fixed section. After the workpieces in channel reach the designated position through the vertical fixed section, the left cylinder returns, and the right cylinder performs the corresponding action. Figure 4 shows a partial enlarged view of the automatic feeding mechanism of the LVR. The LVR workpiece channel consists of two left and right curved fixed sections, two vertical moving sections and one vertical fixed section. The two curved fixed sections are fixed on the fixed vertical plate by bolts, and the curved fixed section inlet is fixedly connected with the outlet of the bowl vibratory feeder for transporting the workpieces oriented by it, and the outlet of the curved fixed section is aligned with the inlet of the vertical moving section. The two vertical moving sections are fixed to the vertical moving plate by bolts, the outlet of which is aligned with the vertical fixed section. The channel is designed by the method of self-weight feeding of workpiece. The method does not require a power device and has a simple structure. In order to ensure that the workpiece passes smoothly in the channel without losing the orientation state, the cross-sectional dimension of the channel must be correctly determined. According to the geometric relationship of Fig. 5: B = L +e

(1)

Research on High Feeding Speed System of L-Valve Rods …

299

Fig. 5 Channel width calculation diagram

where, B—channel width (mm), L—piece length (mm), e—the clearance between the end face of the workpiece and the inner wall of the channel (mm). Due to the presence of clearance e, it is inevitable for the workpiece to tilt and rotate in channel, as shown in the dotted line below. A torque composed of reaction force N and moment arm α is generated, so that the workpiece has a tendency to continue to rotate. The larger the value of e, the larger the angle of rotation. When the workpiece diagonal C is close to or less than the channel width B, the workpiece is in danger of getting stuck or losing orientation. Therefore, the size of the clearance should be able to ensure that the contact with the side wall, its diagonal C and the horizontal angle θ is greater than the friction angle ρ. In this case, the torque composed of the reaction force N and the moment arm α can prevent the rotation of the workpiece, so that it can be transported smoothly in the correct directional state [3]. From Fig. 5: cos θ =

L +e L +e =√ C L 2 + D2

(2)

L 2 + D 2 cos θ − L

(3)

that is: e=



So the maximum clearance allowed emax should be calculated according to limit case θ = ρ. Due to: cos θ = cos ρ = 

1 1+

tan2

ρ

=

1 1 + μ2

(4)

emax can be calculated as:  emax =

L 2 + D2 −L 1 + μ2

(5)

300

S. Cheng et al.

that is: ⎡

emax

⎤ 2 1 + (L/D) L = D⎣ − ⎦ 1 + μ2 D

(6)

where, μ—sliding friction coefficient between workpiece and side wall,μ = 0.1–0.5 D—workpiece diameter(mm) [4]. Combined with the structural particularity of LVR, the rectangular block composed of the largest size of LVR workpiece is used for calculation in design. L = 31.5 mm, D = 25 mm , emax can be calculated as: ⎡ ⎤ ⎡ ⎤ 2 2 1 + (L/D) 1 + (31.5/25) L 31.5 ⎦ = 5.90 mm − ⎦ = 25 × ⎣ − emax = D ⎣ 1 + μ2 D 1 + 0.12 25 (7) According to this, the channel can be designed. However, due to the center of gravity of LVR is at the big endian. In addition, since the length of the small endian is greater than the length of the large endian, it may occur that the workpiece rotates clockwise in the channel as shown in Fig. 6. Therefore, it is necessary to design the channel according to the characteristics of the workpiece, that is, the channel hole is consistent with the shape of workpiece, which can effectively solve the problem of rotation. To make the L valve workpiece smoothly from one section into another section, each outlet and inlet of sections are designed into a horn shape. Figure 7 shows the design model of vertical fixed section, and 3D printing is used for machining.

3 Analysis of System Vibration and Stress The cylinder of the device pushes the moving vertical plate to stop at the positioning block, and oil pressure buffer is installed to make hydraulic buffer to absorb part of energy before contacting the positioning block. According to the relevant theories of mechanical vibration, the model can be regarded as a single degree of freedom system with viscous damping, the force analysis is shown in Fig. 8 [5]. The viscous damping force Fc is proportional to the velocity V : Fc = −c x˙

(8)

x represents the displacement of the moving vertical plate, and the differential equation can be obtained by applying Newton’s law: F + m x¨ = −c x˙

(9)

Research on High Feeding Speed System of L-Valve Rods … Fig. 6 The workpiece rotates in the channel

Fig. 7 The design model of vertical fixed section

301

302

S. Cheng et al.

Fig. 8 stress analysis diagram of the system

where F is cylinder force. To solve (9), assume that the form of the solution is: x(t) = Cest

(10)

where C and s are undetermined constants, the following characteristic equation can be obtained: ms 2 + cs = 0

(11)

The root of the characteristic equation can be solved as follows: s1 = −

c , s2 = 0 m

(12)

The two solutions of homogeneous differential equation are: x1 (t) = C1 es1 t , x2 (t) = C2

(13)

The particular solution of inhomogeneous differential equation that we can solve is: x∗ (t) = −

F t c

(14)

So the general solution of the equation of motion is: x(t) = C1 es1 t + C2 −

F t c

(15)

where C1 and C2 are obtained from the initial conditions x(t = 0) = x0 and x(t ˙ = 0) = x˙0 , and the motion will decay exponentially over time. Under the condition of variable stress, the main failure form of mechanical parts is fatigue fracture. S-N curve represents the relationship between stress value and cyclic fatigue life, and represents the relationship between fatigue strength and fatigue life of materials under certain cyclic characteristics. Because the designed positioning block is hit periodically by the moving vertical plate, fatigue analysis is carried out on

Research on High Feeding Speed System of L-Valve Rods …

303

the positioning block below. Based on the analysis and comparison of the references, the following power function formula is used to calculate the general materials [6]: Sa N = C

(16)

where a and C are material constants; take the logarithm of both sides of the above equation and sort out: lg N = a + b × lg S

(17)

In the formula, a and b are material constants, N is cycle number, a and b are constants, and σ is equivalent stress. When N is equal to 106 , σ106 = δσb . According to the fatigue characteristic estimation method of domestic materials [7], Positioning block material is aluminum alloy, Take δ = 0.255, k = 0.048, According to the material property sheet [8], the tensile strength of aluminum alloy σb = 310 MPa. By substituting the above values into the formula, the following equation can be obtained: 1 k = − , a = 6 − b × lg(δ × σb ) b

(18)

So a = 45.54, b = −20.8. Substitute into the above equation to get the S-N curve of 6061-T6 (Fig. 9).

Fig. 9 S-N curve

304

S. Cheng et al.

4 Establishment and Experimental of Core Part in the System At present, some prototypes of LVR automatic feeding device system designed in this paper are built, that is, the right half of the two-in-one system is built to carry out the sensor experiment and the part transfer experiment. The prototype is built as shown in Fig. 10.

4.1 Sensor Experimental Analysis The design adopts the feeding method of the workpiece by its own weight. It is necessary to install a sensor on the vertical curved section to detect the number of workpieces passing through. When the quantity reaches a certain value, the cylinder moves with the moving vertical plate. Since the workpiece is in free falling, and each workpiece is arranged next to each other, the gap distance that the sensor can detect is the vertical distance difference between the small ends of two LVR h = 20.5 mm, as shown in Fig. 11.

Fig. 10 Establishment of the core part of the system

Research on High Feeding Speed System of L-Valve Rods …

305

Fig. 11 Calculation of workpiece gap

According to the free-falling body formula, the response time t of the sensor can be calculated as follows:   2h 2 × 20.5 × 10−3 t≤ = = 0.065 s = 65 ms (19) g 9.8 The design adopts fiber optic sensor for detection. The model is BF-5R-D1-N digital fiber sensor and FD-320-05 fiber head made by AUTONICS. This sensor can detect up to 20,000 times per second and has a resolution of 1/10,000. The light source adopts red LED modulated light to be transmitted to the fiber head through the optical fiber, and the modulated light interacts with the measured workpiece to change the intensity of the light. After signal processing, the amount of light received is calculated, and compared with the set threshold, the workpiece reaches the sensor fiber [9]. Within a certain distance of the head, the value of the amount of reflection will be greater than the set threshold, indicating that the workpiece reaches the specified position feedback signal to the controller [10]. Figure 12 shows the difference in the amount of light received by the sensor through the sensor in different modes.

4.2 Loading Rate Test Experiment After completing the sensor test experiment, this section tests the loading rate of LVR, counts the number of detection signals of the sensor, that is, each time the detected part passes, the count is incremented by one, and the cylinder actions when

306

S. Cheng et al.

Fig. 12 Sensor light absorption curve

the count value reaches the set value. The feeding experiment sets the target value to 5, 12, 15, and 18 respectively, and each target value is carried out 8 times. The feeding time (ms) is counted as shown in Fig. 13.

Fig. 13 Results of feeding rate test

Research on High Feeding Speed System of L-Valve Rods …

307

5 Conclusion This paper designs the overall structure of LVR high feeding speed system, and analyzes the key components of the system. This includes the workpiece channel, positioning device and sensor device, and realizes the feeding of LVR in turn, to achieve the feeding mode of the two channels in one. This paper introduces the structural design of the system, and finally carries out the debugging and experiments of the prototype. The experimental data shows that the system can output the disordered LVR in the correct direction and accurately position it to the specified position. At the same time, it can greatly improve the feeding efficiency to meet the feeding speed requirements of the assembly line.

References 1. Li, X. (2016). Automatic feeding system design and experimental study on automobile fuel injection pump plunger (pp. 9–10). Southeast University. 2. Sun, C. (2017). A study on key technologies of automatic feeding system of magnetic tile (p. 7). Southeast University. 3. Han, L. (2011). Electronic precision mechanical design (4th ed., pp. 115–117). Southeast University Press. 4. Xu, X. H. (1986). Electronic precision mechanical design (pp. 83–84). National Defense Industry Press. 5. Rao, S. S. (2009). Mechanical vibrations (4th ed., pp. 94–99). Tsinghua University Press. 6. Li, K. (2013). Research on the mechanical components fatigue design method and its application under impact load (p. 23). Hefei University of Technology. 7. Wu, K. J., Yu, X. H., & Qian, R. M. (2006). Mechanical design (pp. 67–74). Higher Education Press. 8. Zhao, S. B. (1994). Anti-fatigue design (pp. 339–346). China Machine Press. 9. High performance single/double digital display fiber amplifier. Retrieved from https://www. autonicschina.cn/series/3000437. 10. Huang, J. R. (2018). A study on the key technology of automatic inlay equipment for popular ornaments (pp. 35–37). Southeast University.

Conclusion

Although their topics cover a diversity of fields, these papers show the application of practical testing to the task of theoretical development. Mechatronics and its sub-field of robotics will never cease to provide a wealth of interesting research problems.

© Springer Nature Switzerland AG 2021 J. Billingsley and P. Brett (eds.), Mechatronics and Machine Vision in Practice 4, https://doi.org/10.1007/978-3-030-43703-9

309