OPTICAL FEEDBACK BASED ON THE PHOTOMETRYBY ELLIPSOIDAL REFLECTOR IN BIONIC FINGERS APPLICATION

FINGERS APPLICATION Background. The investigation focus is the specificity of effective recognition of surface types during the interaction between finger prosthesis and manipulation objects, and organization of optical feedback system in the control module of bionic limb. Objective. The purpose of the paper is development and testing of method of optical feedback organization in the systems of bionic prostheses of human hand fingers. Methods. The developed system of optical feedback is based on the microcontroller measuring device with infrared (IR) optical emitter and sensor, ellipsoidal reflector, and artificial neural network. Results. During the research the comparison of application effectivity of optical feedback system with ellipsoidal reflector and without one was performed. The analysis was performed by classification of set with twelve, eleven, and ten types of surfaces of investigated specimens with the means of artificial neural network. The obtained accuracy of surface recognition for the system without an ellipsoidal reflector was 77%, 82% and 87%. In turn, the accuracy of surface recognition in the application of the ellipsoidal reflector was 94%, 98%, and 100%, correspondingly. Such results prove the possibility of further use of the photometry system by ellipsoidal reflectors for organization of constructive parts or complete feedback modules of the bionic fingers. Conclusions. The organization system of optical feedback based on optical emitter and sensor, ellipsoidal reflectors and artificial neural network for recognition of kinds of separate surfaces, with which can interact bionic prostheses fingers is proposed. The proposed system proved the certainty in recognition of limited set of investigated specimens. The efficiency of such system can be improved by using sensors array and wider data set


Introduction
Nowadays, the development of bionic devices is very progressive and useful branch. Modern biologically compatible inventions allow renovating some critical functions of human organism for comfortable everyday life. For example, some technologies like artificial eye, nose, or limbs are already in development by investigators [1][2][3][4].
Almost all bionic devices consist of several structural parts or modules. The presence of control blocks, measuring sensors or executive mechanisms [5][6][7][8] in many cases defines the success of final realization of bionic system (BS). Moreover, for reaching of the natural sensitivity of BS there applies different feedback kinds. For example, vibro-tactile, optical, ultrasonic, and force feedbacks already uses for increasing of efficiency of modern prosthesis devices [6,9,10].
One of the main criteria of successful realization of feedback in prosthesis is the comfortable use of it without any painful senses or problems with patient skin. Generally, all kinds of feedback in prosthesis devices can be divided into two main categories: feedback for self-dependent, automatic control (SAC); and feedback with the influence on the user body (HBI).
SAC feedback category uses combined type of information about the jointning force and prosthesis position for its automatic interpretation by control block only, without delivering of any irritants to patient body [11,12]. Fore xample, such feedback type can be used for the calculation of instantaneous grip force or the control over the process of objects slipping out of the prosthesis fingers. In addition, SAC feedback category uses for determination of level of opening/closing, flexion/extension of artificial wrist also [9,13,14].
Unlike SAC, the HBI feedback category foresees the direct influence of various stimuluses on the patient body. Among the transferring methods of stimulus considers invasive and non-invasive ones. Invasive stimulation method is closest to the natural method of interaction between the human body and artificial limb.
For irritants transferring this feedback organization method often uses the direct stimulation of afferent nerves or effect of extended physiological proprioception [12].
In case of non-invasive stimulation method, the intensity of organism feelings directly depends on the intensity of selected irritant. From the other side, the efficiency of irritant strongly depends on its position, simultaneous quantity of active stimulation centers, amplitude, and impulse duration. In addition, during the use of non-invasive feedback it is necessary to consider the influence of adaptation processes in the human body. To avoid the adaptation, the selected irritants usually apply during specific time intervals [15]. The most frequently applied methods for organization of non-invasive feedback is vibro-tactile stimulation, electrotactile stimulation (ETS), and method of extended physiological effect (EPE). During the EPE the precise amplitude values, which are measured by force sensors of feedback system, should be converted into the output irritant and applied to the patient body [12,16]. At the same time, the ETS is the method, during which cathode or anode stimulation of user body is used as the instrument for transferring of irritant. The efficiency and change of feelings in this case depends on the pulsation frequency, intensity and width of the output impulse [17]. However, ETS have a disadvantageit is the possibility of appearance of painful senses, which directly depends on the skin state, size and characteristics of stimulating electrodes. Contrary to ETS, vibro-tactile stimulation has significantly lower injury influence. However, the efficiency of feedback in such case depends on the frequency, position and the size of stimulation mechanisms, and hair cover of the skin. The patient feeling on that time can vary from small vibration to acute pain [18].
Usually, for implementation into the feedback scheme the measuring sensor should correspond to set of requirements. These requirements include the spatial distribution, sensitivity, frequency characteristic, hysteresis, flexibility and reliability of construction, and other [9]. Piezo-resistive, piezo-electric, capacitive, barometric, optical, and quantum tunnel effect sensors are the main sensor types, which can be used as measuring ttransducers for prosthesis feedback system [9,13]. Moreover, for feedback realization inventors often use the combination of several sensors in one device. For example, in article [7] force and position sensors uses for spatial coordination and regulation of output compression force in prosthesis device. In the research [8] the sensitive to wrist flexion output power vibrational system is organized by simple sensors of cable tension. And in the research [19] there described the organization of the vibro-tactile feedback, which is realized by means of the position controller. The example of combined use of different sensor types is represented in the research [13]. In the current research authors applied 15 sensors based on the Hall effect, 5 sensors of cable tension based on the strain gauges, 4 sensors for current and 4 optical tactile sensors.
The interesting solution for realization of prosthesis feedback system is application of optical and optical-electronic components as the scanning sensors. Usually, optical scanning is based on the registration of the reflected light from the surface of different mediums with corresponding optical properties. Optical-electronic tactile sensors allow detecting of changes in intensity of light beams [14], and typically consist of infrared (IR) LEDs and photodetectors. In this sensor type the intensity of received by photodetectors light beams is proportional to the value of force influence on the measuring surface [9]. Optical feedback (OF) uses to ensure the contactual interaction of artificial finger with the environment, and also to perform the lateral scanning of objects and detection of their shape, texture, and even softness [14,20].
On realization of measuring optical-electronic systems it is important to reach the maximum level of useful signal, which is especially important during the registration of diffuse and/or collimated component of light, reflected from objects with different surface structure and near-surface layer. The using of ellipsoidal reflectors as a solution for increasing of specific weight of registered light during gathering of scattered light in the range of body angle 2π is successful. It proved its efficiency after interaction with rough surface of technical specimens [21], and optically turbid biological environments [22][23][24][25].
Considering the effective solution of analytical tasks and signals classification (especially in feedback systems and bionic prosthesis [26]), there successfully applies artificial neural networks (ANN) [27][28][29][30]. Their advantages in feedback organization and recognition of different surfaces allowed reaching the significant practical results. For example, in [31] neural networks were used for organization of optical feedback with application of photochromic and luminescent compounds. In researches [32,33] ANN is the instrument for recognition of surfaces and physical visions.

Problem statement
Considering the mentioned information, the goal of the research is development and aprobation of method of organization of optical feedback for systems of bionic finger prostheses for human wrists. Such method is based on the microcontroller measuring device with IR optical emitter and sensor, ellipsoidal reflector, and artificial neural network.

Initial data
he proposed system of optical feedback organization belongs to SAC category (Fig. 1), and can be a suitable solution for application in inexpensive bionical prostheses of fingers or wrists. The photometric core design of optical feedback module can be based on one of the suitable scheme-technical solutions for measuring systems with ellipsoidal reflectors [34].
The application of ellipsoidal reflectors in the specific element base would allow optimizing the construction of bionic prosthesis of human wrist finger. Fig. 2 represents the generalized structural scheme of optical feedback organization with ellipsoidal reflector (ER).
Optical feedback is realized as follows. The radiation (in correspondence to the selected optical emission configuration and receiving-registering system) is directed to the investigated object. The reflected signal, which can contain collimated and diffusive components, is gathered by the ER and concentrates in the sensitive zone of radiation receiver. Proportional to optical flux electric signal reaches the preliminary treatment block. After digitalization in analog-to-digital converter (ADC) and preliminary treatment by microcontroller, there occurs further transition of measured signal through the wireless connection to PC. The further recognition and signals classification (which were received during interaction with investigated object) occurs by using of software with built-in algorithm of neural network.

Methods and technologies
For simplification of practical realization and comparing of results, during the development of construction of bionic finger with optical feedback with ER, there was used optical couple CNY70 (Vishay Intertechnology, Inc.) as an optical sensor. Fig. 3 represents the bionic finger construction with ER application and without one.
The constructive parameters of ER (focal parameter 1.47 mm, eccentricity 0.31, reflector elements reflection coefficient ρ = 0.92, wavelength λ = 940 nm) and other elements foresees the positioning scheme, which is represented in Fig. 3 The used reflector has rotational (around the big semi axis) ellipsoid shape with internal mirror surface, which is cuted on focal planes orthogonal to the axis of rotation.
The measuring optical couple was placed immovable in distal phalanx of artificial finger. The finger was fixed in the special cleat horizontally to the investigated specimen, and ellipsoidal reflector was fixed stationary closely to the optical couple   (Fig. 3, b). From the ergonomic consideration, it is convenient to place the ER inside the phalanx of the bionic finger prototype, and in such case optimizing the reflector parameters and prosthesis construction.
Operational algorithm of module is following. After turning on power in the device and supply of IR radiation from CNY70, the registered by photodetector signal (due to back scattering) of radiation from the investigated object is delivered to the analog-to-digital converter of microcontroller Atmega 16. In the microcontroller (considering the critical value of reference voltage (RV) of the value 4.2 V, which was received from the separate Li-Pol battery) the received signal is recalculated into the corresponding amplitude level. On the next step the determined amplitude level of signal is transmitted to the software on PC through the module UART-RF of transmitter HC-12. The further interpretation and classification of the signal occurs by the software with specially created neural network. At the same time, the delay between the transmissions of each part of information to PC equals to 1 msec, the transmission speed UART -9600 bod, and the working microcontroller frequency -2 MHz.
As empirical parameters of feedback signal in the "time range" of 50 msec, there can be defined two indicators of time characteristics: amplitude peak P of measured signal for test specimen, and average absolute deviation MAD, which shows the deviation of measured set of values from its average value [5,26,35].
The general estimate of classification accuracy performs with using of confusion matrix (EM) with the size N per N classes. The matrix columns correspond to expert answers, and rows are the classification decision. During the specimen classification from the experimental dataset the answer, which is situated in the column (determined by the classificatory class) correlates with the answer in the row (class, to which the object really belongs).
The classification efficiency is evaluated by three parameters: Precision, Recall and F-score [36][37][38], and the values of them are calculated using the equations where TPthe quantity of true positive answers (this means that the surface was correctly classified); FPthe quantity of false positive results (the specimen was classified as the one surface, however really it refers to the other one), and FNquantity of false negative answers. The Precision (1) parameter can be interpreted as the ratio of objects, which were called "positive" by classificatory, to that ones, which are actually positive. The Recall (2) parameter represents the quantity of objects of "positive" class from the total quantity of correctly determined ones by classificatory. The F-score (3) parameter is harmonizing average between Precision and Recall, and is considered by the research authors as the main parameter for classification accuracy in the current research. It is important to note that F-score tends to zero, if the Precision or Recall equals to zero.
The created neural network (ANN) (Fig. 4) consist of three layers (Input, Hidden Layer, Output Layer), and is subject of scaled errors correction, which is conducted with the use of Levenberg -Marquardt algorithm [39,40]. It is used for identification of 12, 11, and 10 types of specimen surfaces based on the analysis of back scattered optical radiation signal.
The network performs "No/Yes"-type evaluation for determination of corresponding category for a b Fig. 3. Investigation scheme without reflector (a) and with reflector (b): 1 -bionic finger; 2 -optical couple; 3 -investigated specimen; 4 -ellipsoidal reflector The work distance The educational and testing dataset for the network equals to 600, 550, and 500 values for P and MAD for 12, 11, and 10 specimens correspondingly. At the same time, the values from the test dataset were not considered in educational one. For the development of ANN model there was applied Matlab Neural Network Toolbox.

Investigation results
As the dataset there was used the value of intensity of reflected (back scattered) light for twelve types of materials as both the biologically compatible and artificial structure (Fig. 5). The specimens, made of wood, plastic, metal, glass, ceramics was selected considering that these are the main materials, which are used in the production of household goods. For example, the metallic and wood specimens were used as the prototypes of materials for handrail production. The ceramic surface is used as the base material for cup or other kitchen utensil.
For conditional ex-vivo researches of biologically compatible surfaces there were selected specimens of hen muscle tissues, pork skin, egg shell, banana and orange skin, and the tree green leaf also. As the example of delicate surface (with which can interact the prosthesis user in everyday life) there was selected silk tissue.
The first measurement series was performed during the calibration on the daylight illumination, and the specimen was initially placed at the distance 1 sm from the distal phalanx of artificial finger with the built-in measuring sensor (see Fig. 3, a). The second measuring series foresee the application of ellipsoidal reflector in the artificial finger construction. This reflector acts as the concentrator of diffusive-scattered light from the investigated sur-faces (see Fig. 3, b). Fig. 5, af illustrates the measurement without ER for surfaces of following types: "wood", "plastic", "metal", "glass", "cloth", "ceramic", and Fig. 5, gl illustrates the photometry by ellipsoidal reflector on the rest of investigated surfaces ("hen muscle tissue", "pork skin", "egg shell", "banana skin", "orange skin", "green leaf ").
The investigation algorithm includes the data measurement in five iterations per a single specimen. A single itearation involve the displacement of the artificial finger from the initial position in the direction to the specimen until the moment of direct contact of distal phalanx with the investigated surface. At the same time, the final data set for training and testing of neural network consists of twenty-five iterations each. The received by the optical couple signal intensity was measured in the "time range" 50 msec. Fig. 6 represents the confusion matrices for classification of 12, 11, and 10 surface types. Fig. 6, a, c, and e represents the efficiency of specimen classification for the first measurement series; and Fig. 6, b, d, and ffor the second measurement series with the application of feedback system based on the ellipsoidal reflector.  Fig. 6 represents the significant increase in the classification accuracy during the second measurement series. Thus, according to the efficiency estimation reading F-score, the difference between the first and the second series equals from 13% to 17% for different quantity of measured surfaces. Fig. 6, a represents the confusion matrix for classification of twelve specimens, which were investigated by means As it can be observed on figure, the neural network was able to determine all kinds of investigated surfaces, and correctly classify only six of them. The error quantity differs in classes where the specimens were determined incorrectly. The biggest uncertainty observes for following surface types: "wood", "metal", and "green leaf". In these cases, the percent of false positive (FP) results equals to 80% for the set for 25 testing iterations for each surface. For the specimen of "plastic" type such percentage equals to 60%, and for the "ceramic" and "banana skin" -20%. At the same time, the confusion matrix for measurement series with application of ER (Fig. 6, b) during the classification of twelve surfaces demonstrates the significantly higher indicators: Precision = 0.93, Recall = 0.95, F-score = 0.94. Similar to the represented in Fig. 6, a case, classificator (neural network) determined all kinds of the investigated surfaces also, however, the quantity of false results for wrongly classified specimens decreased sharply. The biggest quantity of false positive results the network made during the recognition of "banana skin" specimen -60%, and also "cloth" -20%.
As the F-score value for measuring system with ER was significantly higher, and FP results were observed only for two specimens, authors of the current research decided to continue the further testing of the created neural network on the eleven surfaces. From the testing and educational data set there was previously excluded the specimen, which provide the biggest quantity of false positive results of classificator ("banana skin"). As a result, numerical values of evaluation parameters for classification of eleven specimens without ER (Fig. 6, c) equals to: Precision = 0.75, Recall = 0.9, F-score = 0.82. At the same time, the quantity of FP results for the "metal" specimen had reduced to 40%, for "wood"to 60%, and the "plastic" surface was classified without errors. However, the quantity of FP results for "ceramic" and "green leaf" surfaces didn't change. Contrary to the results, represented in matrix on Fig. 6, a, the "pork skin" surface received 80% of FP results. The results of classification of eleven surfaces by the optical feedback system with ER is represented in Fig. 6, d. As it can be determined, the numerical values of parameters equal to: Precision = = 0.98, Recall = 0.98, F-score = 0.98. Due to the previous exclision of the "banana skin" from the testing and training data set, the total quantity of FP results (part of which were received for this particular specimen) decreased also. Thus, the value of F-score parameter had increased. The quantity of FP results for "cloth" specimen didn't change.
During the classification of ten surface types authors decided to exclude the "cloth" specimen from the testing and training data sets, and perform testing of the developed network for system with ER and without one. Similarly to the results, represented on Fig. 6, a and c for systems without using of ER, there was observed increasing of numeric values of parameters: Precision, Recall and F-score. Specifically, for the matrix, represented on Fig. 6, e, the numeric values equals to: Precision = 0.84, Recall = 0.9, F-score = 0.87. The biggest quantity of false positive results was observed during recognition of surfaces of "wood" and "ceramic" types -60%, the "metal" specimen had 40% of FP results. Contrary to the data from matrix, represented on Fig. 6, c, the "green leaf" specimen was determined without faults. The results of classification of ten surface kinds with application of ER is represented on Fig. 6, f. The numerical values of matrix evaluation parameters equal to: Precision = 1, Recall = 1, F-score = 1. Such results ensure that the accuracy of specimen recognition equals to 100% for each iteration from the test data set.
Thus, it can be concluded that the results, received during the investigation, proved the following hypothesis: the relatively simple model of ANN, which is used together with the optical feedback system based on the ellipsoidal reflector can be used for classification of particular surface types, and its capability is sufficient to the general identification of limited quantity of test specimen.
The projected optical feedback system based on ER and ANN represented the high classification results (F-score = 0.94 for twelve surface types) even with only one measuring sensor and relatively small quantity of neurons h 12 M  in the hidden layer of neural network. As the continuation of investigation, authors consider to test the system in more significant test data set with the increased quantity of test specimens. At the same time, for increasing of efficiency of the projected optical feedback system it is necessary to investigate the possibility of its combining with the array of several optical sensors, which are functioning on different wavelengths, and also the sensor, which allow determination of shape and irregularities of the surface. The information, received from such array, allow increasing of the quantity of output parameters for training of neural network and, as a result, ensure additional possibilities for increasing of quantity of recognition surfaces.

Conclusions
In the current research authors propose a method for organization of optical feedback module for bionical prostheses of human wrist fingers. It is based on the IR optical couple, ellipsoidal reflector, and a simple model of neural network.
Electro-optical scheme of module was tested on specimens of natural and artificial structure with using of ellipsoidal reflector and without one. The investigation results reveal that the use of measuring system with ellipsoidal reflector allow reaching of accuracy values during surface determination up to between 94% and 100% depending on the quantity of testing specimens (from 12 to 10 ones), which on average is 13% better then the testing results of such system without ellipsoidal reflector.
Based on the received results it can be assumed that the proposed optical feedback module with microcontroller-based measuring device is suitable solution for using in inexpensive autonomous bionic finger prostheses with SAC-feedback type system.
As the continuation of the current research, authors propose the testing of the proposed module jointly with force sensors. The implementation of such solution will provide additional information about surface structure of separate specimens, thus increasing the classification accuracy in cases of more significant quantity of investigated surfaces.