Computerized pragmatic assessment of Prakriti Dosha using tongue images- Pilot study

Objective: To design an intelligent system to identify Prakriti of a person based on tongue image analysis and machine learning algorithms. Method: Tongue images were captured using webcam and processing was done using Raspberry Pi development board. The algorithms were developed using OpenCV python libraries. Thirteen geometry features, two non-geometry features, and two texture attributes were extracted from each tongue image. These features were used to identify prakriti Vata, Pitta and Kapha. Performance of three classifiers namely, KNN, Neural network and Decision tree was verified for the precision in identifying class of the test image. Findings: KNN provided sensitivity of 42.85% for Vata, for Pitta and Kapha prakriti it was 55% and 45% respectively. With Neural network sensitivity was improved to 62.5% for Vata and Pitta and for Kapha Prakriti it was 60%. Decision tree exhibited better sensitivity of 83.33% for Vata, for Kapha and Pitta prakriti it was 75% and 71.42% respectively. During blind validation to identify prakriti, each physician was told to analyse images for said classes. This procedure resulted in sensitivity of 81.25% and 84.61% respectively.


Introduction
represents exemplary Tridosha images from our dataset. Figure 3 shows attributes of the tongue in various disease conditions. For exterior conditions, tongue examination is done at the coating. Its thickness gives an indication of the strength of the external pathogenic factor. For interior conditions tongue body is examined. Its color and shape will evidently reveal the presence of any internal health issue. In cold conditions, the tongue body color is pale and has white coating. For conditions of heat, tongue body color is red and coating yellow.  • Color -Uniformly pink or pale-red • Shape-Neither too thick nor too thin, has even width , oval shape • Moisture-healthy tongue is moist • Coating-thin transparent or white coating • Movement-when sticking out it is straight rather than veering to one side  Table 1 represents Prakriti dosha and its attributes, the tongue characteristic used to determine Vata, Pitta and Kapha prakriti. Careful analysis of the tongue body color, shape, tongue coating color, thickness and distribution can provide a strong reflection of the disease pattern.

Tongue inspection and problems faced during tongue diagnosis
Ayurveda physician carefully examines tongue following the guidelines, to identify disease specific features in different parts of tongue. These guidelines are: 1. Tongue to be stretched out naturally with muscles relaxed. Over extension of tongue to be avoided as it affects blood circulation. 2. Tongue inspection starts from tip to root. 3. Tongue examination should be done in day light. When observing tongue at night, the fluorescent lamp can be used because it is similar to natural light In examining tongue to extract the visual information, physician come across various problems like, • Atlas-There are no standard tongue topographic representations in various body conditions. • Subjective Variability of diagnosis-All practitioners do not have the same expertise in tongue examination.
• Protocol -opinion of expert varies based on the time at which tongue diagnosis is performed. https://www.indjst.org/

Related Work
We performed literature review in order to understand arena of this research topic and following are the observations: In (5) authors highlighted the integration of information and communication technologies and modern science and engineering techniques with Ayurveda. They have emphasised that tongue is a strong diagnostic tool for looking at one's systemic health. Tongue analysis can be used to supplement pulse, nail and facial diagnosis in order to perform a more thorough evaluation of a patient's health.
In (6) authors presented a computerized tongue diagnosis method for the diagnosis of diabetes mellitus. This is based on the pathological changes on the surface of a tongue. They have used chromatic and textural features to build the mapping. Region growing algorithm has been used for extracting the shape of the tongue. Authors also used Level set Algorithm for tongue image segmentation. Further, shape detection, pimple detection and crack detection was done with help of color intensity extraction method. Then Mean and Entropy value of normal and abnormal tongue was compared to identify diabetes mellitus. In (7) authors presented a method to identify shape of a human tongue using geometry features. Thirteen geometry features based on measurements, distances, areas, and their ratios were extracted from tongue images. Classification of the shapes was carried out with a decision tree. Experimental results showed that the extracted geometry features are effective at tongue shape classification with average accuracy of 76.24% for all shapes.
Authors proposed systematic tongue color analysis system for medical applications in (8) . They have used the tongue color gamut to extract tongue foreground pixels and performed classification using color features. Database consisted of 143 Healthy and 902 diseased tongue images. Both KNN and SVM classifier exhibited 91.99% accuracy.
In (9) authors proposed tongue colors classification with a standardized tongue image acquisition process and color correction. They compared the mean value of L * a * b * of different tongue colors and evaluated the effect of the tongue color classification by machine learning methods. Random forest method showed better performance than SVM in classification.
In (10) authors have proposed a method to detect white coating, cracks, buds and pimples. Color image filtering and enhancement for identifying the true nature of colors on different parts of the tongue has been proposed. This brings out white coating. Contrast enhancement and image sharpening has been implemented on grey scale image for highlighting the shape, cracks, buds and pimples. (11) Provides methods to detect the white coating, true color, cracks, pimples and buds of the tongue. Edge filtering and bilevel thresholding steps are used to filter out the unwanted edge and threshold edge image. Authors have proposed a heuristic initialization and active contour model for final segmentation.
In (12) a method to classify tongue image as healthy or unhealthy has been proposed. It is based on application of linear wavelets and statistical evaluation. Tongue image enhancement for extracting the true nature of colors and textures on different parts of the tongue has been described. Later wavelets transforms and the statistical features were computed. Authors reported 92% accuracy in classifying healthy and unhealthy tongue.
In (13) authors have proposed NI LABVIEW for tongue image analysis. Standard deviation and mean values were calculated for tongue portions corresponding to Vata, Pitta and Kapha Comparison between standard deviation and mean values of a healthy and a diseased tongue was performed.
In (14) authors first established a tongue color spectrum with 12 colors which could be present on the tongue. Later to identify they calculated geometric details based on the thirteen features like texture distance, area and their ratio etc. Authors have used these findings to detect whether a person is positive with diabetes or not.
In (15) authors proposed use of color gamut boundary descriptor using one-class SVM algorithm. Centers of 12 tongue color categories were defined. Color distributions of several typical tongue features were obtained to build a relationship between the tongue color space and color distributions of various tongue features. With the obtained tongue color space, a new color feature extraction method has been proposed for diagnostic classification purposes and validation on large tongue image database.
In (16) authors performed experiments on tongue images provided by the Laboratory of Traditional Medical Syndromes, Shanghai University. Double geo-vector flow (DGF) was proposed to detect the tongue edge and segment the tongue region in the image. The geodesic flow was evaluated in the lower part of tongue and geo-gradient vector flow was evaluated in the upper part. The DGF exhibited better results with true-positive volume fraction attainment of 98.5%, false-positive volume fraction 1.51%, and false-negative volume fraction of 1.42%.

Main focus and motivation
Recently, this area has attracted researchers to understand tongue reading and propose new computerised methods which will help in analysing disease specific characteristics.
https://www.indjst.org/ The processing of tongue image is a multifaceted task which needs basic understanding about tongue image characteristics. The possible limitations with existing techniques and scope identified for research in this area are: 1. Tongue diagnosis practice is subjective in nature. Therefore, to large extent validity of results is highly influenced by expertise in tongue reading. 2. Non-Availability of authentic tongue image databases for research 3. Generally experiments are carried out with insufficient samples; hence efficacy of such algorithms is an issue to use these algorithms in application development. 4. Research work on organ specific traits for the identification of disease/condition needs further work.
The process of visually understanding the tongue image by medical practitioners is challenging. There is strong need of using advanced image processing and machine learning algorithms to develop a system which will be useful in automated tongue diagnosis.
The broad objectives of this research work are, • Develop a portable real time system to process tongue images and computerised classification of Prakriti.
• Many times, it is necessary to perform tongue examination in certain portion of tongue. Therefore, there is a need to develop region of interest based image processing modules which will highlight white coating, cracks, pimples/buds, shape of the tongue etc., to get more valuable information which is not available by direct visual examination of the tongue. • Perform feature engineering to implement machine learning algorithms to identify Prakriti dosha Vata, Pitta and Kapha.
• Implement comparative analysis of machine learning algorithms to decide the best suited for real time tongue analysis. • Ambiguity and subjectivity largely affect diagnostic results. Therefore, blind validation by experts is essential to check subjective variability of the results and compare it with system's performance.

Proposed methodology
Steps involved in Tongue Image Analysis for Prakriti Dosha identification are as denoted in Figure 5

Tongue image acquisition-
In the proposed system for tongue image analysis, tongue image acquisition can be done in two ways: 1. Real time acquisition and analysis-Direct image capture using webcam connected to raspberry pi. 2. Offline analysis -Use of tongue images captured using mobile phone camera by physician. Figure 6 shows experimental setup to capture tongue images. https://www.indjst.org/

Study population and protocol used during data acquisition
Tongue images were collected in the morning. Subjects were adults and not under any treatment. Total 120 tongue images (one image per participant) were collected from the participants. The images were labelled by a physician as Vata, Pitta and Kapha.

Tongue image Pre-processing
The acquired tongue images were pre-processed to remove noise. Performance of median filter with different mask sizes has been analysed. With increase in mask size edge effects seen stifled. Power signal to noise ratio (PSNR) was calculated to decide upon the mask size, as shown in Table 2. Based on results obtained median filter with mask size 3 has been fixed. Figure 7 shows results of median filter with various kernel size.

Region of interest processing using Grab-Cut algorithm
For precise segmentation and to identify attributes of health condition on certain portion of the tongue and exclude other the background information like lips, teeth etc, Grab cut algorithm was implemented. Algorithm 1. Three functions were developed to draw a bounding box and get the region of interest (ROI) from the tongue image.
(a) EVENT_RBUTTONUP -When the right mouse button is clicked, the (x,y) coordinates of that particular point gets stored which indicates the starting point of the rectangle. (b) EVENT_MOUSEMOVE -to detect the movement of mouse and if done so, it records the top-left (x,y) coordinates and bottom-right (x,y) coordinates with its width and height. (c) EVENT_RBUTTONDOWN -Updates the list of points containing the ROI when the right button is released.
2. For clustering foreground and background pixels, Gaussian Mixture Model is used. It uses probabilistic approach to cluster the pixels based upon its relation with the other hard-labelled pixels in terms of color statistics. 3. Then weighted graph is obtained as the minimum sum of weights of edges to segment. 4. Finally, function the on mouse to be called to display the input image and ROI. Figure 8 represents the results obtained for region of interest based segmentation and Figure 9 gives the flow of Grab Cut algorithm.

Tongue feature engineering
The human tongue encompasses several topographies. Traditionally, medical practitioners with good experience would diagnose correctly based on certain features (7) . But these diagnostic results also have ambiguity and subjectivity associated with it. To eliminate these qualitative facets, tongue images can be tangibly analyzed, which offers a new way to disease diagnosis. In various disease condition, specific features are seen on human tongue. These characteristics will play crucial role in the analysis of tongue images.  In this pilot study, we have considered attributes shown in the first row and second row of Table 3 (those attributes are highlighted).
Following are the steps used for extracting features from tongue images.
1. The original RGB tongue image converted to a gray scale as certain features prominently seen in this component. 2. Identification of cracks and pimples -Edge detection is implemented using the canny edge operator to detect sharp changes in the intensity values in the selected (ROI) part of the tongue. Erosion followed by dilation is used for linking the detected edges. 3. Texture -this was identified in terms of mean and standard deviation of the foreground ROI 4. Shape of the tongue: Shape of the tongue was computed in terms 13 geometry features as shown in Table 4    • Smaller Half Distance is either length or width whichever is short. • Centre Distance is the distance from the y-axis centre point to the centre point of length and centre distance ratio is ratio of centre distance to length of tongue. For other features, location of the centroid was located on the tongue portion. Then circle, square and triangle area in terms of number of pixels were computed. • Circle Area (CA) -Tongue area is the area of the entire tongue body. The total area ratio can also be used to describe the degree of similarity of a shape to a regular shape. Tongue shape was identified by expert and this was one of the strong input for analysis. If a tongue resembles circular shape, the horizontal width and vertical length will be same, but its circular area ratio will be closer to difference between Euclidean tuple edge point coordinate right and left side. For precise results this was computed after detecting contours using canny edge operator as shown in Figure 11.
• Rectangle Area (RA)-A rectangle tongue's vertical length is long, but its Horizontal width remains relatively constant. To calculate this Euclidean tuples identified at the extremities as shown in Figure 12 Rectangle Area Ratio= Rectangle area/ /Tongue area (2) • Triangle Area (TA) -If the tongue shape is triangle, its vertical length is greater than its width; steadily decreasing as it approaches the tip of the tongue as shown in Figure 13 Triangle Area Ratio (TAR) = Triangle area/Tongue area (3)

Fig 13. Triangle area of tongue
• Circle area ratio, square area ratio and triangle area ratio were calculated by taking ratio of respective geometry shape area to number of foreground pixels in the segmented foreground tongue area. • Cracks and pimples on the Tongue -A cracked tongue is frequently seen in a clinical practice. These cracks can vary greatly in number and depth. They can be barely visible lines or extremely deep fissures. However, sometimes a tongue develops a wrinkled appearance with Pines on its surface. If the tongue is pale and has cracks, it indicates Spleen deficiency. Short, irregular cracks indicate Stomach dryness with depletion of fluids and heat from excess inside. Algorithm is explained in the section below. Results obtained are as shown in Figures 14 and 15 Algorithm The algorithm developed to detect cracks has following steps.
• Histogram equalization was applied on the gray scale image to highlight cracks on the tongue. Then, various first order and second order derivative filters were implemented to analyze the performance in detecting edge map. Performance of Canny, Second order derivative filter, Laplacian and Laplacian of Gaussian (LoG) has been verified. It is observed that Canny filter is the best suited to detect edges and pimples on the tongue image. Threshold value was adjusted experimentally for our image dataset. • Detection of Coating-In clinical practice, the tongue coating is frequently displayed in two or more different colors simultaneously. This is because in the course of a disease's pathological evolution the coating may only partially change color. Different parts of the tongue may display older or more recent coatings. Experienced physician can gauge its past and future development by analyzing the different coating colorations on different parts of the tongue. White coloration is present on every human's tongue. But excess of white coating indicates Kapha imbalance or mucus accumulation.
https://www.indjst.org/  • Tongue Texture -tongue texture is typically formed by different types of papillae which are microscopic structures on the upper surface of the tongue. Texture feature extraction from tongue images is presented in this section. Tongue texture was identified in terms of mean and standard deviation for total area of tongue. Table 5 provides calculated mean and standard deviation values for sample tridosha images.

Classification
Tongue Image dataset: The tongue image database was composed of 120 images (one image per person). During data acquisition stage, these images were labelled as healthy and Prakriti dosha prominent. Table 6 gives details of classes and number of images in each class, according to physician's ground truth labelling. Total 107 images were used for classification excluding normal tongue images.  Table 7 gives details of feature space used and Table 8 provides details of criteria used for labelling, coating and cracks/pimples on tongue images. During classification, tongue image dataset of 107 images was divided randomly into 80:20 proportion. We have checked our results with other proportion as well, but the precision was very low for 60:40 and 70:30 proportions.
To classify tongue images into three classes as mentioned earlier, performance of the following classifiers have been verified.

K-nearest neighbors(KNN) algorithm 2. Neural Network 3. Decision Tree Classifier
Class Labelling used in the classification is as indicated in Table 9 Table 9. Class labelling Class Class labelling Vata 00 Pitta 01 Kapha 10 Result and discussion: Tables 10 and 11 gives confusion matrix and machine learning metric for KNN classifier, based on Manhattan distance metric.  Vata  3  2  2  Pitta  1  5  3  Kapha  3  3  5 https://www.indjst.org/ 27 images were used for testing to determine performance of the classifier. Proportion of correct class identification was 42.85% for Vata, and for other two classes it was 55% and 45% respectively. We have also verified the performance of the KNN classifier for Euclidean distance metric and it was almost same. The results obtained with KNN algorithm were not encouraging; therefore we decided to go for supervised learning algorithm. Performance of neural network with back propagation algorithm was verified for learning to rate 0.001 and Hyperbolic Tangent activation function. The analysis was performed with respect to one hidden layer. Tables 12 and 13 denote the results obtained.  With neural network improved performance was observed in terms identifying true positives, but more number of epochs were needed for training.
Finally, we verified results for Decision Tree classifier as it is one of the popular classification algorithms for interpreting results in real time. Its training time is faster compared to the neural network. The goal of using a Decision Tree is to create a training model that can be used to predict the class, by learning simple decision rules inferred from the training data. Categorical Variable Decision Tree and Gini index algorithm has been used in this work. In the decision tree chart, each internal node has a decision rule that splits the data. Gini ratio, measures the impurity of the node. Table 14 represents the confusion matrix achieved for the decision tree classifier and Machine learning metric is given in Table 15 . Out of 20 % samples used for testing (27 images), ten samples were truly positive belonging to class Vata, six to Pitta and five samples to Kapha prakriti. Accordingly, machine learning sensitivity per class obtaned is 83.33% 75%, 71.42% for Vata, Pitta and Kapha prakriti respectively.
Neural network and Decision tree classifier, are very different from the way they look to the way they find relationship within variables. The neural network is an assembly of nodes, looks somewhat like the human brain. While the decision tree is an easy to follow top down approach of looking at the data. One of the attractive feature of the Decision tree is, it inherently "throws away" the input features that it doesn't find useful, whereas a neural net will use them all unless some dimensionality reduction is used as a pre-processing step.
Further, image based blind validation was performed by two physicians who had not done labelling. For doing blind validation, tongue images were shown to them separately and were asked to identify Prakriti. As a part of blind validation we only showed the tongue images captured in front view as tongue images were not captured in other profiles. Therefore, this blind validation is not performed in all dimensions. According to inputs provided by physicians, during examination the tongue was divided into two parts; Examination of the tongue body and Examination of the tongue coat. They observed certain noticeable attributes like, coating, color, pimpls, cracks and neighboring tissue colors, texture, fur, thickness, fissures, swellings, and bleedings. According to them other features seen on tongue to lateral sides, are equally important and hence we have also been advised to captur tongue pictures in other profiles as well. When tongue examination is done physicians do ask certain questions to patients to get indirect inputs, and we were advised to prepare general and disease specific questionnaire to further improvise the results. These valuable inputs in will be part of our future work planned on disease specific tongue image analysis. Table 16 represents results obtained during blind validation by two tongue diagnosis experts. The blind validation results depict that number of true positives are more in both cases. That means, those images which were labelled as Vata, Pitta or Kapha Prakriti have been correctly recognised to their true classes. Validation by Physician2 showed improved sensitivity as well as specificity. These results are based on comparison with our ground truth labelled dataset. We also believe that there is also a need to do this exercise other way round. In this exercise number of true positives are more; this may be because, in our feature engineering step we have not considered all the attributes but when physicians look at the image they have also noticed few characteristics like fur, thickness, fissures, swellings, bleedings etc.; these additional features could be the measure point useful in improvising performance of the system designed.
Hence, to summarize: feature engineering will be the main component of an intelligent tongue diagnosis system. One of the main challenges in bringing high-tech innovations in tongue diagnosis lies at the core of the Ayurvedic practice. Ayurveda is the science evolved by practice. While deciding about Prakriti of a person, experts follow their traditionally gained experience and try to look for certain tongue characteristics which must be quantitatively extracted in feature engineering step. Subjective Variability of diagnosis plays vital role in validation. All practitioners do not have the same expertise in tongue examination and hence validation procedures need standard protocols based on disease specific visible tongue characteristic features on front and lateral side of tongue surface. There will never be a replacement for a highly trained, knowledgeable Ayurvedic doctor or practitioner, their diagnostic methods, and the traditional teaching strategies by which this knowledge is shared. But technology should be a welcome partner in this process.

Summary and Conclusion
The need for scientific evaluation of Ayurveda has been recognized for a long time. This research work is a pilot effort in the research planned for tongue image analysis for specific disease/condition identification. Tongue images were captured using webcam and processing was done using Raspberry Pi (Broadcom BCM2837, 4× ARM Cortex-A53, 1.2GHz) development board. The tongue image analysis algorithms were developed using OpenCV python libraries. Thirteen geometry features, two non-geometry features and two texture attributes were extracted from each tongue image. These features were used to identify prakriti imbalance Vata, Pitta and Kapha.
Performance of three classifiers namely, KNN, Neural network and Decision tree was verified for the precision in identifying class of the test image. KNN provided sensitivity of 42.85% for Vata, and for Pitta and Kapha imbalance it was 55% and 45% respectively. With Neural network sensitivity was improved to 62.5% for Vata and Pitta imbalance and for Kapha Prakriti https://www.indjst.org/ detection it was 60%. Decision tree exhibited better sensitivity of per class; 83.33% for Vata imbalance; for Kapha and Pitta prakriti it was 75% and 71.42% respectively.
During blind validation, each physician was told to analyse images for said classes. This procedure resulted in sensitivity of 81.25% and 84.61% respectively, in identifying prakriti imbalance. We found that the number of true positives in manual examination (physician1) are more whereas for Physician 2 true negatives are increased and hence specificity which is percentage of people who are correctly identified as not having particular prakriti. Validation by Physician2 showed improved sensitivity as well as specificity. Subjective Variability of diagnosis plays vital role in validation. All practitioners do not have the same expertise in tongue examination and hence validation procedures need standard protocols based on disease specific visible tongue characteristic features on front and lateral side of tongue surface.
This research work is in experimental stage towards an effort to develop a system which could be used for real time tongue analysis. The results obtained herein are encouraging to further continue this research work with well-defined protocol. We have realised that feature engineering step will play crucial role in improving results Shape of the tongue is again subjective in nature. The computation of tongue area needs further guidelines and strong mathematical basis which will further contribute in improving efficiency of the classifiers. With this pilot effort in identifying prakriti imbalance, we strongly bring out few observations like, unlike other research works with different imaging modalities, it is not a common practice to create repository of authentic tongue images in an open access database, thereby limiting the scope of research in this area. There should be a benchmark datasets of classified tongue images in the open access database to broaden the horizon of tongue diagnosis. Without a standard dataset, it is difficult to perform comparable experiments, verify and validate the results; and further advancements in tongue diagnosis. There is also a need of documentation based on traditionally gained experience with respect to tongue image quality, image acquisition guidelines and tongue features specific to disease. Therefore, there are limitations in achieving the accuracy, robustness and reliability during experimentation. While developing automated tongue analysis algorithms, supplementary inputs about disease specific features and their quantification through computational methods will play a crucial role in achieving good diagnostic accuracy.