360-Degree Panoramic Image Stitching for Un-Ordered Images based on Harris Corner Detection

Objectives: This work aims to provide a system to arrange images automatically and produce the High-resolution panoramic images of a 360-degree wide range. Methods: This work consists of many stages which start by arranging the set of unordered input images. The Harris corner detector is used in the process of detecting key-points. Euclidean distance measure is used to compute the distance between points and determine matches. The stage of stitching consists of several steps which are detecting key-points, matching, estimating the shifting matrix between images by using Random Sample Consensus (RANSAC) algorithm, aligning, and blending. Findings: Experiments show that this method is able to arrange different sets of images, where samples taken are between 15–18 images. Depending on the threshold values in the verification process applied to the matched points, which range from (0.4 to 0.6), the number of extracted features and the degree of accuracy can be controlled. Also, the process of blending images based on the taking of a different percentage of the overlapping areas according to the proximity to the contact line gave a distinctive blending result. Application: Production of panoramic images are used in real estate photography, the creation of an attractive experience on viewing all location details, and lots of other uses.


Introduction
Panoramic photography is one of the most commonly used techniques which are concerned with imagining the virtual environment that depicts the actuality of the real environment. So, it is an obvious expression of virtual reality 1 . The word "Panorama" originates in Greek. It means wide-angle viewing and is used in space representation and drawing. The word "Panorama" dates back to the 18th century.
It is used by the English painter Robert Parker to describe the panoramic paintings 2 . Panoramic image is consistency from many real-world overlapping images, and generally is constructing a wide Field Of View (FOV) image up to a "full view", namely 360 degrees. It can contain complete information about the scene from certain views and is also characterized as a high-resolution image to be a more intuitive user experience 1,3 . The automatic creation of panoramic image stitching is one of the out-standing contributions of research 4 . Image stitching is the process of concatenating two or more images that are relatively collective to generate a large panoramic image without overlapping 5,6 . The key to image stitching is image registration, with regard to the diverse techniques of image registration. In general, image stitching can be classified into two categories 7,8 . One of a possible approach is just too directly match pixel intensities. These methods provide very accurate registration because they have a major use for all input data, but if they do not have a near initialization property, they will fail. The other major approach is the feature-based approach. This registration does not need any type of initialization, but it includes the conventional approaches for feature matching techniques, which doesn't have the properties of invariance. The feature-based approach consists of general steps, which are extracting distinctive features, such as point, line, and edge, then matching individual key-point to make a global correspondence, and finally estimat-ing the geometric transformation among the images 9,10 . Image stitching involves many advanced techniques like computer vision, computer graphics, and digital image processing 11,12 . In this study the generation of panoramic images is described as follows: The series of images is first captured. Each image is arranged in the sequence, and then is warped onto a special surface is the cylinder. The panorama is therefore generated by locating the approach overlapping region among the adjacent images in the series and using image stitching for the sequence of images 3 . The rest of this paper is recognized as follows: Section two presents a discussion of Harris corner detectors. Section three briefly discusses of the stages of a panoramic creation. Section four shows the experiment results. Finally, section five presents the conclusions arrived at from the study.

Harris Detector
The Harris algorithm, also called Harris operator, was introduced to localize image features. It uses the corner and edge combined, based on the local auto-correlation function to measure cornerness 13 . The strength of the response can be determined by computing the following Equation: Where I x and I y : The image derivatives along x and y directions.
k: Constant parameter with an exemplary value is 0.04. λ 1 , λ 2 : The eigenvalues of the autocorrelation matrix (M).
w (u, v): Representing averaging of window weight, either a Gaussian window or square window or another rectangular window which gives weights to pixels underneath. I x 2 , I y 2 , and I I x y are smoothed by using the box filter according to the following Equations as depicted in Figure 1.

Panoramic Image Stitching Model
This section offers one of the models of image stitching based on feature-based techniques. Some minor changes occur on other models, but they share the main steps 11,12 . This model consists of two main stages, each of which consists of sub-stages that are described in Figure

Images Ordering
Most panorama creation projects impose limitations on the way images are inserted. These restrictions are either by forcing a sequential entry of images or the entering of serial images that is done manually by the user. In this work, a method will be provided to arrange the input images automatically without user intervention. The proposed method depends on calculating the amount of similarity between images based on the matched key-point. The search for matches is done with the technique similar to K-nearest neighbor (KNN) technique that is adopted to select the nearest key-point descriptor and retrieve it. In each comparing process, the feature descriptors of one image are compared against all feature descriptors for the second image and the number of matching between them is retrieved. The same thing is repeated between one image and all dataset images. After completing matching between all images, the most similar image returns. It keeps the maximum number of similar of matched features.

Input Unordered Images
The proposed system takes a set of unordered images from the dataset. These images are captured from different places in the scene. However, there are two important things. The first is enough overlapping area between them, and the second is that it must be captured as one horizontally level as much as possible. Depending on particular features in overlapping part between images, the system selects images for the scene from a variety of unordered of image shots.

Feature Extraction
Instead of depending on the whole image, some points are selected according to predefined criteria to represent the most important things in images. This is done by using feature detection and extraction 16,17 . As mentioned above, the Harris algorithm will be used to extract the corner which represents the key-points.

Feature Matching
After extracting the pair of the image feature, the matching is the next step. It corresponds to extracted features with each other 16 . This depends on the amount of similarity between the features which is calculated by Euclidian distance metric. In most cases, there is a certain limit under which a match can be accepted. This is called "matchingthreshold". In this system, different values will be given to this threshold and show the results that relate to them. After comparing the robust features between the first and second images, if the features are matches, the locations will be specified.

Ordering Flat Images into Set
After calculating the number of matching points between each image and all images in the set, they compare each  time to the previous comparison result. If they are higher than the previous ones, they will be replaced, otherwise they will be retained. At the end of each repeat, the image that is most compatible with the previous image will be obtained.

Image Stitching
Image stitching can be defined as the process of merging an arbitrary number of photographic images with overlapping fields of view to reproduce a panoramic image 9 . It is sometimes used to increase the image resolution, where the images are aligned and blended 8,11 . The mathematical point of view of image stitching algorithms is a collection of mathematical concepts that result in large image matrix by processing and merging multiple image matrices. Digital images are nothing, but a pure matrix with different intensity values. However, different intensity values in each image's pixel represent a different type of information or objects in real-life 18,19 . Image stitching processes two or more images which may have been taken at different times for the same scene. Image stitching is the second major stage in this proposed system. It consists of several steps which are explained below:

Cylindrical Warping
Image warping is the process that prepares the image to more forth processing which increases the alignment among matching key-points. It applies a finer-transform that maps image points from one position to corresponding coordinates in a reformatted image 20 . This transformation gives outcomes and thus, is increasing alignment in the overlapping region.
To construct the image under the effect of the cylindrical warping, there are many equations that are developed as in the Equations (7) and (8). It maps each point in the perspective image p = (x, y, f) to 2D cylindrical screen coordinates (θ, v).
Where (θ) is the panning angle and (v) is the scanline. The main benefit of cylindrical projection is that the resulting image can be expanded into a rectangular 2D image. So, it is easier to collect image information than other projection.

Image Registration
Image registration is a local alignment process for every two images in a particular scene. This requires searching for accurate point-to-point correspondence between these two images and finding a match. This corresponding may be a simple relationship and in some cases, may extend to complicated operations 11 . This study will rely on featurebased registration, which are the edges and the corner used to estimate the image-to-image mapping. If the registration is not correct, the resulting panorama composition will not be accurate. This step of processing the images is done through the following sub-steps which are executed in a loop: feature extraction and description, feature matching, mismatch removal, and transformation estimation.
The extraction and matching of features have been mentioned earlier in the ordering stage and here, the same techniques will be used to detect key-point from warped images. After matching the features, the array of distances is arranged to ascend and the two lowest values that represent the closest match are selected. It is called local optimal. Mismatch removal included as a verification step of matching. As a result of the existence of some pairs of features which do not correspond in the same specific scene? These erroneous match points must be eliminated because it absolutely does not represent the matching 21 . These points are called "the exterior point" and have the inferior effect when calculating the model parameters. It is excluded as described in Equations (9) The last step is the transformation estimation using RANSAC. It is shifting according to x and y coordinates which are calculated among every matching points. They form a matrix of displacements that each value in them represents the amount of transition between the previous and subsequent image which can be adapted to align them. In the beginning, to estimate the model parameters, the RANSAC algorithm selects two pairs of points from the matching matrix. It calculates the distance between them to find "shift" value, as in Equation (11).
Where p is a point in the reference image and p' is a point in the sensed image. This value is compared with all other differences, as in Equation (12), to select the best shift later.
If the distance between each point and the model parameter that was created from two pairs of points below the predefined threshold (thr dis ransac _ = 3) is classified as an inlier, as in Equation (13), and if the distances are greater than the threshold, the values of the points are classified as an outlier.
A set of points (only the lowest of thr dis ransac _ ) are counted to represent the maximum number of inliers. The value of max_inliner for each iteration is compared with each other to select the (shift) value which is related to that iteration.

Blending
Image blending is very important for image stitching with visible seams. The blending is used to stitch the image to handle the overlapping regions. In addition, image stitching is a blending of interlocking areas between images to create an attractive panorama. An alpha blending technique is used 22 . It is also called feathers blending. The original algorithm takes the average of pixels in the overlapping regions, as both of the overlapping regions are distributed equally between the two stitched images. In this work, the different ratios have been taken from each image in every stitching step. Here depending on the location of pixels and the nearest from the line of concatenation. The values of pixels in the overlapping region are calculated as in Equations (14) and (15). respectively, and w2 is alpha blending window, which selected equal 5.

Global Alignment and Cropping
Panoramic images consist mostly of aligning and blending a wide range of images. Adjusting local parameters between each pair of images is not enough to stitch all pairs of images. Global alignment is the most appropriate choice by using the bundle adjustment 11 . Global parameters are set for the purpose of providing a panoramic image with the least amount of registration error. The composition of wide range panorama requires selecting an image such as first, as a referenced image and then transforming the other sensed image on it. So, this process is repeated with the rest of the images to form a panorama. The amount of the shift calculated using the RANSAC is used to determine the magnitude of the increase in the width of the panoramic image at each step. After that is the implementation of the cropping process that involves the removal of unwanted black edges. This is done by looking into the image pixels on black values which are located on the top, bottom, left and right. It must be at the lowest, highest, leftmost, and rightmost of these locations respectively. From these four points, the rectangle that contains more important information about the image is formed.

Results and Experiment
The current study is concerned with creating a 360-degree panoramic image. The experiment is executed on a dataset of images taken from the internet, which have an appropriate overlapping between them. These images are of the following sizes: 384 × 512. The implementation of the algorithms and techniques used in this work is done by using the Python 3.6 development environment. The setting of the system is windows 7. The processor manufacturer is Intel with Core i7 Pentium fourth generation running at 2.1GHz. The capacity of RAM is 8 GB.
The number of features detected plays an important role in image ordering, stitching and the completion of precise image panorama. Perhaps, the greater the number of features detected leads to increasing matches between images, thereby providing a sufficient number of features that are finally used to calculate the transformation matrix. The threshold used in Equations (9) and (10) are the most important threshold values used in this work for its great impact on the number and accuracy of the resulting matches as shown in Table (1

Conclusion
The arrangement and stitching of images not only depends on the number of the extracted features, but the features must be accurately expressed in the image content. This means the representation of the contents of the image accurately.
The use of the RANSAC algorithm with the verification equations to prove the nearness of the corresponding points of the intended feature gave good results in excluding the points representing a false match, and thus providing a sufficient and accurate number for the process of constructing the shifting matrix which contain the best values used in the process of stitching an image with another image.