Deep learning framework for precision grading and non-invasive Apple sweetness evaluation

Figure 1 is a diagram showing the multispectral imaging systems being put together. A light source that can emit spectral wavelengths of possible broad range, traversing visible and non-visible spectra, is selected. This light passes through a multispectral filter disk, which takes out specific wavelengths for observation. An apple sample is exposed to each of these sequential wavelengths, and the image is captured using a dedicated camera; the disk turns around in different spectral bands for comprehensive data collection to be conducted. The camera contains no infrared filter so that it can take various wavelengths. The captured images are processed using a convolutional neural network (CNN) algorithm called AppleNet to judge the fruit by its multispectral properties.

Figure 1:

Setup of multispectral imaging.

CNNs consist of multiple layers interconnected and designed to process input data hierarchically, extracting features and patterns relevant to specific tasks. AppleNet comprises a series of layers, each fulfilling a different role in turning raw image data into meaningful output values. The layers are carefully structured so the network can effectively categorize and analyze multispectral images. Detailed descriptions of each of the layers in AppleNet's architecture are given below:

The architecture begins with the image input layer, which is designed to accept 128 × 128 × 3-pixel images. While multiband images typically have more than three bands, the network treats them like a common RGB dataset with modifications to make these spectral band images compatible with current CNN frameworks. This is done to allow them both to remain within the familiar environment of existing CNN frameworks and simultaneously achieve an integration of multispectral data.

The convolutional layer is the backbone of AppleNet. Each convolutional layer performs a convolution operation on the input image data, which will be presented as a 3D array of numbers. Filters or kernels are used in this layer to extract essential features from the input data, such as edges, textures, and patterns. In AppleNet, each convolution operation uses a 5 × 5 filter that slides across the image, producing feature maps. These spatial hierarchies are captured in the feature maps and are crucial to understanding complex patterns in multispectral images. The convolutional layer also has padding and stride settings to optimize the extraction of features while preserving image resolution. Keep as much relevant spatial information as possible, and do not complicate things more than necessary. With several convolutional layers stacked on top of one another, AppleNet gradually learns more abstract and complex representations of the input image data.

Following each convolutional layer, there is a rectified linear unit (ReLU) activation function. The ReLU layer operates on each element of the stacked feature maps, and any negative value is set to zero. This operation introduces nonlinearity into network models, allowing for more complex relationships with input data and output predictions. The rectified feature map generated by the ReLU layer are made available to the next layer, enhancing the network's ability to learn hierarchical features. This step is key in mining subtle differences, sometimes seen as variations in quality or sweetness for multispectral scan data like fruit samples.

Their spatial dimensions are reduced with pooling layers to down-sample the feature maps. This means that the computational requirements are further reduced. Also, it helps to stop overtraining by taking out extra information that is not included (in 1D format), which smooths out the maps into a steady data stream.

The image dimensionality is reduced by selecting the Most Valuable Element of each area filter captured in different Feature maps [17]. Compared with the last convolutional layer, this process further reduces pixels in an output.

The max pooling layer's output is established by connecting it to a fully connected layer consisting of three output layers for image classification and recognition. The fully connected layer connects all neurons in the previous layer. Technically termed a feature vector, the result predicts the output—the AppleNet inputs data to pass through a transformation such as the max-pooling layer. The classes are set up in this place to learn the relationship between extracted features and target labels. The relations that are established by this layer can be understood. The output of the fully connected layer is further connected to multiple output layers. These are arranged in a way that is designed explicitly to distinguish any given image into one of the four Apple classes.

The Softmax layer is an integral part of the classification process in AppleNet. The output from the fully connected layer enters this layer. It converts those results into probabilities for each class, ensuring that the probabilities sum up to 1.0 [19]. By so doing, it offers a simple question: Which class should be chosen most probably when seeing a particular picture taken before? Another helpful function is that, in training, applying the probability viewpoint to learning speeds up convergence rates.

Features are passed to the final classification layer after a series of processes and calculations. According to the probabilities computed at the Softmax layer for each type, the layer identifies the input image belonging to one of the four output classes—Class 10, Class 12, Class 13, and Class 15. It is the final stage of the neural network's classification process [20].

AppleNet is specifically made to encode the unique challenges of multispectral imaging data. Its hierarchical structure allows hierarchical sharing between the complex spatial and spectral relations, sometimes decisive for apple sweetness grading. Furthermore, nothing is left to chance by using dedicated layers like ReLU, max pooling, or Softmax to ensure robust feature extraction and classification in the face of high-dimensional input data.

Besides, AppleNet's ability to adapt to different spectral features makes it a versatile tool for tomorrow's fruit grading applications. And its seamless integration with existing pipelines of software and hardware makes it equally well-suited for real-world uses. AppleNet is a CNN network used to grade apples by sweetness with the help of multispectral imaging. The AppleNet analyzes the multispectral data of the apples by combining convolutional layers, pooling operations, and advanced activation functions. It also represents AppleNet's overall coherence as a network. By its robust architecture and capable classification, AppleNet promises to be a critical tool for automatic fruit grading systems: its advent makes way for less demanding, more accurate methods.

For imaging process, the following steps are used.

• In this step, please add a plate or internet sign and place the apple onto it.

• Make sure the dome has an LED ring at the top and is fixed with a camera in its center.

• Make sure that the apple is in the right place and is not moving anywhere while doing any work, even when capturing it. Also, the chamber must be tightened to keep out all air and light.

• Connect USB cables from the camera to the computer—one for the camera and one for your lighting system.

• The first thing to do is turn on the LED ring light to illuminate the chamber.

• Use the camera application on the computer; invoke it by pressing the Windows button. You'll notice it is all black because of the darkness. The chamber overpowers any light source your camera does have with its complete darkness.

• Capture the initial black image as part of your imaging sequence.

• Turn on the white light to view the picture inside the chamber. Snap your first photograph here.

• Press the RGB button to switch the lighting color and capture another image.

• Continue for eight different colors, letting each color settle before taking a shot

• For automatic mode, follow the steps above; for manual mode, turn off power between each image capture.

• Three positions will be used for photographing the apple: horizontal, vertical, and at rest on a table.

• Do the whole thing three times to allow for noise and anomalies.

• Check to ensure that 27 images of the apples (3 times 9 photos each, including 1 in black and 8 in color) are all there in their designated folders.

This will present a complete dataset with minimal variation amongst photographs for verification purposes.

To provide a merged picture of the apple fruit in all the spectral bands, follow the following steps in MATLAB:

• Go into MATLAB and create a Merge folder to store all nine images sequentially.

• Next select all images by pressing Ctrl + A, then right-click on the first one and rename it a(1). The other eight are named a(2), a(3), a(4), a(5), a(6), a(7), a(8) and a(9). This is done for ease of use as well.

• The images a(1) to a(9) should be read in MATLAB and assigned to variables i through i8 using the accompanying MATLAB code, making sure that the correct path is specified for the “merge” directory.

• After that, combine all of them from i to i8 into the variable i9, as shown by the MATLAB code.

• Then display that “concatenated” image i9 by calling MATLAB's show command.

• This process suffices for merging those photographs in MATLAB and the final merged photo can be seen in its entirety.

Figure 6A–C depict concatenated images of Red Delicious USA, Royal Gala, and Washington Apple, respectively.

Figure 6:

(A) Red Delicious USA, (B) Royal Gala, and (C) Washington.

The dataset is prepared by focusing on Apple fruit. Eight wavelengths are chosen, with six images at each wavelength, including a black reference image across this range. So, the data set consists of 81 photos of each type of fruit at one moment and these eight different wavelengths. A Refractometer is used, which will manually record data on each apple's sweetness in % Brix. Fruits shall be sorted by solid sugar content and other approved methods. The apple types selected for solid sugar content-based grading included the following five varieties: Red Delicious USA, Red Delicious New Zealand, Royal Gala, Washington, and Kinnaur. After repeating the above procedure for all five types of apples, the Lookup table for sugar content in apples is obtained, as shown in Table 1.

• For Grading by Sweetness: 1,539 images

The proposed work employed data augmentation techniques to enhance the proposed dataset further and improve model robustness. This is a standard practice in deep learning to expand the training dataset artificially, expose the model to various image variations, and improve generalization.

While the raw number of capture sessions may appear low, combining multiple images per capture and data augmentation techniques allows us to meet the data requirements for practical deep-learning model training and testing. This approach balances the practical constraints of data collection with the need for a comprehensive dataset, ensuring that proposed models are trained on a diverse and substantial set of images.

Convolutional Layer: The core operation in the convolutional layer is described by the convolution formula: f(i,j)=(I*K)(i,j)=Σ_mΣ_nI(i-m,j-n)K(m,n) {\rm{f}}({\rm{i}},\,{\rm{j}}) = ({\rm{I}}*{\rm{K}})\,({\rm{i}},\,{\rm{j}}) = \Sigma \_{\rm{m}}\,\Sigma \_{\rm{n}}\,\,{\rm{I}}({\rm{i}} - {\rm{m}},\,{\rm{j}} - {\rm{n}}){\rm{K}}({\rm{m}},\,{\rm{n}})

Where:

• I - then input image (128 × 128 × 3)

• K - the kernel (5 × 5)

• f (i, j) - the output feature map

The output size of the convolutional layer is given by: output_height=(input_height-kernel_height+2*padding)/stride+1output_width=(input_width-kernel_width+2*padding)/stride+1 \matrix{{{\rm{output}}\_{\rm{height}} = ({\rm{input}}\_{\rm{height}} - {\rm{kernel}}\_{\rm{height}} + 2*{\rm{padding}})/{\rm{stride}} + 1} \hfill \cr {{\rm{output}}\_{\rm{width}} = ({\rm{input}}\_{\rm{width}} - {\rm{kernel}}\_{\rm{width}} + 2*{\rm{padding}})/{\rm{stride}} + 1} \hfill \cr}

ReLU Activation Function: Applied element-wise to the output of the convolutional layer: ReLU(x)=max(0,x) {\rm{ReLU}}({\rm{x}}) = {\rm{max}}(0,{\rm{x}})

Max Pooling Layer: The max pooling operation is defined as: MaxPool(X)=max(X_i,j)for(i,j)inthelocalregion {\rm{MaxPool}}({\rm{X}}) = {\rm{max}}({\rm{X}}\_{\rm{i}},\,{\rm{j}})\,{\rm{for}}\,({\rm{i}},{\rm{j}})\,{\rm{in}}\,{\rm{the}}\,{\rm{local}}\,{\rm{region}}

The output size after max pooling is: output_size=(input_size-pool_size)/stride+1 {\rm{output}}\_{\rm{size}} = ({\rm{input}}\_{\rm{size}}\, - \,{\rm{pool}}\_{\rm{size}})/{\rm{stride}} + 1

Fully Connected Layer: The operation in the fully connected layer is given by: y=σ(Wx+b) {\rm{y}} = \sigma ({\rm{Wx}} + {\rm{b}})

Where:

• W - the weight matrix

• x - the input vector

• b - the bias vector

• σ - the activation function (ReLU in this case)

Softmax Layer: The softmax function for the final classification is: softmax(z_i)=exp(z_i)/Σ_jexp(z_j) {\rm{softmax}}({\rm{z}}\_{\rm{i}}) = {\rm{exp}}({\rm{z}}\_{\rm{i}})/\Sigma \_{\rm{j}}\,{\rm{exp}}({\rm{z}}\_{\rm{j}})

Input Layer: 128 × 128 × 3 (RGB image)

Convolutional Layer:

Number of filters: Typically, 32 or 64

Kernel size: 5 × 5

Stride: 1

Padding: “same” to maintain spatial dimensions

Max Pooling Layer:

Pool size: 4 × 4

Stride: 4

Fully Connected Layer:

Number of neurons: Typically, 128 or 256

Output Layer:

Four neurons (for the four apple classes)

Learning Rate: Typically start with 0.001 and implement learning rate decay

Batch Size: Usually 32 or 64, depending on memory constraints

Number of Epochs: Start with 100, implement early stopping

Optimizer: Adam optimizer with β1 = 0.9, β2 = 0.999, ɛ = 1e-8

Regularization: L2 regularization with λ = 0.0001

Dropout Rate: Typically, 0.5 for fully connected layers

Additional Convolutional Layers: Add 1–2 more convolutional layers to capture hierarchical features. Each additional layer would follow the same mathematical formulas as the first convolutional layer.

Batch Normalization: Apply batch normalization after convolutional layers: BN(x)=γ*(x-μ_B)/sqrt(σ_B^2+ε)+β {\rm{BN}}({\rm{x}}) = \gamma *({\rm{x}} - \mu \_{\rm{B}})/{\rm{sqrt}}\,(\sigma \_{\rm{B}}\wedge 2 + \varepsilon) + \beta

Where μ_B and σ B^2 are the mean and variance of the mini-batch, respectively, and γ and β are learnable parameters.

Data Augmentation: Implement transformations like rotation, flipping, and zooming. This doesn't change the network architecture but expands the training data.

Residual Connections: To practice skip connections: H(x)=F(x)+x {\rm{H}}({\rm{x}}) = {\rm{F}}({\rm{x}}) + {\rm{x}} where F(x) is the output of a stack of layers and x is the identity function.

More profound Architecture: Increase depth while managing vanishing gradients using techniques like residual connections.

Transfer Learning: Utilize pre-trained models as feature extractors, fine-tuning the last few layers for apple classification.

Each parameter and architectural decision in AppleNet resulted from extensive experimentation and iterative refinement. Our goal was to maximize accuracy specifically for Apple grading tasks.

While AppleNet doesn't introduce fundamental modifications to the CNN structure, its value lies in its specialized configuration for Apple quality assessment, which may not be directly evident from the network architecture alone.

The main contribution of AppleNet is not in proposing a new neural network type but in demonstrating how careful optimization of a CNN can yield superior results for a specific agricultural application.

The comparison now includes detailed performance metrics like precision, recall, and f1-score, as shown in Table 3.

The table above (Table 4) compares results from the proposed system with recent research conducted by various researchers.