1. Class Activation Mapping (CAM)

In Convolution Neural Network (CNN) classification model, the moment the last convolutional layer flattens the output and passes it to the fully connected layer, the information held by the filter disappears. However, the last filter of the CNN can be preserved by using Global Average Pooling (GAP) instead of flattening. On the CAM model, like a normal CNN architect, class prediction is made from softmax output through a fully connected layer with weights from the last convolution and fully connected layers. Weights from the fully connected layer become the feature weight of each class, and each weight has a feature vector from each class. As a result, once the feature map from the last convolution layer and weights from the fully connected layer are multiplied, the image made from the Class Activation Mapping (CAM) model shows which area of the image helps to classify with a given class. The expression of the area of class objects through the preserved filter is regarded as a defect. CAM can extract defects through unsupervised learning if sufficient accuracy of the CNN classification model is secured.

Since CNN model is known as the most powerful tool for an image classification, we consider CAM model to be both effective defect classifier and segmentation extractor.

2. U-Net

U-Net is End-to-End, fully convolutional network-based model. As shown in Fig. 4, it was named U-Net because of the shape of the network (‘U’). It has shown good performance in the field of tumor detection with a small amount of training data in medical electron microscopy images. Moreover, the segmentation image made by this network also has the advantage of being sophisticated. To collect overall context information of the image and exact localization, the architect of U-Net is symmetrical. On the left side of the U-Net architect shown in Fig. 4, the contracting path is to downsample the given image by using a normal CNN layer. With a contracting path, image context can be extracted. On the right side of the U-Net architect, expanding path is to localize the given context.

As cell image shown at Fig. 4, medical electron microscopy image looks similar with wafer image we’re dealing with. Moreover, as suggested on U-Net paper ^[1] shown 30 images with excessive augmentation successfully trained U-Net model which is similar to our circumstance, we consider U-Net to be suitable for semiconductor field. However, to train U-Net model, only supervised learning is possible with a label mask produced.

Fig. 4. U-Net Architect [1].

1. Data Collection and Preprocessing

For data collection, all verified defect lot lists were collected at the SiN/SiO$_{2}$ film deposition inspection step performed on the SK hynix WLPKG TSV Line, from the defect list, true negative defect images were selected by the process engineer in charge. After defect images were selected, we transformed imaged to be grayscale and resized image to 256x256x1. To train the CNN defect classification model for CAM, we augmented total of 22 defect images to 472 images in order for the CNN model to train various types of defect modes and to solve the imbalance problem of the dataset. Since augmented images were made by flipping and shifting, the wafer lookalike images were produced with the region of the defect itself was changed. Based on the domain knowledge of the process engineer in charge of the collected defect image and the defective limit sample, ground truth segmentation was performed, as shown in Fig. 5, after the defective label to create a segmentation mask for the defective image.

Fig. 5. Defective Image Preprocessing for U-Net Example.

2. Model Fitting

Prior to performing CAM, it is necessary to secure a high-performance CNN model. CAM was implemented with ResNet architect, which has the highest performance record among existing image classification models. Due to the nature of P&T, defect image is very rare. In order to solve data imbalance between normal/defect images, as shown on Table 1, defect images were augmented from 4 images to 328 images for training model, from 8 images to 64 images for validating model, and from 10 images to 80 images for test model to train CNN model.

And for U-Net model, since we had to train only segmentation mask for defect region, we augmented 58 defect and segmentation mask images to 6000 images by using Keras.ImageDataGenerator with argument as following: 0.2 rotation, 0.05 width & height shift, 0.05 shear, 0.05 zoom range, horizontal & vertical flip and fill mode so that location of defect-like image on wafer varied as shown on Fig. 6.

After augmented defect images, we trained the CNN model with a train and valid data set with 1,000 epochs. Since we lacked defect images, we used pretrained ResNet-50 with binary classification instead of multi-class classification for CNN model and we could get 96.8% for test accuracy. After training, we changed the architect of the CNN model by changing the flattened layer to a global average pooling layer which helped us to get a segmentation image of the defect.

For U-Net training, we used same network structure from U-Net paper shown on Fig. 4 but changed size of first layer of network since we used 256x256x1 image. Also, we only used defect images with segment masks for training. While training, we used binary cross entropy as a loss function and set 10,000 epochs. As we trained the model, we found although loss from binary cross entropy was well saturated with the Mean IoU value grown, after a certain point, loss stopped falling, and the Mean IoU value stopped increasing, which we found our model was well saturated as shown in Fig. 7.

While training the U-Net model, we checked if the model was well trained by making a segmentation image from the model we trained. With certain points with binary cross entropy and mean IoU were not fallen, although early stopping paused training, we found the model had to be trained more because segmentation images were not clear before saturation, as shown in Fig. 8.

For segmentation image made by CAM model, to compare with segmentation image with U-Net model properly, we converted image to grayscale image.

After both CAM model and U-Net model were trained, we could get segmentation image of defect areas.

Fig. 6. Augmented Image Example.

Fig. 7. Training U-Net Model.