Mid Sweden University

miun.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Enhancing Machine Failure Detection with Artificial Intelligence and sound Analysis
Mid Sweden University, Faculty of Science, Technology and Media, Department of Computer and Electrical Engineering (2023-).ORCID iD: 0000-0002-8262-2414
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

The detection of damage or abnormal behavior in machines is critical in industry, as it allows faulty components to be detected and repaired as early as possible, reducing downtime and minimizing operating and personnel costs. However, manual detection of machine fault sounds is economically inefficient and labor-intensive. While prior research has identified various methods to detect failures in drill machines using vibration or sound signals, there remain significant challenges. Most previous research in this field has used manual feature extraction and selection, which can be tedious and biased. Recent studies have used LSTM, end-to-end 1D CNN, and 2D CNN as classifiers, but these have limited accuracy for machine failure detection. Additionally, machine failure is rare in the data, and sounds in the real-world dataset have complex waveforms that are a combination of noise and sound.

To address these challenges, this thesis proposes modern artificial intelligence methods for the detection of drill failures using image representations of sound signals (Mel spectrograms and log-Mel spectrograms) and 2-D convolutional neural networks (2D-CNN) for feature extraction. The proposed models use conventional machine learning classifiers (KNN, SVM, and linear discriminant) or a recurrent neural network (long short-term memory) to classify three classes in the dataset (anomalous sounds, normal sounds, and irrelevant sounds). For using conventional machine learning methods as classifiers, pre-trained VGG19 is used to extract features, and neighborhood component analysis (NCA) is used for feature selection. For using LSTM, a small 2D-CNN is proposed to extract features, and an attention layer after LSTM focuses on the anomaly of the sound when the drill changes from normal to the broken state. The findings allow for better anomaly detection in drill machines and the development of a more cost-effective system that can be applied to a small dataset.

Additionally, I also present a case study that advocates for the use of deep learning-based machine failure detection systems. We focus on a small drill sound dataset from Valmet AB, a company that supplies equipment and processes for biofuel production. The dataset consists of 134 sounds that have been categorized as "Anomaly" and "Normal" recorded from a drilling machine. However, using deep learning models for detecting failure drills on such a small sound dataset is typically unsuccessful. To address this problem, we propose using a variational autoencoder (VAE) to augment the small dataset. We generated new sounds by synthesizing them from the original sounds in the dataset using the VAE. The augmented dataset was then pre-processed using a high-pass filter with a passband frequency of 1000 Hz and a low-pass filter with a passband frequency of 22,000 Hz, before being transformed into Mel spectrograms. We trained a pre-trained 2D-CNN Alexnet using these Mel spectrograms. We found that using the augmented dataset enhanced the classification results of the CNN model by 6.62% compared to using the original dataset (94.12% when trained on the augmented dataset versus 87.5% when trained on the original dataset). Our study demonstrates the effectiveness of using a VAE to augment a small sound dataset for training deep learning models for machine failure detection.

Background noise and acoustic noise in sounds can affect the accuracy of the classification system. To improve the sound classification application's accuracy, a sound separation method using short-time Fourier transform (STFT) frames with overlapped content is proposed. Unlike traditional STFT conversion, in which every sound is converted into one image, the signal is split into many STFT frames, improving the accuracy of model prediction by increasing the variability of the data. Images of these frames are separated into clean and noisy ones and subsequently fed into a pre-trained CNN for classification, making the classifier robust to noise. The efficiency of the proposed method is demonstrated using the FSDNoisy18k dataset, where 94.14 percent of 21 classes were classified successfully, including 20 classes of sound events and a noisy class.

Place, publisher, year, edition, pages
Sundsvall: Mid Sweden University , 2023. , p. 59
Series
Mid Sweden University doctoral thesis, ISSN 1652-893X ; 395
Keywords [en]
Machine Failure Detection, Machine Learning, Deep Learning, Sound Signal Processing, Audio Augmentation
National Category
Computer Sciences Signal Processing
Identifiers
URN: urn:nbn:se:miun:diva-49212ISBN: 978-91-89786-30-1 (print)OAI: oai:DiVA.org:miun-49212DiVA, id: diva2:1792962
Public defence
2023-09-29, C312, Holmgatan 10, Sundsvall, 09:00 (English)
Opponent
Supervisors
Available from: 2023-09-01 Created: 2023-08-30 Last updated: 2023-09-27Bibliographically approved
List of papers
1. Drill Fault Diagnosis Based on the Scalogram and Mel Spectrogram of Sound Signals Using Artificial Intelligence
Open this publication in new window or tab >>Drill Fault Diagnosis Based on the Scalogram and Mel Spectrogram of Sound Signals Using Artificial Intelligence
2020 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 8, p. 203655-203666Article in journal (Refereed) Published
Abstract [en]

In industry, the ability to detect damage or abnormal functioning in machinery is very important. However, manual detection of machine fault sound is economically inefficient and labor-intensive. Hence, automatic machine fault detection (MFD) plays an important role in reducing operating and personnel costs compared to manual machine fault detection. This research aims to develop a drill fault detection system using state-of-the-art artificial intelligence techniques. Many researchers have applied the traditional approach design for an MFD system, including handcrafted feature extraction of the raw sound signal, feature selection, and conventional classification. However, drill sound fault detection based on conventional machine learning methods using the raw sound signal in the time domain faces a number of challenges. For example, it can be difficult to extract and select good features to input in a classifier, and the accuracy of fault detection may not be sufficient to meet industrial requirements. Hence, we propose a method that uses deep learning architecture to extract rich features from the image representation of sound signals combined with machine learning classifiers to classify drill fault sounds of drilling machines. The proposed methods are trained and evaluated using the real sound dataset provided by the factory. The experiment results show a good classification accuracy of 80.25 percent when using Mel spectrogram and scalogram images. The results promise significant potential for using in the fault diagnosis support system based on the sounds of drilling machines.

Keywords
Deep learning, machine fault diagnosis, machine learning, sound signal processing
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:miun:diva-40655 (URN)10.1109/ACCESS.2020.3036769 (DOI)000590435900001 ()2-s2.0-85102866677 (Scopus ID)
Available from: 2020-12-03 Created: 2020-12-03 Last updated: 2023-08-31
2. A deep learning approach for detecting drill bit failures from a small sound dataset
Open this publication in new window or tab >>A deep learning approach for detecting drill bit failures from a small sound dataset
2022 (English)In: Scientific Reports, E-ISSN 2045-2322, Vol. 12, article id 9623Article in journal (Other (popular science, discussion, etc.)) Published
Abstract [en]

Monitoring the conditions of machines is vital in the manufacturing industry. Early detection of faulty components in machines for stopping and repairing the failed components can minimize the downtime of the machine. In this article, we present a method for detecting failures in drill machines using drill sounds in Valmet AB, a company in Sundsvall, Sweden that supplies equipment and processes for the production of pulp, paper, and biofuels. The drill dataset includes two classes: anomalous sounds and normal sounds. Detecting drill failure effectively remains a challenge due to the following reasons. The waveform of drill sound is complex and short for detection. Furthermore, in realistic soundscapes, both sounds and noise exist simultaneously. Besides, the balanced dataset is small to apply state-of-the-art deep learning techniques. Due to these aforementioned difficulties, sound augmentation methods were applied to increase the number of sounds in the dataset. In this study, a convolutional neural network (CNN) was combined with a long-short-term memory (LSTM)to extract features from log-Mel spectrograms and to learn global representations of two classes. A leaky rectified linear unit (Leaky ReLU) was utilized as the activation function for the proposed CNNinstead of the ReLU. Moreover, an attention mechanism was deployed at the frame level after theLSTM layer to pay attention to the anomaly in sounds. As a result, the proposed method reached an overall accuracy of 92.62% to classify two classes of machine sounds on Valmet’s dataset. In addition, an extensive experiment on another drilling dataset with short sounds yielded 97.47% accuracy. With multiple classes and long-duration sounds, an experiment utilizing the publicly availableUrbanSound8K dataset obtains 91.45%. Extensive experiments on our dataset as well as publicly available datasets confirm the efficacy and robustness of our proposed method. For reproducing and deploying the proposed system, an open-source repository is publicly available at https://github.com/thanhtran1965/DrillFailureDetection_SciRep2022.

Place, publisher, year, edition, pages
Nature Publishing Group, 2022
National Category
Computer and Information Sciences Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:miun:diva-45181 (URN)10.1038/s41598-022-13237-7 (DOI)000809441100062 ()35688892 (PubMedID)2-s2.0-85131903828 (Scopus ID)
Projects
AISound – Akustisk sensoruppsättning för AI-övervakningssystemMiLo — miljön i kontrolloopen
Available from: 2022-06-10 Created: 2022-06-10 Last updated: 2023-08-31Bibliographically approved
3. Separate Sound into STFT Frames to Eliminate Sound Noise Frames in Sound Classification
Open this publication in new window or tab >>Separate Sound into STFT Frames to Eliminate Sound Noise Frames in Sound Classification
Show others...
2021 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Sounds always contain acoustic noise and background noise that affects the accuracy of the sound classification system. Hence, suppression of noise in the sound can improve the robustness of the sound classification model. This paper investigated a sound separation technique that separates the input sound into many overlapped-content Short-Time Fourier Transform (STFT) frames. Our approach is different from the traditional STFT conversion method, which converts each sound into a single STFT image. Contradictory, separating the sound into many STFT frames improves model prediction accuracy by increasing variability in the data and therefore learning from that variability. These separated frames are saved as images and then labeled manually as clean and noisy frames which are then fed into transfer learning convolutional neural networks (CNNs) for the classification task. The pre-trained CNN architectures that learn from these frames become robust against the noise. The experimental results show that the proposed approach is robust against noise and achieves 94.14% in terms of classifying 21 classes including 20 classes of sound events and a noisy class. An open-source repository of the proposed method and results is available at https://github.com/nhattruongpham/soundSepsound.

Keywords
Sound separation, Sound classification, Short Time Fourier Transform, Transfer learning.
National Category
Computer Sciences Signal Processing
Identifiers
urn:nbn:se:miun:diva-43839 (URN)10.1109/SSCI50451.2021.9660125 (DOI)000824464300302 ()2-s2.0-85125765216 (Scopus ID)
Conference
IEEE Symposium Series on Computational Intelligence (IEEE SSCI 2021), Orlando, USA, [DIGITAL], December 5-7, 2021.
Available from: 2022-03-20 Created: 2021-11-24 Last updated: 2023-08-31Bibliographically approved
4. An artificial neural network-based system for detecting machine failures using a tiny sound dataset: A case study
Open this publication in new window or tab >>An artificial neural network-based system for detecting machine failures using a tiny sound dataset: A case study
2022 (English)In: Proceedings - 2022 IEEE International Symposium on Multimedia, ISM 2022, IEEE conference proceedings, 2022, p. 163-168Conference paper, Published paper (Refereed)
Abstract [en]

In an effort to advocate the research for a deep learning-based machine failure detection system, we present a case study of our proposed system based on a tiny sound dataset. Our case study investigates a variational autoencoder (VAE) for augmenting a small drill sound dataset from Valmet AB. A Valmet dataset contains 134 sounds that have been divided into two categories: "Anomaly"and "Normal"recorded from a drilling machine in Valmet AB, a company in Sundsvall, Sweden that supplies equipment and processes for the production of biofuels. Using deep learning models to detect failure drills on such a small sound dataset is typically unsuccessful. We employed a VAE to increase the number of sounds in the tiny dataset by synthesizing new sounds from original sounds. The augmented dataset was created by combining these synthesized sounds with the original sounds. We used a high-pass filter with a passband frequency of 1000 Hz and a low-pass filter with a passband frequency of 22 000 Hz to pre-process sounds in the augmented dataset before transforming them to Mel spectrograms. The pre-trained 2D-CNN Alexnet was then trained using these Mel spectrograms. When compared to using the original tiny sound dataset to train pre-trained Alexnet, using the augmented sound dataset enhanced the CNN model's classification results by 6.62%(94.12% when trained on the augmented dataset versus 87.5% when trained on the original dataset). For reproducing and deploying the proposed method, an open-source repository is available at https://gitfront.io/r/user-1913886/MKyfLWwTPm87/Paper5/ 

Place, publisher, year, edition, pages
IEEE conference proceedings, 2022
Keywords
Alexnet, audio augmentation, machine failure detection, variational autoencoder
National Category
Computer Sciences
Identifiers
urn:nbn:se:miun:diva-47576 (URN)10.1109/ISM55400.2022.00036 (DOI)000964457800030 ()2-s2.0-85147542959 (Scopus ID)9781665471725 (ISBN)
Conference
24th IEEE International Symposium on Multimedia, ISM 2022, 5 December 2022 through 7 December 2022
Available from: 2023-02-14 Created: 2023-02-14 Last updated: 2023-08-31Bibliographically approved
5. Denoising Induction Motor Sounds Using an Autoencoder
Open this publication in new window or tab >>Denoising Induction Motor Sounds Using an Autoencoder
2023 (English)In: 2023 IEEE Sensors Applications Symposium (SAS), IEEE, 2023, p. 01-06Conference paper, Published paper (Refereed)
Abstract [en]

Denoising sound is essential for improving signal quality in various applications such as speech processing, sound event classification, and machine failure detection systems. This paper proposes an autoencoder method to remove two types of noise, Gaussian white noise, and environmental noise from water flow, from induction motor sounds. The method is trained and evaluated on a dataset of 246 sounds from the Machinery Fault Database (MAFAULDA). The denoising effectiveness is measured using the mean square error (MSE), which indicates that both noise types can be significantly reduced with the proposed method. The MSE is below or equal to 0.15 for normal operation sounds and misalignment sounds. This improvement in signal quality can facilitate further processing, such as induction motor operation classification. Overall, this work presents a promising approach for denoising machine sounds using an autoencoder, with potential for application in other industrial settings.

Place, publisher, year, edition, pages
IEEE, 2023
Keywords
Autoencoder, Denoise sound
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:miun:diva-49392 (URN)10.1109/sas58821.2023.10254150 (DOI)001086399500082 ()2-s2.0-85174071313 (Scopus ID)
Conference
2023 IEEE Sensors Applications Symposium (IEEESAS), Ottawa, Canada, 18-20 July, 2023
Available from: 2023-09-27 Created: 2023-09-27 Last updated: 2023-11-10Bibliographically approved

Open Access in DiVA

fulltext(2754 kB)127 downloads
File information
File name FULLTEXT02.pdfFile size 2754 kBChecksum SHA-512
0afbdb0c8650f44c053305a8bc67a46f423be958918a7ac247a63fa872b75dc8dc829fcf178309c865e5a74ee89b569599d67d8e7db7a72b34d6264b3f473ccf
Type fulltextMimetype application/pdf

Authority records

Tran, Thanh

Search in DiVA

By author/editor
Tran, Thanh
By organisation
Department of Computer and Electrical Engineering (2023-)
Computer SciencesSignal Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 130 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 921 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf