Combining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor Detection

Changhee Han, Leonardo Rundo, Ryosuke Araki, Yudai Nagano, Yujiro Furukawa, Giancarlo Mauri, Hideki Nakayama and Hideaki Hayashi

Introduction

The use of Convolutional Neural Networks (CNNs) in medical image analysis requires large amounts of annotated training data, which is often difficult to obtain. To overcome this problem, researchers have explored Data Augmentation (DA) techniques, such as geometric or intensity transformations of original images. However, these transformed images have limited performance improvement since they intrinsically have a similar distribution to the original dataset. To address this issue, researchers propose using Generative Adversarial Networks (GANs) as a DA technique. GANs can synthesize new and diverse samples that fill the real image distribution uncovered by the original dataset. In this article, the authors propose a two-step GAN-based DA approach that generates and refines Brain Magnetic Resonance (MR) images with and without tumors separately. The first step uses Progressive Growing of GANs (PGGANs) to generate realistic/diverse 256×256 images. The second step uses Multimodal UNsupervised Image-to-image Translation (MUNIT) to refine the texture/shape of the PGGAN-generated images to fit them into the real image distribution. By combining noise-to-image and image-to-image GANs, the proposed approach significantly outperforms the classic DA alone in tumor detection and other medical imaging tasks. The authors thoroughly investigated CNN-based tumor classification results, also considering the influence of pre-training on ImageNet and discarding weird-looking GAN-generated images. The main contributions of this article are the use of whole image generation, a novel two-step GAN-based DA approach that combines noise-to-image and image-to-image GANs, and a misdiagnosis prevention strategy. The proposed approach can generate whole 256×256 images, instead of regions of interest, for more robust classification. Along with classic image transformations, the novel approach may become a clinical breakthrough.

GAN-based Data Augmentation for Tumor Detection

The scientific article discusses the use of a combination of Noise-to-Image (PGGAN) and Image-to-Image (MUNIT and SimGAN) Generative Adversarial Networks (GANs) to augment brain MR images for tumor detection. The authors use PGGAN to generate synthetic brain MR images, which are then refined using MUNIT or SimGAN to improve the realism and diversity of the synthetic images. The refined synthetic images are then used in data augmentation to improve tumor detection in ResNet-50, a 50-layer residual learning-based Convolutional Neural Network (CNN).

Evaluation of GAN-based Data Augmentation

The authors train the PGGAN, MUNIT, and SimGAN models and the ResNet-50 CNN on a dataset of brain MR images and evaluate the effectiveness of their augmentation method using a Visual Turing Test and t-SNE visualization. The results of the study show that the two-step GAN-based data augmentation method significantly improves tumor detection sensitivity compared to traditional data augmentation methods. The study also finds that the use of PGGAN generates realistic and diverse synthetic images, which can be effectively refined using MUNIT or SimGAN.

Conclusion and Future Work

Overall, the study demonstrates the potential of using GANs for medical image data augmentation and highlights the importance of combining different types of GANs for effective image refinement. Researchers from Korea University have developed a two-step GAN-based data augmentation (DA) technique to improve the sensitivity of tumor detection in brain MR images. The technique combines the use of noise-to-image and image-to-image GANs to generate synthetic images and refine them, respectively. The synthetic images generated by PGGANs exhibit a high degree of similarity to real MR images, including texture and tumor appearance. The refined images produced by MUNIT or SimGAN preserve the overall shape of the synthetic images, while enhancing texture and contours. The resulting images were evaluated by an expert physician, who found that the synthetic images successfully captured tumor/non-tumor features. The GAN-based DA technique was then compared with classic DA techniques, which geometrically transform images to cover global features. The GAN-generated images were found to non-linearly cover local tumor features with less tumor/non-tumor overlap when combined with classic DA techniques. MUNIT-based GAN-based DA technique was found to significantly improve sensitivity over the best-performing classic DA technique, while also reducing the risk of overlooking tumor diagnosis. While GAN-generated images may include odd artifacts, the researchers found that discarding them was helpful only without pre-training. The technique has potential to minimize annotation requirements and improve other medical imaging tasks, such as object detection and segmentation, as well as serve as a training tool for medical students and radiology trainees. The researchers plan to further optimize the GAN loss function to explicitly improve classification results.

Link to Article: Combining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor Detection | IEEE Journals & Magazine | IEEE Xplore

This article was summarized by an AI tool that uses natural language processing. The tool is not perfect and may make mistakes or produce inaccurate or irrelevant information, but is reviewed by the post’s author prior to publishing. If you want to learn more about the article, please refer to the original source that is cited at the end of the article.