DeepImage-an-Image-to-Image-technology

中文版 | English Version

This warehouse contains the pix2pixHD (proposed by Nvidia) algorithm, and more importantly, the universal image generation theory and practical research behind it.

This resource includes the TensorFlow2 (Pytorch | PaddlePaddle) implementation of image generation models such as pix2pix, CycleGAN, UGATIT, DCGAN, SinGAN, VAE, ALAE, mGANprior and StarGAN-v2, which can be used to systematically learn to Generating Adversarial Network (GAN).

---

Content of this resource

1. What is DeepImage?
2. Fake Image Generation and Image-to-Image Demo
3. DeepImage Algorithm: Normal to Pornography Image
4. NSFW: Pornography to Normal Image, Pornographic Image Detection
5. GAN Image Generation Theoretical Research
6. GAN Image Generation Practice Research
7. DeepImage to DeepFakes
8. Future

---

What is DeepImage?

DeepImage uses a slightly modified version of the pix2pixHD GAN architecture, quoted from DeepImage_official. pix2pixHD is a general-purpose Image2Image technology proposed by NVIDIA. Obviously, DeepImage is the wrong application of artificial intelligence technology, but it uses Image2Image technology for researchers and developers working in other fields such as fashion, film and visual effects.

---

Fake Image Generation Demo

This section provides a fake image generation demo that you can use as you wish. They are fake images generated by StyleGAN without any copyright issues. Note: Each time you refresh the page, a new fake image will be generated, pay attention to save!

• Click to generate fake person image
• Click to generate fake cat image
• Click to generate fake waifu image

Image-to-Image Demo

This section provides a demo of Image-to-Image Demo: Black and white stick figures to colorful faces, cats, shoes, handbags. DeepImage software mainly uses Image-to-Image technology, which theoretically converts the images you enter into any image you want. You can experience Image-to-Image technology in your browser by clicking Image-to-Image Demo below.

Try Image-to-faces Demo

Try Image-to-Image Demo

An example of using this demo is as follows：

In the left side box, draw a cat as you imagine, and then click the process button, you can output a model generated cat.

---

:underage: DeepImage Algorithm

DeepImage is a pornographic software that is forbidden by minors. If you are not interested in DeepImage itself, you can skip this section and see the general Image-to-Image theory and practice in the following chapters.

DeepImage_software_itself content:

1. Official DeepImage Algorithm(Based on Pytorch)
2. DeepImage software usage process and evaluation of advantages and disadvantages.

:+1: NSFW

Recognition and conversion of five types of images [porn, hentai, sexy, natural, drawings]. Correct application of image-to-image technology.

NSFW(Not Safe/Suitable For Work) is a large-scale image dataset containing five categories of images [porn, hentai, sexy, natural, drawings]. Here, CycleGAN is used to convert different types of images, such as porn->natural.

1. Click to try pornographic image detection Demo
2. Click Start NSFW Research

---

Image Generation Theoretical Research

This section describes DeepImage-related AI/Deep Learning theory (especially computer vision) research. If you like to read the paper and use the latest papers, enjoy it.

1. Click here to systematically understand GAN
2. Click here to systematically image-to-image-papers

1. Pix2Pix

• Berkeley 2017 paper Image-to-Image Translation with Conditional Adversarial Networks
• Pix2Pix homepage
• code pix2pix
• Run in Google Colab pix2pix.ipynb

Result

Image-to-Image Translation with Conditional Adversarial Networks is a general solution for the use of conditional confrontation networks as an image-to-image conversion problem proposed by the University of Berkeley.

<details> <summary>View more paper studies (Click the black arrow on the left to expand)</summary>

2. Pix2PixHD

DeepImage mainly uses this Image-to-Image(Pix2PixHD) technology.

• NVIDIA 2018 High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
• Pix2PixHD homepage
• code pix2pixHD

Result

Get high resolution images from the semantic map. The semantic graph is a color picture. The different color blocks on the map represent different kinds of objects, such as pedestrians, cars, traffic signs, buildings, and so on. Pix2PixHD takes a semantic map as input and produces a high-resolution, realistic image. Most of the previous techniques can only produce rough, low-resolution images that don't look real. This research has produced images with a resolution of 2k by 1k, which is very close to full HD photos.

3. CycleGAN

• Berkeley 2017 paper Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• CycleGAN homepage
• code CycleGAN
• Run in Google Colab cyclegan.ipynb

Result

CycleGAN uses a cycle consistency loss to enable training without the need for paired data. In other words, it can translate from one domain to another without a one-to-one mapping between the source and target domain. This opens up the possibility to do a lot of interesting tasks like photo-enhancement, image colorization, style transfer, etc. All you need is the source and the target dataset.

4. UGATIT

• NCSOFT 2019 paper Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• code UGATIT-TensorFlow | UGATIT-Pytorch
• UGATIT principle analysis
• UGATIT experimental recurrence

Result

UGATIT is a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. UGATIT can do both image conversions that require Holistic Changes, and image conversions that require Large Shape Changes. It can be seen as an enhanced version of CycleGAN, a more efficient general image conversion framework.

5. StyleGAN

• NVIDIA 2018 paper A Style-Based Generator Architecture for Generative Adversarial Networks
• code stylegan

Result

Source A + Source B (Style) = ?

StyleGAN can not only generate fake images source A and source B, but also combine the content of source A and source B from different strengths, as shown in the following table.

Style level (from source b)	Source A	Source B
High level (coarse)	all colors (eyes, hair, light) and details facial features from Source A	inherit advanced facial features from Source B, such as posture, general hair style, facial shape and glasses
Medium level	posture, general facial shape and glasses come from source a	inherits the middle level facial features of source B, such as hair style, open / closed eyes
High level (fine)	the main facial content comes from source a	inherits the advanced facial features of source B, such as color scheme and microstructure

StyleGAN2

Without increasing the amount of calculation of StyleGAN, while solving the image artifacts generated by StyleGAN and obtaining high-quality images with better details, StyleGAN2 implements a new SOTA for unconditional image modeling tasks.

• NVIDIA 2019 paper Analyzing and Improving the Image Quality of StyleGAN
• code stylegan2
• Run in Google Colab stylegan2

6. Image Inpainting

• NVIDIA 2018 paper Image Inpainting for Irregular Holes Using Partial Convolutions and Partial Convolution based Padding.
• Paper code partialconv。

Result

In the image interface of Image_Inpainting(NVIDIA_2018).mp4 video, you only need to use tools to simply smear the unwanted content in the image. Even if the shape is very irregular, NVIDIA's model can “restore” the image with very realistic The picture fills the smeared blank. It can be described as a one-click P picture, and "no ps traces." The study was based on a team from Nvidia's Guilin Liu et al. who published a deep learning method that could edit images or reconstruct corrupted images, even if the images were worn or lost pixels. This is the current 2018 state-of-the-art approach.

7. SinGAN

ICCV2019 Best paper - Marr prize

• Israel Institute of Technology 2019 paper SinGAN: Learning a Generative Model from a Single Natural Image
• SinGAN homepage
• code SinGAN-Pytorch
• Video Show SinGAN for single image animation
• SinGAN principle analysis

Result

We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.

8. ALAE

• 2020 paper Adversarial Latent Autoencoders
• code ALAE

Result

Although studied extensively, the issues of whether they have the same generative power of GANs, or learn disentangled representations, have not been fully addressed. We introduce an autoencoder that tackles these issues jointly, which we call Adversarial Latent Autoencoder (ALAE). It is a general architecture that can leverage recent improvements on GAN training procedures.

9. mGANprior

• 2020 paper Image Processing Using Multi-Code GAN Prior
• mGANprior homepage
• code mganprior

Result

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, applying trained GAN models to real image processing remains challenging. Previous methods typically invert a target image back to the latent space either by back-propagation or by learning an additional encoder. However, the reconstructions from both of the methods are far from ideal. In this work, we propose a novel approach, called mGANprior, to incorporate the well-trained GANs as effective prior to a variety of image processing tasks.

10. StarGAN v2

• Clova Research 2020 paper StarGAN v2: Diverse Image Synthesis for Multiple Domains
• Video
• code stargan-v2

Result

A good image-to-image translation model should learn a mapping between different visual domains while satisfying the following properties: 1) diversity of generated images and 2) scalability over multiple domains. Existing methods address either of the issues, having limited diversity or multiple models for all domains. We propose StarGAN v2, a single framework that tackles both and shows significantly improved results over the baselines.

11. DeepFaceDrawing

• Institute of Computing Technology, Chinese Academy of Sciences 2020 paper DeepFaceDrawing: Deep Generation of Face Images from Sketches
• DeepFaceDrawing homepage
• Video
• code DeepFaceDrawing-Jittor

Result

Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch.

</details>

---

Image Generation Practice Research

These models are based on the latest implementation of TensorFlow2.

This section explains DeepImage-related AI/Deep Learning (especially computer vision) code practices, and if you like to experiment, enjoy them.

1. Pix2Pix

Use the Pix2Pix model (Conditional Adversarial Networks) to implement black and white stick figures to color graphics, flat houses to stereoscopic houses and aerial maps to maps.

Click Start Experience 1

2. Pix2PixHD

Under development... First you can use the official implementation

3. CycleGAN

The CycleGAN neural network model is used to realize the four functions of photo style conversion, photo effect enhancement, landscape season change, and object conversion.

Click Start Experience 3

4. DCGAN

DCGAN is used to achieve random number to image generation tasks, such as face generation.

Click Start Experience 4

5. Variational Autoencoder (VAE)

VAE is used to achieve random number to image generation tasks, such as face generation.

Click Start Experience 5

6. Neural style transfer

Use VGG19 to achieve image style migration