2023: The best AI papers - A Review 🚀

A curated list of the latest breakthroughs in AI by release date with a clear video explanation, link to a more in-depth article, and code.

With the creation of a whole new field called "Generative AI", whether you like the term or not, research hasn't slowed its frenetic pace, especially the industry, which has seen its biggest boom in implementation of AI technologies ever. Artificial intelligence and our understanding of the human brain and its link to AI are constantly evolving, showing promising applications improving our life's quality in the near future. Still, we ought to be careful with which technology we choose to apply.

"Science cannot tell us what we ought to do, only what we can do."<br/>- Jean-Paul Sartre, Being and Nothingness

Here's curated list of the latest breakthroughs in AI and Data Science by release date with a clear video explanation, link to a more in-depth article, and code (if applicable). Enjoy the read!

The complete reference to each paper is listed at the end of this repository. Star this repository to stay up to date and stay tuned for next year! ⭐️

Maintainer: louisfb01, also active on YouTube and as a Podcaster if you want to see/hear more about AI!

Subscribe to my newsletter - The latest updates in AI explained every week.

Feel free to message me any interesting paper I may have missed to add to this repository.

Tag me on Twitter @Whats_AI or LinkedIn @Louis (What's AI) Bouchard if you share the list! And come chat with us in our Learn AI Together Discord community!

👀 If you'd like to support my work, you can check to Sponsor this repository or support me on Patreon.

Watch a complete 2023 rewind in 13 minutes

<img src="https://imgur.com/U69O8qQ.png" width="512"/>

---

The Full List

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers [1]
• InstructPix2Pix: Learning to Follow Image Editing Instructions [2]
• MusicLM: Generating Music From Text [3]
• Structure and Content-Guided Video Synthesis with Diffusion Models [4]
• PaLM-E: An Embodied Multimodal Language Model [5]
• Segment Anything [6]
• Key-Locked Rank One Editing for Text-to-Image Personalization [7]
• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold [8]
• Neuralangelo: High-Fidelity Neural Surface Reconstruction [9]
• TryOnDiffusion: A Tale of Two UNets [10]
• StyleGANEX: StyleGAN-Based Manipulation Beyond Cropped Aligned Faces [11]
• 3D-LLM: Injecting the 3D World into Large Language Models [12]
• METAGPT: META PROGRAMMING FOR MULTI-AGENT COLLABORATIVE FRAMEWORK [13]
• Visual Instruction Tuning [14]
• MVDream: Multi-view Diffusion for 3D Generation [15]
• Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling [16]
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets[17]
• Paper references

---

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers [1]<a name="1"></a>

Last year we saw the uprising of generative AI for both images and text, most recently with ChatGPT. Now, within the first week of 2023, researchers have already created a new system for audio data called VALL-E.

VALL-E is able to imitate someone’s voice with only a 3-second recording with higher similarity and speech naturalness than ever before. ChatGPT is able to imitate a human writer; VALL-E does the same for voice.

• Short Video Explanation:<br/> <img src="https://imgur.com/aEvCsR0.png" width="512"/>
• Short read: VALL-E: An AI Generating Voice from Text!
• Paper: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Code

InstructPix2Pix: Learning to Follow Image Editing Instructions [2]<a name="2"></a>

We know that AI can generate images; now, let’s edit them!

This new model called InstructPix2Pix does precisely that; it edits an image following a text-based instruction given by the user. Just look at those amazing results… and that is not from OpenAI or google with an infinite budget.

It is a recent publication from Tim Brooks and collaborators at the University of California, including prof. Alexei A. Efros, a well-known figure in the computer vision industry. As you can see, the results are just incredible.

• Short Video Explanation:<br/> <img src="https://imgur.com/1nUPDhb.png" width="512"/>
• Short read: Image Editing from Text Instructions: InstructPix2Pix
• Paper: InstructPix2Pix: Learning to Follow Image Editing Instructions
• Code
• Demo

MusicLM: Generating Music From Text [3]<a name="3"></a>

We recently covered a model able to imitate someone’s voice called VALL-E. Let’s jump a step further in the creative direction with this new AI called MusicLM. MusicLM allows you to generate music from a text description.

Let's not wait any longer and dive right into the results... what you will hear will blow you away!

• Short Video Explanation:<br/> <img src="https://imgur.com/7Y1QEJJ.png" width="512"/>
• Short read: Generating music with AI!
• Paper: MusicLM: Generating Music From Text
• Listen to some results

Structure and Content-Guided Video Synthesis with Diffusion Models [4]<a name="4"></a>

Runway have created a system called GEN-1 that can take a video, and apply a completely different style to it in seconds. The model is a work in progress and has flaws, but still does a pretty cool style transfer from an image or text prompt into a video, something that would've been impossible a few years or even months ago. Even cooler is how it works...

• Short Video Explanation:<br/> <img src="https://imgur.com/yBCKVaa.png" width="512"/>
• Short read: Gen-1, the future of storytelling?
• Paper: Structure and Content-Guided Video Synthesis with Diffusion Models
• Try it

PaLM-E: An Embodied Multimodal Language Model [5]<a name="5"></a>

PaLM-E, Google’s most recent publication, is what they call an embodied multimodal language model. What does this mean? It means that it is a model that can understand various types of data, such as text and images from the ViT and PaLM models we mentioned, and is able to turn these insights into actions from a robotics hand!

• Short Video Explanation:<br/> <img src="https://imgur.com/YkDCiBE.png" width="512"/>
• Short read: Google’s New AI Robot Can See and Understands Language! (PaLM-E)
• Paper: PaLM-E: An Embodied Multimodal Language Model
• More results!

Segment Anything [6]<a name="6"></a>

Segmentation - it's like the photo world's equivalent of playing detective. This superpower allows you to identify anything and everything in an image, from objects to people, with pixel-perfect precision. It's a game-changer for all kinds of applications, like autonomous vehicles that need to know what's going on around them, whether it's a car or a pedestrian.

You also definitely know about prompting by now. But have you heard of promptable segmentation? It's the newest kid on the block, and it’s really cool. With this new trick up your sleeve, you can prompt your AI model to segment anything you want - and I mean anything! Thanks to Meta's incredible new SAM (Segment Anything Model), there's no limit to what you can do.

If you're curious about how promptable segmentation and the SAM model work their magic, then you won't want to miss out on my video. In it, you'll learn all about how this amazing new technology is changing the game when it comes to image segmentation. So sit back, relax, and let me take you on a journey into the world of promptable segmentation with SAM. Trust me, you won't regret it!

• Short Video Explanation:<br/> <img src="https://imgur.com/kHsMafX.png" width="512"/>
• Short read: Meta's new Segment Anything Model Explained
• Paper: Segment Anything
• Code

Key-Locked Rank One Editing for Text-to-Image Personalization [7]<a name="7"></a>

Imagine creating stunning Instagram images without leaving home or snapping photos! NVIDIA's new AI model, Perfusion, advances text-to-image generation with enhanced control and fidelity for concept-based visuals.

Perfusion is a significant improvement over existing AI techniques, overcoming limitations in generating images that remain faithful to the original content. This model can accurately create these "concepts" in a variety of new scenarios.

Perfusion builds on Stable Diffusion with additional mechanisms for locking onto and generating multiple "concepts" in new images simultaneously. This results in unbeatable quantitative and qualitative performance, opening exciting possibilities across diverse industries.

🚧 While not perfect, Perfusion is a significant step forward for text-to-image models. Challenges include maintaining an object's identity and some overgeneralization, as well as requiring a bit of prompt engineering work.

NVIDIA's Perfusion sets the stage for an exciting future of AI-generated images tailored to our desires.

• Short Video Explanation:<br/> <img src="https://imgur.com/ecWaTs4.png" width="512"/>
• Short read: Perfusion: Stable Diffusion but more Controllable!
• Paper: Key-Locked Rank One Editing for Text-to-Image Personalization
• Code (coming soon as per the authors)

Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold [8]<a name="8"></a>

Drag Your Gan prioritizes precise object dragging over image generation or text manipulation. The AI realistically adapts the entire image, modifying the object's position, pose, shape, expressions, and other frame elements.

🐶🌄 Edit expressions of dogs, make them sit, adjust human poses, or even alter landscapes seamlessly. Drag Your Gan offers an innovative and interactive way to experiment with image editing.

How does it work? Drag Your Gan leverages StyleGAN2, a state-of-the-art GAN architecture by NVIDIA. By operating in the feature space (latent code), the AI learns how to edit images properly through a series of steps and loss calculations.

Even though the results are fantastic, as you will see below, it's essential to note that Drag Your Gan has some limitations, including only being able to edit generated images for now. Images are part of the distribution. Other limitations are that the selection of points is based on pixel colors and contrast, so you cannot really drag anything. If you take a part of a red car and move it staying on the red car, it might not understand that you move it at all.

Can't wait to try it out? The authors mention that the code should be available in June. Tune in to the video (or article) to learn more about this new image manipulation style with DragYourGan!

• Short Video Explanation:<br/> <img src="https://imgur.com/QuULqbR.png" width="512"/>
• Short read: Image Manipulation with Your Mouse! Drag Your Gan Explained
• Paper: Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
• Code

Check out the What's AI podcast for more AI content in the form of interviews with experts in the field! An invited AI expert and I will cover specific topics, sub-fields, and roles related to AI to teach and share knowledge from the people who worked hard to gather it.

Neuralangelo: High-Fidelity Neural Surface Reconstruction [9]<a name="9"></a>

Neuralangelo is NVIDIA's latest breakthrough in image-to-3D AI. This new approach builds upon Instant NeRF, enhancing surface quality and providing highly realistic 3D scenes from simple images in just seconds.

Neuralangelo aims to overcome the limitations of its predecessor, Instant NeRF, such as the lack of detailed structures and a somewhat cartoonish appearance of the AI-generated 3D models.

The secret behind Neuralangelo's improvements lies in two key differences: using numerical gradients for computing higher-order derivatives, and adopting a coarse-to-fine optimization on the hash grids controlling levels of detail, which we dive into in the video.

This optimization process results in a smoother input for the 3D model reconstruction, allows more information to be blended, and creates a perfect balance between consistency and fine-grain details for a realistic outcome.

The quality of Neuralangelo's 3D models is truly astounding, but the AI does face challenges with highly reflective scenes. Nonetheless, its potential real-world applications are vast and exciting!

• Short Video Explanation:<br/> <img src="https://imgur.com/1QD1QUK.png" width="512"/>
• Short read: The Best Image-to-3D AI to date: Neuralangelo
• Paper: Neuralangelo: High-Fidelity Neural Surface Reconstruction
• More results!

TryOnDiffusion: A Tale of Two UNets [10]<a name="10"></a>

In this week's episode I decided to explore a new research called TryOnDiffusion, presented at the CVPR 2023 conference. This innovative approach represents a significant leap forward in realistic virtual try-on experiences. By training AI models to understand input images, differentiate clothing from the person, and combine information intelligently, TryOnDiffusion produces impressive results that bring us closer to the ultimate goal of a perfect virtual try-on.

If you're intrigued by the intersection of AI and fashion, join us as we unravel the inner workings of TryOnDiffusion and its potential impact on the future of online shopping. Whether you're an AI enthusiast, a fashion lover, or simply curious about the latest technological advancements, the video offers valuable insights into the cutting-edge world of virtual clothing try-on.

We will dive into the world of diffusion models, UNets, and attention, where all those incredibly powerful mechanisms combine forces with helping the field of fashion and online retail. Of course, this work has limitations, but (as you will see) the results are just mind-blowing and very promising.

• Short Video Explanation:<br/> [<img