Revolutionizing Creativity

The Journey of AI from Image Captioning to Art Generation.

Seven years ago, in 2015, a significant breakthrough in AI research was the development of automated image captioning. Machine learning algorithms had already been able to label objects in images, but they advanced to the point where they could now create natural language descriptions of those images. This progress led a group of researchers to wonder: What if they could reverse the process? If AI could translate images into text, could it also generate images from text?

This was a more challenging task. The researchers didn’t want to retrieve existing images like a Google search would. Instead, they aimed to create entirely new scenes that didn’t exist in the real world. For example, they asked their computer model to generate something it had never encountered before, like a red or green school bus. Despite the fact that all the school buses it had seen were yellow, the model managed to produce a 32 by 32 pixel image of a green school bus though it was just a vague blob of colors.

The researchers tested other prompts, such as “a herd of elephants flying in the blue skies,” “a vintage photo of a cat,” “a toilet seat sits open in the grass field,” and “a bowl of bananas is on the table.” While these early results were far from museum-worthy, the 2016 paper they published hinted at what might be possible in the future.

Fast forward to today, and the progress has been staggering. The technology has advanced at a pace that’s almost impossible to fully grasp. In just a few years, AI-generated images have gone from crude blobs to creations that can elicit awe and surprise.

You might recall the AI-generated portrait that sold for over $400,000 at auction in 2018, or the morphing portraits that Sotheby’s sold the following year. These early examples required artists like Mario Klingemann to painstakingly collect specific datasets of images and train their models to mimic that data. If Klingemann wanted to create landscapes, he had to gather landscape images; if he wanted portraits, he needed a dataset of portraits. Each model was specialized and couldn’t easily switch between creating different types of images.

The recent breakthroughs in AI art come from a new approach: instead of training models on narrow datasets, developers now use enormous models trained on vast and diverse datasets. These models are so large and complex that individuals like Klingemann can no longer train them on personal computers. But once these models are trained, they can generate almost anything, from simple landscapes to complex, surreal scenes, all from a single line of text.

This revolution in AI art began in January 2021 when OpenAI announced DALL-E, a model named after the artist Salvador Dalí and the character WALL-E. DALL-E could create images from text prompts across a wide range of concepts. More recently, OpenAI announced DALL-E 2, which promises even more realistic results and seamless editing capabilities. Although these models haven’t been released to the public, a community of independent developers has built text-to-image generators using other pre-trained models that are accessible online for free.

One of the most notable developments comes from a company called MidJourney, which created a Discord community where users can turn their text prompts into images in under a minute. This accessibility has opened the floodgates, with people spending hours creating and refining their prompts to produce thousands of images. The art of crafting these prompts, known as “prompt engineering,” has become a new skill, blending creativity with a deep understanding of how to communicate with AI.

The process of generating these images involves feeding the model a massive dataset, consisting of hundreds of millions of images scraped from the internet along with their text descriptions. These descriptions often come from alt text used on websites for accessibility and SEO purposes. But rather than simply copying pixels from the training data, the model operates within a “latent space”—a mathematical construct where it learns to generate entirely new images based on the patterns it has identified.

This latent space is vast and multi-dimensional, far beyond our ability to visualize, with over 500 dimensions representing different variables that the model uses to distinguish between objects. For instance, it might cluster all the features of “banana-ness” in one region and those of “snowglobe-ness” in another. When you give the model a text prompt, it navigates through this space to generate an image, which it then refines through a generative process called diffusion. This process begins with random noise and iteratively arranges the pixels into a coherent image that aligns with the prompt.

Interestingly, the model doesn’t just mimic specific images it can emulate an artist’s style simply by including their name in the prompt. For example, by mentioning Salvador Dalí in a prompt, the model can produce images that resemble his style without directly copying any of his works. This raises important ethical questions, particularly about the rights of artists whose styles are being replicated by these models.

James Gurney, an American illustrator, has become a popular reference in text-to image models. He believes it’s crucial for the public to know the prompts and software used to create these AI-generated works. Gurney also advocates for artists to have the option to opt in or out of having their work used in training datasets.

As the technology evolves, it will inevitably provoke more questions, especially around the biases inherent in the datasets used to train these models. The latent space of these models can harbor associations and patterns that reflect societal biases, such as stereotypes related to gender, race, and culture.

Despite these challenges, the potential of AI-generated art is undeniable. It removes the barriers between ideas and their visual representation, allowing anyone to create images, and eventually, videos and virtual worlds, with just a few words. This is more than just a technological shift it’s a fundamental change in how humans imagine, create, and interact with culture. The long-term consequences, both positive and negative, are still unfolding, and we are only beginning to understand the implications of this new era in art and technology.