The Mystery of AI’s Hand Problem:

A deep dive into the Challenges of models

Generative You’re asked to create a post-apocalyptic giraffe astronaut. Done. Genghis Khan playing a guitar solo in pixel art? Generated. But then you request a simple image: a man holding a delicious apple. Suddenly, the hands look… strange. Why is it that AI art models can create detailed images of Abraham Lincoln dressed like glam David Bowie, but struggle with something as basic as a woman holding a cell phone? This isn’t just a random glitch; the struggle with hands reveals something fundamental about how AI art works.

To get to the bottom of this, I spoke with an artist who has taught thousands of people how to draw hands from imagination. Before someone formally trains as an artist, they rely heavily on pattern recognition—just by living in the world, you see countless hands and start to understand what they look like. AI is similar but with some key differences. Imagine an AI as someone who’s been trapped in a museum from birth, only able to learn from pictures and short descriptions. While you can hold and examine an apple from every angle, the AI can only look for another picture of an apple in its “museum.”

When artists draw hands, they simplify them into basic shapes—like the palm being a blocky form—before adding style, texture, and details. AI, on the other hand, knows how things look, but not how they work. It can beautifully replicate textures and light but may make bizarre shapes because it doesn’t simplify forms the way humans do. Trapped in its museum, it just guesses where the “hand-like” pixels should go, without a true understanding of how hands function.

However, this isn’t the whole story. AI can create stunning images of skyscrapers despite not understanding construction. To dig deeper, I spoke with two experts who work with generative art models: Yilun Du, a grad student with a background in robotics, and Roy Shilkrot, who has been teaching generative art since 2018. They identified three key challenges: data size and quality, the complexity of hand movements, and the low margin for error.

Back to our museum analogy: The AI’s museum has many rooms dedicated to faces, but not so many for hands. The AI might have 70,000 face images to learn from, but only 11,000 hands, and those might not even be detailed enough to understand how hands work. Hands also move in countless ways, unlike faces that stay relatively static. A face is almost always centered in an image with the eyes in the same spot. Hands, however, can do many different things—grasp, point, wave—which complicates matters further.

Because hands do so much, AI struggles to get them right. The AI might generate hands that look like hands but are not exactly hands. This is why it’s more noticeable when something is off—like with the man holding an apple. While the AI can create a freckly face with no problem, the standards for hands are much higher.

You might think that AI art is inherently flawed and will never improve, but that’s not entirely true. Midjourney version 5, the latest AI art generator, has made progress with hands, but it’s still not perfect. Improving AI art involves looking at more images and refining the models, but this takes tremendous computing power. Another approach might be to bring more people into the “museum”—similar to how ChatGPT uses human feedback to improve. If people could rate the quality of AI-generated images, many of these issues might be resolved.

The challenges aren’t limited to hands. AI struggles with any pattern or object that involves a specific quantity, like teeth or abs. It doesn’t understand the rule of “there are this many” because it’s trained on different amounts. But as AI continues to learn and evolve, it might get better at creating images that satisfy even our toughest standards.