AI Pet Magic: A Look Inside The DrawnBy AI Model

Welcome to this week’s Deep-fried Dive with Fry Guy! In these long-form articles, Fry Guy conducts an in-depth analysis of a cutting-edge AI development or developer. Today, our dive is about DrawnBy.ai, an AI tool which allows users to upload photos of their dogs and generate unique artistic renderings. We hope you enjoy!

*Notice: We do not gain any monetary compensation from the people and projects we feature in the Sunday Deep-fried Dives with Fry Guy. We explore these projects and developers solely for the purpose of revealing to you interesting and cutting-edge AI projects, developers, and uses.*

Do you wonder what your dog would look like as an astronaut, or as the captain of a ship? Maybe your dog dreams of taking laps around the moon or sailing the seven seas. Artificial intelligence (AI) is bringing you and your pet’s dreams to life.

DrawnBy.ai is a fun AI tool that allows users to upload photos of their pets and generate unique, professional artistic renderings within seconds.

THE BIRTH OF THE DRAWNBY-AI MODEL

Matt Lea and Dominic DiMarco are the developers behind this innovative AI model. They met about twelve years ago through various tech groups in Madison, Wisconsin, and began working on a variety of different projects together.

Matt Lea, a Web Application Architect, has been dabbling in AI integration for years now, and has recently developed an interest in Stable Diffusion (text-to-image models) and large language models (LLMs). His expertise is in hosting applications on Amazon Web Services (AWS) and he’s recently ventured out into hosting AI models on AWS as well. As Lea says, “I try to bridge the gap between those who have marketable models and are looking for ways to make it available to 10,000 or a million people.” He started his own company, Schematical Consulting, where he specializes in doing just this.

Dominic DiMarco is a software developer by trade, but he has spent the better part of the last 10+ years on the management side of technology. Recently, he had a desire to swing his career back towards tech, so he completed a data science bootcamp where he started to connect with Lea and align interests. As they found complementary strengths, they wanted to do something cool with AI in the form of a tool.

At the time, Lea was working on retooling his skillset to make sure he could service his machine learning and AI-based clients, and he and DiMarco figured there was something they could take a swing at. Lea began by developing a Stable Diffusion (image generation) model and inputting a variety of images from pictures of his dogs, mother, and himself to see what it could generate. He describes, “I wanted to be able to train these models and fine tune them so they could generate my face … so I could say, ‘I want a picture of me on the beach in watercolor’ and it could do that.” Lea was most impressed by what this revised Stable Diffusion model could do with photos of his dog. From here, Lea and DiMarco began tinkering with the model and building the web infrastructure and business plan to make this possible.

HOW DOES IT WORK?

DrawnBy.ai is designed in a way that can be used by people with no coding or technical experience. To begin, the user is asked to upload 10 pictures of their dog. This allows the model to learn the animal from different angles. In fact, it is encouraged the user uploads 15-20 pictures to get the most realistic, detail-specific image generation. The process is truly a matter of “quality in, quality out.” DiMarco describes, “You want to upload pictures that positively impact the output … it’s quality over quantity.” Lea adds, “If you only upload four images, and they are all very similar, you’re not going to get any creativity with the [outputted] poses opposed to if you input photos of higher quality and more variety.”

Image editing has been around for a while now, and AI has demonstrated the ability to edit images into other images. However, DrawnBy AI’s model allows for the generation of completely new artwork from the input of a number of images featuring the actual dog fed into the model. In this way, it is different from anything else available to the public.

By implementing You Only Look Once (YOLO) visual recognition, the model is able to detect and flag images that don’t fit the others or don’t resemble a dog, making the site more consistent and secure, also preventing it from spam. At this point, the images go into the queue and are used to generate a model which is saved on the cloud and can be reused to generate more photos on the fly.

The DrawnBy AI tool uses a base Stable Diffusion model that is trained with a variety of methods on hundreds of images of dogs to produce what are called intermediate models. DrawnBy AI then uses these intermediate models to train the model again on specific dogs to get the artistic output of them. So the intermediate model that is used has been trained on tens of thousands of pieces of art from all over the world, and it is then tailored to the user’s dog.

DrawnBy AI currently has nine different “inferences” or output designs for images. These include things such as your dog as an astronaut, a cartoon, or a pencil drawing. The team is presently working on training the model to develop more inferences, such as holiday designs.

Going forward, Lea and DiMarco are looking for ways to give the user more of a hand in the creative process rather than using pre-designed inferences. However, they highlight that training these Stable Diffusion models can be difficult. DiMarco points out, “If you’ve played around with language models or image generators enough, you realize you can’t use simple prompts and get deep, textured responses from it. So that astronaut [for example] is about 25-30 words to create the astronaut image.” Not to mention, DiMarco noted that some prompts work super well for a particular dog or image set. But, when this same prompt is used on 20 or 30 other image sets of varying qualities, you get very different outputs. So the process of creating these prompts is not only based on what works for one type or dog or image set, but what renders particularly well across dozens of image sets of varying qualities and types of dogs. DiMarco emphasizes, “There are a lot of tools out there that give the user the box—’type what you want into this box’—but unless you’ve really spent time with these prompts and understand how to add the texture and depth to the prompt to get back a satisfactory result, you’re going to be really disappointed in it.”

One way Lea and DiMarco are looking to implement user creativity into the image generation process is to provide the user with a set of descriptors, scenes, and props. For example, “I want a picture of my dog… in front of the Eiffel Tower, wearing a beret, in watercolor.” This would allow the user to check major descriptors and mix and match their images, but behind the scenes would be these professionally hand-crafted prompts formulated by Professional Prompt Engineers.

THE FUTURE OF IMAGE CREATION

DrawnBy AI is an extraordinary platform for the ordinary person to use a highly advanced technology, bridging the gap between users and an innovative AI capability.

AI-generated pictures of your dog are fun, and in the future, we might expect to see these AI-generated pictures of dogs be made into tangible products, like sweatshirts, mugs, blankets, and more. DiMarco and Lea are certainly onto something big here, but it’s not only in terms of their commercial project—they are also revolutionizing Stable Diffusion and image creation.

As of the moment, large AI image generators don’t allow users to custom train or fine tune models to specific faces or dogs. As Lea says, “There is no amount of text prompt that you can feed [current AI image generation tools] that is going to make it know what your dog actually looks like.” DrawnBy.ai offers a level of customization in this way which is not being offered by many big tools, making it a revolutionary breakthrough in the space.

WATCH THE FULL INTERVIEW:

Our team at FryAI sat down with Matt Lea and Dominic DiMarco to talk about DrawnBy.ai. Check out the full interview here:

Need help with your AI project?

If you are looking for a way to scale your AI solution or need help with back-end architecture, reach out to Matt Lea at Schematical.com.