An Exclusive Conversation with Bryan Chiang of RizzGPT: The Innovator Behind Live-Assist AI Glasses

Welcome to this week’s Deep-fried Dive with Fry Guy! In these long-form articles, Fry Guy conducts an in-depth analysis of a cutting-edge AI development. Today, our dive is about a smart monocle device called, “RizzGPT.” We hope you enjoy! 🙂

Do you have rizz? Bryan Chiang does! This young developer created “RizzGPT,” a smart monocle device which recognizes faces and tracks live conversations, all while granting the user access to the features of ChatGPT’s expansive database.

RizzGPT has the potential to provide real-time feedback during conversations which can provide the user with prompts on what to say in certain social situations. This project could be useful in job interviews, conversations with a friend, or even when trying to flirt with a cutie at the coffee shop!

WHAT INSPIRED THE PROJECT?

Bryan Chiang grew up in Cupertino, CA. From a very young age, he was interested in programming, tinkering, and building stuff. He has since done extensive research about how to apply machine learning to health-related fields and has recently been exploring the emotional aspect of AI interactions and societies. Chiang came up with the idea for RizzGPT when he was studying at Stanford. He had been working on Augmented Reality (AR) and Virtual Reality (VR) by designing headsets and holographic displays—similar to those currently being commercialized by Meta and Apple—and RizzGPT emerged as a side, hackathon project with a few of his friends.

One of the biggest trends in AI over the past year has been the use of large language models, such as OpenAI’s ChatGPT LM. Chiang’s goal was to combine this sort of LM with an AR component. Chiang, who has a passion for exploring uncharted territory, said, “People were building LM stuff all over, but nobody had really combined that with the AR component yet, so I thought it would be really fun to create a proof of concept experiment about what this future might look like.”

HOW DOES RIZZGPT WORK?

Chiang and his team used mostly publicly available tech to create RizzGPT. The conversation audio is picked up by a microphone and transcribed to text using Whisper, an automatic speech recognition system trained on 680,000 hours of multilingual and multitask supervised data. The response is then generated using the GPT-4 application program interface (API). The glasses used are regular glasses, and they utilize a Brilliant Labs monocle attachment. This monocle, which contains a small camera and microphone, provides the display for the user to read. The monocle, in its current model, communicates to the user’s phone or laptop through Bluetooth.

The most strenuous part of development, says Chiang, was the process of textual display. To display text on the monocle, Chiang and his team had to interact with the libraries for the monocle hardware itself, which took a lot of manual, tedious coding. As Chiang said, “That part was not a very fun time, to say the least.”

IMPROVEMENTS TO BE MADE

Chiang believes there is a lot of room for this project to be extended and improved upon. For example, after his original creation of RizzGPT, he made a few more demos with enhanced camera support which improved recognition functions. This extended the uses of the device. For instance, it allowed the user to use RizzGPT to recognize items on a menu and give recommendations based on one’s preferences or to scan books at a library and help the user pick out the one that would be most interesting to them. Chiang believes these sort of features could continue to be extended and improved upon.

One complication RizzGPT faces, from a hardware perspective, is that it is bulky. Although these glasses could be quite the fashion statement, Chiang is aware that the majority of people would not actually want to wear them, as they are, in social settings. He believes this issue will soon disappear, however, as hardware experts have been working on improving this to make it look like a regular pair of glasses.

On the software side, Chiang said some of the biggest complications to the project came with the speech-to-text feature. He explains, “Part of the software is that it has to turn your current conversation into text, and we were using API to perform this conversion of the audio into text. And because that relies on some external cloud service, the speed at which you could transcribe a conversation was highly correlated to the actual strength of the WiFi that we were using.” Because of this, it made conversations using RizzGPT more difficult in areas with bad WiFi. This hindered response time and made for choppy conversations. In some cases, this stifled the abilities of the monocle completely. Chiang believes the solution to this issue is to have transcription happen locally on the monocle hardware itself, either on small piece of edge hardware such as the microcontroller within the monocle or the field-programmable gate array (FPGA). Chiang has “no doubt this will be fixed” in the future.

WHAT IS THE FUTURE FOR RIZZGPT?

Though he remains skeptical about taking early steps into the consumer hardware space, Chiang believes that RizzGPT is just a fraction of what the future holds for AI wearables. He said, “I think it is inevitable that we are going to end up in a future where everyone has these AI-powered wearables that help them out with every decision.”

It is fun to imagine what it would be like if everyone had their own pair of RizzGPT glasses or something similar, but it does cause one to question whether this would hinder authentic human interactions and personalities. It could be the case that people would miss out on developing relational and conversational skills that are an important part of human interaction. However, it is clear this has already been happening in the form of cell phones and other mobile devices, which are changing the way people communicate and live their day-to-day lives. It seems this sort of transformation is not necessarily reducing human communication and interaction, but is nonetheless reweaving the fabric of it.

As AI wearables continue to evolve and grow more popular in the consumer space, it will provide people with hands-on access to real-time information and conversation prompts. We might imagine the uses this could have not only in everyday conversations, but also in some of the following fields:

• Business interactions and sales: AI wearables could provide salespeople with the necessary prompts to help stimulate client interest and improve sales prospects.

• The service industry: AI wearables could be worn by servers at restaurants, for instance, to keep guests engaged and identify needed tasks, such as refills or empty plates to remove from a table.

• Education: AI wearables could help students gain quicker access to information that could be useful in studying, and this could help teachers better address questions from students and read the body language of their students to more effectively gauge understanding.

• Medical enterprises: AI wearables could give nurses and doctors real-time advice on patient needs and available treatment methods.

• Mental health: AI wearables could provide therapists with prompts which might help them approach difficult patient situations. It could also help people better articulate the way they are feeling.

Although there might be potential drawbacks to these benefits, it is clear that this sort of technology could have a wide range of uses that would truly transform our social landscape.

Chiang is excited about the future of AI and is full-steam ahead on creating new innovations which could dramatically transform the space. As he says, “We’ve barely scratched the surface of what AI can do to transform society.”