The AI Football Coach: D.J. Lee's Play Prediction Algorithm

Welcome to this week’s Deep-fried Dive with Fry Guy! In these long-form articles, Fry Guy conducts an in-depth analysis of a cutting-edge AI development. Today, our dive is about an algorithm that could change football coaching forever. We hope you enjoy!

The championship football game is going down to the wire. The away team is down four points, and it is 4th down and goal from the five yard line. There is one second left on the clock. The entire season comes down to this one play …

What if there was a way for the coach of the home team to predict which play their opponent was going to run in this situation? Suppose there is a way for the coach to know, say, that there is a 90% chance the team is going to pass the ball to #87 on the right side of the field?

This type of prediction model might not be too far off, as a professor and many of his students have been working on an AI algorithm that might eventually have the ability to predict plays with acute accuracy.

WHO IS BEHIND THE PROJECT?

D.J. Lee has been a professor at Brigham Young University (BYU) for 22 years. Before that, Lee worked in the computing industry for over eleven years. His research has always been in computer vision and visual information processing. About twelve years ago, he started using machine learning for his visual information processing, and he has been doing that since. Lee remarks, “When people talk about AI, they talk about ChatGPT. But it is so much more than that—there are other things also. And one of those other things I do is using AI for video information processing. That is my expertise and my passion.”

Lee has worked on several projects over the past 35 years that utilize video information processing—a sort of information analysis that utilizes input from a camera. Because the use of camera has become so common in the past decade, it has made it much easier to get videos and images for processing and analysis. In the early days of his video processing exploration, Lee used the technology to inspect food, agriculture, robots, and other things that require visual information. In the past decade, he has switched from computer vision to more AI-based vision, which he has successfully applied to food and agriculture industries. He has also taught classes about using AI vision to aid self-driving cars. Recently, Lee has been focusing on variations of facial motion authentication. Lee explains that he has been exploring ways where “you don’t just use your face; you also use a specific motion as your password. So you could just wink or you could say, ‘hello’ or ‘open sesame.’” This would allow the visual recognition feature to read your facial expressions for authentication purposes to enhance security.

D.J. Lee loves football, both college and professional, and about three years ago, he was watching BYU play. Given his background in visual recognition, he wondered, “Maybe there is a way I can help BYU to play better and win.” He talked to the coaches and got some videos for exploration of how visual recognition might help film study and possible play predictions. This started his journey towards development of the football algorithm which he undertook with some of his students.

HOW DOES IT WORK?

There are relatively simple steps undertaken for the use of the football detection algorithm. It begins with detecting players, determining their position on the field, and using those features to determine a team’s formation.

The first step is detecting players, differentiating them from anything in the background, such as a logo on the grass or fans in the stands. This is done through a deep learning neural network. The neural networks learns by analyzing image data, such as a “football player” and a “fan” for instance, and the more data the algorithm has, the more accurately it can determine what is a “football player” and what is a “fan” or a “yard marker,” for example. In this way, the model learns what the camera should look for. Currently, the model can recognize players with 95-96% accuracy. If a player is not detected—for example, if the neural network only detects 21 players on the field instead of 22—it will most always be able to recognize which player is missing, such as the center or left guard, or the fullback, for example.

After the players are located or detected on the field, another neural networks is applied to determine what positions they are playing. Based on their location, the neural network can determine their position, for example, whether a given player is a quarterback, running back, wide receiver, lineman, defensive player, etc.

The third step involves determining a team’s formation. By recognizing the players, their positions, and their locations on the field, the neural network can determine the team’s formation. In football, there are different formation families based on where the players are positioned at the start of the play. This often gives a strong indication of the play type they will run, based on where the players are positioned. As of now, this is what the project is capable of.

WHAT ARE THE LIMITATIONS?

The project is limited to the data that is fed into the neural network. Because football data is sometimes difficult to acquire, Lee used a video game (Madden 2020) to collect data for the algorithm. He said, “We hired students to play the video game, and we collected a lot of data and images. We used those images to train our algorithm.” In order for this algorithm to be used effectively for real teams, there will have to be a large amount of camera footage from a high angle. The current problem is that most teams don’t film from a high enough angle, where the neural network is able to detect all the players on the field due to them being off the screen or blocked by fans, people on the sidelines, etc. Although the neural network can predict where certain players might be, this can be difficult for the model’s training purposes.

If teams are willing to adopt this model, they will need to find a way to get camera angles which can view all of the players on the field each play. Additionally, the model will work better with more data, so the more film the model is presented with, the better it will perform. This will require teams to have a lot of film on a given team in order for the algorithm to perform ideally.

WHAT IS NEXT FOR THE PROJECT?

In the Madden video game that is used to train the algorithm, users have the ability to access and record replays. Lee plans to use this sort of data to begin tracking player motion, so not only will the algorithm spit out the formation of the players, but also detect what play is being performed based on player motion. This would also allow for the algorithm to detect the circumstances, such as the score and the down—for example, 2nd and 3 on the 40 yard line, winning by seven with five minutes left in the first quarter. The hope is that, with enough data, the algorithm might eventually be able to predict which play or play type will be performed in a given situation. For example, with enough data, the algorithm might be able to predict that a given team will run a certain play with 82% probability in a situation where they are on the opponent’s 30 yard line, losing by three points with five minutes left in the game.

The implications of this algorithm have the potential to expand beyond teams as a whole to individual players as well. Once movements are able to be predicted, this will help train individual players on how they should approach certain assignments, such as covering a given players. We might imagine, for example, a wide receiver who runs towards the middle of the field 77% of the time on third down. With this information, the defender will be able to more effectively scout and cover that receiver.

Lee, given the great attention his algorithm has gotten, has a group of students who hope to continue the development of this project as it progresses to the next phase. Lee’s hope is that, once the algorithm can start recognizing and predicting plays in the Madden video game, he can begin to apply this to actual football games, beginning with high school teams—where the plays which are ran are more simple and easily recognizable than college or professional teams.

The idea is not that this will take the human element out of coaching, but rather that—at a time of an increased attention to analytics—the coach will be given more data to help inform his decision. As Lee remarks, “I don’t imagine this ever replacing coaches, because that takes away the fun. Outsmarting your opponent is fun, and using all sorts of tools can help you do that.”