In this letter, we report on the use of the UC-Berkeley Pacman Capture the Flag competition as a major assessment component in two large artificial intelligence (AI) courses. Teams of students are to integrate AI techniques to control the behavior of agents in an adversarial version of the well-known Pacman game. We discuss how the contest was designed, set up, and used for grading, and reflect on some of the outcomes achieved, including student perception. While the contest is very challenging (and, at times, frustrating) for students, it was successful in keeping them engaged. More importantly, a significant majority of students reported enjoying and learning from it. A powerful software tool to automate the competition for large number of teams is made freely available.