UMD students work across disciplines and majors on four-year research project

How can we remove bias from artificial intelligence (AI) systems designed for everything from talent acquisition to online shopping to surveillance systems? 

Ten University of Maryland undergraduates came together to answer this question for their Gemstone honors research project. One of those students was Philip Mathew, a junior mathematics and computer science double major.

“I knew I wanted to focus on AI bias because of research I’d done on diabetic retinopathy at Johns Hopkins Applied Physics Lab,” Mathew said. “When I was a freshman intern, I learned that with diabetic retinopathy, your melanin count does play a difference in how your retinal scans look—and we saw that underrepresentation of people of color in the training data led to bias against accurate diagnoses for those people groups.”

Historical human bias—against people of color, women and other marginalized groups—causes artificial intelligence bias today. That fact became glaringly evident in 2018 when Amazon scrapped an AI and machine learning-based recruitment program after figuring out that the algorithm was biased against women.

Amazon’s AI model was programmed to vet candidates by observing patterns in resumes submitted to the company over a 10-year period. But because those hired during that period had been predominantly men, the system deduced that male candidates were preferred over female candidates. This prominent example is part of what inspired the Gemstone team, aptly named Project DeBIAS, to look at hiring algorithms.

“If you train AI on data that already has this systematic disadvantage against a group of people, it’s going to find and replicate those trends,” Mathew said. “The issue is that a lot of coders will think the AI system works fine without trying to understand the distribution of their data in the way of protective attributes such as gender and race.”

Developing the Methodology

Early on, the Project DeBIAS team hypothesized that resumes for people of color are being ranked disproportionately lower on hiring websites like Indeed due to AI bias. To test their hypothesis, they collected 59 anonymized resumes that included attendance at a Historically Black College or University (HBCU) and 304 resumes that did not. Then, Mathew and his teammates created an AI model to simulate how an autonomous hiring system would function, training it based on Indeed’s publicly available hiring data. 

“What we’re trying to ensure is that these systems don’t have something baked in where HBCUs are discriminated against,” Mathew said.

The team’s preliminary data analysis shows a trend that aligns with their hypothesis—resumes from individuals who went to HBCUs ranked lower in the hiring system. 

“We’re determining whether this is some sort of spurious correlation or a proper inverse correlation between what the status of your college is and what your ranking is in this Indeed resume ranking algorithm,” Mathew said. “Based on our preliminary findings, we do think we’re going to find an inverse correlation where people who went to HBCUs are ranked lower—and then we plan to find a way to solve for that in the ranking algorithm.”

Becoming Better Researchers

Mathew and junior mathematics majors Seth Gleason, Johnny Rajala and Daniel Zhu contributed their statistics know-how to understand the distribution of data and analyze the components of hiring algorithms. 

“I’m now better able to talk about and understand things from a statistical perspective, like why data is distributed in a certain way,” Mathew said. “And while it may seem obvious, linear algebra ended up being a huge help because computers use matrices, so you kind of need to know how matrices work.”

As they worked through their research work plan and received Institutional Review Board approval for resume data collection, the team worked closely with their team librarian Kate Dohe from UMD Libraries and their advisor, Steve Sin, an associate research scientist in the National Consortium for the Study of Terrorism and Responses to Terrorism at UMD.

“The Project DeBIAS team has become very agile over the last two and a half years,” said Sin, who has worked on detecting bias in emerging technologies. “One of the things they got really strong at is putting together a work plan with branches that says, if A then B. Their work plan helped them continually evaluate whether they were on track, adjusting course as needed.”

Sin noted that the students on the team have grown by “leaps and bounds” since their freshman year in sticking to timelines and collaborating across both the STEM and social sciences aspects of the project.

“Really, all of us are principal investigators on this research because every single one of the 10 of us gets a say on how we carry out the research,” Mathew said. “Not only are we getting experience working on an interdisciplinary team, we also can say that as undergrads we have shaped this research from beginning to end, which is an opportunity I’m pretty grateful for.”

Another important lesson the students learned was how to pivot when their initial research plan wasn’t working. When selecting a proxy for racial demographics, the Gemstone team initially planned to examine redlining—the systematic denial of providing financial services to residents of communities associated with a certain racial group. However, their initial research revealed that very few job applicants include their home addresses on their resumes. 

“Once we saw that redlining wasn’t offering enough data, we pivoted to looking at whether the highest level of education listed was from an HBCU,” Mathew said. “We used that as a proxy for seeing how marginalized populations get treated by these AI systems.”

In April, the Project DeBIAS team presented their research at UMD’s Undergraduate Research Day. Next, they’ll further analyze their preliminary findings, prepare their Gemstone thesis and submit their research results to a journal—and, hopefully, make a difference. 

 “Part of the goal of this whole research is to get it out there and let it add to the field,” Mathew said. “We really want to show that, yes, AI bias is an actual problem—and we might just have a way to fix it or evaluate it.” 

Written by Katie Bemb

A Brin MRC Distinguished Lecture delivered by Ofer Zeitouni will also take place in the fall.

A message from Doron Levy, Brin Mathematics Research Center Director and Department of Mathematics Chair

The Brin Mathematics Research Center (Brin MRC) was established in 2022 through a generous gift from Michael and Eugenia Brin and the Sergey Brin Family Foundation. We plan to begin hosting activities in the Brin MRC in Fall 2022. The center is located on the fourth floor of the Computer Science Instructional Center, which is within a short walk from Kirwan Hall, in space that we are in the process of renovating.

Since the Center was announced, we formed an advisory board composed of five professors from the Department of Mathematics: Sandra Cerrai, Dima Dologpyat, Dio Margetis, Bill Goldman and Richard Wentworth. We also identified an exciting lineup of activities for the center’s first year, including 9 workshops and conferences and 2 summer schools:

• Workshop on low-dimensional topology and homeomorphism (organized by three of our new faculty members: Lei Chen, Dan Cristofaro-Gardiner and Boyu Zhang)
• Two workshops organized together with UMD’s Brain and Behavior Institute
• Workshop on rare events (organized by Maria Cameron and Pratyush Tiwary)
• Two workshops on probability and stochastic partial differential equations (organized by Lenya Ryzhik from Stanford, Sandra Cerrai, Dima Dolgopyan, Yu Go and Leonid Koralov)
• Two workshops on partial hyperbolicity (organized by Amie Wilkinson from the University of Chicago, Frederico Rodriguez Hertz from Penn State and Dima Dolgopyat)
• Two summer schools are planned for summer 2023: one on partial hyperbolicity and a second school on fluid dynamics (organized by Huy Nguyen and Hussain Ibdah) 

The Brin MRC will provide many opportunities for short-term and long-term visitors who are interested in interacting with faculty members at the University of Maryland and in experiencing the metropolitan Washington, D.C. area. A Brin MRC Distinguished Lecture will be delivered in Fall 2022 by Ofer Zeitouni.

This is only the beginning and it is all very exciting. We will share updates on the Brin MRC in the math department’s biannual newsletters. However, we invite you to visit the Brin MRC website and follow us on Twitter to get more up-to-date information. If you have questions about Brin MRC activities, please email .

Prior to UMD, he was a research director at the National Center for Scientific Research (CNRS) in Paris.

From the time the École Polytechnique in France named his Ph.D. dissertation the best in the year 2000, University of Maryland Mathematics Professor Bassam Fayad built a reputation as a world expert in the theory of dynamical systems.

Fayad has traveled across continents to collaborate on research with colleagues in Brazil, Italy, Sweden, China and the U.S. In 2018, he was invited to lecture at the International Congress of Mathematics in Rio de Janeiro. Until recently, he was a research director at the National Center for Scientific Research (CNRS) in Paris.

Now, he has taken another prestigious position here at UMD. Fayad has been named the Michael and Eugenia Brin Endowed Chair in Mathematics. Established in 2015, the Brin Chair is “chosen strictly on the basis of demonstrated exceptional mathematical ability, achievement, potential, and leadership in mathematical research and education, in a vital field at the heart of mathematics.”

Known for his work in Hamiltonian dynamics, Kolmogorov-Arnold-Moser (KAM) theory, Liouville phenomenon and homogeneous dynamics, Fayad sees great potential in holding the Brin Chair. 

“In terms of possibilities to engage in far-reaching activities, Maryland is on a very remarkable upward slope,” he said.

He’s especially excited about funding that will be available through the recently established Brin Mathematics Research Center. 

“In dynamical systems, one of the strongest groups in the world is concentrated here at Maryland,” Fayad said. “We can host workshops. We can invite specialists in all areas of dynamics from around the world to give lectures or crash courses. We can invite colleagues and collaborators as well as their students. If you add our colleagues from Penn State and the University of Chicago and Northwestern and New York, we can function as a hub, because all these places are quite near and people interact a lot.”

It was the late Penn State mathematician Anatole Katok—who Michael Brin also studied under—who drew Fayad to his first experience in the U.S. After serving on Fayad’s Ph.D. jury, Katok extended an offer to Fayad for a one-year post-doc at Penn State. Following the post-doc, Fayad returned to Paris, where he lived since he was a teenager—moving there from Lebanon with his elder brother to prepare for the entrance exam for École Polytechnique. 

“This happened in the summer of a very hard last year of war in Lebanon,” Fayad said. “My father calculated that schools would not open, and he was right. Two days before classes were supposed to open, my brother’s school in France agreed to enroll me. I was in crisis and started crying for a whole week. I was young and not prepared and not convinced. The plan was that I would become an architect in Lebanon like my father. The story ended up with both of us at Polytechnique.”

Mathematics was also an unplanned path for Fayad. 

“I said I will do Polytechnique, and then I will do architecture,” he said. “And then I will go back to Lebanon. But math was also fun and that’s what I ended up doing.”

Fayad’s research today ranges from abstract ergodic theory to topics intimately connected to real-world mathematical physics, such as Hamiltonian dynamics and their applications to the N-body problem.

“The research field of dynamical systems has this advantage of being very close to physics,” Fayad said. “Part of our job, for example, is to look at the planetary system and study all the possibilities. Will this system behave in a tame way with all planets gravitating on ellipses around the sun, while their satellites do the same around them? Under which conditions will it behave in a completely different manner and maybe in a chaotic way? For example, can a planet escape? Can we lose the moon?”

When Fayad received the offer to join UMD, the pandemic had just started.

“Businesses were shutting down everywhere and the world seemed to be collapsing,” he said. “And on my side, it was all working well. I was signing a contract for a new life. However, the pandemic did delay everything by a couple of years.”

Fayad finally arrived on campus at the beginning of 2022. He’ll continue to travel to France to work with some of his Ph.D. students there, while he works on attracting students to his research group at Maryland. 

“The department is also very active in the process of hiring high-quality professors in all disciplines in mathematics. And we have a lot of projects,” he added. “You feel good to embrace this new adventure, where you know that people are putting in hope and energy and means to do things for our field that I think will make a difference.”

Written by Ellen Ternes