The College of Computer, Mathematical, and Natural Sciences hosted a Reddit Ask-Me-Anything spotlighting natural language processing and empirical machine learning research.
University of Maryland Computer Science Assistant Professor Sarah Wiegreffe participated in an Ask-Me-Anything (AMA) user-led discussion on Reddit to answer questions about artificial intelligence transparency on August 12, 2026.
Wiegreffe studies the explainability and interpretability of deep learning systems for language, with a focus on understanding how language models make predictions to make them more reliable, safe and transparent to human users.
She is currently leading a new research effort to test whether the reasoning processes used by advanced AI systems will remain transparent. At the center of her research is a widely used technique known as chain-of-thought reasoning, in which AI models generate step-by-step explanations of how they reach their answers.
This Reddit AMA has been edited for length and clarity.
If a model divulging its chain-of-thought reasoning relies on the model providing said chain by also using reasoning, how do you assure that the process isn't entirely circular?
That's a good question—and a very open area of research! Generally, chain-of-thought reasoning is hypothesized to boost model performance because it is sort of like a 'stream of consciousness' that provides additional computation for solving harder problems. So we are usually not explicitly asking the model about how its reasoning functions after it provided said reasoning for the reasons you stated; that would be what is called a "post-hoc rationalization." That being said, there is a line of work trying to understand how "faithful" chains of thought are to the model's internal computations, and many examples where they fail to capture all of the reasoning in the text in a way that is understandable to humans.
I would recommend checking out literature on chain-of-thought faithfulness and monitorability. The general rule of thumb for training models to produce chains of thought is not to put any optimization pressure on the reasoning itself, so that the model uses it purely for computation rather than a human-palatable explanation. This is partially why the reasoning/chain-of-thought you see from Claude or ChatGPT is not the actual chain of thought; it's a rewritten summary. There are a few examples of "real" chains of thought extracted via a jailbreak attack.
I've heard some AI experts say we are at the point where we don't know how AI arrives at conclusions. What do they mean by that?
This is a classic problem in deep learning more generally, and specifically in the field I work in, large language model (LLM) interpretability. These systems are very complex, have billions of parameters, and are optimized on a lot of data that is fairly hands-off. This makes understanding the internal mechanisms—and specifically how and why they learn certain behaviors and produce certain responses—very challenging compared to classical machine learning models or deterministic software algorithms.
In my view, what these AI experts mean by that is that our understanding has only gotten worse as the size of the systems and the amount of data they are trained on have rapidly scaled. We have more cases of models exhibiting surprising or unintended behaviors than ever before (emergent behavior). For example, see the recent Hugging Face hack by OpenAI coding agents. Many of these are genuinely surprising because even experts are surprised by how well our current paradigms have taught models things we didn't explicitly reward or intend. At the same time, we have greatly improved the suite of interpretability tools and findings for better understanding what is going on. Interpretability and understanding are always playing catch-up to the state-of-the-art in the field.
How do we leverage trustworthiness and overreliance on AI models, especially for our most impressionable section of society?
I think this is a huge problem, and one that psychologists, neuroscientists and human-computer interaction researchers have been studying to quantify the negative effects of AI use on the human brain. Ideally, there will be incentives not to use AI for tasks where it is not needed and also broader education about the pros/cons of using AI (what is called AI literacy). But this is a problem where we need better AI policy and governance initiatives.
It's also important for people to be able to recognize when they are interacting with AI content. I hope we will end up with policies similar to what the European Union recently enacted, which requires AI-generated content to be labeled. Both Anthropic and OpenAI have announced they will now watermark content generated by their models.
What are some truly open problems in language modeling and processing that are yet unsolved?
This answer is going to differ a lot depending on who you ask. On one hand, many classical natural language processing problems and tasks are considered "solved" by LLMs. On the other hand, there are still many failure cases that can affect specific user groups, such as culturally aware generation, domain-specific generation or efficient customization.
There are also a lot of open questions about the "science" of language modeling. We now have proprietary systems that can do a lot of these tasks. That said, the general public and the scientific community do not know how this has been accomplished and to what extent it is based on scientific breakthroughs versus a lot of engineering on the backend.
If AI is using bad data, how can its reasoning get "back on track"?
Models are influenced by two things: training data and what is given in the prompt. Both of these things can be contaminated, either intentionally or unintentionally. In the case of bad training data, we hope that the "bad" data is in the minority, which is often the case. Models can be put "back on track" by continuing to train on newer or more updated data. (Generally, more recent training data will dominate older training data.)
Prompting can also be used to "correct" errors either in the model's parametric knowledge (i.e., what it learned from the training data) or in the user's prompt. To correct errors in parametric knowledge (such as outdated information), we use tools and retrieval methods (retrieval-augmented generation or "RAG") to find relevant, up-to-date sources with correct information that are then given to the model with your query.
To correct errors from the user's prompt, it is very common for developers to include a hidden "system prompt" that is combined with your prompt when you query a language model. Models are taught that the instructions in this system prompt should take precedence over the user's query, so this can help keep the model on track. They are also explicitly trained to refuse certain types of user queries.
There has been a notable trend of large data-sharing companies submitting Public Information Act requests to the State to harvest data for LLM purposes. If a legislative committee were to call you in for expert testimony, how would you either a) defend these requests, b) clarify the intention behind these requests, and/or c) propose certain guardrails for either the government or tech companies to prevent over-harvesting?
My personal stance is that the state should take measures to protect citizens' privacy, and that there should be a clear case for why training on this data would benefit citizens—for example, if the company is building a tool that will help people navigate bureaucratic systems and users have opted in.
If the data is legally public, I am not sure what the basis would be for denying these requests. But there have certainly been cases of people being doxxed or targeted based on public information, such as voting records, that used to be much harder to access/search. AI tools are only going to make this process far easier. This seems to be an example where the laws' assumptions about how and why information could be accessed no longer hold in the age of automatic AI agents.
Policymakers should be aware that government agencies could be spammed at higher and higher rates, since companies (as well as individuals) may be employing skilled AI agents to submit requests and fill out paperwork on their behalf.
Do all these models converge to the same point? Using AI writing as an example, will all models eventually converge to write similar outputs?
In terms of the technical definitions of convergence, no. The chance of different models being identical is infinitesimally small. I don't work in optimization, but there is an entire field focused on the specific type of optimization algorithms that make neural networks learn well (non-convex optimization). Usually, generalization refers to how well a model works on new, unseen examples after it has been trained. Current models generalize very well, even if they do not produce identical text or possess identical behaviors.
However, especially with the amount of text available on the internet for training being exhausted and the fact that companies are using very similar methods to train their models, you might see increasingly similar behavior from different models. (Though I am unsure whether any recent research has concretely shown this.) A common method for making models better is to train on outputs from a stronger LLM (distillation), which leads to even stronger similarities in their outputs. It's hypothesized that this is why some smaller open-source models may sound like Claude, for example.
I don't know whether model outputs will start sounding more similar or more different; this probably depends on economic and legal factors influencing tech companies, which is outside of my expertise. The companies want to have a differentiating model, but they also want to have the best model.
What's the most surprising thing your research has uncovered about how language models reason?
Models use surprisingly simple and sparse structures to represent certain concepts, and we can steer models using these structures to provide a form of control (even though they have the capacity to do things in fancier ways). A concrete example of this is the fact that we can predict and control certain model behaviors—such as how they recall basic facts—with simple linear approximations. This is generally what's called the linear representation hypothesis. We have also found simple and sparse representations for how models refuse dangerous queries or solve math problems with multiple steps.
In our work, we have found that these simple linear approximations work best for facts that the model saw a lot over the course of training. While intuitive, it's fascinating that models allocate their limited capacity to memorize facts based on how often people have written about them, even though we never explicitly instruct models to do this. It's a very efficient way to store information, which could explain how the same model can be capable of doing so many things.
What worries you most about where AI is headed with gathering information and reasoning answers, and what is the role of consumers in making AI more accurate and more transparent?
I worry the most about the psychological offloading of tasks to AI to the extent that people will be easily persuaded and/or fall out of the practice of verifying answers and using their common sense. This is especially poignant as model outputs/interactions are becoming increasingly long and complex, making them harder to evaluate with our limited time and attention. Providing explanations or interpretations can often increase the burden on users to perform verification!
I am not sure what role consumers have to play in increasing accuracy and transparency, apart from "voting with their dollars" towards either open-source models or model providers that take model development and deployment seriously (e.g., proper handling of environmental, safety, copyright, dual-use concerns, etc.).
Do you feel optimistic or pessimistic about the future of AI transparency over the next 5 or so years?
I am very optimistic about open-weight model transparency! We have made substantial progress in the last ~5 years. Our knowledge is increasing rapidly, so I am very excited to see what we will understand another 5 years from now. Chain-of-thought reasoning is very powerful for understanding, yes, but it has shortcomings and is far from the only important model understanding method in our toolbox. I predict these other tools will be more important soon as chains of thought become overly long—and, in some cases, illegible—making monitoring the text alone increasingly difficult.
I am pessimistic about the transparency of closed models built by the big companies, though—I predict that it will only get worse, as companies become more secretive due to economic pressures and the AI race heating up further.
Do you have any advice for final-year Ph.D. students on navigating the job market?
I really enjoyed doing a postdoc, and I think that's a great way to expand your research direction beyond your dissertation and get experience doing some aspects of a faculty job.
Some advise doing a limited faculty search one year as a "practice run," but I personally think it's best to wait and then do the full search when you're ready so you can compare all of your options at the same time. It can be good to visit some universities you're interested in and give talks six months to a year before you go on the market.
It's very important to have a brand, and ideally one that differs in some meaningful way from your Ph.D. advisor. Universities want to know that you have a research agenda that you are passionate about and capable of executing once removed from your Ph.D. environment.
