By Daniel Dunaief

Good parenting, at least in mice, is its own reward.

No, mice don’t send their offspring to charter schools, drive them to endless soccer and band practices or provide encouragement during periods of extreme self doubt.

What these rodents do, however, protects their young from danger.

When a young mouse wanders, rolls or strays from the nest, it becomes distressed, calling out mostly to its mother, who is the more effective parent, to bring it back to safety.

Responding to these calls, the mother mouse carries the young back to the safety of the nest.

This behavior involves a reward system in a region of the mouse brain called the ventral tegmental area, or VTA. When the mouse effectively retrieves its young, the VTA releases the neurotransmitter dopamine, which is the brain’s way of saying “well done!”

In a paper published in December in the journal Neuron, Cold Spring Harbor Laboratory Associate Professor Stephen Shea and his postdoctoral researcher Yunyao Xie, who worked in the lab from 2019 to 2021, likened the release of dopamine in this area to a neurological reward for engaging in the kind of behavior that protects their young.

The research “proposes a mechanism that shapes behavior in accordance with that reward,” Shea said. The connection between dopamine in a reward system is an established paradigm.

“There was plenty of smoke there,” he said. “We didn’t pull this out of thin air.”

Indeed, in humans, mothers with postpartum depression have disrupted maternal mood, motivation and caregiving. PPD is linked to dysfunction of the mesolimbic dopamine system, which is a neural circuit that involves the VTA, Xie explained.

“Studies using functional magnetic resonance imaging (fMRI) revealed that the reward brain areas including VTA in healthy mothers have higher response to their own babies’ smiling faces than those in mothers with PPD,” Xie added.

What’s new in this research, however, is that it is “a study of how these signals use mechanisms to shape behavior and social interaction,” Shea said.

How the process works

The feedback loop between dopamine in the VTA and behavior involves a cumulative combination of dopamine interactions.

Dopamine is not at its highest level when the mouse mom is engaging in effective pup retrieval.

“Dopamine is shaping future, not current behavior,” Shea said. “If dopamine was driving the mouse on a current trial, a high dopamine level would be associated with high performance. The trial found the opposite: a low dopamine level was associated with high performance in a given trial, and vice versa.”

Like a skater laying her blades down effortlessly and gracefully across the ice after spending hours exerting energy practicing, the mother mouse engaged in the kind of reinforcement learning that required less dopamine to lead to effective pup saving behavior.

As the performance increases, dopamine diminishes over time, as the reward is “more expected,” reflecting a nuanced dynamic, Shea said.

To test the correlation between dopamine levels in the VTA and behavior, Shea and Xie created an enclosure with two chambers. They put a naive virgin female mouse, which they called surrogates, on one side and played specific sounds behind a door on each side of the chamber. The test mice initially had “no experience in maternal behaviors,” Xie explained.

As these surrogates became more experienced by either observing mothers or practicing on their own, the amplitude of the VTA dopamine signals got smaller.

To provide a control for this experiment, Xie monitored a group of naive virgin female mice who spent less time with pups and had to figure out how to retrieve them on their own under similar neurological monitoring conditions. The dopamine signals in this group stayed elevated over days and their performance in maternal behaviors remained poor.

Through these experiments, Xie and Shea concluded that “there is a negative correlation between the dopamine signals in the VTA and their performance in maternal behaviors,” explained Xie.

‘Mind blowing’ moment

In her experiments, Xie used optogenetic tools that allowed her to inhibit the activity of dopaminergic neurons in the VTA with high temporal precision.

Shea appreciated Xie’s hard work and dedication and suggested the discoveries represent a “lot of her creativity and innovation,” he said.

A native of China, Xie said her grandparents used to have a garden in which they taught her the names and morphologies of different plants during her childhood. She enjoyed drawing these plants.

In graduate school, she became more interested in neuroscience. She recalls how “mind-blowing” it was when she learned about the work by 1963 Nobel laureates Alan Hodgkin, Sir Andrew Fielding Huxley and John Eccles, who established a mathematical model to describe how action potentials in neurons are initiated and propagated.

In the study Xie did with Shea, she found that the dopamine signals in the VTA encoded reward prediction errors in maternal behaviors that was consistent with the mathematical model.

In the bigger picture, Xie is interested in how neural circuits shape behaviors. The neural circuits of most natural behaviors, such as defensive behaviors and maternal behaviors are hard-wired, she added.

Mice can also acquire those behaviors through learning. She is interested in how pup cues are perceived as rewards and subsequently facilitate learning maternal behavior. She found a great fit with Shea’s lab, which focuses on the neural mechanism of maternal behavior.

Xie enjoyed her time at Cold Spring Harbor Laboratory, where she could discuss science with colleagues by the bench, at the dining room or at one of the many on site seminars. She also appreciated the opportunity to attend neuroscience seminars with speakers from other schools, which helped expand her horizons and inspire ideas for research.

Next steps

As for the next steps, Shea said he believes there is considerable additional follow up research that could build on these findings. He would like to apply methods that measure the activity in individual neurons. Additionally, with a number of targets for dopamine, he wants to figure out what areas the neurotransmitter reaches and how the signals are used when they get there. More broadly, he suggested that the implications for this research extend to human diseases.