It was recently peer review week in the academic world, and one article was making the rounds: ‘Substandard and unworthy’: why it’s time to banish bad-mannered reviews. The piece highlighted a recent video showcasing highly successful scientists…and the sometimes nasty reviews that they have received.
“Much of what has been reported in this work is useless.”
“This is rubbish…”
When I posted about it on Bluesky, the piece got a massive response from scientists. Because we have ALL had this experience, every one of us.
I will never forget one Thanksgiving, I was in grad school, and working on a paper at my aunt’s house (this will come as no surprise to anyone in academia, of course). We’d just gotten the reviews back. I ended up explaining the process of peer review to my aunt, and showed her the review. She handed it back a few minutes later with tears in her eyes.
“How could they be so mean?” she asked.
The paper was accepted with minor revisions. These were “nice” comments. But they were comments that also questioned the scientific ideas, the methods, and my own intelligence.
Peer review is an essential part of scientific research.* Before a paper can be published in a journal, the editor of the journal sends it to other scientists in your field (they will avoid sending it to your greatest scientific enemies, in theory). The scientists read over the paper (hopefully in detail), and point out problems that the scientists may have missed, keep an eye out for fraud, etc.
This is hugely important because scientific research is massively detailed, and often very much in the weeds. We spend years on a single set of experiments, in the bog of drug dosing and troubleshooting the stupid cameras on the stupid maze, praying to any and all gods to please make the HPLC machine work for once. Deep in these studies, pursuing one set of experiments, it can be easy to miss alternative angles or explanations. Peer review helps us to see the forest instead of the trees. It helps us to make sure we haven’t missed a statistical method that will help us understand the data, and to make sure we really have proved our point.
But of course, peer review is also made up of our peers.
And we, and our peers, can be absolute jerks.
When I posted about this, several people asked how anyone could be this way? Science is supposed to be collaborative! We’re all supposed to be seeking truth!
Several scientists answered that the roots lie in the culture of modern (in my case biomedical) science. I would agree, and I’d like to talk about one of the ways I feel that comes to be.
(As per usual: This is just my opinion. It might not be yours! It’s also possible that other people went through the same training I did and came out with different impacts! We all approach the world through our own lens. That’s ok. It doesn’t make either of us bad people. So like, take that into account.)
It’s time to talk about journal club
Being a student in science isn’t just about memorizing formulas, running experiments, learning the scientific method. It’s also about learning the culture of science, how scientists are expected to behave to each other, what we are supposed to value, how we are supposed to think and write and approach the world.
One of the ways we learn this is in journal club. Grad students and postdocs get together, and someone selects a paper they want to talk about.
We all read it ahead of time, and then sit in the conference room (if we’re lucky there is pizza). We then tear that paper to shreds. Limb from limb.
In my journal clubs, I learned quickly that you cannot say you liked a paper. That’s not what journal club is for. It’s for finding every single flaw that a paper can possibly contain—even and especially flaws in papers published in some of the biggest journals in the world.
This is supposed to be a learning experience, of course. By seeing flaws in other people’s papers, we learn not to do that ourselves. Learning that a one-tailed t-test is not appropriate for most studies you do in stats class is one thing. Seeing use of it eviscerated in journal club? That’ll put you off it for all time.
Journal clubs teach you how to read and analyze papers. As a now-journalist, this is one of the most important things I learned. I know how to read papers critically and find issues, unanswered questions. I know how to spot uncited factoids and interpret ill-made graphs.
But journal clubs also teach students about scientific culture. They teach you how “real” scientists behave. The postdocs and senior grad students I admired attacked those papers with the same gusto they used to attack the pizzas at the back of the room. By the time they were finished, I’d feel horrible for even having picked the paper in the first place. To find a flaw was to be clever. To have a good, clear scientific eye. If I didn’t see that error—how could I have missed it? What was wrong with me? I must be a horrible scientist.
“Real” scientists, I learned, cared about facts. Real biomedical scientists didn’t have feelings. They should be able to dispassionately separate themselves from the years of late scientific nights, the subjects they went into because they cared about other people’s suffering. In “real” science, anything you did right was what you should have been doing in the first place and unworthy of praise. What you did wrong? A catastrophe. Pathetic. A sign you are not a good scientist. Takedowns were best if they were clever, disdainful. After all, it was just journal club. The authors of the paper were never there in the room.
Some of the takedowns I saw were, frankly, also jealousy. If you know, deep down, that your work is never going to end up in Science, if you hide a tiny kernel of rage over being at a non-Ivy League school, knowing that the vast majority of funding and tenure track positions will go to those fortunate few, it can be cathartic to show that your well-published colleagues have scientific feet of clay.
Of course we’d all like to think we’re not really like this, that we can dole out praise where it’s due. Many scientists I know are deeply caring people, people who do not go through their lives being cruel to other people for kicks.
But in the lab? This is what we should do. Be so abundantly critical of our own work that no one else could possibly be more critical, and translate that criticism to others for the good of the scientific enterprise. I absorbed this idea of what scientists should do. I dished it out. I ripped research that I could never, in a million years, have conducted myself. Because that is what real scientists do. It’s not mean. It’s science.
The line between cleverness and cruelty
Another moment from grad school stands out in my mind. A friend of mine was defending their dissertation. They were nervous, of course! They were presenting their work, and in the process, said ‘reward’ when they should have said ‘reinforcement.’
In addiction pharmacology, reward is one thing, and reinforcement another. This is something we all learn in our first week of grad school. The defending student absolutely knew this, it was what their entire dissertation was, in fact, about. It was just a verbal typo, brought about by nerves (I mean c’mon do they BOTH have to start with “r”?).
But that did not stop another student raising their hand, and saying with undisguised hauteur “Did you mis-speak? I think you said ‘reward,’ when you should have said ‘reinforcement.’”
And we all had to watch as the defending student fought back tears on stage on what was supposed to be one of the biggest days of their life. Because another student had (I hope temporarily) confused cruelty and cleverness.
No one said anything. As far as I know, no one ever told that student that what they had done was cruel. I would not have been surprised if other students in their lab smirked over the clever “gotcha” moment. I’m sure they felt they were being clever. Just like in journal club. Pointing out an obvious mistake that anyone should know better than to commit.
This cruel cleverness even had a point. We learned quickly that our colleagues were also our biggest competitors. There are only so many jobs to go around. Research is not a pure disinterested peering into reality. It’s also your future on the line. Your future job prospects. Your identity. We are encouraged to say “I AM a scientist.” “I AM what a scientist looks like.”
So if you do flawed science, then…what are you?
The more holes you can poke in someone else, then, the better you look. You saw all the flaws in research you didn’t do. You obviously deserve that fancy post-doc, that tenure track job.
It’s easy to see how this translates into peer review. We have all the incentive in the world to be critical, and very little incentive to be kind. Many of us have learned that cruelty is clever, and that impatience with other people’s mistakes is a sign of intelligence. We’re often reviewing papers on little sleep, while fighting for tiny scraps of funding, and in constant terror that we’ll be out of a job next year. We’ve been taking hits, the person writing this paper should take them too.**
We could all use a compliment sandwich
The culture of science should be critical. Of course it should. We don’t want bad science making it out there into the world, potentially causing harm to patients. Bad science makes it through as it is! We do, in fact, need to be critical, to find those flaws, to do everything we can to ensure the truth.
But I wonder if my life in science would have been different if I had learned that criticism doesn’t have to be mean. This was the point of the Nature article on banishing bad reviews.
Not everyone shrugs off insulting remarks. Their impact on self-confidence, productivity and career trajectories can be significant, says Laura Feetham-Walker, IOPP’s reviewer-engagement manager, who led the video project. Here, she explains why mean-spirited peer-review comments should be challenged, and why the science community needs to discuss this commonplace humiliation of its younger members.
I think a lot of the much-maligned compliment sandwich, where you say a thing you like, then drop your criticism, and then end with a thing you liked. I think it’s a start. I did get some peer reviews like this, where the reviewer had the grace to say “this is obviously a lot of work, and a good question to ask.” It really did take the sting out. They recognized that a person, a student, did that work. Those few words said “I’ve been where you are. I want you to do good work.”
But the reason compliment sandwiches end up maligned is because there’s no amount of compliment bread will make up for a turd in the middle. And a compliment sandwich is no good if the compliments are just window dressing, and the meat is mean.
So if we’re going to serve sandwiches…we need to serve them with kindness. We need to remember that every paper has a person behind it. A person like us who spent years working late nights in rooms humming with equipment, praying over their HPLC, gritting their teeth at the messiness of data and trying not to cry with stress.
There’s nothing wrong in approaching a peer review with the knowledge that the scientists on the other end aren’t trying to pull one over on us, probably. They’re trying to do good science. Peer review is trying to help them do it.
It’s not just science, is it?
I’m not just saying this for other scientists out there. I’m saying it for me.
I did this too. I turned in peer reviews that were impatient, terse, and honestly picking nits. And I still do it, sometimes, on the internet, especially these days. We’re all so stressed, so afraid. We’re doomscrolling and we can’t look away. Every good feeling is undermined with another feeling of guilt. How dare we be happy at a time like this.
I do this. I get up in my feelings, and mistake cruelty for cleverness. I’d rather be right than kind. Cruelty, after all, gets clicks. We love a hot take. I assume ill intentions and trolling where most often there is mere misunderstanding.
When everyone else around you is in the same boat, it can honestly feel like self-defense.
So what this post is is a reminder. It’s a reminder to myself, that it doesn’t have to be this way. That culture is mutable. That we can change. That not every interaction needs to have a surge of adrenaline behind it.
I’ve unlearned much of how I critique by interacting with wonderful editors. Editors who point out flaws with kindness. Who offer suggestions and dole out sincere praise and criticism in equal measure. These editors make my job a joy, and make me want to be a better writer and reporter. The result is good work—and it didn’t require humiliation to get there.
I can do this. I can try to remind myself to be better. To ask if this comment might ruin someone’s day, or make it better. We can’t (and shouldn’t) take peer review out of science, or out of other interactions! But we can put some humanity back in.
Where have you been?
Is it reading this delightful letter from the King of the Rats to Mayer Eric Adams, because it slayed me. “How many signups did you get for Rat Squad, Mr. Mayor? How many T-shirts did you give out to your “elite rat-fighting squad”? How many students have matriculated from Rat Academy? And of those, how many are familiar with the term “fuck around and find out"?” By Emily Flake.
Maybe it’s reading this useful piece from Self magazine on why yes, it is still a good idea to avoid COVID actually. By Erica Sloan.
Or maybe it’s learning about this really cool little firefly that lives only in Delaware, Maryland, and Virginia? It’s got my name! And it’s not doing great. Save my firefly! By Dino Grandoni.
Maybe it’s learning why we might want to bury our wood. Hehehe. Bury it DEEP. Bury the BIG LOGS. By Jonathan Lambert
Or maybe it’s reading this important piece on the PTSD from Covid that is still afflicting many medical providers. By Amy Maxmen.
Maybe it’s reading this piece on amoeba pellets. You’ve heard of owl pellets. But amoeba pellets? By Jake Buehler.
Or maybe it’s voting in fat bear week!!!! GO BEARS GO! GET YOUR CHONK ON. And no, no one weighs the bears. Do you wanna do it? By Natalie Compton.
Where have I been?
This is one of those times where several articles I’ve been working on for months all end up published at the same time!
I got to write about a scientist who studies eyes to find out what your brain is up to.
I got to write about all the different kinds of electricity in your body! Like the electric signals in your bones. Yes. Your bones.
I also have an explainer on how your heartbeat stays steady. It doesn’t need your brain.
And I also explain how cells full of chemicals can create electric current.
And I wrote about how much we admire variation in male athletes that makes them super fast or super strong. But when women have similar variations, suddenly we wonder just how “female” they really are. Why? It’s got nothing to do with biology.
Finally, I’m leading some sessions at the National Association of Science Writers meeting in November (and virtually in October). Join us for networking at Science Writer for Hire, and learn about implementing trauma-informed reporting in your work with the workshop I’m running with Emily Sohn!
Anti-Discourse Actions
Why fight on the internet when you could use your energy to do stuff instead?
I donated some classroom supplies for teachers this year! IMO, I want teachers and finance bros to switch salaries, the bros can make a base of $30K and make it up with being good at finance. Teachers? Give them $400K and a pension because they hold the future in their hardworking hands. I’ve never met a teacher who wasn’t wholly and deeply dedicated to their work. Glad to help these wonderful humans make their classrooms great places for their students (and also give the students nice shiny new supplies. If you are like me, there is no joy like the joy of a perfect new notebook).
I contacted my government critters about KOPSA, an online safety act that is truly anything but.
I donated to a fundraiser for a friend who lost a lot in Helene. Western NC is a beautiful place full of really good people, and it breaks my heart how many have lost so much.
We HIT 500!!! 500 Vote Forward letters went out this week. There was a small party for sealing and stamping.
*Of course peer review has many problems, independent of people being jerks. That’s just not what I’m talking about right now. There is no ethical consumption under capitalism. Please go touch some grass.
**Yes these are first-world problems. See star above.
Have I given mean reviews? I expect that the authors would think so, yeah. [The one that always sticks with me: from a psychologist who wrote a paper making claims about brain metabolism but repeatedly described glucose as a protein and claimed that visual processing did not require energy, among other equally ludicrous elements.]
I don't regret eviscerating it.
Review isn't just mean in science, it can be really mean in creative writing classes too. I teach beginners level creative writing and I always point out the positives in everyone's work and then ask if they want any feedback on what could be improved. But many tutors apparently aren't so kind, which can be very detrimental to beginners! Obviously, serious scientific work and writing that is being put forward for publication need to be critiqued, but it's always possible to be kind.