How Science Really Works: Meta-Research, Publishing, Reproducibility, Peer Review, Funding | John Ioannidis | 212

Short Summary: An insider’s look at the messy realities of scientific research with Stanford’s Dr. John Ioannidis. The good, the bad, and the ugly about how scientific research actually works.

About the guest: John Ioannidis, MD, PhD is a professor at Stanford University in medicine, epidemiology, population health, and biomedical data science, with an MD from the University of Athens and a PhD from Harvard in biostatistics. He directs the Meta-Research Innovation Center at Stanford (METRICS), focusing on improving research methods and practices. Renowned for his paper “Why Most Published Research Findings Are False,” he’s among the most cited scientists globally, tackling biases and reproducibility in science.

Note: Podcast episodes are fully available to paid subscribers on the M&M Substack and everyone on YouTube. Partial versions are available elsewhere. Full transcript and other information on Substack.

Episode Summary: Nick Jikomes dives deep with John Ioannidis into the nuts and bolts of scientific research, exploring the replication crisis, the flaws of peer review, and the $30 billion publishing industry’s profit-driven quirks. They unpack Ioannidis’s controversial COVID-19 infection fatality rate estimates, the politicization of science, and the gaming of metrics like publication counts. The chat also covers NIH funding woes, administrative bloat, and Ioannidis’s current work on bettering research through transparency and new metrics.

Key Takeaways:

* Science’s “replication crisis” isn’t new—it’s baked into how tough and bias-prone research is, hitting all fields, not just “soft” ones like psychology.

* Ioannidis’s famous claim, “most published findings are false,” holds up: stats show many “significant” results are flukes due to weak studies or bias.

* Peer review’s a mixed bag—only a third of papers improve, and unpaid, tired reviewers miss a lot, letting shaky stuff slip through.

* Publishing’s a $30 billion game with 50,000+ journals; big players like Elsevier rake in huge profits from subscriptions and fees, often over $10,000 per paper.

* Researchers game the system—think fake co-authorships or citation cartels—boosting metrics like the H-index, which tracks papers with matching citation counts.

* Ioannidis’s early COVID-19 fatality rate (0.2-0.3%) was spot-on but sparked a firestorm as politics warped science into “clan warfare.”

* NIH funding’s clogged by red tape and favors older researchers, starving young innovators and risky ideas that could shake things up.

* He’s building tools like a public database of scientist stats (4 million downloads!) to spotlight gaming and push for transparent, fair research.

Related episode:

* M&M #100: Infectious Disease, Epidemiology, Pandemics, Health Policy, COVID, Politicization of Science | Jay Bhattacharya

*Not medical advice.

* Full audio version: [Apple] [Spotify] [Elsewhere]

* Full video version: [YouTube] [Rumble]

* Support M&M if you find value in this content.

* Episode transcript below.

Episode Chapters:

00:00:00 Intro

00:06:21 The Replication Crisis Explained

00:13:12 Replication in Science: How Much and Why?

00:18:14 Why Most Published Research Findings Are False

00:25:13 Peer Review: Strengths and Weaknesses

00:33:07 The Explosion of Journals and Predatory Publishing

00:41:40 The Business of Scientific Publishing

00:48:45 Open Access Costs and the Funding Dilemma

00:57:00 Preprints & Potential Solutions

01:04:34 Gaming the System: Metrics and Misconduct

01:11:08 COVID-19 & Politicization of Science

01:18:31 Revisiting the Infection Fatality Rate

01:25:48 NIH Funding & Leadership Changes

01:32:13 Directs vs. Indirects in Research Grants

01:40:56 Hopes for NIH Reform with Jay Bhattacharya

01:46:37 Current Projects & Closing Thoughts

Full AI-generated transcript below. Beware of typos & mistranslations!

John Ioannidis 1:53

I'm a professor at Stanford in the Department of Medicine, of epidemiology and population health and biomedical data science. I'm running the meta research innovation center, or metrics at Stanford, which is a center focused on studying research and its processes, its practices, how we can make research methods and practices better. And I've worked in different fields, in evidence based medicine and other areas that it's very common to see problems with methods, with biases, with making errors, including prominently my own, I guess, and trying to be sensitized by them and try to see how we can improve efficiency and

Nick Jikomes 1:16

To find all of my content. Foreign.

Thank you for joining me.

John Ioannidis 2:39

eventually get the most out of this fascinating enterprise that we call science.

Nick Jikomes 2:43

Yeah, and it really is an enterprise. There's lots of parts to the scientific research process. So you you've done a lot of work studying the scientific research process itself. You research research.

John Ioannidis 2:57

It is research on research, and sometimes research on research and research. So there's no end to the method transformation.

Nick Jikomes 3:05

And so how did you, how did you even get into this? Why did this come to be a focus of your work?

John Ioannidis 3:11

I think from my very early steps in trying to do research, I was very interested on methods. So methods tend to be the fine print section that most people are not so interested in. People probably focus more on results, and results are great but, but I was fascinated by the machinery, and I tried my hands on different types of research. So I I did some research that was wet lab basic science. I did some research that was clinical, some research that was population based epidemiology, some work that was more mathematical and statistics that the common denominator of what I found most attractive was the difficulty of doing good research, how much effort and dedication and commitment and being aware that error was just waiting for you to creep in. And biases are so prevalent, starting from our own biases, and also seeing that play out in pretty much the majority of papers that I was reading to try to inform my evidence base and try to see what I would do next. So increasingly, I was interested to try to understand the problems in the methods and the machinery, rather than test the specific result, which might be interesting and fascinating to pursue. But I thought that unless we can improve our methods and the way that we run our investigative efforts, our chances of getting reliable results would be pretty dismal. Yeah,

Nick Jikomes 4:53

and you know, I want to dig into a lot of different things with you, including, you know, how we identify whether results. Are likely to be reliable. What markers of unreliable work are give people a sense for some of the problems and the biases and the replication issues that are out there. But also, I think one thing that we'll come to is, you know, I've also gotten interested in sort of the research process itself. And, you know, especially as we're entering this new age where we have AI tools, there's a lot of AI based large language models and other AI based search tools that you know are helping people search through and synthesize the literature. One of the things I become fascinated by is there are problems in the literature that aren't it's not just that there are individual studies that I say are poorly done or under sampled. There can be systematic biases in entire fields for decades at a time, sometimes, and the technologies that we build on top of them are sort of going to inherit those biases based on how they work. And I want to dig into a lot of this. Let's just give people some sense for one of the big issues here. So a lot of people talk about the replication crisis. They will often talk about this in the context of specific fields like psychology. But of course, it's not specific to any one field. There's a lot of big replication issues that have become more known to people I think in recent years, what would you say the replication crisis is? Is it something that spans most scientific disciplines, and what is sort of the extent of this crisis?

John Ioannidis 6:21

I think the replication crisis is not necessarily something new. It is something that is inherent to the way that we have been practicing science, and replication is a fundamental component of how science should be run. It's a way to verify that what we're doing is reproducible. We can put some more trust to it, and we can build on it with with more confidence. I think that the term crisis has been coined in the last 1015, years because more attention has focused on that aspect of research. And I think that our calibration and expectations of how likely our results might be correct has been a bit overblown, and when people started looking systematically into that question, we had to recalibrate our expectations. I think that for a while we had forgotten that science is so difficult and so bias prone, we thought that we were probably overconfident, and then we have these large reproducibility efforts suggesting that a large share of our published results cannot be reproduced, and that leads to that terminology crisis. But it's not that this is new. You know, it was there. I think that people, at least the majority probably did not have that as a top priority in their thinking, and probably it moved up the ladder of priorities and prime considerations in doing science and thinking about science and interpreting science and translating science. It's it's a problem, and it is also inherent in the scientific method. So it's both a challenge and an opportunity. I don't see it as something negative. Realizing that there is that challenge could be a good way to try to think about how we can improve some of our performance and some of the track record of how our research could replicate or not.

Nick Jikomes 8:23

Is there? How are replication issues distributed across fields? So you know, people often talk. People often sort of think in terms of the spectrum from soft sciences to hard sciences, usually based in terms of how rigorously quantitative they are. Is are the replication issues confined mostly to so called softer sciences, or is this something we see in in harder sciences and even physical sciences as well? I

John Ioannidis 8:50

think that they can arise in any scientific field, provided that it is a scientific field. So a field that is always 100% correct is probably not a scientific field, it's probably some dogmatic religion, political, but not scientific effort that would be always correct. I think that also it depends on how difficult the questions are that we ask and how difficult are the odds of success. So some fields that seemingly have lower replication rates may actually be dealing with more interesting questions, with more high risk questions, with more difficult issues, that success is likely to be less compared to others. So I don't believe in a hierarchical view of sciences as these are the best sciences, and this is the second class citizen, and this is the the worst scientific fields, because each scientific field has its own performance characteristics, has its own goals and and targets. It has very different priors compared to others in terms of the likelihood of success. And it would be a pity if we, if we try to kind of put one scientific field against the others, and create some sort of competition of shaming each other that you're not reproducible and you're not replicable. We have seen problems with replication across practically any field that someone wanted to take a serious look. And it is not something that should be so surprising, even with rigorous measurements, even with fields that are very structured, things can go wrong or the types of questions are such that the success rate is bounded by some percentage. And you know, maybe that's the best that you can get in some cases. But that's okay if you have a way to filter eventually the credible information and move forward.

Nick Jikomes 10:58

Yeah. So a low replication rate, whatever exactly we choose that to mean it could indicate something negative or bad, like, you know, people are doing sloppy work. But it can also be an indicator of, you know, being on the cutting edge or actually doing high risk interesting questions. So it's not necessarily a negative thing, inherently,

John Ioannidis 11:19

exactly. I mean, if you study whether the sun is going to rise next day, probably the replication rate will be 100% but it's not an interesting question. Conversely, if you work in a very high risk field, many of the of the leads that are discovered are likely to be false and be refuted. But this doesn't mean that that field needs to be abandoned. So it's an it's a question of efficiency that needs to be calibrated against what is the likely value of the information. What are the consequences of true discoveries? What are the consequences of false discoveries, especially false discoveries that take a while to be refuted, and they stay with us and we build on them and go down the wrong path, or even get translated into interventions and policies that may be detrimental. So it's it's a complex system. I don't think that we should be oversimplifying into single percentages of success and failure, and we need to look at the big picture and try to see whether we can get something that that makes more sense, is more useful and is is more credible,

Nick Jikomes 12:35

and whether or not there's something like a replication crisis in any given field, If we, you know, if we have any sense for what the replication rate is, that implies that people have actually tried to replicate experiments. And of course, it's very natural, you know, if you're, if you're doing research, you know, it's, it's, it's more fun and more exciting and more advantageous from a career advancement standpoint, to make a novel discovery, not to just check if someone else's discovery is accurate. So how much emphasis is there out there on actually replicating results that are already in the literature?

John Ioannidis 13:12

So there's quite some debate about that, both regarding how much replication is out there and also how much replication is desirable. If you take scientific papers at face value, the vast majority of them are trying to say that what I have done here is something noble or has something new to say. And the reality is, nevertheless, that many papers don't really do something noble, and they're just replicating, perhaps with some minor twists, some experiments or studies or knowledge that already exists. One way to document this is to look at meta analysis, systematic reviews and meta analysis that try to revisit what we know on a given scientific question, and in medicine, for example, the average meta analysis finds seven to nine studies. There are some topics that we have 100 plus studies, and, of course, others that we have zero but but on average, we do have a number of studies that systematic reviewers believe are similar enough to put in the same forest plot, consider them as attacking more or less the same or similar question that they can be summarized together. Now, are they replications of each other? I think the people who do them don't think themselves as replicators. They see themselves as investigators, contributing knowledge. But in fact, they're pretty similar, so they do belong to the same bin of information, to the same bin of knowledge. So there's more replication than we are knowledge. Probably it varies a lot across scientific fields. I use the example of meta analysis, even when whether we perform since. Medical reviews and meta analysis has wide heterogeneity across science. In most biomedical fields, there's a lot of meta analysis. In some fields, there's more meta analysis than primary studies. And in other fields, there's very few, and some fields still have done hardly any such efforts to synthesize the evidence and see where do we stand? Where do we know? Where have we done a lot of studies, and where have we done no studies at all? So there is wide divergence on both the extended replication and even the willingness to look at how much replication has been done. That goes even a step further when you think about what is the desirable level of replication? Because an argument that is raised very often is that you cannot just try to replicate everything, and that would mean that you will have a lot of waste, and you're then trying to replicate waste, and that would be a loss of time and resources and effort. And we need to move forward and give priority to discovery. This is partly true. Probably there are some experiments or some studies that are so horrible and so useless that nobody should care about them and we should just put them to rest. But at the same time, I believe that replication is integral to discovery. It's not necessarily separate from discovery. I think that the replication and reproducibility check is integral in solidifying discovery and making sure that that we're dealing with something that we can put some trust and and move forward. And without it, we may be wasting even more effort and even more resources going down the the wrong path because we trusted something that was not to be trusted. So this is an open debate, and obviously the the best answer has to be operationalized differently in different circumstances, depending on feasibility. Sometimes, you know, if you have a study that took 20 years to do, and how is it to replicate that wait another 20 years versus here's a study that replicating it means just clicking a few buttons on your laptop to to run some analysis on some existing data sets that can tell you right away whether you get the same signal or not, and it would cost nothing versus it would cost a billion. So it's not a one size fits all, but there's unevenness in both how much replication we have, and also how much replication would be desirable.

Nick Jikomes 17:48

One of your most highly cited papers, and you're a very, very highly cited researcher yourself, one of the most highly cited papers you have is titled Why most published research findings are false, and it says right on the first page, it can be proven that most claimed research findings are false. Is that, can you unpack that for us? Is that meant to be taken literally or not? It

John Ioannidis 18:14

is literal. I mean, it's a bold statement, perhaps, but it is basically modeling the chances that, if you come up with a discovery based on, let's say, some statistical threshold of significance, and you say, I have found some signals, some treatment effects, some association. How likely is that to be truly so and not a false positive finding? So it tries to see what the impact of different factors would be, including the power of the study to discover effects that might be out there to be discovered, the number of investigators who try to attack that question, or similar questions, and do not join forces to just do a single analysis of all their data sets, but each one of them is trying to outpace the others, and Also the extended bias that may creep in and may turn some of the non significant results into significant results. So why these things are happening? Of course, can be due to multiple reasons and multiple influences on the research agenda, the research design, the sponsors, conflicts of interest, knowledge of research methods or lack of knowledge of research methods, sloppiness, sometimes fraud, hopefully not that common, but even that exists, so if you build their. Composite impact under most circumstances, where you can think of how research is done in most fields and most types of designs and most types of questions that we ask, the probability of a statistically significant discovery to be a true positive is less than 50% and in some cases, it's much less than that, especially when we talk about a very lenient statistical significant threshold of a p value of less than 0.05 which is what most fields traditionally have used. So I don't think that it would be surprising, and it is actually congruent with what we see with empirical results when we try to to reproduce, replicate empirical investigations. I want

Nick Jikomes 20:56

to talk a little bit about statistical significance and and what that means and how we define it and how it's used. Of course, you know, if you just sort of think in, you know, everything worked the way that we hope it worked. You know, scientists do scientific research. They submit their research to the peer review process, so other people that have similar expertise have to actually check their work, and if they agree that the work is is good enough, it gets published. And then, of course, after it's published, it's hopefully going to be open to anyone to look at, and then other people can look at it and and discover the results. So for example, journalists will often report on the scientific results that are published in the peer reviewed literature for a lay audience. Obviously, that work is published in journals by technical experts. If everything's working as it should, then, you know, a journalist is going to look at it, and they should be able to take it at face value, right? They should be able to say, this was published in Nature. Therefore, it's a very reputable journal. It's been peer reviewed. Two or three experts have looked at it and checked it, and it says, right in the paper, statistically significant, of course, as you know, and anyone who's in the research world knows, of course, that's not, not exactly how things work. Just because something's published doesn't mean it's rigorous or replicable. Just because something statistical statistically significant doesn't mean it's actually true for people who don't know what a p value is. Can you explain that concept and what we mean by statistical significance in normal speak?

John Ioannidis 22:28

Okay, I think that any effort to simplify these concepts probably will lead to a wrong definition, unavoidably, to some extent. But one simple way to put it is that it is one way to provide some sense about what would be the probability of finding some result that is as extreme as we have found, or even more extreme, so deviating from the the null, finding, you know, like there's no difference, or no treatment effect, or no treatment benefit, or no harm, or no signal, no association, so we find some signal, what is the probability of finding that signal, or even a stronger signal, if actually there is no signal. And if actually you know that is kind of silenced, there is no bias. This is taken out of the the usual way of of thinking about it based on on what we have done so if we do see a small p value, this doesn't mean that we have a very small chance that this is not true, because it much depends on what are the starting chances that there is some signal. And if we're looking in a field that has no signals to be discovered, and let's say we were very unlucky, and we selected to go into a scientific field where there's nothing to be discovered, then no matter what the p value is, we're stuck, you know, we will get some very nice looking p values, very small p values, but they will mean nothing. And it's always a challenge to understand and calibrate what is the field that I'm working on and how much is there to be discovered? There now a paper being published in a peer reviewed journal is is better than not being published in a peer reviewed journal, and we know that peer review does improve scientific papers and roughly about a third. 100 of scientific papers become better through peer review, maybe 5% become worse. The editors and the reviewers managed to make the paper worse.

Nick Jikomes 25:09

And how do we specifically know that? And what exactly do you mean by better or worse? So

John Ioannidis 25:13

there are some studies, for example, like some that Sally Hopewell and others have have done, where we had access to the original version of a paper and the final published version, and also the reviewer comments, and let's say independent scientists, try to arbitrate very carefully to see whether the interventions of the peer reviewers and editors improve the paper in some tangible way, not improving commas or full stops or just a little bit of the esthetics of the language. But you know, some equation was wrong and it was corrected, or some data had been miscalculated and that was fixed, or some information was missing and that was really added. Conversely, some important information was there, but it was removed, which makes the paper worse. So roughly, based on this type of empirical evaluations, we know that, as I said, about a third of the papers get better, about 5% get worse, and about two thirds are not really touched materially by by peer review. And now these that are not touched materially does not mean that they were perfect. It's very unlikely that they were perfect, based on what we have seen in terms of biases and errors and misrepresentations and flaws, very likely most of them have problems that simply were not detected by peer review. Reviews are over fatigued. They have very limited time. They don't get paid for what they do. Sometimes you ask 20 people to get a couple to agree to write a review that is usually 100 words or 200 words, and half of that is not really saying anything. And maybe there's a few points that might be making a difference at best. So without saying that peer review is a bad idea, it's not a panacea, and a lot of flawed papers will go through the system. There's also lots of journals that have little or no peer review. There's predatory journals. There's other journals that have very little peer review, they will publish practically anything or almost anything. So having a paper published does not necessarily mean much. Having it published in a in a prestigious, highly competitive journal like Nature or Science in basic science, or new journal medicine or Lancet in in medicine does not mean necessarily that it is more credible. I think that this is a misconception. There's opposing forces here. I think that some papers that end up in these top tier journals that have the highest competition for their pages and acceptance rates of 5% or less, I think that, yes, they may attract some of the best work, some of the most kind of anticipated and expected and work that a lot of resources and a lot of thinking and a lot of brain power have been invested, and everybody's waiting for their results, and it is a well done study, very well designed and very well supervised, and lots of people, in a way, have reviewed it beyond the couple of peer reviews who see it at the end. So yes, that that type of research probably is going to be more credible on average, although sometimes even that may not be the case and if there's a conflicted agenda, for example, behind it, so it has to be seen on a case by case basis. But then the majority of papers that would end up in these journals are not necessarily these widely expected kind of central studies that hundreds of people are reviewing somehow and putting effort into. They're mostly studies that tend to have a very strong surprise factor. They are studies that find some result that is extreme, really novel or seeming to be novel, and effect sizes are larger than than average, and and in that case, there's two possibilities. One is that, yes, this is a discovery of something that is really so extreme and so so nice and so beautiful, large effect. And the second possibility is that this is winners curse, that even simply by chance, someone has found this very strong signal. But actually the real signal is much smaller, if not nonexistent at all. And I. If you think in this way, if you have a small study, that there's million such studies being done, and it gets an average result, it's not going to make it in these journals. Yeah, yeah. The only chance that it will make it in in one of these journals is if it has found something that's really extreme.

Nick Jikomes 30:18

So if, by chance, something that's not that extreme is studied a bunch of times. The person who finds the extreme result is actually going to be biased to be the one who publish on publishes on it first. Exactly

John Ioannidis 30:28

so. So you have a winner scarce that is likely to affect these journals far more than the average, let's say good, respectable journal that is willing to publish more average type of results, and we have documented that we have run empirical analysis looking at topics that have been assessed by studies that were published in top tier journals like New England Journal medicine Jama and Lancet, and also in specialty journals on the very same intervention the same type of question. What we saw is that when we're talking about large studies, then the effect sizes are about the same in the top tier journals and in the specialty journals that published on that topic. But when you have small studies published in these top journals, their results are hugely inflated compared to similar studies asking the same question and published in special journals, and these inflated results typically just get washed away when someone looks at them again and realizes that that's not such a huge benefits, such as huge signal as what has been published now. So you know, nothing is perfect. No journal is perfect. Guarantee that what is published there is going to be impeccable.

Nick Jikomes 32:00

You mentioned, there's a lot of places we can go with this, I guess, I guess the the essence of what you were, you were saying just now, for people to understand is, you know, statistical significance is an important concept. It has to be computed the right way. There are assumptions to go into it, which may or may not be justified. The sample sizes and the effect sizes matter. I'm not sure we need to go into too much technical detail on those things, but just for everyone listening, the size of your samples, how good your data is when it's collected, matters, the assumptions, being cognizant of the assumptions that are made to do the appropriate statistical tests, all of that matters. And sometimes people do it correctly and rigorously, and sometimes they don't. It's a mixed bag. All these things matter. I want to talk about next something that you mentioned, which is that there are a there's a lot of journals out there. There has been a massive proliferation of the number of journals over the years. And you also mentioned that some of these journals are predatory journals. Let's take those one at a time. How many journals are out there? Can you give us a sense for that, and why are there so many?

John Ioannidis 33:07

There's different estimates, but let's say that there's more than 50,000 journals out there. 50,000 50,005 zero 1000. Yes, so yes, indeed, that's a huge number, of course, that covers all science, and there's a very large of scientific fields. So it could be that, if you break it down per scientific field, for some fields, there's only one or two that really published the majority of papers in that particular domain. And some other fields, there's just a very large number that could accommodate the literature that is coming out of scientific efforts. So it's not that every paper can go to any of these 50,000 journals. Some papers have a more limited target journal space compared to others. But clearly there's a lot and and we don't even know the full number, because, as I said, there's some that are less visible. There's many that are predatory, which means that they are just small, or more than small businesses that you pay some money and the paper will get published practically with with no peer review. There's a growing array of mega journals that publish 1000s of papers every year. The typical journals in the past used to publish 100 200 papers every year, some of them a bit more, but not much more. And mega journals by definition, they publish more than 2000 papers every year. Some of them publish more than 20,000 papers every year. So they have peer review. Their acceptance rates are not 100% unless they're predatory, but they may be accepting 3040, 50, 60% of of what they receive. And. Then you have a very wide array of all sorts of specialty journals and with different business models, with a very large publisher industry that is making lots of money out of that process. So the publishing industry is roughly $30 billion annual turnaround.

Nick Jikomes 35:32

That's 30 billion in revenue per year, in revenue,

Speaker 1 35:35

and the profit margin for the big publishers is in the range of 30 to 40% which is, if not the highest, among the highest, compared to any other really legitimate enterprise.

Nick Jikomes 35:48

That would be a product, you know, that would that's as good or better than the profit margins of like apple.

Speaker 1 35:54

It is. Yes, it is so Elsevier, for example, and Wiley have better profit margins than Apple. Yeah,

Nick Jikomes 36:01

and those are two of the larger publishers that own, journals that people probably heard about, brand name journals like sell

Speaker 1 36:09

Exactly. There's five publishers that publish the lion's share of the scientific literature, and now we have these mega journal publishers that are also pushing the frontier of one of them is called frontiers actions,

Unknown Speaker 36:26

yeah,

Speaker 1 36:29

that are, you know, pushing the numbers of papers. So,

Nick Jikomes 36:33

so 50,000 journals, give or take, doing 30 billion with a B, dollars in revenue per year with profit margins that are among the highest of any private company in any industry. So this is a big business in every sense.

Speaker 1 36:51

It is, it is, and one wonders whether we're getting what we pay for.

Nick Jikomes 36:58

Yes, and why is this such a big business, how does this connect to so so obviously there's issues here. On the supply side, there are a lot of academics doing a lot of research, and so there's a lot of potential papers to publish. There's a demand, there's a demand side to this, and we can talk about that. And then there's also the issue of costs, and the costs for these publishers, it would be an interesting area to talk about, because you already mentioned previously that. So we're talking about publishing papers that go through peer review. So peer review is a key component of the manufacturing process, so to speak, but the peer review is done for free. In most cases, that is that true.

Speaker 1 37:40

That's true. There's very, very few exceptions where peer review is is paid. And even those, the amount being paid is so little that it doesn't make a difference. Like, you know, $5 $10 you know, maybe $50 I'm involved in one journal that was launched recently, where the model is to pay reviewers $500 and it's, it's an experiment. I don't know if it would work to improve the quality and rigor of of the peer review process. Yeah, just

Nick Jikomes 38:11

Martin cold, or if that's the journal that you're referring to, yeah,

Speaker 1 38:14

yeah. So, so I see it as as an experimental effort. We don't know what is the best model for supporting peer review. To be honest, we have run some randomized trials, not as many as I would wish, but there are some randomized trials trying to randomize different modes of review, like open peer review, blinded peer review. There's some trials of training for peer review. There's some that look at having a statistician, peer reviewer, look at at the papers, the the effects tend to be modest or even null. I think the the clearest signal would be for having a statistician look at at the papers that clearly seems to improve the subsequent versions that get published. But then the the challenge is, how? Where do we find statisticians to look at 7 million papers that are published every year? Right? There's not that big a workforce of statisticians and methodologies. They have their own things to do and work on. We cannot hope to engage them to review 7 million papers. So there's there's lots of incentives to publish. I have nothing against publication and productivity. In fact, for many decades, I have been struggling, as many other scientists, against publication bias and non publication of negative results. So I would be the last to say that we should not publish. We need to publish. We need transparency. We need openness. We need to communicate, then we need to communicate with even more transparency and with more details about what we do. But it is, it is a huge business, as you say, and it is exploited by those who are centrally placed to make profit out of it.

Nick Jikomes 40:27

Let's talk about that a little bit, because, I mean, people in this world often know, at least to some extent, how some of this stuff works. Almost no one in the research world who I've ever met when I was in academia, almost no one is satisfied with the way peer review works for those listening, you know, when you're in the academic world and you're doing research, and you know all of your friends and colleagues are doing research too, when I think about every single time I heard someone talk about, oh, I've got my paper. It's under review right now. Oh, how's it going? Almost 100% of the time the answer is some version of it sucks. It's terrible. The reviewers aren't doing it right. Blah, blah, No, I've never heard anyone once in my life say, you know, the reviewers had some critical things to say, and they're absolutely right, and I changed my mind, and, you know, the paper is not getting in, but I learned a lot. And then, of course, there's sort of the business side of this and the exploitative nature that many people would say that that the big journals operate with. Can you start to talk about how this became such a big business? How? Where are they generating all this revenue from? Is it from subscriptions? How is this such a big profitable business?

Speaker 1 41:40

So just to close our discussion on peer review, I think that there are peer reviews that are constructive. I don't want to be so dismissive. Personally, I have had peer reviews on many of my papers where I felt that the paper did improve. I feel that I really benefited from that input, sometimes even very negative input. But goodness, you know, thank you very much. You know, you pick some error that I had made and I hadn't noticed, and now I can, I can fix this so, and some papers do get substantially improved. So yeah, so I'm uneasy about just discrediting peer review, and I do notice, nevertheless that it is a very suboptimal system. It leaves lots of possibilities of things to go wrong. Now the the publishing system has evolved over the years, and it has become more massive, both in terms of number of journals and in terms of number of papers, and also it is highly hierarchical, so journals have very strong prestige factors attached to them, and this has been largely been a journal impact factor business, which basically looks at the average number of citations in The first two years after the publication of a paper, it is a flawed metric by many different ways of looking at it. There have been many efforts to try to dismantle it, to say, No, we're not going to look at it. But goodness, everybody's looking at it, yeah, and that reinforces both the prestige ladder, and also the gaming of the system. So people, journals, editors, publishers, are just struggling to prevail in a in a gaming system that is not necessarily aligned with with better science, with better research, it is trying to optimize some numbers that are surrogates, and most of the time they're very poor surrogates, or capture very little of the essence of what we should be interested in. How do you get rid of that? I think that we need to experiment. We need to try many different ways. Let many flowers bloom and see whether some of them may be more successful. It is not like a uniform system. We have new ideas. We have new concepts. We have pre prints, for example, where people can post their work practically for free, a model that has been very successful in fields like physical sciences. And I think that now it starts becoming more successful, or at least more popular in biomedical sciences as well, we have models like E life, where journals are seen mostly as a platform to perform review, hopefully good review, and then the authors may decide to publish their papers regardless of what reviews they get, but they will have the reviews along with with what they publish. And I think that that we need to study peer review rigorously to to understand what it does, what it does not do. Also study who is doing what you know, some very fundamental questions like, what kind of editors do we need? For example, most of the high profile journals, they don't have editors, people who are highly credential scientists, who are in the top of their fields. They have professional editors who they only have experience as editors. You know, they may have done some background training, you know, maybe got a master's degree, some of them maybe got even a PhD, but they never, kind of did a lot of scientific work themselves. They're editors, and that's what they do. Is that better? Is it worse? I mean, they will defend the model as being more objective, that they don't care that much about who is submitting and what they say they're kind of remote. But, I mean, how remote can you be and still be relevant to a field, right? I think that that's a challenge. So, so even very fundamental questions, things should be open to scrutiny and assessment in terms of, do they improve or do they make things worse?

Nick Jikomes 46:17

But, you know, just just getting down to the nitty gritty here to make this super concrete for people. So we got a $30 billion industry. So there's a lot of revenue being generated with high profit margins by these big name journals. Like, where is the money coming from? Like, like, what is the specific revenue that's being generated? Is it? Is it a high cost to publish that the researchers are paying? Is it subscriptions to the journal that many people are are subscribing to, or is it so

Speaker 1 46:44

some of the revenue comes from subscriptions. A large part of the revenue comes from subscription, which means that universities and other institutions need to pay these publishers a lot of money in order to be able to have access to the papers that their researchers are producing,

Nick Jikomes 47:05

and how are those prices determined? Is there sort of a free market where the prices are set by supply and demand? Or, well,

Speaker 1 47:13

in theory, it is a free market. But as I said, there's really very few big publishers, and that creates a situation that is probably more close to a cartel. So if you really had 50,000 players that are independent, truly independent, probably the prices would go down. But we don't have 50,000 players. We have five, you know, maybe six, seven, if you add also the new mega journal publishers that are rising quickly, and therefore, somehow, the prices have been staying at at pretty high levels and allow for these large profit margins for For these big players. So it's, it's a it's mostly, how do you kind of change that cartel situation? And to a large extent, it is about subscriptions. There's an extra layer, which is the publication cost, the article processing fees, where typically to publish your work in open access, and some journals are entirely open access. Others are moving to become entirely open access. Some are mixed, but people may still wish to have their work published as open access. They have to pay several $1,000 to do that, and in some of the top journals, they need to pay more than 10,000 more than $12,000 you know, nature, for example, to have a single paper published.

Nick Jikomes 48:45

So so just to be clear for those listening, if, in some cases, if you see a paper out there published in a place like journal, a journal like nature or another big name journal, if it's open access, which it may or may not be, if it's open access, that often means that the scientists who did the study, who led the study, paid 1000s of dollars just to have it published. So this open openly available, exactly

Speaker 1 49:09

yes, which means that that money is removed from that scientist budget to do research. You pay research or salaries or consumables or or other types of resources and effort. Often it would come from the funders. If it's public funding, it would come from public funding, meaning taxpayers. So there's a lot of double dipping here that people who profit from the system. They're, they're gathering money from from different sides and different steps in that process. Yeah,

Nick Jikomes 49:51

you're actually asking scientists to do peer review for free. It's like, it's like, literally having an employee that you don't pay, and then they have to pay you to have it published. And. Pay more if it's open access. And then it actually becomes even more absurd, you might say, when you start to think about it in terms of well over time, right? We used to have physical journals. Used to physically buy a journal, and they would have to physically print it. But that's really not the case so much anymore. So the cost of the journal should be going down in that sense, and yet, I imagine these fees have not adjusted,

Speaker 1 50:22

indeed, and if you have a cartel, it will not get adjusted everything. It will remain at a very high profit margin moving forward, no matter what improvements we have in technological transformation of the of the publishing enterprise, which indeed now can cost less because we can just do everything online, we can have things done much more efficiently. Yeah, so it is a big problem. It is a big black hole, in a sense, in in the vicinity of science. It is absorbing a lot of of scientific resources. And scientists, by default, are just offering more to that black hole. We're offering free peer review, as you mentioned. We are offering funds to make a paper open access. We're offering funds from our universities to to pay for subscriptions

Nick Jikomes 51:19

and so forth. So there really is probably, like a collective action problem here, where, if every scientist, every researcher, just simultaneously demanded, hey, we need to be compensated for a time, even if it's modest, because that's just what we're demanding from you, something would happen. But of course, you need a critical mass of people to do that, and for various reasons, that's very unlikely indeed. Yeah, and

Speaker 1 51:43

perhaps we sh