Brian Nosek
speaker
65 appearances
2 recordings
1 series
first heard Dec 2024
last heard Jan 2025
Brian Nosek’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
We feel like when we're in cultures that there is no way for any of us to change the culture. It's a culture. My God, how could we change it? But we also recognize that cultures are created by the people that comprise them. And the notion that we collectively can actually do something to shift the research culture, I think, has spread.
And that spreading has actually accelerated the change of the research culture for the better.
Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career.
I asked Nosik how he thinks this culture can be changed. So, for example, we have to make it easy for researchers to be more transparent with their work. If it's really hard to share your data, then adding on that extra work is going to slow down my progress. We have to make it normative. People have to be able to see that others in their community are doing this. They're being more transparent.
They're being more rigorous so that instead of us saying, oh, that's great ideals and nobody does it, you say, oh, there's somebody over there that's doing it. Oh, maybe I could do it too. We have to deal with the incentives. Is it actually relevant for my advancement in my career to be transparent, to be rigorous, to be reproducible? And then we have to address the policy framework.
If it's not embedded in how it is that funders decide who to fund, institutions decide who to hire, journals to decide what to publish, then it's not going to be internally and completely embedded in the system.
Yeah, so the idea is you register your designs and you've made that commitment in advance. And then as you're carrying out the research, if things change along the way, which happens all the time, you can update that registration. You can say, here's what's changing. We didn't anticipate that going into this community was going to be so hard and here's how we had to adapt. That's fine.
You should be able to change. You just have to be transparent about those changes so that the reader can evaluate. And then those data are timestamped together? Exactly. Yeah. You put your data and your materials. If you did a survey, you add the surveys. If you did behavioral tasks, you can add those.
So all of that stuff can be attached then to the registration so that you have a more comprehensive record of what it is you did.
It makes fraud more inconvenient. And that's actually a reasonable intervention. I don't think any intervention that we could design could prevent fraud in a way that doesn't stifle actual legitimate research. We just want to make visible all the things that legitimate researchers are doing so that someone that doesn't want to do that extra work has a harder time.
And eventually, if everything is exposed, then the person who would be motivated to do fraud might say, well, it's just as easy to do the research the real way. So I guess I'll do that.
So in the standard publishing model, I do all of my research. I get my findings. I write it up in a paper and I send it to the journal. In that model, the reward system is about the findings. I need to get those findings to be as positive, novel, and tidy as I can so that you, the reviewer, say, OK, OK, you can publish it.
That's dysfunctional and it leads to all of those practices that might lead the claims to be more exaggerated than the evidence.
The registered report model says to the journal, you are going to submit, Brian, the methodology that you're thinking about doing and why you're asking that question and the background research supporting that question being important and that methodology being effective methodology. We'll review that. We don't know what the results are. You don't know what the results are.
But we're going to review based on, do you have an important question?
Exactly. And the key part is that the reward, me getting that publication, is based on you agreeing that I'm asking an important question and I've designed an effective method to test it. It's no longer about the results. None of us know what the results are.
Yeah, so the commitment that the journal makes is we're going to publish it regardless of outcome, and the authors are making that commitment too. We're going to carry this out as we said we would, and we'll report what happens.
Now, an interesting thing happens in the change of the culture here in evaluating research because you said, well, if it's an uninteresting finding, do we still have to publish it? It turns out that when you have to make a decision of whether to publish or not before knowing that the results are –
The orientation that the reviewers bring, that the authors bring, is do we need to know the answer to this? Regardless of what happens, do we need to know the answer? Is the question important, in other words? Exactly. Is the question important enough that we need evidence, regardless of what the evidence is? And it dramatically shifts what ends up being published.
So in the early evidence with registered reports, more than half of the hypotheses that are proposed end up not being supported in the final paper. In the standard literature, comparable type of domains, more than 95% of the hypotheses are supported in the paper. You wonder in the standard literature, if we're always right, why do we bother doing the research, right?
Showing 1–20 of 65 · page 1 of 4
Next →