You are a very busy, very important professor publishing very important work. Do you
a) just publish the code and data along with the paper because you know your work will survive close scrutiny and you have better things to do
b) spend your time handling individual data requests, negotiating over the scope of data shared, and re-describing individual analysis steps that are unclear in the methods
4 Boosts
0 Likes
@jonny I made _gigabytes_ of raw data available when publishing my Internet measurement studies _in the early 2000s_. Back then, deciding to download gigabytes of data was a commitment! It can absolutely be done, and I applaud communities (like many ACM SIGPLAN venues) that make this a requirement for publication -- either show an artifact or explain (convincingly) why you cannot.
I have seen scientist code so gnarly, data formatting so chaotic, that you simply cannot [edit: faze*] me. I know how that goes, grad students get no training in any of that. That's my baseline expectation, your code and data simply can't be bad enough that I would care about how bad you think it is - I would rather have an incomprehensible jumble of notebooks and .mat files than nothing at all, and if its in a repo I can pull into, I'll help you clean it up along the way. There is no downside to posting code and data.
*but also you cannot phase me and cause me to shift into a different state of matter or shift my relative synchronization with anything either
If it didn't mean that I would become a pariah and be the end of my career, I would have submitted a paper with absolutely zero experiments or data underlying it where the paper is the sole artifact and you have to just take my word for it years ago just to prove a point. Without code and data, there is no difference between high-concept science fiction with very strong Academic Genre trappings and actual science. Convincingly faking a paper at the figure level is easy, convincingly faking a paper all the way down to the raw data and the commit history in an analysis repo is hard. You are making a claim about how the world works, and it is your responsibility to convince me, not my responsibility to believe you.
I hope its clear that all of the above does not refer to cases where code and data is not relevant or important to the interpretability or believability of a work, can't be shared, or otherwise is not "an empirical work that involves the collection, analysis, and interpretation of a large sum of empirical data, of which only summaries are presented in the paper." I am not demanding all my qual folks turn over their entire interview transcripts including the confidential participant information sections. Maybe I am over broad with "papers" in the statement "I dont believe papers without code and data," but its sort of a chore to accompany every statement with an appendix that clarifies all the things I do not mean that I usually hope is taken care of by context.
@jonny I’m all for the practical benefits of data and code sharing, but in some fields there are hard ethical and legal boundaries, i.e. sharing participant-level health data. Some friction exists for good reasons. Also, reproduction of figures by rerunning published code is not the same thing as replicating findings to validate truth claims, which is far more important IMO. If your field is non-experimental, like you characterize thousands of stone tools, open code won’t necessarily help much
@jonny Obviously neither, because the paper speaks for itself to the correct audience?
@jonny it's faze, yo
@devlin
What on earth, faze came first??
phase(v.) ... Earlier as a bad spelling of faze.
https://www.etymonline.com/word/phase
Learn something new every day
@jonny Same. After the abstract, it's ctrl+f -> "package" -> "library" -> "git" -> "repo" -> "code" -> "http". If nothing hits, I'm moving on.
@erc_bk i just want to look at the actual thing you did by reading the code in parallel to the prose! it's a normal thing to want!
@jonny I want to be able to use it. Could I take what's in some of these papers and code it myself --- depends. I'm not wasting my time to find out though, because chances are it's an incremental improvement on whatever best practices method/algorithm is already out there.
@jonny I'm more suspicious if I don't detect in the text of the paper some genuine attempt at trying to check their idea might be wrong (comparing against multiple baselines, null hypotheses, parameter sensitivity checks etc). Honestly, doing all of that is so rare that if someone does it I'm inclined to believe them even without the code. And if they don't do those things, I struggle to believe it even with the code.
@neuralreckoning there's often a great deal of creativity that goes into constructing a null, especially since most neuroscientists don't have a strong statistical background and can't tell when some permutation shuffling test is not the relevant null to test against. but yes agreed with the general sentiment - the most trust-inducing thing you can do is strongly argue against your own ideas, and i would take that further: allow others to argue against your ideas directly on the work itself via direct annotation and linked responses
@jonny completely agree. I wrote an article around this idea actually:
https://thesamovar.github.io/zavarka/transparent-scrutiny/
"By sticking their neck out and subjecting themselves to increased scrutiny, they increase the value of their work if it survives this more rigorous process. They can do this by allowing their work to be openly reviewed in an ongoing way. They can make their raw data and code open. They can even go really far beyond even the most rigorous of proposed standards, and invite scientists who have doubts about their results to come and visit their lab and see how the data was collected, and then publish their own reflections on what they saw. These are all signals that the authors believe in their results. If others don’t find major problems despite everything being open to inspection, that should hugely increase our trust in the results and reflect well on the authors.
This is what would happen to science if we moved towards a culture that explicitly recognised the value of thorough and transparent scrutiny."
@jonny I'll try to answer, although it was probably a rethorical question but I think it's useful to see the point of view of experimentalists..
First, what data are we talking about? The widely-processed, very simplified tables that you use at the last step of analysis just to make your figures? Or the raw data that hasn't been spike-sorted, checked, or corrected when needed? Or something in between?
If we're talking about the first one, sure, it doesn't cost much to share it, but it's also very easy to fake, and checking that you can remake the figures from those is not going to catch any kind of error, since the author would have very likely noticed if there was an error at that stage. Just looking at the figure-making code would probably be more useful to catch errors. So: does it really change anything to publish that data? I'm not sure.
If we're talking about raw data, not only that's going to be much more difficult to share (e.g. size might be a problem, storage might have a cost, reuse might be a problem without also providing an explanation of all the processing steps done on the data), it's also going to take the time of the person dealing with the sharing. There is the added problem that the author might want to analyse more aspects of the data and sharing exposes them to being scooped. Of course that wouldn't be a problem if people analysing your data were sharing authorship with you (but they generally don't) or if the system wasn't stupidly geared towards encouraging high-impact, non-collaborative publications.
Sharing something in between raw a final will have problems from either stage, so once again: is it worth it?
So, you ask:
"Why would you pique suspicion in your reader and signal something all may not be as it seems by not publishing the data? Why would anyone believe a paper that doesn't have code & data?"
-> it might be worth more to the author to use their time & money for something else. Plus, do we really think that scientists who don't share data are faking the data? I don't. Do we think that scientists who share the data never fake the data? I don't either.
If a scientist wants to bend results some way there are many ways to do this other than faking data (p-hacking, cheating at the experimental stage, hiding results, over-inflating results..). I think it's better to read the reasoning of the paper, the methods, what kind of analyses were done, and look at the individual data distributions in the figures (these should definitely be shown).
Overall: obviously, it is better, if everything else was equal, to have as much data shared as possible. Unfortunately, once you take other parameters into account, there is a question of cost-benefit and it's not always in favour of sharing the data.
PS: it is almost never the professor that actually deals with data-sharing, but I'm sure you probably know that
@elduvelle
yes i am talking about the raw data. there is a gradient there too, and given the state of the tooling it's often very hard to reproduce the "first step" of the analysis like extracting traces/spikes from raw recordings because it involves a lot of manual clicking around that isn't easily captured in code, and i have a bit of grace there since that's not the author's fault, that's the state of the tools. but yes, the complete, raw data should be shared including videos and raw recordings. even if that 'first step' cant be perfectly and automatically replicated, it's usually possible to get close enough for replication.
e.g. size might be a problem, storage might have a cost, reuse might be a problem without also providing an explanation of all the processing steps done on the data
again neuroscientists are never given training in this, but the data is almost never as big as they think once the data are properly encoded and compressed, and repositories like zenodo and dandi are effectively free-forever unlimited storage archives, as they should be. i'm not expecting perfectly curated analysis pipelines and am expecting to have to fight through the thicket a bit, that's all completely fine -- as long as it's there!
There is the added problem that the author might want to analyse more aspects of the data and sharing exposes them to being scooped.
I've never seen or heard of this happening, where open data caused some future result to get scooped, but a) it's fine to just say that in a data availability statement: "the data and code will be made available in 3 months after we finish up a secondary analysis of it," b) you have a huge head start so someone would have to be very quick to do that, and c) i actually think that's a good thing: it's ridiculously wasteful (and cruel, in the case of animal research) to need to re-do every experiment every time someone wants to investigate a phenomenon. someone re-analyzing and publishing on open data is a good thing. if i was an editor or reviewer, if i saw such a paper, i would require that the source of the data is properly cited and ask if the secondary authors had made contact with the primary authors and what their view on the secondary paper was. if the primary authors are like "hold up we're writing this exact thing" i'd suggest merging efforts.
it might be worth more to the author to use their time & money for something else
my standard is mere existence, i don't have an estimate of how much time and money it costs on an average case, but given that standard i would suspect "not much" since all im' asking for it "give it to me however you have it"
Plus, do we really think that scientists who don't share data are faking the data? [...] If a scientist wants to bend results some way there are many ways to do this other than faking data/
this is exactly the point. the argument isn't that the data is fake, but that the fine details of the analysis are cherrypicked to tell a particular story, and if the data is not shared, my first thought is that the authors don't want me looking too closely at it or else i'll see all the little tweaks and choices made in service of that story. the best way to demonstrate that your conclusions hold up to scrutiny and are not the result of some carefully picked garden path analysis is to share the data.
it is almost never the professor that actually deals with data-sharing
usually, yes, but oddly i am in the middle of a request where that is the case.
@jonny yeah, I agree with 80% of your points, but I still think the main obstacle to generalised data-sharing is that there is no obvious, predetermined path showing how to do it, and without that obvious path, and despite what you say, preparing the data and actually uploading the data to an online platform does take a lot of time. I'll explain my thought process because I would also like all neuro data to be shared and all labs to be able to easily and efficiently do this, but this is not currently the case.
First, the size: the raw data for the most recent dataset I collected is about 2.7Tb and even just copying it from a disk to a networked "filestore" took several days where I couldn't close or restart my computer. So, uploading that online would be a pain. I am sure that compressing is a good idea but it would also hog my computer's resources for a long time and I don't know what kind of compressing is best for my data, which has video files, proprietary ephys files, text files and some other random stuff.
Then, where to upload it to? In the past I've used EBRAINS which was part of the human brain project but I'm not sure now if it works for non-HBP datasets.
little parenthesis to explain that process: in that case, I didn't share the raw data, only an intermediary dataset which contained only the processed data used in the paper. It took a while to organize the data nicely (which was done by a collaborator) and then deal with the data curation and uploading process. The data has been used by at least one other lab for now, who contacted us to ask about how to use it but didn't propose any kind of authorship. Overall, I am glad we shared it but it did take time and effort, with no immediate gain for my career, and that's not how I like to think but we all have to take that cost/benefit into account.
Is there any easier way to upload data? I looked into the repos you suggested:
- Zenodo says it doesn't have a size limit but at the same time has some kind of 50gb "packet"/ account size limit, you can upload different "packets", but that obviously complicates things.
- DANDI may not have a size limit (?) but it looks like data has to be in a specific format, like NWB, and obviously converting things to a specific format takes time, effort and expertise, so that's another cost. It also seems that you have to input some medatada info, at least I had to do that with eBrains, and it did take several hours of work for a relatively simple dataset. Of course it makes sense that the dataset is more useful with the metadata and I'm glad they ask for it but it again adds to the cost of sharing.
is there any other way to easily - and at no or low cost - dump a huge dataset for sharing?
scooping: let's keep this one for another discussion
Conclusion: having done it once, I don't think it's fair to say that data sharing is quick and easy. It has a cost and if we want more people to do it we have either to lower that cost or increase the benefits.
[Post edited because I clicked send too early]
@jonny @elduvelle I might have been unlucky, but every time I've asked for data to the authors of a paper who said that data was available on request they either ignored me or the data had issues ...
Anyway, two interesting reads about this
Why don't we share data and code? Perceived barriers and benefits to public archiving practices - Gomes et al 2022
https://royalsocietypublishing.org/doi/10.1098/rspb.2022.1113
Why data is never raw - On the seductive myth of information free of human judgment
https://www.thenewatlantis.com/publications/why-data-is-never-raw
@jonny "why wouldn't you just dump whatever you have on zenodo and remove an easy avenue for someone to dismiss your work?"
Because the company the data is from will sue you out of a job for... ever.
I know you will not like this answer, but I've been there, done that. Working with *some* insurance and/or finance data will make you jump some hoops no scientist should have or want to jump, but it is my conviction that in some cases there *is* something to learn from the exercise.
@ftranschel
OK sure, yes, caveats apply like if your data is sensitive PII/health data or proprietary licensed data, but that's not what I'm talking about.
Here is a view from 1979, at https://pubs.acs.org/doi/pdf/10.1021/ci60017a012
"We restrained from giving all small details of the algorithm to keep the paper concise, working under the assumption that persons interested in the details would contact us directly."
@jonny Heh, I have had this conversation at work. And yeah, everyone wants to share data but almost no one wants to share code.
I have even had to fight writing which packages were used for stats or transformations in the methods 😉
@jonny c) ignore all requests
My flow chart for reading a paper is like
Abstract -> Data availability
Why would you pique suspicion in your reader and signal something all may not be as it seems by not publishing the data? Why would anyone believe a paper that doesn't have code & data? You claim to have gotten a bunch of data and done a bunch of things to it in code, and you had to have done that in an organized enough way to yield a paper, so even if its not pretty, why wouldn't you just dump whatever you have on zenodo and remove an easy avenue for someone to dismiss your work?
Edit: clearly, caveats apply like if the data is privacy-sensitive health/PII or data under a strict license where you'd get sued if you post it. That's not what I'm talking about, and its fine if you explain that and post whatever derived data you can ethically and legally post. I'm talking about most primary research in my field which has no such limitations.