Can one demand to see code used to generate an article?

I believe you are asking the wrong question.

You are asking: Can one demand to see code used to generate an article?

To which I would answer: Assuming you're in a country with free speech, you can "demand" the code. Just make sure you do not threaten the author or otherwise break the law with your "demand". However, you are unlikely to be successful in forcing the author to share the code with you. A "demand" for code to me would come off as arrogant and I would ignore or decline a "demand".

I think a better question is: How can I get the code from an article I am interested in looking at or using?

To which I would answer: It depends upon your subfield and why the author will not share her or his code. Some subfields now require/expect authors to share code and data. For example, the American Statistical Association recommends sharing code to create reproducible results. Here are the steps I would follow to get the code from an article I wanted:

  1. I would first try reaching out to the author and asking nicely if she or he would be willing to share their code with you.
  2. If the author declined, I would ask the journal editor if they could get the code to support the conclusions article because you wish to reproduce the article's results. Also, some journals now have open code/open data requirements. Check the policy of the journals the author published in.
  3. If you end up in hostile situation with the author, and you think they are being misleading based upon their findings, I would contact the research integrity and ethics office at the author's university (e.g., Illinois or Rochester).
  4. If you truly suspect scientific misconduct as the reason behind not sharing code, you could take this farther by writing a response article questioning their findings and state that they refused to share code with you. For example, watch how this author investigated a cancer scandal. The end result was prison time for the fraudulent scientist.
  5. If the author's funding agency requires open access to code, you could contact them and request they force the author to share their code. For example, all data and code development funded by the US Government is required to be open.

I would suggest working with the intellectual property office or similar office at your institute if step 1 fails. They can guide you on what options, if any, you have.

Last, you might need to accept that you cannot force someone to share his or her code with you.


The answer to your question is very simple, but it is not the one you are clearly hoping to get: No, you cannot demand (and hope for your demand to be met or to be seen as even remotely appropriate, that is) to see the code the author used, either from the author or the journal where they published.

The reason is also simple to explain. Academics don't make "demands" of each other. The way academia works is that each researcher does their research and releases their work in the way that they think is appropriate, and the community evaluates them and gives them credit for it as it sees fit. Do I think researchers should release their code? Absolutely! I also think journals should make that a condition for any work whose validity depends on the code to be published. But I don't get to decide how journals and other researchers conduct their business.

On the other hand, I do get to decide whether I trust a result described in an article; whether I recommend papers I am asked to referee for publication; who to invite to give a talk at my department or at a conference I'm organizing; who to write a recommendation letter for for their tenure case or job application, etc. And I am also free to express my opinion to others that a result can't be trusted if it can't be verified. That is precisely how things should be, and the result is that people who don't release their code, and their results, will (generally speaking) have a lower level of credibility and lower reputation than those who do.

So, unfortunately you need to accept that you cannot make "demands" or force an author to give you their code when they don't want to. But you can, and should (if you feel that strongly about it), make your voice heard in your research community that you believe releasing the code underlying scientific work is a good practice, should be the norm, and should be a precondition to having the work refereed and/or published. If enough people express such opinions, we will start to see some progress in this increasingly important area.

Edit: here's another way of looking at this issue. Availability of code is just one of many factors that make some research products more useful than others. So I think it may help to frame the question more generally: If an author published a paper that describes their research results, but the description has a flaw that makes it less useful than it might have been otherwise, do I as a reader have a right to demand that the author fix the flaw? For example:

  • A mathematical proof is presented with "some trivial details left to the reader as an exercise" (the author forgets to mention that filling in those details takes 2-3 weeks of frustrating calculations, at the end of which the diligent reader is still uncertain if the result is true);

  • A paper is written by an author who doesn't speak English well, to the extent that reading the paper becomes difficult and in certain parts the author's intent seems impossible to decipher.

  • A physics paper cites certain results from the literature in a vague and imprecise way that makes a mathematically inclined reader wonder which of the claims in the paper are theorems, which are conjectures, which are heuristic derivations not meant to be taken literally, etc. A look at the cited literature does nothing to clear up the confusion.

  • Etc etc.

The answer in all of those cases is again a resounding no: it is completely inappropriate to demand that authors improve their papers (unless you are a referee, and even then you cannot demand it, you can at most make it a condition for recommending publication). Researchers already have every incentive to want to improve their papers as much as they can; if they don't do so, either they are incapable of it, or they can do it but decided that doing something else is a better use of their time. You as a reader need to have the maturity to accept that some papers you read will be useful and trustworthy, and others will be less useful and less trustworthy, and that the world is not going to bend to your demands and adapt any research study to your vision of what the ideal study looks like. That applies to code as much as to any other aspect of the research.

I'll add that after reading more of your thoughts in the comments you posted, I have to say your entitled tone is off-putting to say the least. The fact that you think authors should make available materials that you decided need to be released, and that if they don't do so that represents some kind of moral outrage, makes me personally think somewhat less of you (and it might make others reach similar conclusions). Since you are posting using your own name, I think this should worry you; hopefully it will also cause you to reflect about your unreasonable approach to this issue.


Reproducibility is an important issue, and there is a growing trend to make more raw data available. Many journals actually require the authors to deposit certain kinds of data in public repositories, for example when publishing a paper with an X-ray or NMR structure of a protein, it is very likely that the journal requires the authors to deposit those into the PDB database. There can be an embargo period, but in the end the data has to be public.

I looked at the author guidelines for Nature and Science, and found passages about code in both.

Nature:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

Science:

After publication, all data and materials necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data or materials must be fulfilled. Any restrictions on the availability of data, codes, or materials, including fees and restrictions on original data obtained from other sources must be disclosed to the editors as must any Material Transfer Agreements (MTAs) pertaining to data or materials used or produced in this research, that place constraints on providing these data or materials.

Your first step should in any case be to ask the author politely for the code. If that fails, you can contact the journal, if they have policies similar to the ones I quoted.

Tags:

Code