How we reached out
We brought the petition to arXiv through two channels: through Prof. Yuri Tschinkel at the Simons Foundation, and through the arXiv Scientific Advisory Council (SAC). We met with the SAC on October 6 and want to share what we learned.
First, a reminder: arXiv is a small nonprofit organization that maintains an enormous, publicly accessible body of scientific work. Its value to science is hard to overstate. We are grateful to the many people, a large number of them volunteers, who have built and sustained this common good, and to the Simons Foundation for its financial support.
The ability of bots, especially those used to train large language models, to scrape arXiv brings new opportunities and new challenges. Some of these were addressed in our meeting.
What we heard from the SAC
The SAC said it would bring the petition to an upcoming discussion with arXiv leadership. The main points raised during the October 6 meeting were:
- a.The SAC is an advisory body only.
- b.Some participants felt it may already be too late to stop bots.
- c.arXiv already has a file of instructions for crawlers (robots.txt; see background), but it may be out of date and may not cover many newer LLM crawlers.
- d.Much of arXiv's traffic comes from AI agents rather than from model training.
- e.Models can also be trained on arXiv content obtained in bulk from sites such as Hugging Face.
- f.Authors keep the copyright to their papers on arXiv. There have been informal discussions with Creative Commons about licenses that would prevent models from training on papers.
- g.arXiv wants to hear from authors on these issues and suggested the arXiv blog as a place to leave comments.
Our requests
In the spirit of the petition, and following up on the points above, we ask that:
- c.arXiv update its robots.txt file to block current AI crawlers as well as older ones, where needed.
- e.arXiv stop sharing data with bulk-download sites that make papers available for training ML models.
- f.Legal experts interested in working with us to protect arXiv authors from AI crawlers contact us directly at elmos@mit.edu.
- g.arXiv publish a post responding to the petition and related issues, where researchers can share more detailed and nuanced views, or provide a web form or email address for this purpose.
Each request is lettered to match the point above that it responds to.
If you have not yet signed, please consider adding your name.
Read and sign the petition