Hello All!
Mayhem Heroes Program Phase II completed on March 31, 2023 and we
integrated 1,954 Open Source Software projects and found over 110,000 defects!
That’s over 50 defects per project!!
And the best thing is that Mayhem is continuously running these targets and finding more and more defects autonomously.
But we are not done!
We are delighted to announce that we are launching Mayhem Heroes Phase III in April, 2023.
This phase will run until July 31, 2023. Here are the frequently asked questions about the program.
Frequently Asked Questions
Q: Which GitHub repos qualify for Heroes Program Phase III integration?
A: The requirements on the new GitHub repos that qualify for integration are the same as in Phases I and II, namely:
- It’s a public repository on github.com. Private repositories are currently not qualified.
- Has 100 or more stars.
- All code is open source and available in source code format.
- Code is considered Open Source if the entire code base is licensed under one (or more) of the recognized Open Source Initiative Licenses.
- The project is not already a part of OSS-Fuzz; you can find a list of integrated repositories here.
- The project has been active in the last 6 months.
- The target isn’t inappropriate (e.g. an integration of fetlang (NSFW) 6 was submitted and, despite meeting the criteria above, was rejected).
Q: What are the Heroes Program Phase III integration rules?
A: You can integrate a new Github repository with Mayhem or improve an existing integration. If you choose to integrate a new Github repo, the repo needs to meet the criteria as described in the question Which GitHub repos qualify for Heroes Program Phase II integration. You can also choose to improve an existing repository that was already integrated by another developer (you can find a list of already integrated projects here).
Q: How will my submission be scored if I am integrating a new GitHub repo?
A: If you are integrating a new GitHub repo, the repo needs to meet the criteria described in the question Which GitHub repos qualify for Heroes Program Phase III integration. The scoring system is as follows:
- Mayhem was correctly integrated and generated at least 10 test cases - 1 point
- Mayhem generated more than 100 test cases - 2 points
- Mayhem generated more than 100 tests/second - 2 points
- You created harnessing that exercises additional code - 2 points
- Mayhem found at least 1 defect in the fuzzed code - 3 points
1 point = $100; max payment $1000 (per repo)
Q: How will my submission be scored if I am improving an existing integration made by another developer?
A: If you are improving an integration previously performed by another developer, you will do so by writing a harness that exercises new code, previously not exercised by Mayhem. Ideally, this will result in new bugs being found, but it’s not necessary for scoring points. The scoring system for improving existing integrations is as follows:
- You created harnessing that exercises additional code - 2 points
- Mayhem found at least 1 defect in the fuzzed code - 3 points
1 point = $100; max payment $500 (per improved repo)
Q: Are there special rules for participation in this program for members of US Service Academies, such as USNA and West Point?
A: Yes, special participation rules have been shared with students at the military academies. If you have any questions about this, please email support@forallsecure.com
Q: Can I improve my submission after it’s been submitted?
A: You can continue improving your submission until you get notification that your submission is accepted and ready for payment.
Q: Can two people improve the same integration?
A: Yes, as long as these two people are improving / writing harnesses for the different parts of code being integrated. In the unlikely scenario where they happened to harness the same code, the person who submitted their work first will be paid for their submission.
Q: Can one person submit improvements to the same repo more than once?
A: We will only accept one submission improvement from one person, although an exception can be made for a very large codebase at ForAllSecure’s discretion. If you have any questions about this, please email support@forallsecure.com
Q: When does Heroes Program Phase III start and finish?
A: Heroes Program Phase III started on April 1, 2023 and will finish on July 31, 2023.
Q: If I harness and fuzz different parts of the same GitHub repo, should I create multiple pull requests for each component or one pull request for everything?
A: You must submit one pull request for all changes to one repo to simplify processing.
Q: Is there a limit on how many projects one program participant can submit?
A: Yes, to provide an opportunity for as many people as possible to participate in the program, each Phase III participant can only submit up to 20 projects.