Sophos Principles of Third-Party Testing
Testing of security products should be fair, rigorous, and transparent. Security vendors should be open to good-faith testing conducted in accordance with these principles.
Fairness
All vendors being tested should have equal opportunity to be successful in accordance with the strength of their products. Tested products should be configured optimally or in a way that is equivalent across vendors (e.g., all products use default settings, all products be configured by the vendor to reflect best practices, etc.). Vendors should be given a reasonable opportunity to guide and verify proper setup and configuration and to review and dispute results.
Rigor
Tests should be designed to evaluate, as accurately and comprehensively as possible, the ability of the tested products to accomplish the stated goal. For example, protection tests should attempt to exercise the full attack chain, and performance tests should strive to model real-world conditions. Measurements, scores, and statistical analyses should be mathematically valid and represented in accordance with research and evaluation best practices. Published reports should communicate the limitations of the methodology to help readers contextualize the results.
Transparency
The testing methodology and product configurations used should be referenced in published test results. Public and commissioned test reports should also indicate whether vendors were able to participate actively, which chose to do so, and which declined. Vendors should also have the option to provide statements on both the methodology and the execution, and these statements should be made available to consumers of the reports.
Collaborative
Customers, testers, analysts, and vendors acting in good faith and in accordance with these principles should have the opportunity to test products and publish factual results. Vendors should remove arbitrary and excessive restrictions to this freedom.
Call to Action
Sophos is calling on cybersecurity vendors and testing organizations to:
- Embrace and contribute to the ongoing review and improvement of such projects as the AMTSO Testing Protocol Standard and MITRE’s ATT&CK Framework, which respectively promote transparency and fairness in security testing and the rigorous mapping of attack lifecycles
- Remove arbitrary and excessive restrictions on the use of security products for the purposes of comparative testing and publication of factual results
- Actively participate, when practical, in high-quality independent tests, especially those that are aligned with these principles and follow the AMTSO Standard
- Advocate for more fair, rigorous, transparent testing that provides the answers prospective customers need to make better informed decisions about security products