4. Evaluation campaigns: benchmarking, challenges and competitions
4.1 Benchmarking campaign
The aim of a benchmarking campaign is to carry out a comparative evaluation (parangonnage) of the performance of different AI systems designed to automate a specific task, at a given time and place.
-
LNE's past experience in organizing assessment campaigns has led to the identification of best practices for guaranteeing the value of the results obtained. In particular, this means ensuring that the campaign is :
scientific: while preserving the demonstration aspect typically associated with evaluation campaigns, these must be based on the scientific criteria of objectivity of evaluation, repeatability of performance measurement and reproducibility of experiments,...
The Ultimate Scientific and Technical Reference
This article is included in
Software technologies and System architectures
This offer includes:
Knowledge Base
Updated and enriched with articles validated by our scientific committees
Services
A set of exclusive tools to complement the resources
Practical Path
Operational and didactic, to guarantee the acquisition of transversal skills
Doc & Quiz
Interactive articles with quizzes, for constructive reading
Evaluation campaigns: benchmarking, challenges and competitions
Conclusion
Bibliography
Standards and norms
- BIPM: International Vocabulary of Metrology – Fundamental and general concepts and associated terms (VIM) 3rd edition - JCGM 200 - 2012
https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf
Events
METRICS project (2020-2023, funded by H2020) – Metrological evaluation and testing of robots in international competitions
The aim is to organize intelligent robot competitions in four fields: healthcare, agri-food, infrastructure inspection and maintenance, and agile production. In particular, the aim is to build a permanent structure bringing together all European skills to jointly provide a satisfactory...
The Ultimate Scientific and Technical Reference
