SciPost Submission Page
Signal region combination with full and simplified likelihoods in MadAnalysis 5
by Gaël Alguero, Jack Y. Araz, Benjamin Fuks, Sabine Kraml
This Submission thread is now published as
Submission summary
| Authors (as registered SciPost users): | Jack Araz · Benjamin Fuks · Sabine Kraml |
| Submission information | |
|---|---|
| Preprint Link: | https://arxiv.org/abs/2206.14870v2 (pdf) |
| Code repository: | https://github.com/MadAnalysis/madanalysis5 |
| Date accepted: | 2022-10-19 |
| Date submitted: | 2022-08-31 14:28 |
| Submitted by: | Kraml, Sabine |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approaches: | Theoretical, Computational, Phenomenological |
Abstract
The statistical combination of disjoint signal regions in reinterpretation studies uses more of the data of an analysis and gives more robust results than the single signal region approach. We present the implementation and usage of signal region combination in MadAnalysis 5 through two methods: an interface to the pyhf package making use of statistical models in JSON-serialised format provided by the ATLAS collaboration, and a simplified likelihood calculation making use of covariance matrices provided by the CMS collaboration. The gain in physics reach is demonstrated 1.) by comparison with official mass limits for 4 ATLAS and 5 CMS analyses from the Public Analysis Database of MadAnalysis 5 for which signal region combination is currently available, and 2.) by a case study for an MSSM scenario in which both stops and sbottoms can be produced and have a variety of decays into charginos and neutralinos.
Author comments upon resubmission
We hope that with these clarifications and small modifications our paper is now suitable for publication in SciPost Physics.
List of changes
* ATLAS-SUSY-2018-04: added a remark at the end of the last paragraph discussing ATLAS-SUSY-2018-04 to clarify why we can reproduce the expected limit but fail with the observed one:
"For the expected limit, we note that the MadAnalysis5 results shown in the right panel of figure 2 are pre-fit, while the ATLAS expected limit curve seems to be post-fit. The difference between pre-fit and post-fit background numbers (cf. last paragraph of section 2.1) turns out to compensate the higher acceptance x efficiency values from the recast code."
* CMS-SUS-16-039: updated figure 5 including also the off-shell region in the MadAnalysis5 recast.
* CMS-SUS-19-006: in reply to the referee's concern regarding the over-exclusion in particular in the expected limit, we modified the last sentence of the relevant paragraph ("Despite the small over exclusion ...") the following way:
"For example, at $m_{\tilde\chi_1^0}=100$~GeV, the observed limit on the gluino mass improves from 1950~GeV in the best-SR approach to about 2260~GeV with combined SRs, to be compared to the official CMS limit of 2180~GeV. Despite the small over exclusion with combined SRs, this improves the reinterpretation potential of this analysis. We note, however, that for the expected limit, the over-exclusion with combined SRs is as important as the under-exclusion with the best SR only."
Published as SciPost Phys. 14, 009 (2023)
Reports on this Submission
Report 2 by Nishita Desai on 2022-9-20 (Invited Report)
- Cite as: Nishita Desai, Report on arXiv:2206.14870v2, delivered 2022-09-20, doi: 10.21468/SciPost.Report.5725
Strengths
1. First public code to use full likelihoods published by ATLAS. Analyses with Simplified likelihood from CMS also have been implemented. This will encourage more experiments and analyses groups to publish this kind of data products.
2. Clear demonstration of improvement in reach from using combinations of signal regions with respect to best sensitivity methods used before.
Weaknesses
I expect this paper will become a reference for future users to make use of published full likelihoods. In that case, it would be good to have a validation summary (see report).
It would also help if a short appendix for usage of this new functionality can be provided.
I do not see any significant deficiencies otherwise.
Report
This paper describes a comprehensive implementation of using published ATLAS and CMS likelihoods which has been a longstanding desirable for phenomenological studies. It clearly demonstrated how this information, if published by experiments, can be used by the wider community.
The paper is well written and contains detailed implementation notes and explanation of discrepancies when they are seen. However, in places where there is significant discrepancy with published experimental limits, it would be good for the user to quickly know this without having to read through each implementation. In places where MA5 fails conservatively compared to published limits, this is acceptable. However, there are situations (e.g. CMS–SUS–19–006 and ATLAS–SUSY–2018–04) where MA5 seems very aggressive and several sigma in excess of experimental limit. It would be good to include a table with analysis name and topology of each of the searches saying whether the validation is within 1(2) sigma or there is a higher discrepancy and whether it is under or over estimating. It would also be good to have a rule of thumb for where there are known problems with using published likelihoods (i.e. known missing efficiencies that the authors recommend be supplied).
Requested changes
1. Table of validation summary for each implemented search
2. Short appendix with user instructions
3. A line or two in the conclusions about where the authors would like more input from experimentalists that is currently missing and causing difficulties in reinterpretation.
Report
The authors have addressed my questions and modified the draft adequately.
Author: Sabine Kraml on 2022-09-08 [id 2797]
(in reply to Report 1 on 2022-09-08)We thank the referee for the speedy assessment (1 week!) and are looking forward to the publication of our paper.
Author: Sabine Kraml on 2022-09-21 [id 2836]
(in reply to Report 2 by Nishita Desai on 2022-09-20)Thank you for taking the time to review our paper and for the (non-anonymous!) report. We are very pleased by the positive assessment and the high importance and high quality that are accorded to our work. The requested changes are, however, a bit misaligned with the spirit of our paper, and somewhat orthogonal with its main message (so that they may dilute it, which we want to avoid).
1. Table of validation summary for each implemented search
The purpose of our paper is to show the value of detailed information on the statistical modeling from the experiments, not the validation of individual recast codes (which is already published elsewhere). Including a validation summary would, to our mind, draw attention away from the main point of the paper and thus dilute its message and impact.
Apart from this concern, the exact difference to the result derived by the experimental collaboration depends on a number of things, in particular the location in phase space; whether the quantification in terms of sigmas is based on the cut-flows or on the exclusion line*); whether, for any point along the exclusion line, one considers the difference in x or y direction; and whether “1 sigma” includes experimental uncertainties only, or also theory uncertainties. (When including both theoretical and experimental uncertainties, MA5 is nowhere “several sigma in excess of the experimental limit”.) Given these factors of arbitrariness and the above concern about diluting the message of the paper, we wish to refrain from adding a validation summary.
*) Another way to quantify the difference to the official experimental result is the ratio of MA5/official (official=ATLAS or CMS) excluded cross sections; plots of this ratio were requested by the first referee and are available in the supplementary material to version 1 of the submission. However, this cannot be phrased in terms of standard deviations.
2. Short appendix with user instructions (for usage of the new functionality)
There are only two new commands
and
They are explained in Section 2. The defaults are “on” and “apriori”, so in principle there’s nothing to do for the user to invoke the new functionality, everything is automatic. We do not think that this merits an appendix.
3. A line or two in the conclusions about where the authors would like more input from experimentalists
The conclusions summarize our work and give an outlook to future developments. We could repeat the question regarding pre-fit versus post-fit background numbers and uncertainties (raised in section 3) at the end of the conclusions. However, this would weaken the statement in the last paragraph about the extraordinary scientific value of full statistical models. So we prefer to keep the conclusions as they are, with a very clear message.
For the same reason, we do not think that complaints about too approximate reconstruction efficiencies or missing HEPData entries for some analyses would be appropriate in the conclusions. By the way, regarding a “rule of thumb for where there are known problems with using published likelihoods”: missing efficiencies are a problem on their own, they have nothing to do with the usage of published likelihoods.
For completeness we also want to point out that MadAnalysis5 is not the first public code to use full likelihoods published by ATLAS: while it is the first simulation-based public framework that provides this functionality, the first public tool to do so was SModelS, already two years ago.