Search for Higgs boson pair production in the two bottom quarks plus two photons final state in pp collisions at √s =13 TeV with the ATLAS detector

Searches are performed for nonresonant and resonant di-Higgs boson production in the bb¯γγ final state. The dataset used corresponds to an integrated luminosity of 139 fb^-1 of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. No excess above the expected background is found and upper limits on the di-Higgs boson production cross sections are set. A 95% confidence-level upper limit of 4.2 times the cross section predicted by the Standard Model is set on pp→HH nonresonant production, where the expected limit is 5.7 times the Standard Model predicted value. The expected constraints are obtained for a background hypothesis excluding pp→HH production. The observed (expected) constraints on the Higgs boson trilinear coupling modifier κ_λ are determined to be [-1.5,6.7] ([-2.4,7.7]) at 95% confidence level, where the expected constraints on κ_λ are obtained excluding pp→HH production from the background hypothesis. For resonant production of a new hypothetical scalar particle X (X→HH→bb¯γγ), limits on the cross section for pp→X→HH are presented in the narrow-width approximation as a function of m_X in the range 251 GeV≤mX≤1000 GeV. The observed (expected) limits on the cross section for pp→X→HH range from 640 fb to 44 fb (391 fb to 46 fb) over the considered mass range.