On a Rayleigh-Faber-Krahn inequality for the regional fractional Laplacian

We study a Rayleigh-Faber-Krahn inequality for regional fractional Laplacian operators. In particular, we show that there exists a compactly supported nonnegative Sobolev function u 0 that attains the infimum (which will be a positive real number) of the set { ∬ { u > 0 } × { u > 0 } \ u ( x ) − u ( y ) \ 2 \ x − y \ n + 2 σ d x d y : u ∈ H σ ( R n ) , ∫ R n u 2 = 1 , \ { u > 0 } \ ≤ 1 } Unlike the corresponding problem for the usual fractional Laplacian, where the domain of the integration is R n × R n , symmetrization techniques may not apply here. Our approach is instead based on the direct method and new a priori diameter estimates. We also present several remaining open questions concerning the regularity and shape of the minimizers, and the form of the Euler-Lagrange equations.