Question

How to prove a function is convex or non-convex?

Answer 1

are you asking about convex lens
then your answer is here_

convex lens is also called converging lens
means it converge all the ray of light at one
point.

so by showing this conversion you can prove

Answer 2

Convexity testing for a large and arbitrary function is typically non-trivial. One test for convexity is to check the function's Hessian. A continuous, twice-differentiable function is convex if its Hessian is positive semidefinite everywhere in interior of the convex set.