Question

How to justify whether a polygon is concave or convex by mathematically?

Answer 1

convex polygon is defined as a
polygon with all its interior angles less
than 180°. This means that all the
vertices of the polygon will point
outwards, away from the interior of the
shape. Think of it as a 'bulging'
polygon. Note that a triangle (3-gon) is
always convex.
A convex polygon is the opposite of a
concave polygon.

Answer 2

Convex polygon = A polygon called if its diagonal like inside the polygon
concave polygon= A polygon called if its diagonal like outside the polygon.