Question

What happens to the output voltage if bypass capacitor is removed?

Answer 1

Besides gain being reduced, input impedance raises as well as output impedance.
Common emitter transistor amplifiers are basically transresistive, that is they have low input impedance and high output impedance. By eliminating the emitter bypass the amplifier becomes more transconductive. And that is not completely bad because having some resistance on the emitter makes the amplifier gain more predictable and stable and less dependent on transistor, temperature, ageing, supply voltages. Also you get more bandwidth.

Besides, electrolytic capacitors are one of the less reliable electronic parts. They may leak, and all dry out; the hotter the fastest. Plus they are not linear. Good designs try to avoid electrolytic capacitors like the plague.