The percentage % of modulation is an indicator of the extent to which it is modulated.
Here is a picture of an unmodulated sine wave.

By comparison here is a picture of an Amplitude Modulated sine wave.

Formula
The percent modulation is given by
%M = ((VMAX-VMIN)/(VMAX+VMIN)) * 100
Calculator
Example
In the following example, VMAX = 2.5, VMIN = -0.5. Using these numbers in the calculator gives 150% modulation. This is an example of an overmodulated signal.

If VMAX = VMIN, the sine wave is not modulated and the percentage modulation is 0.