The percentage % of modulation is an indicator of the extent to which it is modulated.
Here is a picture of an unmodulated sine wave.
By comparison here is a picture of an Amplitude Modulated sine wave.
Formula
The percent modulation is given by
%M = ((VMAX-VMIN)/(VMAX+VMIN)) * 100
Calculator
Example
In the following example, VMAX = 2.5, VMIN = -0.5. Using these numbers in the calculator gives 150% modulation. This is an example of an overmodulated signal.
If VMAX = VMIN, the sine wave is not modulated and the percentage modulation is 0.