More, mostly relevant, information.
If you go to Settings > Sounds you will see a switch under the volume slider. If the switch is set to Change with Buttons, then 'all' sounds including system sounds (alerts, notifications, alarms, ringers, keyboard clicks, etc) will change with the main volume (using the volume buttons). If it is off, the system sounds will remain at the level set by the slider, no matter what the the general (app) volume level is set to. In that case you need to use the Mute option to turn off system sounds.
The ability to set two volume levels for system and general app sounds is so that you can make sure your alarms can be heard, even if you turned your music way low for some reason.
The Mute option is so that you can turn off system sounds and prevent the iPad from making (unintentional) rude noises when it is inappropriate; like theaters, meetings, and dinner with the in-laws. But, you can still whip it out and show them the video of their grandchildren (with sound) and not worry about the settings.
Summary: To stop all sound on an iPad.
With the Change with Button turned on, all you need to do is lower the volume to zero with the buttons.
With the Change with Buttons turned off, you need to lower the volume to zero and mute the iPad.