You delete the magazine's app the same way you delete any other app. Tap and hold on one until they all start to jiggle, then tap the small (x) that appears at the icon's corner. Press the Home button to exit this mode.
However, this does not remove your subscription.
To remove the subscription (paid or not):
- Open the Newsstand.
- Tap the Store button at the bottom left. (may be somewhere else if you are not on iOS 7)
- Scroll all the way to the bottom of the store's page.
- Tap on the Apple ID: (your email address) button.
- Choose View Apple ID
- Enter your Apple ID password when requested.
- Tap Manage under the Subscriptions section.
Here you will see all your active and expired subscriptions.
- Tap on the magazine subscription(s) you want to stop and turn off Auto-Renew.
Any magazine you already paid for will still be available. No way to un-pay, even if you have not downloaded it yet, but you won't receive (or be charged for) any new magazines from that point forward.
You can also re-subscribe to magazines from here (provided they still exist), if you change your mind later and don't want to search the Newsstand store for them.
Note: You'll get the same options if you go to any of the store apps, tap your Apple ID, and chose View Apple ID. Just incase you have trouble going through the Newsstand app/folder oddity.