Yes, that's the case if you don't go into settings and switch private browsing off AND confirm delete open tabs. This can also initiate a crash problem. I've been able to replicate this problem every time.
If you run into the crash issue, here's how to resolve it while you're in the Private Browsing mode...
- Close your private tabs.
- Open up another tab.
- Open Settings > Safari and turn Private Browsing "Off"
- On the close open tabs confirmation pop-up tap "Keep All"
- Open safari (Close the tab you left open if needed)
Basically it boils down to tapping the "Keep All" option to avoid issues. That's a real pain since you have to delete any you don't want to return when you stop Private Browsing, but I can confirm that it does work. Apparently the problem arises when all of the tabs are closed.
Of course you can always clear Safari's history and cache (Settings > Safari > "Clear History" & "Clear Cache") but that's seems counterproductive and negates the Private Browsing option.
There are obviously a couple of bugs that need to be fixed by Apple and hopefully that will come in the next update to iOS 5...and soon.