With how many people die and are hurt in cars now, on top of just car damages, we'd still be better off with self-driven cars. The biggest problems are inattention or risky driving on the driver's part. With self-driven cars, you could address that. Of course there's no sense putting them on the road before they're ready, but that doesn't mean we shouldn't look ahead.
I expect small affluent communities and countries will be the trial bunnies first.