I found a fix. I had the same frustration. Problem solved when the invitee logs in to photos on a Mac being logged in to their iCloud account. Then you can "Accept" the invitation from the Shared stream "Activity". That took care of it for me.
I could not get it to work trying to accept through...