It's probably a hardware issue with the keyboard. At a guess, I'd say that the process of opening/closing the iPad keyboard case is causing the Caps Lock key to be pressed, either through an internal short, or otherwise.
A quick question, when you open the case, do grab it anywhere near the Caps Lock key? A long shot, but really the only thing I can think of that doesn't mean the keyboard itself likely has a fault.
I'd also suggest a close look at the key for any deformity or other indication that it might be physically off, but your picture of it looks good, so probably not.
There is, of course, a small chance that this is a software malfunction. Since you've done a restart the next step would be a complete system restore; an awfully big hammer to use when the chance of fixing the issue is so small.
As you've discovered, there is setting that effects the default state of Caps Lock. As far as I can tell the Enable Caps Lock setting only effects the touch keyboard. It allows you to toggle Caps Lock on by double tapping the Shift key.
The last thing I can think of is also a long shot. Assuming you are using a U.S. keyboard, make sure you've selected the U.S. keyboard under the keyboard hardware settings. A mismatch of hardware and this setting 'might' do something unexpected. I don't know.
Not very helpful, I know, but thats all I can think of at the moment.