LE secure connection doesn't work

Ahh, yes, I didn't mention that but only certain combinations for setSecurity work and unfortunately the Nordic SDK doesn't always report back when some settings don't make sense.

Any clue why I get this "public key is not valid" error?

I'm afraid not. I guess it's not referring to the passkey you entered? Maybe Nordic Devzone has some ideas.

The odd thing is I'm pretty sure I have had this working, because the passkey pairing on Central code was implemented for a user that needed it for his project.