Remote on multiple devices

but it reconnects immediately by itself.
It would be great if I could disconnect/forget all devices from the Puck itself.

I think Puck is the passive side in this and it is the phone that does it. You could refuse connection on the Puck when specific device connects but that would not help much.

Simplest solution that may work is changing MAC address of the Puck, you may then have different addresses for different devices so switching to right address will let specific device work with the Puck. Check this NRF_setAddress