I am actually chatting with Samsung support at this very minute. I have the exact same issue with my Note 4. The default camera app takes 10-30 seconds to focus for the first time - once finally focused, it re-focuses with ease. Occasionally, entering the settings menu and exiting it seems to allow the lens to focus. I have also tried Google Camera and see the same results. The interesting (and worrisome) thing is that my wife also purchased a Note 4 and her device does not exhibit the same behavior. This leads me to believe it may be a hardware problem...
I'll report back here if I receive any helpful advice from Samsung. So far, not looking too promising.