He's in your network the moment he knows a working MAC address (which is easy). If you assume that someone unauthorized has your passphrase, you either should change it or move to 802.1X.
Changing the PSK is too hard? That's the reason it is only meant to be used in residential setups and not in a company.
If you found this post helpful, please give it Kudos. If my answer solves your problem, please click Accept as Solution so others can benefit from it.