Found the root cause. Ubuntu, for a long time now, randomizes MAC address when scanning for WiFi availability. When it scans for WiFi, the connection will break.
Not sure if it's Meraki's problem, network-manager problem, wpa_supplicant problem or it's the integration problem, but the bottom line is that connection will break.
The solution is to disable MAC address randomization when scanning for WiFi availability. This is longer explained here: https://askubuntu.com/a/905019 and to avoid data loss caused by link rot, I will paste the solution here too:
Open a terminal and run:
sudo tee /etc/NetworkManager/conf.d/wifi.scan-rand-mac-address.conf > /dev/null <<EOF
sudo service network-manager restart