Your office is a crutch
Your physical office acts as a crutch, helping your organization forward in spite of your failures as a manager.
When you fail to communicate goals and mission, people recover by talking to each other on their way to lunch. Unclear documentation gets fixed with a tap on the shoulder. Poor performance reviews matter less when you end up promoting whoever is most popular around the office. Colleagues build rapport on their own by being social in the office and sometimes even outside of it. Our gregarious human nature of mingling and gossiping turns into a safety net against organizational gaps and management failures.
Once you go remote, that net disappears. Onboarding, documentation, rapport, performance evaluation — nothing will be caught by the proverbial water cooler anymore. You have to be mindful about every process and build them so these things actually happen.
Initially that means more work, but being intentional when designing your processes, observing and tweaking them as you go (and as you grow) will make for a much better experience for the team and likely a more productive way of operating.