As a pool owner, you have a responsibility to others to ensure it is safe. Whether you invite someone to swim or not, you are liable for injuries that occur. Therefore, it is in your best interest to make sure you implement safeguards for the area.
The U.S. Consumer Product Safety Commission explains you can take certain steps to avoid accidents, including someone wandering into your pool and drowning.
You should make it impossible for someone to get into your pool area without your knowledge. This includes someone coming in from outside your home and those inside your home. To do this, you need to install proper barriers.
You want to have a fence around the area that has a gate with a lock. Ensure that the lock is secure and nobody can easily enter the gate. You also want to put safety barriers around the area that prevent someone from getting to it from inside your home. Alarms work well for this as they will alert you to someone entering the area and serve not only to keep a child from wandering to the pool from inside your home but also act as a backup to your fence to stop outsiders.
Make sure you have set rules about pool use. Teach children to never go in the water without an adult present. Set boundaries for guests. Also, enforce the rules at all times.
Finally, make sure you know how to do CPR and can react should someone accidentally get into the pool and need help.