Yep, this happened to me too. We changed the deadbolt to a key lock on the inside. The key was attached to a magnet at the top of the door. (Make sure the key is always near the lock, so you don't get locked in during a fire.) There are lots of different ways to secure the door. Find one that works for you.
It is unlikely that anyone will call CPS. However, if they do, CPS is aware that toddlers run. If some idiot calls CPS on you, they'll drop by, make a 5 minute investigation, and note the complaint as "unfounded." Unless, of course, you're passed out drunk on the floor. :)
Kidnapping is not a significant risk. Kidnapping is incredibly rare and children under 5 are the single lowest risk group. Cars are a *very* significant risk. Worry about cars, not people.
As much as possible, get to know your neighbors. Make sure they know you've got an escaping toddler, and to bring her back home if they see her on the loose. Tell them that they have your permission to pick her up and haul her home, over her objections if necessary.
Especially get to know any teenagers or adults who are often hanging out on the street. They're more likely to see a tot on the loose, and usually happy to bring them home. Everyone welcomes the chance to be a hero, even people who are otherwise making rather poor choices with their lives. Saving a baby from getting hit by a car definitely goes in that category of hero.
Consider getting her a ID bracelet with all your phone numbers on it: cell, home, etc.
Finally, train her to a call-and-response. Mine is a sing-song chant "Where is my Sam?" to which he responds "I'm right here!" The words and the tone are always identical. He is eight and we still use it. It cuts through background noise and distractions, and his response is pure reflex. Pick something that works for you and use it constantly in the apartment until she responds perfectly. This makes it a lot easier to find her when she takes off. Holler and listen. Even if she's under a bush inspecting earthworms, you'll be able to hear her and find her.
Good luck!