Okay, conclusion:
Asbestos is very nasty stuff if you breathe it in, and exposure now may kill you down the line - so if you want to live to see your kids or grandkids, it makes sense to take reasonable precautions.
Those would be to learn what asbestos looks like, and to avoid it. If it cannot be avoided, then a proper mask (PP3-rated, not a decorator's mask) should be worn. The filters should be chucked after use, as should any clothing that could have collected asbestos fibres. It wasn't uncommon for shipworkers' wives to get asbestosis from their husbands' clothes. Even if you don't care about yourself, care about your family.
But don't worry too much - most places, even if they contain asbestos in the construction, do not contain asbestos fibres in the atmosphere. Don't go into places where asbestos removal is in progress, and don't smash stuff up.