An observer in a very dark place will have eyes that are fully adjusted to night vision mode (it takes a few minutes for them to switch over). They will be able to see the slightest amount of white light. Eyes are especially sensitive just off the centre of vision. However night vision is much less sensitive at the deep red end of the spectrum. If you use the right type of red filter (on the edge of IR and letting no shorter wavelength light through) the observer will not be able to see it anywhere near as well as white, even for the same brightness. (The downside is that neither will you, but you are a lot closer to the torch, so have more light to play with.)
A second effect is that using a red light will not switch off your night vision, so you will be able to see better in the dark and consequently can use less light from the torch, making you less visible again.