I may have climbed the odd chimney in the past, both for work and play. If you are using a permanent access system (ladders and platforms) the thing to bear in mind is they are not used very often so things like gratings being unattached, ladder rungs and handrails corroded through and fixings to the chimney itself may have failed.
The obvious thing is you will stand out like a sore thumb while ascending the structure unless it has internal access.
The flue gasses can be an issue, but usually just on the downstream side of the prevailing wind, and even then only in the upper few meters. Also if you are daft enough to put your head over the top. I have had my head over the top on power station chimneys on the upwind side, both on coal and gas power stations. Glass works there is al sorts coming out the chimney at very high temperatures, again have worked at the top of these while they were on line, fair enough we had a gas meter and knew when it was too hot, the wooden ladders or scaffold boards would start charring and smoking.
Like any high structure the main hazard would be a fall from height, permanent access is infrequently used and also exposed to the elements, both the weather and any corrosive material in the flue gasses.
You can't beat watching the world from a high vantage point, a couple of photos from the same chimney. A 650ft coal power station, working in the flue gasses, but they had been put through a FGD (flue gas desulphuration plant, so were essentially just warm steam, plus the view from the top.