I can distinctly remember what first caused it in me; some friends of my parents lived (and still live) in Lincolnshire. The lady of the house (J) was a blue badge tourist guide for the county, and visiting them was always a fascinating insider's guide to the county's landmarks. One day, we went to visit Gainsborough Old Hall, a medieval English Heritage property with an imposing tower.
On the trip there, J recounted the story of the ghost of Gainsborough Old Hall - the Grey Lady. She was, apparently, the daughter of a local nobleman who had fallen in love with a farm worker and was subsequently locked up in the Hall by her father, where she was apparently driven to suicide. Since then, her ghost has been seen wandering the corridors, and doing that clever ghosty trick of walking through walls. There are also tales of another young lady that has been seen there, and a quick google search reveals an article in the local paper from 2008 that shows the appearance of a mysterious orb .
Suffice to say, being young and easily frightened, by the time we reached the property, I had a severe case of the heebie-jeebies. This was not helped by a sign in the "haunted" corridor recounting the story again in some detail, with a helpful sketch of what the Grey Lady might look like were to encounter her. (Perhaps unsurprisingly; grey and ghost like).
For years afterwards - well into my teens - the story of the Grey Lady stayed with me. More than anything, it was the one thing that convinced me at the time that ghosts were real. More than anything it was the one thing that made me sick with fright if I had to go to the toilet at night.
These days of course, I am way too old and cynical to believe in any of that nonsense. Every now and then, however, I still hesitate in a dimly-lit corridor...