Where did the signalman say he had seen the narrator before?

At the beginning of the story, the signalman is certain that he has seen the narrator before:

"I was doubtful," he returned, "whether I had seen you before."

"There?" I said.

Intently watchful of me, he replied (but without sound), "Yes."

When asked, the signalman says that he has seen the narrator at the red light, just a short distance from his signal box. The narrator denies having ever been at the red light, but the signalman does not seem convinced, and he has a good reason for such a feeling. As the narrator later learns, the signalman has been haunted by the specter of a man at the red light. In addition, the signalman heard this mysterious figure say the exact same words as the narrator:

Halloa! Below there!

While the signalman wants to believe that he saw the narrator at the red light, he is, in fact, very much mistaken. This mysterious figure is a ghost who, in a tragic turn of events, is warning the signalman of his own demise. Both the signalman and the narrator, however, will learn this lesson too late.