Question

what could phantom listeners referred to

Answer 1

Answer:

It's an actual spirit of a dead person. The other definition is that a phantom is a figment of the imagination. In relation to this poem, that means the phantom listeners are likely the ghostly spirits of the house's former inhabitants, but the phantoms could be merely figments of the visitor's imagination.