Question

why does the sight of daffodils come to the poet's mind later in the poem?

Answer 1

The poet Wordsworth was so much moved by the beauty of those flowers that they left an everlasting impact on his mind. That is why whenever he is in pensive or vacant mood, the daffodils “flash upon that inward eye” of the poet. 
 
Actually the poet is a nature lover and beauty of nature attracts him more than anything else. He is somewhat obsessed with the fair aspects of nature. This is why it happens to him. He loves to remind those daffodils and be happy with them whenever he is sad. He finds some solace in those flowers.

Answer 2

The sight of daffodils come to the poet's mind layer in the poem since he describes that whenever he is sad, the sight of the daffodils in his mind makes him happy again. He loves being around nature, he enjoys natural things