Your consciousness may be within my space, but you aren't truly here. Not in flesh and blood.
I can be affected by you because I've willed your presence. Your... imprint has become as much a part of my domain as anything else.
You can see and follow me. Hurt me even, if you were so inclined.
But I can't do the same to you. Not like this.
Though this does not explain why he never noticed or spoke to me, or why he has no recollection of you.
Nor does it explain why you too have no memory of our prior meetings.