Remembering takes a lot of imagination.
In the book Stumbling on Happiness, author Daniel Gilbert talks about how science is showing that memory is not a video that we play back, but that it is constructed with parts that we know or believe to be true already. To demonstrate Gilbert describes a scenario where a person approaches a stranger to ask for directions. During their conversation two people pass between them carrying a large wooden door. Unknown to the stranger the person asking for directions is changed to someone else who has made no effort to look like the first. The conversation continues as if nothing has happened.
If it is difficult for our brains to hold information that describes a person in a brief encounter like this one, it may be that our memory for details of our own past are not as sharp as we believe it to be.