Exactly the same for me. In fact, some childhood memories which i cherish happened in a different viewpoint - even today, my memory of those times is played back in my mind only from that particular viewpoint (not necessarily my default/current viewpoint).
If I close my eyes and try to remember that viewpoint, with enough concentration, i can 'trace' and bring my present location to that viewpoint (though doing that takes a lot of brain processing which can leave me exhausted).
Do you have to see it from the correct viewpoint as well?
For example let's label the 4 different views of the house you grew up in: A, B, C, and D.
Try to remember an event from your childhood that happened in view A, from the perspective of view C... it isn't there right?
For me it isn't there because it didn't happen in that parallel world/viewpoint, it happened in this one...
I have to think that through, though i can say for sure that any such event wouldn't ring the same bells in my memory if it were not associated with that particular viewpoint.
btw, sometimes excessive viewpoint flipping makes me a bit nauseous - anything like that for you?