I have always had a theory that only the current incarnation of the Doctor will remember the events that happen with a multi-Doctor interraction.
Let me explain, In the Three Doctor's, well, the three of them got together, they had an adventure, however, we never heard Two tell them, "Oh, I remember this event when I was #1." Likewise, Three never said, "Well, I remember when I was #1 AND #2."
I just think that when they all go back to their time-lines, only the current incarnation carries the memory of their meeting. So at the end of the Three Doctors, only #3. remembers the encounter, 1 and 2 do not.
Likewise in the Five Doctors, #5. would remember the last time this happened! (When HE was # 3. And if the 4th Doctor had been there, he too would have remembered the event)
But, when the other 4 Doctors get put back 'in place' only the 5th Doctor would remember what happened. The Fourth Doctor would still remember what happened when HE was # 3, but this current memory would be erased from his memory, as only # 5. would carry it... So on, such forth.
Oh, wait.... Time Crash! "You remember being me, watching you...."
Crap, just thought of that. Well, not gonna delete the post now that I've come so far on it! :lol