Serge having both memory sets is alluded in one of the cutscenes.
Anyway, what I'm saying is being the Arbiter, the link between man and Lavos (who transcends time, if you remember) He remembers changes to the timeline without being a time traveler. Because of this, time mechanics splits time. This also applies to the Entity at Lavos' crash. It remembers a timeline without Lavos falling (Reptite timeline) and the Lavos Timeline. Thusly, a split O_o