My assumption, having finally understood the animation sequence(it looks odd without the blade animation attached), is that it was meant to be a sort of Iai style attack...
The basic premise of Iaido is, unsheath your weapon and strike a single fatal wound with it in one swift motion, and then resheathe it... Since your strike is a guaranteed kill if executed properly, you've no more need of a weapon and the battle is over.
This type of attack is common in anime, and is generally accompanied by a 'failure to see the strike hit, but then sudden wound appearing anyway' style of animation. Another example of that is the Hien style attack, where two figures rush toward each other and are shown attacking, seemingly missing... and once they rest for a moment, one dies... often again accompanied by a delayed wound appearance.
If you notice his strike, we don't see his blade, however, we do see a large energy 'curtain' appear in roughly an arc spreading from behind Lavos' mandibles. This to me is meant to represent the effect of the strike, since the blade is never seen.
If we assume his strike was meant to be of the Iai variety, he likely practiced that technique for years (as it is a dangerous maneuver to use unless completely mastered). It would certainly be surprising if your time tested masterful execution of a sure-kill strike... failed to do any real harm.
Also, I believe Janus was present to witness Lavos once, wasn't he? Be that the case, he could certainly have ingrained the structure of the beast into his mind and expected the severing of the "head" to be an effective attack.
Also, you call it a mouth, but I see no opening... I see an eyeball inside a well protected eye socket. Lavos, for whatever reason, reminds me of flea or mite...only of a meteorically large stature and covered in spiny plates... It also appears to be covered in vesicles of some sort, which may contain its offspring or be the method it uses to consume/absorb matter (in this case probably energy and DNA).
(Don't cite the anime cutscenes, they don't hold water in my book)
Whatever the case, being that it is at the front, contains a mouth/eye/something, its reasonable to assume just by looking at it that this is a 'head.' Furthermore, the primary neural bundle is generally located at the dorsal anterior of a creature's head... or in other words, at the base of the head, and generally close to the top or back of the creature depending on its body orientation... If we assumed that was Lavos' head, we could reasonably get the impression that the space above and behind is where the spinal cord would be... We know Magus is proficient at shadow magic, including attacks that use vacuums or attempt to banish things to some other space... I doubt he would try to do such a thing to Lavos' entire body mass, as such a spell would likely require more energy and preparation time than he could afford... so instead he may have augmented his physical attack (opening up the shell) with an intense vacuum/negative energy/pressure or simply tried to banish the nerve column itself... thus we see the energy I previously mentioned as an effect of his strike...
As I recall, his weapon is normally bladeless in battle, and when he is recruited is that same red color(or it may be yellow and that red is the color of his sprite without a palette attached). We could make up some even less well based fanon than my explanation and say he prepared a special dreamstone weapon... but if that were the case, why then would he not have it when we recruit him?
My opinion is that it was an artistic style of showing his fighting skill and possibly his ability to use magic and physical attacks in unison.
He is surprised it didn't work because he put his whole being into mastering the 'perfect, instantaneous, surprise attack.'
Consider... Indiana Jones: In a scene, he is confronted by a highly skilled swordsman, and is seemingly defenseless for a moment...(Seeing the parellel yet?) Before the sword-wielding man can move to attack, the ever hatted hero quickdraws a pistol and shoots-him-dead, surprising the crowd and audience, as that was a completely unexpected move, and was completely decisive at the same time.
This is essentially Magus' plan... or his effective action at least... Now consider how surprised Indie would be had the blade-using man he shot not been affected at all by it... Generally people die when you kill them.
Lavos did not die when it was killed. This is the matter of contention for Magus.
That is how I see it.
(Before realizing Magus attack was actually one animation without a blade effect sprite drawn over it... it just looked to me like the game eff'ed up and draw a bunch of random Magus sprites while depicting magic which was clearly not cast using the casting animation... I always hated the scene for being so confusing... before anyway.)
