I wouldn't be surprised if they did it mathematically. You have to remember that back in those days video game music wasn't a simple matter of writing notes. All of it was a ballet of frequencies and amplitudes, tiny little static tones played in rapid succession to generate the sound of musical instruments playing at symphony. Back then, music composers for video games also had to be capable sound engineers, and a ridiculous amount of mathematics went into their work.
What I'm saying is that anybody who could figure out how to make static blips sound like trumpets and violins, would also have been able to design equations for even the most hideous yowls from the otherworld. Who knows, it may even have been easier.
This is just my speculation; I don't have any inside information on how Mitsuda wrote Lavos' scream.
