In an unarmored context, which applied very often throughout history, the spear is easier to use and especially lighter, which makes it a better and more nimble weapon. Spears can also be much longer than heavier pole arms whilst remaining usable, keeping the danger further away from the user
The speed at which one can move a spear tip is impressive and getting stabbed by it has large stopping power. The spear can also parry attacks in a large sweeping area, which makes it hard for anything else than another spear to get through
The more complex pole weapons start to shine in an armored context, where stabbing someone at the end of your long pointy stick becomes harder. Then, the hook-y, chop-y and spike-y bits of the halberd can really help tackle the armor
While the halberd's blade offers a lot of options for hooking your opponent, it equally gives them an opportunity to hook your polearm and manipulate it themselves. The simpler spearhead is also lighter than that of a halberd, allowing for more sustain when fighting for long periods.
I would definitely prefer a halberd (or billhook) over a spear given the choice, but it's not as much of a direct upgrade as you might think.
I just read about swiss mercenaries a while ago and wondered the whole read why they didn't have a short range weapon like a knife or axe too. Probably because of this.
Btw, they needed a whole century to equip the army with spears. Things sure moved slower back then.