In the course of the war with Midyan, the Israelites killed the five Midyanite kings, and likewise Bilam, in accordance with whose advice the Midyanites acted. The Sages describe how Bilam attempted to utilize his supernatural powers to save himself, but ultimately failed.

When Pinhas and all the military units that accompanied him went to Midyan, upon seeing Pinhas the wicked Bilam transformed his two arms into stone boards and he flew and ascended upward by using the ineffable name of God. When Pinhas saw that Bilam was flying and ascending, he made his own two arms into two stone boards and flew and ascended, pursuing him, until Pinhas found Bilam prostrating himself before the Throne of Glory.

Immediately, Pinhas donned the diadem of the Holy One, blessed be He [worn by the High Priest, upon which the words “sacred to the Lord” were written], and he seized Bilam and brought him before Moses. The Israelites judged Bilam in the Sanhedrin and killed him. (Yalkut Shimoni, Matot 1:785)