Answer:
Leviathan was a great sea serpent known in Biblical times and was also associated with evil, much like a serpent or dragon is associated with the devil.
What that person was doing was literally saying, I rebuke you evil spirit!
Good question, and here are some verses below where this is mentioned in the Bible.
Job 41:1 Canst thou draw out leviathan with an hook? or his tongue with a cord which thou lettest down?
Psalms 74:14 Thou brakest the heads of leviathan in pieces, and gavest him to be meat to the people inhabiting the wilderness.
Psalms 104:26 There go the ships: there is that leviathan, whom thou hast made to play therein.
Isaiah 27:1 In that day the LORD with his sore and great and strong sword shall punish leviathan the piercing serpent, even leviathan that crooked serpent; and he shall slay the dragon that is in the sea.