An incorrectly merged ticket cannot be unmerged via the user interface, but all is not lost!
This is RT, all is never lost!
It's never recommended to work directly on your RT database, but in this instance that is the only way to recover from this mistake. If you're in any way unsure of what you're doing, don't do it.
You may wish to take a backup of your database, just in case.
Within your database 'tickets' table, change the effective ID of the incorrectly merged ticket back to it's original ticket number and the ticket will be unmerged. Be careful to do this before adding any more transactions to the ticket or the ticket that the transactions will be associated with may become confusing.
You could use the following SQL command assuming that this was the only ticket merged into the new one ...
... where is 11002 is the number of the merged into ticket ... but don't forget to remove any new users from a wrong merge as well.