"The launcher doesn't work" is pretty consistent with "64 bit Windows with 32 bit Java." Unless you made a point of going to the "Not the right operating system? See all Java downloads." link, you got 32 bit Java. (You can test this by setting the max memory to 1024 MB. If it works then, that's the problem.)
As aliasmask says, the best first step is to uninstall all Java, reboot, then install 64 bit Java from https://java.com/en/download/manual.jsp
. Also wouldn't hurt to remove all trace of MapTool and reinstall that, too, from fresh downloads.