How do you set it for more?
In the launcher (launcher.versionnumbe.jar). You are using the launcher to start MapTool, right? If not, you get the default settings for Java, which are pitifully inadequate.
In the launcher, you'll see three text boxes below the logo, for max memory, minimum memory, and stack size. Minimum memory you don't generally need to worry about. Stack size, the experts recommend 3 instead of 2. Don't go much higher, or you get bad things.
Max memory is trickier, especially if you're using Windows. If you're using 32 bit Windows, do not, under any circumstances set it to higher than about 1600, or MapTool will not launch. Better to never go above 1400, and 1024 is about as high as you can depend on being safe. This is because of how Windows handles memory, and how Java uses it.
If you're using 64 bit Windows, you also need 64 bit Java, or you have the same memory restrictions. If you went to www.java.com
and clicked on the big red button to install Java, you got 32 bit Java. There's a link below the "Agree and Start Download" button titled "Not the right operating system? See all Java downloads." that will get you to the 64 bit Java installer. Then, you can go to whatever memory is available, but generally, as aliasmask says, 2048 is plenty. (And if you're running a server, and the other players who connect to it have 32 bit Java, you can create a map that works fine on the server, but they get big red "X"s. Be aware of this.)
If the launcher doesn't work for you (which is rare, but happens), somebody here will explain how to create a batch file that will set the memory settings for you.
If you're interested in the technical details, Java requires contiguous memory, and 32 bit Windows - and 32 bit applications on 64 bit Windows - cannot, ever, under any circumstances, allocate more than about 1.6 GB of contiguous memory, and what is available is often less, as little as 1.2 GB or even less). Because the launcher suppresses the black DOS window when it launches MapTool, you don't see the error message about being unable to allocate enough memory, so it's very, very confusing.