I'd recommend 2048 for maxmem. You're problem is probably related to video drivers. You can try checking the Disable Direct3D Pipeline in the launcher under troubleshooting and if that doesn't work, uncheck it then check the Enable OpenGL Pipeline and see if either of those solves your problem. Also, make sure you have the latest 64bit version of java installed (It's a manually download from a zip file). Uninstalling java, restarting the computer and installing the new version usually works best.
Sorry, but i have no idea where to make these changes, complete rookie here. The troubleshooting tab of the launcher only has options for selecting the logging detail.
Also, I already made sure to install the 64bit java as that was on of my attempted fixes.