Where are the problems happening#There are several different server components, so the first step is figuring out which one is running out of memory. The logs of the different servers that can be accessed through the web admin application will help out determining this (unless you cannot access the web admin application, in which case your problems are likely to be related to the web server).
Solutions#A series of solutions are stated in this section, classified by the component where the problems might be happening.
web server#If it is the web server, you can modify the top-level build.xml file to increase the heap. You would do this in the "run-without-wrapper" target where it starts Java.
Alternately, you can build the server using "ant package", and then run it with a command like:
java -jar dist/Wonderland.jar -Xmx900m my.run.properties
This will start the web server using the setting from my.run.properties with 900 MB of Java heap.
If the Darkstar server is running out of memory, you can modify the run script used to start Darkstar, which is at wonderland/core/build-tools/run-scripts/run-server.xml. You can add arguments to the java invocation in the core-run-server target.
Client problems#The GL_OUT_OF_MEMORY error is due to memory issues on the graphics card. Large textures are the most likely culprit, so reducing the texture sizes should make a difference.
Java typically places its own limits on the amount of memory the VM can use, independent of the operating system. You can configure the maximum available to the Wonderland client by modifying wonderland/web/webstart/web/Wonderland.jnlp. There is a line that looks like this:
<j2se version="1.6+" initial-heap-size="128m" max-heap-size="512m" java-vm-args=""/>
The direct buffer memory (MaxDirectMemorySize which is used extensively by JME as well as the Darkstar IO system) in Java 6 can be configured via JNLP, according to this doc
You should be able to modify Wonderland.jnlp to add:
<resources> <j2se version="1.6+" initial-heap-size="128m" max-heap-size="512m" java-vm-args="-XX:MaxDirectMemorySize=256M"/> [...]