The Java™ heap becomes exhausted when garbage collection cannot free enough objects to make a new object allocation.
Garbage collection can free only objects that are no longer referenced by other objects, or are referenced from the thread stacks (see Memory management for more details).
Java heap exhaustion can be identified from the -verbose:gc output by garbage collection occurring more and more frequently, with less memory being freed. Eventually the JVM will fail, and the heap occupancy will be at, or almost at, 100% (See Memory management for more details on -verbose:gc output).
If the Java heap is being exhausted, and increasing the Java heap size does not solve the problem, the next stage is to examine the objects that are on the heap, and look for suspect data structures that are referencing large numbers of Java objects that should have been released. Use Heapdump Analysis, as detailed in Using Heapdump. Similar information can be gained by using other tools, such as JProbe and OptimizeIt.