This is part 2 of my series on JVM Memory management and debugging. Read part 1 here
In this post, we will cover symptoms of memory issues for JVM-based applications, which tools we can use to diagnose them and how we can fix them.
Symptoms Here are a few symptoms of memory issues:
Poor application performance Abnormal memory usage OutOfMemory errors (OOME) Poor Application Performance Application not performing to expected level Long response times Dropping client requests Stuck threads Service unavailability Large gaps in timestamps in application logs Causes of memory problems: Misconfigured memory Old generation memory space is sized smaller than live-set of objects. This triggers a major garbage collection (GC), resulting in larger pauses. Code cache is smaller than generated compiled code footprint Young generation is not sized appropriately leading to premature promotion of objects PermGen / Metaspace not sized correctly leading to full GC Memory leaks - Unintentional retention of objects in memory spaces Unintentional references to set of objects in heap Not dereferencing classloader instances appropriateky Not releasing native resources appropriately Excessive use of finalizers Objects with finalizers may delay their own GC Finalizer thread needs to invoke finalize() method of the instances before reclaiming them There can only be 1 Finalizer thread. If it does not keep up with rate at which objects become available for finalization, JVM fails with OOME Pending finalizer objects are essentially accumulated garbage Finalizers deprecated in Java 9 Explicit GC calls System.gc() and diagnostic data collections can cause long pauses -XX:+DisableExplicitGC can disable System.gc() calls -XX:+PrintClassHistogram also calls an explicit GC when receiving kill -3 signal OutOfMemoryError Hierarchy : Throwable -> Error -> VirtualMachineError -> OutOfMemoryError (Unchecked exception) Thrown when JVM runs out of space in various memory spaces or cannot proceed further with process execution. Some of the possibilities: Heap space full JVM already invoked full GC but could not free up space Heap may be sized smaller than app footprint or app is unnecessarily holding on to some set of objects in heap GC overhead limit exceeded Too many GCs with very less space claimed Application threads not getting any CPU cycles Requested array size exceeds VM limit PermGen space / Metaspace / compressed class space Full GC invoked but unable to free space in Metaspace and application is attempting to load more classes Metaspace by default “unlimited” but can be controlled by MaxMetaspaceSize. By default, 1 GB reserved for compressed class space Make sure that -Xnoclassgc is not in use as it prevents unloading of classes Native memory - out of swap space / stack trace with native method Native space used for Java thread stacks, loaded jars, zips, native libraries, native resources like files; mem allocated from native code Unable to allocate more native memory or to create new threads or native memory leaks Running 32 bit JVM on 64 bit machine puts 4 GB limit on process size Position of Java heap can put a cap on max size of native heap. Can be controlled by option -XX:HeapBaseMinAddress=n to specify address native heap should be based at CodeCache warnings warning message printed by JVM saying CodeCache full, compiler has been disabled. No OOME when code cache is full Emergency cleanup undertaken by Sweeper. This may discard compiled code and JIT may need to perform optimizations again Ensure appropriate size of CC using ReservedCodeCacheSize option Direct Buffer Memory ByteBuffer.allocateDirect(N) : Direct buffers which are garbage collected using phantom references and a reference queue Unlimited memory by default but can be controlled by -XX:MaxDirectMemorySize=n Used by Java NIO. Heap ByteBuffer for I/O uses temporary direct ByteBuffer Diagnostic Data, Data Collection and Analysis Tools Troubleshooting Memory leaks Confirm memory leak
...