mapreduce - How to see hadoop's heap use? -

I am working as a school for analyzing the use of the pile in the HAUP. It involves the running of two versions of the program to calculate the length of the forum annotations: The first is a 'memory-unconscious' and handles the program in memory with the length of each comment; The second is 'memory-aware' and the reducer uses a very memory-efficient data structure to handle data.

Objective Both programs process data of different sizes and how memory usage goes up fast (unless it ultimately does not exit the memory).

My question is how can I use the heap's pile or reduce the work?

I will include this data to the counter "use of total committed heap" (bytes) ", but I think both versions of the program get back almost identical values.

Regarding the accuracy of the programs, 'memory-unconscious' emits out of memory with a large input and fails, while no one else is able and able to finish. > Thanks in advance

I do not know which memory-sensitive data structure (If you can help it anymore), but most memory data structures use virtual memory, it means the size of the data structure is somewhat increased depending on the additional data element / s on the basis of the data dump in the virtual memory Will happen. Therefore, we do not get the result of out-of-memory error. But if memory unconscious does not do this, in both cases the data structure size will remain the same. That is why you are getting the same size. To achieve real memory usage by Reducer, you can get it:

New feature added. Java 1.5 is the instrumentation interface through which you can get object memory usage ( getObjectSize ). Good article about this:

  / * Returns the amount of free memory in the Java Virtual Machine. Calling GC method can increase the value returned by free memory. * / Long free memory = runtime.gettime (). FreeMemory () / * Returns the maximum virtual memory that will try to use the Java Virtual Machine. If there is no implied limit then the value will be returned to Long.MAX_VALUE. * / Longest maximum memory = runtime.gettimetime (). MaxMemory (); / * Returns the total amount of memory in the Java Virtual Machine. Depending on the host environment, the prices given by this method can vary from time to time. Note that the amount of memory required to keep any type of object may be implementation-dependent * / long memory = runtime.gettime (). Total memory ()