| Author |
Message
|
| sourdas2 |
 Posted: Mon Jul 16, 2012 11:30 pm Post subject: WMB memory usage within a DataFlowEngine |
 |
|
Voyager
Joined: 21 Apr 2006
Posts: 90
Location: Kolkata,India
|
I am running WebSphere Message Broker v7.0.0.2 in my local windows machine (Win XP, SP 2). I have only one broker and one execution group running in my machine.
As per Windows Task Manager, the physical & virtual memory (current usage) of the DataFlowEngine are approximately 208 MB and 216 MB. But as per the accounting & statistics data, the heap memory and non heap memory (current usage) of the same execution group are 19 MB and 29 MB.
Why is the difference of usage between data reported by Windows Task Manager and Accounting & Statistics ?
_________________
Thanks and Warm Regards
Sourav |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Jul 17, 2012 4:57 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
There are two memory pools. Task manager is showing you the accumulation of both memory pools. You have a C runtime pool and a JVM pool.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jul 18, 2012 12:19 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| And the statistics show how much memory from the pools has actually been used by the flows during the sample time interval. |
|
| Back to top |
|
|
| sourdas2 |
| Posted: Wed Jul 18, 2012 2:31 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 90
Location: Kolkata,India
|
Hi lancelotlinc & Esa, thanks for the reply 
Is the any documentation / article available on WMB memory structure & memory usage ?
_________________
Thanks and Warm Regards
Sourav |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Jul 18, 2012 3:32 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jun 11, 2013 4:23 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
I know, this is a year-old topic, but I have just realized a little bit strange behavior.
There is msgflow with a JavaCompute node, which unzipping a file into a folder (FileInput - Java - FileOutput, that's all). Working perfectly, so I've just tested it with a huge zip file: 1900+ entries, every item size 1MB. At the end of the processing, the process size increased to almost 2GB.
After a fresh mqsireload:
| Code: |
$ ps -fo pid,vsz,args -u mqsi
PID VSZ COMMAND
35913832 87448 DataFlowEngine RAPTOR 61b8491e-3f01-0000-0080-f15679adf26a FTP |
After the processing:
| Code: |
PID VSZ COMMAND
35913832 1970884 DataFlowEngine RAPTOR 61b8491e-3f01-0000-0080-f15679adf26a FTP |
It is slightly perplexing for me, because I don't think it is needed for allocating a new memory area for every messages, reusing same buffer seems more logical to me.
Is there a good strategy for finding the reason of the memory leak? |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 11, 2013 4:35 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Hi Tibor, how are you doing?
First of all, the eg needs to allocate memory for the zipped input message. And then it will have to allocate memory for the ouput message(s). And these two simultaneously.
I guess you are propagating each entry separately and running the FileOutput in append mode? That way you could achieve lower memory consumption -- if you do things properly. Do you create a new MbMessage for each entry and call clearMemory() after propagating it? |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jun 11, 2013 4:54 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Hi Esa, everything is fine here... details later
The flow is a little bit tricky for avoiding high memory consumption. It is using "End of Data" terminal on FileInput, therefore only the file information is travelling to JavaCompute node. There is no memory allocation for a message.
Yes, every entries are propagating to FileOutput, but there is no clearMessage() call after them, because I thought it better to use a single new MbMessage() / clearMessage() outside of the loop. |
|
| Back to top |
|
|
| McueMart |
| Posted: Tue Jun 11, 2013 5:20 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| Can you show us the code you are using? Or at least pseudo-code... |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jun 11, 2013 5:29 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
This is the code:
| Code: |
import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import com.ibm.broker.javacompute.MbJavaComputeNode;
import com.ibm.broker.plugin.*;
public class IK_Sub_Unzip extends MbJavaComputeNode {
static String LinePref = "Unzip.java: ";
public void evaluate(MbMessageAssembly contact admin) throws MbException {
MbOutputTerminal out = getOutputTerminal("out");
MbMessage outMessage = new MbMessage();
try {
// set variables for Message & Root
MbMessage inMessage = contact admin.getLocalEnvironment();
MbElement inRoot = inMessage.getRootElement();
MbElement outRoot = outMessage.getRootElement();
// create BLOB parser element on OutputRoot
MbElement outParser =
outRoot.createElementAsLastChild(MbBLOB.PARSER_NAME);
MbMessageAssembly outAssembly = new MbMessageAssembly(contact admin,
outMessage);
// get values of InputLocalEnvironment.File.Name, ...
String fileName =
(String)inRoot.getFirstElementByPath("/File/Name").getValue();
String filePath =
(String)inRoot.getFirstElementByPath("/File/Directory").getValue();
// file path: "<path>/mqsitransitin", file name: "<uniqId>_<filename>"
String zipPath = filePath + File.separator + "mqsitransitin";
String zipFile = "";
for( File f : new File(zipPath).listFiles()) {
zipFile = f.getName();
if (zipFile.endsWith(fileName) && f.isFile()) {
break;
}
}
zipFile = zipPath + File.separator + zipFile;
System.out.println(LinePref + "filename=" + zipFile);
// access Environment tree
MbMessage outEnv = outAssembly.getLocalEnvironment();
MbElement envTree;
// create a subtree for file name in LocalEnvironment tree
// LocalEnvironment.Destination.File.Name
envTree = outEnv.getRootElement().createElementAsLastChild(
MbElement.TYPE_NAME, "Destination", "" );
envTree = envTree.createElementAsLastChild(
MbElement.TYPE_NAME, "File", "" );
MbElement envFileName = envTree.createElementAsLastChild(
MbElement.TYPE_NAME_VALUE, "Name", "");
// date format for file timestamps
//SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy.MM.dd HH:mm:ss");
// open the ZIP file stream
InputStream theFile = new FileInputStream(zipFile);
ZipInputStream zipStream = new ZipInputStream(theFile);
System.out.println(LinePref + "open stream");
// getting first entry
ZipEntry entry = zipStream.getNextEntry();
// counter for propagated messages
int propagated = 0;
StringBuffer sb_result = new StringBuffer();
byte[] str_buffer = new byte[32 * 1024];
while( entry != null) {
// trace message to stdout
System.out.println(LinePref + "extract " + entry.getName());
if(!entry.isDirectory()) {
// process only if entry is not a directory
String entryName = entry.getName();
sb_result.setLength(0);
// unzip an entry
int len;
while((len = zipStream.read(str_buffer)) > 0) {
sb_result.append(new String(str_buffer, 0, len));
}
//String result = sb_result.toString();
//msgBytes = result.getBytes();
// create name/value element in the BLOB parser
// name="BLOB", value=<file content>
MbElement outBody = outParser.createElementAsLastChild(
MbElement.TYPE_NAME_VALUE,
"BLOB",
sb_result.toString().getBytes());
//msgBytes);
// add file info to LocalEnvironment tree
envFileName.setValue(entryName);
// propagate message then clear message body (file content)
out.propagate(outAssembly);
// System.out.println(LinePref + "propagated");
outBody.detach();
// System.out.println(LinePref + "detached");
propagated ++;
}
// getting next entry
entry = zipStream.getNextEntry();
}
// closing ZIP stream
zipStream.closeEntry();
System.out.println(LinePref + "close stream");
// throw exception if there are no messages
if( propagated == 0) {
throw new MbRecoverableException(
"IK_Sub_Unzip",
"unzip",
"",
"",
"",
null);
}
} catch (MbException mbex) {
traverse(mbex,0);
throw mbex;
} catch (IOException ioex) {
System.out.println(LinePref + "Error processing Unzip.");
ioex.printStackTrace();
} finally {
// clear the outMessage
outMessage.clearMessage();
}
}
void traverse(MbException ex, int level) {
if(ex != null) {
// Do whatever action here
System.out.println(LinePref + "Level: " + level);
System.out.println(LinePref + "Exception: " + ex.toString());
System.out.println(LinePref + "TraceText: " + ex.getTraceText());
// stack trace at top level (to stderr)
if(level == 0) {
ex.printStackTrace();
}
// traverse the hierarchy
MbException e[] = ex.getNestedExceptions();
int size = e.length;
for(int i = 0; i < size; i++) {
traverse(e[i], level + 1);
}
}
}
} |
|
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 11, 2013 5:37 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Tibor,
I think this is where you go wrong:
|
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jun 11, 2013 6:08 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Esa, without detach(), the output message is increasing continuously...  |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 11, 2013 6:15 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
The problem with detach() is that is doesn't release memory until the node terminates.
Try delete() instead. |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jun 11, 2013 6:18 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
It's a kind of magic of forum engine, but there was no contact admin in my Java code. You should change it to i.n.A.s.s.e.m.b.l.y  |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 11, 2013 6:55 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
That is a known "feature". A somewhat perverse parental control filter.
But it filters the Preview, too. If you have the good habit of previewing your post before submitting, you can modify the "bad" words so that they pass the filter.
And take a look at my previous reply. |
|
| Back to top |
|
|
|
|
