| Author |
Message
|
| jonny |
 Posted: Fri Apr 15, 2005 1:24 am Post subject: Persistent messages |
 |
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Hi,
MQ log is defined as default, which will give me Circular log with a maximum capacity of 20MB.
I was putting persistent messages to a queue, and I managed to get the queue size (by lookijg at the size of the queue file to more than 50MB. No application is reading from that queue.
My understanding is that for each persistent message, a copy is first written to the log and then to the queue file. How can you explain that I was able to write more than 20MB in terms of message size to the queue, whitout getting an error about log being full.
I am using MQ V5.3 CSD8 on Solaris
Thanks |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Apr 15, 2005 1:48 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Because once the message is put on the queue, it doesn't need to stay on the log any more.
So the message in the log can be erased later, when the circular logging comes around to the same place in the log files again.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Fri Apr 15, 2005 2:32 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
| There is a good explanatoin of how msgs are stored in the queue files and persistent logs in the System Admin manual. |
|
| Back to top |
|
|
| jonny |
| Posted: Mon Apr 18, 2005 7:59 am Post subject: |
|
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Thanks for your replies.
Using Linear logging: when it comes to define the number of primary and secondary files:
1)- I don't see what the point here if I am using linear logging. It wouldn't make a diffeence if I define 3 or 60 primary files, as log files will be created whenever is needed untill there is no more disk space. Also what's the point of secondary files?
2)- If persistent messages will disappear from the log (they could have been overriden) once they are written to queues, how would then MQ know when a queue manager is restarted, which messages were persistent and put them back on the queue? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Apr 18, 2005 8:17 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The number of log files is always the number of active log files, not the total number of log files.
And the number/size of the active logs directly decides how many and how large transactions you can run.
So this makes a big difference if you define 3 or 60 log files.
Please read the System Administration and Intercommunications guides.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| jonny |
| Posted: Tue Apr 19, 2005 9:39 am Post subject: |
|
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Jeff,
I have trun the following tests using persistent messages:
1)- Created a queue manager with 2 primary and 2 secondary files with log size set to 2MB and using linear logging, which will give me a maximum log size of 4MB
So at this stage I only have 2 logs under the log directory
S0000000.LOG and
S0000001.LOG
Log required to start queue manager is S0000000.LOG
I then Put a persistent message to a local queue using rfhutil (not the client version) message size on the queue as shown on the queue is 2,099,869 which is bigger than the the size of the primary files, so I was expecting one secondary file to be created, but there are five logs now:
S0000000.LOG
S0000001.LOG
S0000002.LOG
S0000003.LOG
S0000004.LOG
"Log required to start queue manager is S0000000.LOG"
I restarted the queue manager:
"Log required to start queue manager is S0000002.LOG"
Why did the queue manager had to allocated three more files to process a message of just over 2MB, I would have expected it to allocated just a third one (secondary) i.e.
S0000000.LOG
S0000001.LOG
S0000002.LOG
The message which gets written to the AMQERROR file, indicates the starting point of active files, so from the second message above, I understand that active log files are:
S0000002.LOG
S0000003.LOG
S0000004.LOG
Which of the abive is the secondary file? is it S0000004.LOG? and do secondary files get deallocated once space has been freed from the primary files?
I managed to put a 10MB persistent message to a local queue, why was this possible if the maximum log capacity is 4MB? (log required to start queue manager is now S0000010 after I restarted the queue manager, and there S0000011 exists)
I will carry out some testing tomorrow regarding channels (SDR and RCVR)
Thanks |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Apr 19, 2005 9:47 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
RFHUtil probably doesn't use syncpoint.
If you're not using syncpoint, then messages to local/xmit queues are written directly to the queue file, I believe.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vennela |
| Posted: Tue Apr 19, 2005 9:52 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| If you want to PUT using syncpoint, then you might use the support pac called Q program. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Apr 19, 2005 5:41 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20738
Location: LI,NY
|
As well read up on logs and backup recovery.
Linear logging and circular logging are 2 different animals...
 |
|
| Back to top |
|
|
| jonny |
| Posted: Wed Apr 20, 2005 4:15 am Post subject: |
|
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Thanks Jeff and Vennela,
I have run the foolowing tests
1) - Persistent messages (no syncpoint) and message size is bigger than the maximum log size (I am putting a single message)
Using Circular Log:
--------------------------
AMQ7463: The log for queue manager is full.
Using Linear Log:
------------------------
No problem (as long as there is enough disc space)
****** Why?
2) - Persistent messages (with syncpoint) and UOW is bigger than the maximum log size (I am using Q program to unload messages from a file)
The following message is reported for both circular and linear.
AMQ7469: Transactions rolled back due to log space
3)- Non-persistent messages (no syncpoint) and message is bigger than the maximum log size (I am putting a single message)
No problem for both types of log(as long as there is enough disc space)
****** That's fine, because non-persistent messages are not written to the log
4) - Non-persistent messages (with syncpoint) and message is bigger than the maximum log size (I am using Q program to unload messages from a file)
For both, circular and linear logs, messages were not put onto the queue, and NO error messages were reported. Basically messages were lost.
Here I think it is using shared memory to hold non-persistent messages.
Could someone explain to me why test number (1) for linear log was successful?
Thanks |
|
| Back to top |
|
|
| PGoodhart |
| Posted: Wed Apr 20, 2005 5:40 am Post subject: |
|
|
Master
Joined: 17 Jun 2004
Posts: 278
Location: Harrisburg PA
|
Because there is no "Maximum Log Size" for a linear log. Basically it will just keep adding logs until it consumes all the available disk space. This is one of the reasons usually you want to use linear logs on production systems that make use of persistent messages. (Obviously this isn't even the main reason, that's recoverablity...)
_________________
Patrick Goodhart
MQ Admin/Web Developer/Consultant
WebSphere Application Server Admin |
|
| Back to top |
|
|
| jonny |
| Posted: Thu Apr 21, 2005 2:44 am Post subject: |
|
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Patrick,
But the "maximum log size" for linear logging should be the size of primary + secondary files defined for a queue manager (qm.ini). The manual referes to a concept of active files, the maximum number of active files is limited by the number of primary + secondary files.
Here is a paragraph from the system administration manual:
| Quote: |
If a new checkpoint is recorded in the second, or later, primary log file, the first file
becomes inactive and a new primary file is formatted and added to the end of the
primary pool, restoring the number of primary files available for logging. In this
way the primary log file pool can be seen to be a current set of files in an
ever-extending list of log files. Again, it is an administrative task to manage the
inactive files according to the requirements of your operation.
Although secondary log files are defined for linear logging, they are not used in
normal operation. If a situation arises when, probably due to long-lived
transactions, it is not possible to free a file from the active pool because it might
still be required for a restart, secondary files are formatted and added to the active
log file pool.
If the number of secondary files available is used up, requests for most further
operations requiring log activity will be refused with an
MQRC_RESOURCE_PROBLEM return code being returned to the application.
|
And if you're saying there is no maximum log size for linear logs, why did test (2) above failed for linear logging??
It looks like for linear logging there is a limitation to the size of UOW when using persistent messages, but there is no limitation if persistent messages are used outside UOW, abd that's what confuses me, because persistent messages are written to the log regardless whether UOW is used.
Thanks |
|
| Back to top |
|
|
| PGoodhart |
| Posted: Thu Apr 21, 2005 3:51 am Post subject: |
|
|
Master
Joined: 17 Jun 2004
Posts: 278
Location: Harrisburg PA
|
Keep reading.
There is no limit for linear logging. (We've tested 1 GB persistent segemented messages on linear logs.) I believe that is born out by the manuals, though it is a little obtuse.
There is a difference between "active" logs and just log space.
Under a transaction you may be filling the "active" log space which I believe is the "addressable/active" logs in memory (meaning you just hit a wall). This may be your issue, so linear logging is not limited, but transactions size is.
Honestly I spent a lot of time working out what/how logging should be used on MQ. If you never use persistence, segmentation, or messages over 100 MB then circular logging will work just fine. With segmentation, persistence, and big messages, you want linear logs.
_________________
Patrick Goodhart
MQ Admin/Web Developer/Consultant
WebSphere Application Server Admin |
|
| Back to top |
|
|
| jonny |
| Posted: Thu Apr 21, 2005 5:21 am Post subject: |
|
|
Acolyte
Joined: 03 Jul 2003
Posts: 57
|
Patrick,
Thanks again for your reply.
You've mentioned that you've tested 1GB persiostent segmented messages on linear logs, were they transactional. i.e were they part of one UOW?
The configuration for my tests were: 2 primary and 2 secondary files, with log size set 1MB, in total I will have 4MB.
If there is no limit for linear logging, what's the point of defining the number of primary and secondary files???
Thanks |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Apr 21, 2005 7:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| jonny wrote: |
| If there is no limit for linear logging, what's the point of defining the number of primary and secondary files??? |
How many times do you need to hear an answer to this question?
The number of primary and secondary log files determines the total size of all messages in all open transactions.
This is includes both application transactions and channel transactions. So do not try and send a 100MB message through a channel if your log files are 4MB in size.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
