| Author |
Message
|
| fjb_saper |
 Posted: Thu Sep 16, 2010 7:37 pm Post subject: |
 |
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20763
Location: LI,NY
|
| PeterPotkay wrote: |
| SAFraser wrote: |
The zfs filesystem for sparse zones does synchronous writes to disk. First data is written to buffer, then to disk. Control is not returned to the application until the write to disk is successful. In zfs, this introduces noticeable latency.
|
But PUTs were faster than GETs? I would think both would be equally affected. |
My understanding is that it would not let you read from the buffer until it was written to disk. As you write to the buffer the put operation is not really affected. But you cannot get until the corresponding buffer has been written to disk... so you have io involved a) to write to the disk and b) to read from the disk.... hence the delays...
Maybe I misunderstood SA's explanation... 
_________________
MQ & Broker admin |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Sat Sep 18, 2010 2:11 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| SAFraser wrote: |
| Unfortunately, running without synchronous write protection is not feasible. If the server crashes with messages in the buffer, we believe that the queue manager would not come back up as its logs would not reflect correct information. |
The LogWriteIntegrity parameter in the qm.ini is something that may have an impact here.
If you are feeling solidly that you need to rely on the synchronous write of the file system, then you can consider reducing LogWriteIntegrity to SingleWrite rather than taking the default of TripleWrite.
This would presumably increase performance without sacrificing reliability.
If, on the other hand, you are not comfortable going without TripleWrite, you might find that you trust it *more* than the synchronous write of the file system, and thus can get away with turning that off and relying on TripleWrite. Again, an increase in performance.
It's not my tradeoff to make, but I'd personally trust TripleWrite more than the synchronous write. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Sep 18, 2010 7:06 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9475
Location: US: west coast, almost. Otherwise, enroute.
|
This is one more example of the triad of cost, quality and time. In this triad, you only successfully get two of the three.
So, in order to maximize time, you sacrifice quality. If you/your organization wants to maximize time and quality, it will cost you more.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Sep 18, 2010 9:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20763
Location: LI,NY
|
Moving it off to SAN storage instead of using zfs might also be an alternative here, and is the short term solution SAFraser took
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Sep 18, 2010 4:22 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
...
I think I want to disagree with both fjb_saper and bruce2359.
I think it's a case of deciding whether or not to pay double for the same value - in terms of enabling both synchronous write AND TripleWrite integrity.
There are various options to avoid paying double - one of which is indeed moving the log file system onto a disk service that does NOT include synchronous write.
But that's really the *same* as disabling the synchronous write.
So I'm not sure where there are secondary options than either changing TripleWrite integrity or disabling synchronous write - just differing approaches to making the same choice.
And as for sacrificing quality to achieve time, again I think it's not that. I think it's a question of choosing to NOT use doubly-redundant systems that sacrifice time for a LACK of increase in quality.
TripleWrite integrity ensures that in all cases the data on the log files will be accurate.
Synchronous write ensures that in all cases, the application writing the data will be ful;ly enformed as to whether or not the write operation succeeded, and so it can retry it if necessary.
Retrying the second of three TripleWrite log attempts, because the synchronous write says it failed, is really a waste of effort because the third write was going to retry it already. |
|
| Back to top |
|
|
| mvic |
| Posted: Sat Sep 18, 2010 4:57 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| mqjeff wrote: |
| Retrying the second of three TripleWrite log attempts, because the synchronous write says it failed, is really a waste of effort because the third write was going to retry it already. |
Did you want to phrase it like that?
Triple-write is a mechanism for working around less-than-perfect behavior in the IO system. http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa12640_.htm
It's not a mechanism for retrying failed IO calls. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Sep 19, 2010 4:41 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| mvic wrote: |
| mqjeff wrote: |
| Retrying the second of three TripleWrite log attempts, because the synchronous write says it failed, is really a waste of effort because the third write was going to retry it already. |
Did you want to phrase it like that? |
I'm not certain.
Well, yes, I agree with that much. My point was that with a synchronous write, the program executing the write gets back a return code that says that the write succeeded or failed and can RELY on that result code. It can then consequently decide for itself if it wants to retry it.
I don't know whether the qmgr will attempt to retry a failed write in either case.
But TripleWrite is used when you can't rely on the file system. I guess I'm not certain that the synchronous write in this case does provide the kind of guarantee that is talked about in that section of info center either. But I'm not sure how much good it would be if it wasn't.
It's at least something for SAFraser to have a play with or a further discussion with the system engineers and perhaps MQ support on. |
|
| Back to top |
|
|
| SAFraser |
| Posted: Sun Sep 19, 2010 7:13 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
Sorry for the delay in adding information.... We are thirteen days from go live at the new data center. Sleep optional.
The OS tuning parameter with which we tinkered was zfs_nocacheflush.
The discussion of TripleWrite integrity is certainly valuable. But we had no performance degradation with PUTs. The performance degradation is the GETs. Yet, turning off synchronous writes to the OS fixed the GETs performance problem.
I don't get it.
We are doing the entire load again this week on a machine that has only a global zone. It will be a run of about 700,000 messages and we will use the actual applications to do the PUTs and GETs (rather than qload and amqsget).
Your further thoughts are most welcome. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Sep 19, 2010 9:45 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9475
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| SAFraser |
| Posted: Mon Sep 20, 2010 4:24 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
Bruce, I had never found this URL before -- thank you! Our team will study it.
All -- do you think it's worth a test to reduce the log writes to SingleWrite with the zfs synchronous writes enabled?
We have very few non-persistent messages. Nearly everything is persistent. (Even if they shouldn't be. The term "non-persistent" is apparently frightening.)
Could it be.... on a PUT, there is write/write/write. But on a GET, there is read/write/write/write. And that's why the GET is slower?
I've read and read about how circular logs worked, and I thought I knew it pretty well. I didn't think a GET caused a write operation to the log, but we see that it does. And I guess it would have to, for persistent messages. Is that correct? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Sep 20, 2010 4:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| SAFraser wrote: |
| All -- do you think it's worth a test to reduce the log writes to SingleWrite with the zfs synchronous writes enabled? |
As you seem to be leading the pack on WMQ in Solaris Zones, then yes. I'd personally be interested. Especially as you're the kind of skilled & methodical person who'd do it right and produce useful results.
| SAFraser wrote: |
| We have very few non-persistent messages. Nearly everything is persistent. (Even if they shouldn't be. The term "non-persistent" is apparently frightening.) |
Application people traditionally don't like the idea that their messages are expendable. After all the trouble their little applications go to to generate them. It's a shame, because it makes them trout targets. Especially on sites I adminsiter.
| SAFraser wrote: |
| I didn't think a GET caused a write operation to the log, but we see that it does. And I guess it would have to, for persistent messages. Is that correct? |
Yes, and it must update the queue's file to show the message logically deleted.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Sep 20, 2010 8:48 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
SAFraser,
Confirm the syncpoint options being used for both puts and gets. It can have a substantial effect at the level of detail you are working at.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
