| Author |
Message
|
| exerk |
 Posted: Tue Sep 01, 2009 10:27 am Post subject: |
 |
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Vitor wrote: |
| ...I personally find a training programme based on the... |
...use of a baseball bat on the offender works wonders. Aversion therapy is a useful tool 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Sep 01, 2009 10:46 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9482
Location: US: west coast, almost. Otherwise, enroute.
|
This falls squarely into the realm of sysadmin target-lock. Target-lock is a military term that describes the natural tendency to focus on something (low/out of disk space) to the exclustion of all else. Target-lock leads to irrational thinking and irrational behaviors (behaviours).
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Sep 01, 2009 11:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I have a new phrase to torment admins with! 
This forum is so informative!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Sep 01, 2009 3:34 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| Quote: |
| If the secondaries are not used they are not there and nobody will delete them... |
An odd solution to this problem. As a clue to the clueless, you could name the log filepath /dontdeleteoryouwillbefired/. |
How about putting a "readme" file in each log directory that contains a suitable warning message. It could be cheap insurance if you are nervous about dodgy sys admins.
The best insurance is to make sure the MQ log file system is large enough, and monitoring and alerting is in place to inform the MQ support team if usage of the file system exceeds say 80%.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Sep 01, 2009 3:39 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9482
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| How about putting a "readme" |
Who reads readmes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 01, 2009 8:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
| gbaddeley wrote: |
| if usage of the file system exceeds say 80%. |
I hope you'd alert earlier. Isn't that the threshold where the qmgr starts shutting down?
Our alerts are set at 50% and 75 % (or used to be...) 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 01, 2009 11:26 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Stick it in the alert "Under NO circumstances are any files to be deleted in the directory without confirmation from WebSphere MQ Support!"
Of course, it fails miserably if there is no way to insert it into the alert...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Sep 02, 2009 4:19 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| fjb_saper wrote: |
| gbaddeley wrote: |
| if usage of the file system exceeds say 80%. |
I hope you'd alert earlier. Isn't that the threshold where the qmgr starts shutting down?
Our alerts are set at 50% and 75 % (or used to be...) |
50% is OK, particularly if a rogue application or a volume spike causes a lot of MQ logging and the log file system usage is growing fast, and it takes some time for the alert to propagate and the MQ support person to react. However, you could receive lots of false alerts and it essentially wastes 50% of the file system allocation.
_________________
Glenn |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 02, 2009 4:49 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
Into every /var/mqm and /var/mqm/log dir I put several dummy files each 1 GB in size named FillerFileOKtoDeleteToClaimSpace.
If my disk space threshhold is exceeded for logs or for queues I have several gigs of files I can safely delete to buy time. Or if some target-locked sys admin comes across my log dir, hopefully he will be happy in deleting my dummy filler files and will not delete the log files.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 02, 2009 10:35 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| PeterPotkay wrote: |
Into every /var/mqm and /var/mqm/log dir I put several dummy files each 1 GB in size named FillerFileOKtoDeleteToClaimSpace.
If my disk space threshhold is exceeded for logs or for queues I have several gigs of files I can safely delete to buy time. Or if some target-locked sys admin comes across my log dir, hopefully he will be happy in deleting my dummy filler files and will not delete the log files. |
I tried this, but when 'they' had a purge, deleted them anyway, and to pile on the misery, reduced the size of the file system ! Apparently I had "not got my original requirements correct" and they weren't going to "waste expensive disk by just having it sit there". This was from the same bunch that refused my original request for space, which was based on a max outage time etc. You can guess what happened when a DB went full somewhere...
...and it was still our fault - go figure 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Sep 02, 2009 11:12 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Something nice sysadmins have done for me is to dynamically increase the size of the MQ log file system until the problem had passed, provided they have space disk capacity or other file systems they can temporarily take space away from.
This can be very useful is usage is stuck at 100% and there are no longer any files that I can safely delete and the queue manager refuses to start.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 03, 2009 5:48 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9482
Location: US: west coast, almost. Otherwise, enroute.
|
We are talking about a fairly inexpensive commodity here, aren't we? Disk is far cheaper than our collective effort to manage it.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 03, 2009 5:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
| We are talking about a fairly inexpensive commodity here, aren't we? Disk is far cheaper than our collective effort to manage it. |
You'd be amazed how many management believe our time is cheaper than hardware. Or how many sites can recruit people like us faster than they can buy hardware.
Weird yet true.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Sep 03, 2009 3:32 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| We are talking about a fairly inexpensive commodity here, aren't we? Disk is far cheaper than our collective effort to manage it. |
Acquiring new disk for a production server can be a time consuming and expensive process. You can't just walk down the high street and lay down ~$100 for a new 1TB disk and plug it into a spare esata socket.
I find it bizarre that my laptop has more disk capacity than some of our production Unix servers.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 03, 2009 3:41 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9482
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Acquiring new disk for a production server can be a time consuming... |
I agree completely.
When I was in this loop, I regularly reported to management the current state of disk (and other tech things) utilization; so that they knew months in advance (based on expected and unexpected growth) when we were approaching the need to upgrade.
This can be managed; but not in crisis mode.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
