| Author |
Message
|
| satya2481 |
 Posted: Thu Jul 16, 2015 4:23 am Post subject: Message Broker V7 and Performance in zLinux |
 |
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi All,
We are facing issues with the Non Functional testing with WMB V7 Broker running on zLinux environment. Posting this topic here to know with below Infrastructure and flows what is the TPS we can expect.
The non functional testing happening with 100TPS (Transactions Per Second) for 1 hour. Almost 10GB RAM is consumed and there are many messages getting piled up in the queue. We are able to achieve only up to 25TPS.
Broker details
Version = 7005
Fix pack details = CMVC Level: S700-FP05
Build type = Production, 64 bit, s390x_linux_2
Infrastructure and OS details
Architecture = s390x
CPU op-mode(s) = 32-bit, 64-bit
Byte Order = Big Endian
CPU = 3
Thread per core = 1
Core per socket = 1
Socket per book = 1
Books = 3
Bogo MIPS = 18115
Hypervisor = z/VM 6.2.0
Virtualization type = Full
RAM = 10GB
Implementation details
Number of Brokers = 2 (Broker Queue managers are clustered)
Number of Execution groups = 7 Per Broker
Number of Flow threads = 162 in total
High level functionalities implemented
1. Receive XML Request
2. Transformation1
3. HTTPS Call and receive response
4. Split the message into multiple messages (At the max 50 messages)
5. Transformation2
6. Sending request to External system over MQ
7. Receive response from External system and correlate with the request stored in DB
8. Check if all the responses are received from external system for the particular transaction
9. Send the consolidated response
Any information with which we can improve the performance and get better results will be of helpful.
Regards
Satya |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Jul 16, 2015 4:57 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20700
Location: LI,NY
|
Have you thought moving to IIB9 or IIB10?
You do realize that MB7 will be out of support by end of September? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 16, 2015 4:58 am Post subject: Re: Message Broker V7 and Performance in zLinux |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
| Any information with which we can improve the performance and get better results will be of helpful. |
How do you know it's a problem with WMB? How have you eliminated (for example) slow database response or poor transformational code? If there's a lot of RAM being used and slow response my first thought is a badly written JCN, but there's a certain amount of bias in that statement.
The infrastructure you've described should be easily capable of 100 TPS in "normal" situations, though of course having a normal situation is highly abnormal. You need to dive into the flow processing, work out where the time and memory are being eaten and only then will you be able to develop a strategy to resolve it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Thu Jul 16, 2015 6:23 am Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi fjb_saper
| Quote: |
| Have you thought moving to IIB9 or IIB10? |
Yes we thought of moving to IIB9 but client has to agree for the migration. We have already raised it with the Higher management. We feel it will take atleast next 6 months to decide and finalise.
| Quote: |
| You do realize that MB7 will be out of support by end of September? |
Same answer as above. We know that the support will end soon. Challenges with getting the new version of the software. But we are working towards it.
Hi Vitor,
We have taken care of below points with respect to code and the DB operations.
1. Not used CARDINALITY
2. Used reference variables instead of Index variables for the WHILE Loops
3. Indexes are created wherever multiple SQL Queries are used with different WHERE clauses
4. Entire code is written in ESQL and no Java coding
5. Observed that after the test memory is getting released and it will come back to normal state.
6. Configuration details are restored from table in Shared variables to avoid frequent DB operations.
We are thinking is the issue with the zLinux environment, will it work better in AIX Environment ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 16, 2015 6:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20700
Location: LI,NY
|
Can you be more specific about point 3?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 16, 2015 6:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
Hi Vitor,
We have taken care of below points with respect to code and the DB operations.
1. Not used CARDINALITY
2. Used reference variables instead of Index variables for the WHILE Loops
3. Indexes are created wherever multiple SQL Queries are used with different WHERE clauses
4. Entire code is written in ESQL and no Java coding
5. Observed that after the test memory is getting released and it will come back to normal state.
6. Configuration details are restored from table in Shared variables to avoid frequent DB operations. |
I was hoping for an answer more along the lines of:
"yes, we've examined the resource statistics and event monitoring, and have determined that everything is operating in a linear fashion".
What you've described there is the set of conditions that should lead to a performent system; you've in no way demonstrated that they do and indeed your testing indicates that it isn't. To pick on just 2 points:
- Nothing you've posted proves that the queries in point 3 are actually using the indexes you've defined. It's a common issue where (for a variety of reasons) a query ends up doing a table scan rather than using an index. It's also plausible that your queries are using the indexes, but the database is running like a snail dragging an anvil. You can set up anything you like in configuration but if the server's heavily loaded things will run badly.
- that the memory is released at the end of the test proves it's not a leak; it doesn't explain why these highly optimized ESQL flows you talk about are swallowing 10Gb of memory while they're running! What are they doing in there? Something is taking a lot of time and resource, and that's at variance with your own best practice list. 10Gb is a lot of memory for an ESQL flow.
| satya2481 wrote: |
| We are thinking is the issue with the zLinux environment, will it work better in AIX Environment ? |
Why are you thinking that? Or when you say "thinking" do you mean "hopeful" because simply moving to a different environment will enable you to "fix" the problem without actually expending any effort in investigation.
If you have an AIX environment, try it and see. My personal experience is that a zLinux installation performs better than an equivalent AIX installation (and the definition of "equivalent" can be debated).
From everything you've posted, you have a code or a DB problem rather than an installation or environment (wrong OS) problem. You need to investigate, not just say "well we coded it correctly so it must be working ok so it must be the OS" and cross your fingers that it runs ok on AIX. At least identify what sucks performance in zLinux and prove that AIX doesn't do the same thing.
To close, consider this: if you're right, zLinux is slow & inefficient, and 4 times slower than AIX. Why would anyone ever use it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 16, 2015 7:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
The investigation I've described might also reveal that it's something fairly straightforward in the coding or configuration, such as the flow fighting for the single thread in the core you've configured with something else that someone has running in the zLinux machine. Or an issue with your shared variable handling. Something that's much easier to fix than a port to AIX and might well occur there as well.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 16, 2015 8:06 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20700
Location: LI,NY
|
Finally use support pack IS03 in your performance/test environment and pinpoint for specific flows where the bottleneck actually is located...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| satya2481 |
| Posted: Thu Jul 16, 2015 11:15 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
fjb_saper wrote
| Quote: |
| Finally use support pack IS03 |
IS03 or IS02?
I have installed IS02 support pack in my development environment but unable to view the statistics of the flows. Start statistics is working but view statistics when clicked nothing is visible in the Explorer. I need to further investigate on that and see what is the issue.
But yes, that is something we are working on to find out more details on flow behavior.[/quote] |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 17, 2015 4:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20700
Location: LI,NY
|
| satya2481 wrote: |
fjb_saper wrote
| Quote: |
| Finally use support pack IS03 |
IS03 or IS02?
I have installed IS02 support pack in my development environment but unable to view the statistics of the flows. Start statistics is working but view statistics when clicked nothing is visible in the Explorer. I need to further investigate on that and see what is the issue.
But yes, that is something we are working on to find out more details on flow behavior. |
IS03 gives you a graphical view of the flow statistics, node by node. Each node represented by a square with numbers at each corner.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| satya2481 |
| Posted: Sun Jul 19, 2015 1:44 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
| Quote: |
| as the flow fighting for the single thread in the core you've configured with something else that someone has running in the zLinux machine |
How to identify this? We do not have access to these test servers. So if any information is required need to share the details with the server owner team. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 20, 2015 4:14 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
| Quote: |
| as the flow fighting for the single thread in the core you've configured with something else that someone has running in the zLinux machine |
How to identify this? |
You look at your earlier post:
| satya2481 wrote: |
Thread per core = 1
|
Let me pose the question a different way. If your expectation / requirement is 100 TPS then what information / redbook / experience leads you to think that this configuration of zLinux can achieve this? What led you (or your server team) to use this configuration for testing.
Do not say "because this is what the server team use for testing". I have a laptop on my desk; I couldn't use it to model an atomic explosion in real time because it's not got the power. Likewise I don't use my 4 core R&D Liniux server to estimate how well a flow will perform in my 32 core Prod server.
The investigation (including the support pacs mentioned) will help identify what's happening, including this.
Note, in conclusion, if you do decide the answer is to use AIX not zLinux, you've no guarantee the AIX machine the server team gives you won't be as underpowered as the zLinux one you've been supplied with.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jul 20, 2015 5:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20700
Location: LI,NY
|
What happens if you set:
Thread per core = 255
Sockets per core = 32768
Sockets per book / book per socket => talk to the admin and figure it out...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| satya2481 |
| Posted: Tue Jul 21, 2015 1:56 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
| Quote: |
What happens if you set:
Thread per core = 255
Sockets per core = 32768
Sockets per book / book per socket => talk to the admin and figure it out... |
I can check about this with the Admin team.
Vitor,
Thread per core = 1 : This information I got from Admin team only.
| Quote: |
| If your expectation / requirement is 100 TPS then what information / redbook / experience leads you to think that this configuration of zLinux can achieve this? What led you (or your server team) to use this configuration for testing. |
We do not know if with the above configuration we will be able to achieve 100TPS or not. We are slowly increasing the TPS and carrying out the testing. We are taking a trial and error approach.
Regarding AIX I raised it as a question. If anyone has worked and achieved this TPS. It is very difficult to get the different OS and Server at this stage.
Our production environment is having same configurations. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 22, 2015 4:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
| We do not know if with the above configuration we will be able to achieve 100TPS or not. |
According to your testing, it doesn't.
| satya2481 wrote: |
| We are slowly increasing the TPS and carrying out the testing. |
How are you "increasing TPS"? Telling WMBv7, "ok, now come on, push through the pain barrier, another 25 TPS"? You can't control TPS in v7 so what are you changing?
| satya2481 wrote: |
| We are taking a trial and error approach. |
Why? There's enough publications on zLinux and WMB performance to make an estimate, and again what are you changing? What information are you gathering from the trial to deduce the error.
| satya2481 wrote: |
| Regarding AIX I raised it as a question. If anyone has worked and achieved this TPS. It is very difficult to get the different OS and Server at this stage. |
I've used zLinux with an "average" sort of v7 flow and got 800 TPS out of it. I've done the same, with the same sort of flow, on AIX, Solaris and HP-UX. I've also got 1200 TPS with the same flow by reconfiguring the zLinux and currently am testing a highly optimized IIBv9 flow that yields 3500 TPS from a RHEL x64 virtual machine.
This illustrates almost nothing about your situation.
| satya2481 wrote: |
| Our production environment is having same configurations. |
If you need 100 TPS in Prod, you're likely to be disappointed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
