MQSeries.net :: View topic - MQ FTE V7.0.2 - New Feature

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ File Transfer Edition » MQ FTE V7.0.2 - New Feature - FTP Bridge

Goto page 1, 2 Next

MQ FTE V7.0.2 - New Feature - FTP Bridge

« View previous topic :: View next topic »

Author

Message

JKulcyk

Posted: Wed Oct 07, 2009 12:43 pm Post subject: MQ FTE V7.0.2 - New Feature - FTP Bridge

Novice

Joined: 24 Apr 2009
Posts: 16

I am looking at the announcement letter for MQ File Transfer Edition 7.0.2 (209-300, October 2, 2009). It mentions the ability to bridge to FTP, SFTP, and SSH servers.

"The source or destination agent for a file transfer can be an FTP for SFTP server that bridges the transfer to the non-MQ protocol. File transfers bridged to FTP and SFTP retain auditability and reliability. There is no need for WebSphere MQ or WebSphere MQ FTE to be installed at the FTP or SFTP location. Bridging agents for FTP and SFTP is fully integrated into the graphical, command line, and scripting interfaces."

Does anyone know any more details about this? Do I need to install an FTP server, or do the MQ agents act like one and then pass the file through the MQ network? Can files be sitting on a queue until someone FTP's into the service and is able to pick them up? Can files be directed to different or the same queues depending on the id they log in with?

Availability isn't till Nov 27 so I may be asking too early but I'm curious if anyone has any experience with it yet.
Thanks.
-John

PeteVerdon

Posted: Fri Oct 16, 2009 6:47 am Post subject:

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

Hi John,

An agent configured to do FTP acts like an FTP client. A good way to envisage it is as a normal agent which has had its connection to its local disk replaced with an FTP connection. So if you send a file to the agent it will end up on the FTP server, and similarly you can request a file from the agent and it will actually be taken from the server.

If you wanted to set up a system to which files are delivered over FTP, I'd suggest you set up a normal FTP server and have an agent (not in bridge mode) monitoring the FTP server's directory and forwarding any files that are uploaded. A very similar setup would be used if you wanted a system where files come in over FTE but are then made available for FTP download. FTE does not provide FTP server functionality.

Your last two questions are difficult to answer in an FTE context, since files don't really sit on queues in the same way that messages do in MQ. A file only exists in usable form when at rest on disk - the "wire format" between FTE agents is not public and as far as I know nothing except FTE currently understands it.

Pete

JKulcyk

Posted: Fri Oct 16, 2009 7:01 am Post subject:

Novice

Joined: 24 Apr 2009
Posts: 16

Pete,
Thanks for your reply. I understand how FTE currently works, just a couple of agents putting files on one queue and pulling them off at the other end, using all the goodness of MQ in the middle.

I'm looking at the announcement letter for FTE 7.0.2, it talks about the "Ability to bridge to FTP and SFTP servers. The source or destination agent for a file transfer can be and FTP or SFTP server that bridges the transfer to the non-MQ protocol." We already dump files from FTP into a directory then have a job sweep the directory to pick them up and send them via MQ. It would be nice to have a more seamless connection, from the ftp client to our ftp server to the destination queue without having to watch directories. It looks like 7.0.2 might provide that, but the details are a bit lacking. I might have to wait for GA of 11/27 and play with it.

Thanks!

-John

zpat

Posted: Sat Oct 17, 2009 10:30 am Post subject:

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

This is a feature which I suggested to IBM (and probably others did as well). Good to see it incorporated.

PeteVerdon

Posted: Mon Oct 19, 2009 4:43 am Post subject:

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

JKulcyk wrote:

It's actually not quite that simple. If we were to just pump file data as fast as we can from the disk to the queue, big files would rapidly fill up the MQ network. Imagine if the agent at the other end were down, and you tried to send it a 1000GB file - ask your MQ admin if he minds you keeping 1000GB of file data in his queue buffers until that agent came back up.

Further, to get the reliability needed with that simple approach, you'd have to send all the data messages in persistent mode. Now, the folks over in the next building try very hard to make MQ fast, but even so persistent messages are necessarily slower than non-persistent messages as they have to be hardened to disk at each point. To provide the speed people need, FTE sends the bulk file data as non-persistent messages, implementing its own reliability/recovery protocol on top. That protocol does take advantage of MQ persistence for its control messages, but it's more complex than simply cutting a file up into chunks and pushing them as messages onto a queue.

JKulcyk wrote:

I'm looking at the announcement letter for FTE 7.0.2, it talks about the "Ability to bridge to FTP and SFTP servers. The source or destination agent for a file transfer can be and FTP or SFTP server that bridges the transfer to the non-MQ protocol." We already dump files from FTP into a directory then have a job sweep the directory to pick them up and send them via MQ. It would be nice to have a more seamless connection, from the ftp client to our ftp server to the destination queue without having to watch directories. It looks like 7.0.2 might provide that, but the details are a bit lacking. I might have to wait for GA of 11/27 and play with it.

As you'll see from my location, I have a bit more to go on than the letter alone

The FTP support in 7.0.2 is as I describe - it bridges to (S)FTP servers, as stated in the sentence you quote, by implementing an FTP client. 7.0.2 does not provide FTP server functionality.

Thinking about this, I'm not sure that providing an FTP server interface to FTE would even be possible. In FTP it's the client that is in control of the transfer; if the agent (acting as FTP server) needed to resend some data, or restart a transfer (due to network failures between it and the destination) it would have no way to ask the client to do so.

Pete

JKulcyk

Posted: Mon Oct 19, 2009 9:40 am Post subject: FTE 7.0.2

Novice

Joined: 24 Apr 2009
Posts: 16

Pete,
Thanks for all the info, that's all good stuff to know. I was curious as to how it would work as an ftp server, it seemed like it would require a fair amount of magic to work cleanly.

One more question - Does the fte/ftp client integration require an agent on both ends, moving an existing file out through ftp? Or can I generate data and write it directly to a queue, having it go out through ftp using some id/pw parameters that have been specified. Most of the information travelling over our MQ network is written from applications and moves to an edge server where it gets written to a file and ftp'd/emailed from there. I am hoping we can do it without having to create intermediate files.

Thanks in advance!
-John

PeteVerdon

Posted: Tue Oct 20, 2009 1:55 am Post subject: Re: FTE 7.0.2

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

JKulcyk wrote:

One more question - Does the fte/ftp client integration require an agent on both ends, moving an existing file out through ftp? Or can I generate data and write it directly to a queue?

The answer to this applies both to FTP and normal FTE operation - at present, the link is between agents only. It is not possible with current versions of FTE (or the upcoming 7.0.2) to load data in a usable form onto a queue, or to take arbitrary data from a queue and write it as files.

I believe some people are doing this using WebSphere Message Broker, but we are aware of the requirement to include the function directly in FTE itself. We have many such requirements in a regularly-updated priority queue for implementation; I'm afraid I cannot comment publicly on any individual requirement's position in that queue.

Pete

PeterPotkay

Posted: Tue Oct 20, 2009 4:09 am Post subject:

Poobah

Joined: 15 May 2001
Posts: 7716

PeteVerdon wrote:

JKulcyk wrote:

Consider 2 FTE Client using a single Queue Manager as their FTE server - #1 If the sending FTE client wants to start sending flies, but the receiving FTE agent is down, will the transfer begin?

#2 If yes, they will just queue up on the Queue Manager, so an WMQFTE Admin must plan for this by allocating enough disk space for the queue files to handle potential large queueing?

#3 How much queueing will it allow? Are there settings in WMQFTE, or is it just upping Max Q depth on some WMQFTE specific queues?

#4 Do these answers change when the sending and receiving WMQFTE agents are connected to different QMs?
_________________
Peter Potkay
Keep Calm and MQ On

PeteVerdon

Posted: Tue Oct 20, 2009 5:10 am Post subject:

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

PeterPotkay wrote:

Consider 2 FTE Client using a single Queue Manager as their FTE server

This statement reveals a common misunderstanding about FTE topologies and the difference between client and server. I'd recommend that you have a read of this blog post: https://www.ibm.com/developerworks/mydeveloperworks/blogs/pete_fte/entry/clients_and_servers46

PeterPotkay wrote:

#1 If the sending FTE client wants to start sending flies, but the receiving FTE agent is down, will the transfer begin?

Depends on your definition of "transfer". Some control messages will be sent to prepare the destination agent to receive the transfer, but since the destination agent is down the source agent will not get a reply. It will therefore not send any file data.

PeterPotkay wrote:

#2 If yes, they will just queue up on the Queue Manager, so an WMQFTE Admin must plan for this by allocating enough disk space for the queue files to handle potential large queueing?

No. This is precisely why FTE works the way it does, rather than the way some people assume it does

. Clearly adding FTE to a network will add some traffic, so an MQ network that was on the limits of capacity before will need to be expanded, but in general FTE is designed not to add disproportionate extra load.

PeterPotkay wrote:

#3 How much queueing will it allow? Are there settings in WMQFTE, or is it just upping Max Q depth on some WMQFTE specific queues?

At any moment a certain amount of file data will be in-flight between the two agents. This amount can be tuned - both the size of individual messages and the number of them that can be in-flight at once can be changed in the agent's properties. If the receiving agent is running slower than the sending one, so that messages would build up in the network, the sending agent throttles itself down to a rate that the receiver can accept.

PeterPotkay wrote:

#4 Do these answers change when the sending and receiving WMQFTE agents are connected to different QMs?

I haven't examined this, but I wouldn't expect them to.

I hope this gives everyone a better insight into how FTE transfers data.

Pete

Michael Dag

Posted: Tue Oct 20, 2009 6:27 am Post subject:

Jedi Knight

Joined: 13 Jun 2002
Posts: 2602
Location: The Netherlands (Amsterdam)

PeteVerdon wrote:

I hope this gives everyone a better insight into how FTE transfers data.

Pete

Thanks Pete, interesting stuff
_________________
Michael

MQSystems Facebook page

PeterPotkay

Posted: Tue Oct 20, 2009 7:35 am Post subject:

Poobah

Joined: 15 May 2001
Posts: 7716

PeteVerdon wrote:

PeterPotkay wrote:

Consider 2 FTE Client using a single Queue Manager as their FTE server

Thanks for the link, good stuff. It didn't really change my perception of the topology, so maybe I didn't explain myself properly.
Server 1 needs to send some files to Server2. Both of these servers have no MQ installed, but do have the WMQFTE Client Agent.

Being that this is a very small set up, Server 3 hosts the WMQFTE Coordination QM and it also acts as the the Command QM and the Agent QM for Server1 and Server2.

The Server1 WMQFTE agent will connect in client mode to Server 3, and the Server2 WMQFTE agent will connect in Client mode to Server3. When they start sending files, they will send them thru Server3, right?

PeteVerdon wrote:

PeterPotkay wrote:

#2 If yes, they will just queue up on the Queue Manager, so an WMQFTE Admin must plan for this by allocating enough disk space for the queue files to handle potential large queueing?

No. This is precisely why FTE works the way it does, rather than the way some people assume it does

PeterPotkay wrote:

#3 How much queueing will it allow? Are there settings in WMQFTE, or is it just upping Max Q depth on some WMQFTE specific queues?

Very good info and not at all clear in any of the doc I have read so far. So WMQFTE is not asynchronous. If the sender wants to move a 10 gig file, the WMQFTE infrsastructure will insure the destination is ready and it will "stream" the file. At no point will anywhere near 10 gig of data be inside MQ, right?
_________________
Peter Potkay
Keep Calm and MQ On

PeteVerdon

Posted: Tue Oct 20, 2009 7:54 am Post subject:

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

PeterPotkay wrote:

PeteVerdon wrote:

PeterPotkay wrote:

Consider 2 FTE Client using a single Queue Manager as their FTE server

This statement reveals a common misunderstanding about FTE topologies and the difference between client and server.

Ah, OK, I may have jumped to conclusions when you said "single queue manager as their FTE server". As you'll appreciate, a queue manager is in no sense an "FTE server", but it's a common mistake that people make.

PeterPotkay wrote:

Being that this is a very small set up, Server 3 hosts the WMQFTE Coordination QM and it also acts as the the Command QM and the Agent QM for Server1 and Server2.

The Server1 WMQFTE agent will connect in client mode to Server 3, and the Server2 WMQFTE agent will connect in Client mode to Server3. When they start sending files, they will send them thru Server3, right?

Yes, that's a valid configuration, and all data exchange between FTE components (including sending of files) will happen inside the queue manager on Server3.

PeterPotkay wrote:

Right. A while ago now I had reason to transfer a 1000GB file from an AIX box to a Windows machine. The file source and destination were on external storage (I think attached by fibre or something; not really sure) but the machines themselves were fairly low-powered with small local disks. The queue managers would have been using these local disks, and certainly couldn't have held even a small fraction of the file. But it transferred fine.

Pete

PeterPotkay

Posted: Wed Oct 21, 2009 4:00 am Post subject:

Poobah

Joined: 15 May 2001
Posts: 7716

Pete, a follow up question on the topic of what gets passed between agents thru MQ. In my original scenario, the 2 WMQFTE clients were both attached to the same QM.

What if WMQFTE Client #1 (Server 1) was attached to QMA (Server A) and WMQFTE Client #2 (Server 2) was attached to QMB (Server B).

QMA and QMB are in an MQ Cluster. One of them is the Coordination QM.

I'm assuming the file transfers will work just fine, but here's the question: As in the example where they were both connected to the same QM, the sending client Agent will only continue to send to the receiving client Agent (thru the 2 QMs) while the receiving agent is up and functional? The WMQFTE Infrastructure will be managing the "streaming" of the huge file piece by piece thru the MQ plumbing across 2 QMs, and it is intelligent enough to tell the sending WMQFTE agent to stop if the receiving one is down? Or even if both agents are fine but the QM to QM communication between QMA and QMB breaks down? There are little administrative control messages zipping and zooming thru the infrastructure making sure the streaming is functional and can continue?
_________________
Peter Potkay
Keep Calm and MQ On

PeteVerdon

Posted: Wed Oct 21, 2009 5:51 am Post subject:

Apprentice

Joined: 11 Feb 2009
Posts: 33
Location: Hursley, UK

That's essentially correct, yes. Strictly speaking the control messages are not sent by "the infrastructure" - there is no real infrastructure in an FTE network beyond the agents and standard MQ. The control messages are part of a protocol between agents, controlled by the agent (the source) that received the original transfer request message.

I am not expert in the details of this protocol (it was worked out and tested before I joined the team) but I believe it consists of an initial negotiation between the two agents, followed by periodic acknowledgments of the data received. If there is a gap in the acknowledgments, or the stream of data, due to an agent going down, the other will enter a recovery mode in which it tries to reestablish contact. Similarly, there is provision for throttling if the acknowledgments are not coming back fast enough.

A number of the thresholds for "long enough", "fast enough", "enough data" can be modified to tune for different kinds of networks; the empirical knowledge on how to do this is still evolving based on experience, but I believe that Tech Sales have been doing some work with slow and high-latency links to less developed areas for instance.

Pete

kodiakk

Posted: Wed Oct 28, 2009 7:29 am Post subject:

Acolyte

Joined: 04 Dec 2007
Posts: 58
Location: Jacksonville, Fl.

PeteVerdon wrote:

A number of the thresholds for "long enough", "fast enough", "enough data" can be modified to tune for different kinds of networks; the empirical knowledge on how to do this is still evolving based on experience, but I believe that Tech Sales have been doing some work with slow and high-latency links to less developed areas for instance.

Pete

I would be interested in how this turns out. I'm in the middle of implimenting FTE in nearly 300 retail locations, 5 of which are running on dail-up connections to the point where a 128K pipeline is really all that is available. I've put this in 33 locations so far, and 2 of those were in these "slow latency" stores... Without exception, those two stores had trouble receiving files (transfer is only one way at this point in time, from Corp to the location) every day. In one case, over 20 attempts were started to send an 11MB file, each time dying and starting over. The most I was ever able to send was about 6MB out of the 11, but in 20 attempts, I got a total of 68MB transferred.
Please keep the info coming on this one, as now we are trying to find other solutions to this situation outside of FTE.
Thanks!
_________________
"Don't take life too seriously... you won't get out of it alive..."

Display posts from previous:

Goto page 1, 2 Next

Page 1 of 2

MQSeries.net Forum Index » IBM MQ File Transfer Edition » MQ FTE V7.0.2 - New Feature - FTP Bridge

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP