| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| IIB9: FileInput SFTP |
« View previous topic :: View next topic » |
| Author |
Message
|
| zpat |
 Posted: Thu Sep 03, 2015 1:11 am Post subject: |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
| akil wrote: |
| mqjeff wrote: |
| I believe that the mqsitransit directory is created on the remote server, not on the local server when using FTP/SFTP. |
In 9.0.0.2, the mqsitransit is only on the local (ESB) machine, and not on the remote folder.. |
It's my understanding that broker uses Unix file locking and atomic operations to implement its best efforts attempt at transactional integrity. This is only possible when accessing file systems directly (and not over SFTP).
So if you mount a NFS drive - then yes the mqsitransit folder may well be on the remote machine, but this is not the same thing as using file transfer protocols likes FTP and SFTP (both of which belong in a museum).
So you set the "transactional" file location (where mqsitransit will be created) in the basic properties and if you want to fetch files over FTP/SFTP then you also configure the FTP tab.
But I might be completely wrong about this of course... try it and see 
One of the many problems with using files, is that while the broker plays nicely and uses file locking - most other Unix applications don't. Including FTP/SFTP.
Moral here is to use MQ if you want your data delivered once and only once, not lost or duplicated, or truncated, or corrupted, etc etc
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
 |
| ibmrob |
| Posted: Tue Sep 08, 2015 6:02 am Post subject: |
|
|
Newbie
Joined: 08 Sep 2015
Posts: 1
|
As has been mentioned by others, the FTP/SFTP nodes do not support any sort of locking as the basic protocol does not provide this capability so there is no simple way of doing remote FTP with locking.
There are some ways the same end goal can be achieved:
1) Put a MessageBroker onto the remote FTP server and use it to pick up the files locally and then pass them onto the brokers in a round robin approach using MQ/FTE etc. This is a fairly heavy handed solution for what is a simple problem.
2) Create a small daemon on the FTP server to pick up each file and move it into a sub-directory in a round robin approach and update the brokers to pick up from a specific sub-directory. This has the limitations in that you will get marooned messages if one of the brokers has an outage, so multi-instance brokers will be needed or a mechanism for moving the files around in the event of a broker failure.
3) Remote mount the filesystem with the files on. If all the brokers have the folders mounted it will use the filesystem locking mechanism that is normally used on local filesystems to make sure that only 1 broker is able to process each message. To do this you must use a shared filesystem that supports the locking requirements. I think this will be similar to those of the multi-instance i.e. NFS4 etc. |
|
| Back to top |
|
|
| akil |
| Posted: Tue Sep 08, 2015 7:47 pm Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
We are now doing the following:
1. introduce a database table to record which files are being processed by which broker (broker-uuid)
2. place a filter-node after the file-input, to filter records that are being processed by the other broker.
One downside is that the broker that won't process the file , still reads the entire file ( there seems to be no way to tell the broker , to skip a file that the fileinput has decided to start to process ) ..
_________________
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 09, 2015 4:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Again, the problem is that the FTP protocol is designed specifically to allow multiple simultaneous reads from multiple clients.
So there's nothing Broker can do to handle this on it's own - it doesn't know anything about other Brokers or EGs.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
