ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexIBM MQ API SupportDoes RfhUtil Write No Headers exist in the XMS API?

Post new topicReply to topic Goto page 1, 2, 3  Next
Does RfhUtil Write No Headers exist in the XMS API? View previous topic :: View next topic
Author Message
psychodelic_drumboy
PostPosted: Thu Apr 07, 2022 3:54 am Post subject: Does RfhUtil Write No Headers exist in the XMS API? Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Hello and good afternoon to you fine and learned people...

I have written a tremendous piece of code that pulls iBytesMessage items from an MQ queue, in a spritely fashion.

However, I need to implement the option that you see in RfhUtil, under the write menu, which states "No Headers". The provider of the messages has insisted on the format, and won't alter it...even when I ask nicely!



We didn't historically need to use that option, but now we do, otherwise the resultant file is unusable.

So my question is, where does an equivalent to the "no headers" option in the RfhUtil tool reside within the IBM.XMS API?

I'm using version 9.2.5 of the IBMXMSDotnetClient by IBM, as found via the NuGet route in Visual Studio.

Thanks in advance, I know you guys will probably have the answer to this one..
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Apr 07, 2022 8:12 pm Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

In fact you might be able achieve this without changing your application code at all.

If you change the queue definition in this way:-

Code:
ALTER QLOCAL(q-name) PROPCTL(NONE)


This option will cause any RFH2 user property headers to be removed from the message before they are given to an application which cannot deal with them. Any applications reading from this queue that can handle message properties should be unaffected by this setting.

Having said that, IBM.XMS API is one of those APIs which can handle message properties, so this option may be over-ridden by the the XMS API asking for the message properties.

Worth trying anyway.

If you do want to change your application, you might be looking for the XMSC_WMQ_MESSAGE_BODY property with value XMSC_WMQ_MESSAGE_BODY_MQ, but I am not an XMS expert, so someone else might jump in with a better answer.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
psychodelic_drumboy
PostPosted: Fri Apr 08, 2022 12:11 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Morag, many thanks for your detailed response.

I am trying the .NET code change first, by adding this additional line in my code:

Code:
destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_MQ)


We are now examining the effect on the output files; I will report back as soon as I know.

If this fails, we'll see if we can alter the queue definition using the command you mentioned. I almost certainly don't have access to run that command, so I would need to find the person who can.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Fri Apr 08, 2022 1:34 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Ok, so I ran the code with all three possible WMQ_MESSAGE_BODY options

- Not set (documentation says default is XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED)
- XMSC_WMQ_MESSAGE_BODY_JMS
- XMSC_WMQ_MESSAGE_BODY_MQ

Unfortunately, none of these options resulted in a working set of files.

Interestingly, when I did a binary file compare with the BeyondCompare tool, I actually saw no header data when using XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED, but header data when XMSC_WMQ_MESSAGE_BODY_MQ was used.

I am now obtaining a set of files that definitely work, using RfhUtil, so that:
1 - I can confirm that the messages on the queue are in working order
2 - I can maybe spot differences between the RFHUTIL version and the versions with various WMQ_MESSAGE_BODY options.

I'll let you know what differences I find, but thanks for your help so far.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Fri Apr 08, 2022 1:54 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

psychodelic_drumboy wrote:


1 - I can confirm that the messages on the queue are in working order


They're not! Apologies, I should have performed this stage beforehand.

We've been given duff data. We should see 4 x messages on the queue, where 1,2 + 3 are the same size. That is not the case, so please ignore my findings so far, your code change suggestion may still be the right one.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Wed Apr 13, 2022 12:37 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Ok so we've been sent four new mesages from our supplier, which, size-wise, look valid.

I'm again trying all three possible " XMSC_WMQ_MESSAGE_BODY" options, to see if any of those address our issue.

I will report back shortly - my colleague is now validating the downloads.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Apr 14, 2022 12:44 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Define the destination as "queue://QMGR/QNAME?targetClient=1"
The QMGR name is optional.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
psychodelic_drumboy
PostPosted: Fri Apr 22, 2022 4:07 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

fjb_saper wrote:
Define the destination as "queue://QMGR/QNAME?targetClient=1"
The QMGR name is optional.


Thanks fjb_saper. I will try that. The API certainly accepts the targetClient argument, I can see the messages coming down right now.

I've been provided with a fresh set of messages but my wider team haven't yet verified that they're valid (as I mentioned, the last set were not).
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Tue Apr 26, 2022 5:39 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Ok, so I have tried a number of options, recommended in this thread, to resolve the issue.

To be clear, we are sent IbytesMessages which are actually zip files; files larger than 47.6MB are divded into 47.6 MB segments.

On the queue right now I have the following iBytes messages:
- 1 x 16MB message (that, when renamed to .ZIP, should unzip without error)
- A set of 4 x divided zip messages (3 x 47.6 MB, 1 x 47.4MB)
- A set of 3 x divided zip messages (2 x 47.6 MB, 1 x 44 MB)

We use an EXE utility that glues the divided zips together into one zip file. I didn't write the utility, but I believe it just pulls all the data from the multiple groups of files into a single file and names it .ZIP

I have tried all three XMSC_WMQ_MESSAGE_BODY options:
- XMSC_WMQ_MESSAGE_BODY_UNSPECIFIED
- XMSC_WMQ_MESSAGE_BODY_JMS
- XMSC_WMQ_MESSAGE_BODY_MQ

..but none result in working file sets.

When using the XMSC_WMQ_MESSAGE_BODY_MQ against the 16MB "single file message", this results in an invalid zip, but the other two options result in a file that will unzip.

I have also tried the "queue://QMGR/QNAME?targetClient=1" argument, and tried both UNSPECIFIED and JMS options. Neither resulted in file sets that could be reconstituted. I ruled out XMSC_WMQ_MESSAGE_BODY_MQ due to the finding above.

Lastly, my colleague has manually downloaded the messages with RFHUTIL, using the "No Headers" option in the Write Menu - those messages have been found to be usable, so we have a valid set of messages on our queue - we just can't get them down via the XMS API.

So, we still don't have the answer yet.

Do you have any other questions about what I've tried so far?

Any assistance you can provide on this would be very much appreciated.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Tue Apr 26, 2022 5:58 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

I think at this point it would be useful for me to share my code. I've chosen to write it in Visual Basic.

Please note that this code does not, at any time, throw an exception, but the resultant files (as described above) don't turn out to be valid.

I've stripped out the other message types from the excerpt because that's not relevant to what we're talking about. The messages in this case are all IBytesMessage.

Code:
Imports System
Imports System.IO
Imports IBM.XMS
Imports System.Diagnostics


Public Class clsMQ

    'MQ Connection Setings Specified By User
    Private mstrQueueManager As String
    Private mstrHostName As String
    Private mstrDestinationURI As String
    Private mstrDestination As String
    Private mstrMQErrorMessage As String = ""

    'Local variable to count messages received
    Private mIntMessagesReceived As Integer = 0

    'The specified prefix for each file
    Private mstrFilePrefix As String = ""
    Private mstrFileExtension As String = ""

    'Deletion Property
    Private mBlnDeleteFromQueue As Boolean = False

    Public Enum enBinaryDownloadType
        enWMQ_MESSAGE_BODY_JMS = 0
        enWMQ_MESSAGE_BODY_MQ = 1
        enWMQ_MESSGAE_BODY_UNSPECIFIED = 2
    End Enum

    Private mIbytesDownloadType As enBinaryDownloadType

   
    Public Property iBytesOption As enBinaryDownloadType
        Get
            Return mIbytesDownloadType
        End Get
        Set(value As enBinaryDownloadType)
            mIbytesDownloadType = value
        End Set
    End Property

    Public Sub RetrieveFiles()

        'Create MQ Objects so they can be destroyed in the Finally block
        Dim blnErrorOccurred As Boolean = False
        Dim Consumer As IMessageConsumer = Nothing
        Dim session As ISession = Nothing
        Dim destination As IDestination = Nothing
        Dim connection As IConnection = Nothing
        Dim cfConnectionFactory As IConnectionFactory = Nothing
        Dim strFileNameFromMsg As String = ""

        Try

            'Create an XMS Factory Object
            Dim FactoryFactory As XMSFactoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ)

            'Create a ConnectionFactory Object
            cfConnectionFactory = FactoryFactory.CreateConnectionFactory()

            'this variable will contain the full path of any file downloaded from MQ
            Dim strMQMessageOutputFileDestinationFilePath As String = ""

            'This variable will be used to evaluate whether the MQ Message Output file exists
            Dim fiMQMessageOutputFile As FileInfo = Nothing

            'Set various Connection Factory properties
            cfConnectionFactory.SetStringProperty(XMSC.WMQ_HOST_NAME, Me.HostName)
            cfConnectionFactory.SetIntProperty(XMSC.WMQ_PORT, 1414)
            cfConnectionFactory.SetStringProperty(XMSC.WMQ_CHANNEL, "SYSTEM.DEF.SVRCONN")
            cfConnectionFactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, 1)

   
            cfConnectionFactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, Me.QueueManager)
            cfConnectionFactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, 0)

 
            'Create a new Iconnection object via the Connection Factory
            connection = cfConnectionFactory.CreateConnection()

   
            'Create a sesion via the Connection Object, using ClientAcknowledge mode
            'ClientAcknowledge is being used because it allows us to control precisely
            'when a message should be removed from the queue
            session = connection.CreateSession(False, AcknowledgeMode.ClientAcknowledge)

            'Create a destination using the Session Object
            destination = session.CreateQueue(Me.mstrDestinationURI)

            destination.SetIntProperty(XMSC.DELIVERY_MODE, 1)

            Select Case Me.iBytesOption

                Case enBinaryDownloadType.enWMQ_MESSAGE_BODY_JMS
                    destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_JMS)

                Case enBinaryDownloadType.enWMQ_MESSAGE_BODY_MQ
                    destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_MQ)

                Case enBinaryDownloadType.enWMQ_MESSGAE_BODY_UNSPECIFIED
                    destination.SetIntProperty(XMSC.WMQ_MESSAGE_BODY, XMSC.WMQ_MESSAGE_BODY_UNSPECIFIED)
                Case Else
                    'do nothing

            End Select



            'Create a consumer using the Session & Destination Objects
            Consumer = session.CreateConsumer(destination)

 
            connection.Start()

            'IMessage is the base class that is returned from the Consumer's Receive method
            Dim recvMsg As IMessage = Nothing

            Do

                ' Retrieve message from Queue, using the XML file's specified time out
                'ReceiveNoWait will run until
                recvMsg = Consumer.ReceiveNoWait

                If recvMsg Is Nothing Then

                    'No more messages left to process, exit loop
                    Exit Do

                Else

                    'Increment File Counter
                    mIntMessagesReceived += 1

   
                    'Obtain filename from MQ Message Meta Data (if available)
                    strFileNameFromMsg = If(Not recvMsg.PropertyExists("fileName"), "", recvMsg.GetStringProperty("fileName"))

     
                    If TypeOf (recvMsg) Is IBytesMessage Then
                        'Binary Message
                        Dim msg As IBytesMessage = CType(recvMsg, IBytesMessage)
                        Dim buffer(msg.BodyLength) As Byte

                        msg.ReadBytes(buffer)

                        'The PrepareDestinationFile Function will generate a unique file name for the new file
                        'and ensure that the file does not already exist on the drive
                        strMQMessageOutputFileDestinationFilePath = PrepareDestinationFile(strFileNameFromMsg)

                        'A FileStream object is needed to write a binary array to a file
                        Dim fsZipFile As FileStream = New FileStream(strMQMessageOutputFileDestinationFilePath, FileMode.Create)

                        'Write the contents of the Byte Array to the File via the FileStream object
                        fsZipFile.Write(buffer, 0, buffer.Length)
                        fsZipFile.Close()


                    End If

   
                End If

            Loop 'Next message in MQ Queue

            'Close connection to MQ
            connection.Close()


        Catch mqe As XMSException
            blnErrorOccurred = True
            'MQ Related Error Handler
            'Any errors which occurred during code execution will be logged to the event log as Warnings

            mstrMQErrorMessage = "Reason:" & mqe.Message.ToString & " / " & "Reason Code:" & mqe.Source.ToString()

            Exit Sub

        Catch ex As Exception
            blnErrorOccurred = True

            'General (Non-MQ Related) Error Handler
            'Any errors which occurred during code execution will be logged to the event log as Warnings
 
        Finally

            'Final Event Log entry will be either Information or Error, depending on whether
            'any unexpected errors occurred during code execution
 
            'destroy all objects
            If Not Consumer Is Nothing Then
                Consumer = Nothing
            End If

            If Not destination Is Nothing Then
                destination = Nothing
            End If

            If Not session Is Nothing Then
                session = Nothing
            End If

            If Not connection Is Nothing Then
                connection = Nothing
            End If

            If Not cfConnectionFactory Is Nothing Then
                cfConnectionFactory = Nothing
            End If

        End Try

    End Sub

 
    Private Function PrepareDestinationFile(ByVal strFileNameFromMsg As String) As String

        Try
            Dim fiDestinationFile As FileInfo

            If strFileNameFromMsg.Length = 0 Then

                'Generate unique file name using specified file prefix & suffix, specified in XML config file
                PrepareDestinationFile = Me.MessageFileDestination & Me.FilePrefix & System.DateTime.Now.ToString("yyyyMMddhhmmss") & "_" & mIntMessagesReceived.ToString() & "." & Me.FileExtension

            Else

                'use file destination found in MQ Message
                PrepareDestinationFile = Me.MessageFileDestination & strFileNameFromMsg

            End If

            'Ensure destination file does not exist
            fiDestinationFile = New FileInfo(PrepareDestinationFile)

            If fiDestinationFile.Exists Then fiDestinationFile.Delete()

        Catch ex As Exception

            PrepareDestinationFile = ""

   
        End Try

    End Function

End Class


Do you see any obvious errors here?

The best case scenario would be that I'm simply doing something wrong when either reading the message data from MQ into the "buffer" byte array, or writing the "buffer" ByteArray data into the file (in the fsZipFile object)

It's also apparent that I load the data into a variable called content and then don't use it.

Code:

content As String = Text.Encoding.UTF8.GetString(buffer)


This can be removed.


Last edited by psychodelic_drumboy on Wed May 04, 2022 4:21 am; edited 1 time in total
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Apr 27, 2022 9:37 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

I am not seeing you writing/producing the message. Just consuming the message.
Forget any problems with the message. The errors have already been made.
Forget anything about the destination set up as you are consuming the message and not producing it...
If you are producing it, forget all about setting the destination properties.
The only thing you need to do is:
session.createQueue("queue://qmgr/qname?targetClient=1").
Read the message as an IBytesMessage.
I would however question why your are reading the message as UTF-8?
It's a zip file and not a String!!!
Code:
Dim content As String = Text.Encoding.UTF8.GetString(buffer)
.
Also I have no idea what this does as a pointer to your buffer?
May be comment out that line and see what you get?

Can you share how the message is produced?

_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
psychodelic_drumboy
PostPosted: Wed Apr 27, 2022 11:36 pm Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

Hi,

So to be clear...

Quote:
Just consuming the message.

Correct, the solution only concerns itself with reading messages sent by others.

Quote:
I would however question why your are reading the message as UTF-8? It's a zip file and not a String!!!


Quote:
Also I have no idea what this does as a pointer to your buffer?


This line of code is redundant. You're right that I should remove it; Maybe it's affecting the integrity of the buffer object?

I'll try again, with it removed, and report back. If that's all it is, then that would be great.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Thu Apr 28, 2022 2:15 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

I'm sorry to report that even with the "?targetClient=1" value in the WMQ_HOST_NAME, and with the redundant line of code removed, this still results in files that cannot be reconstituted.

Is there anything I could look for, in the resultant files, that could provide an angle as to what's wrong? It's possible that I could get a copy of the working files downloaded by RHFUTIL, and perform some kind of comparison.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Apr 28, 2022 2:53 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

psychodelic_drumboy wrote:
… this still results in files that cannot be reconstituted.

Please explain “reconstituted”. From what? To what?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
psychodelic_drumboy
PostPosted: Thu Apr 28, 2022 3:09 am Post subject: Reply with quote

Novice

Joined: 07 Apr 2022
Posts: 20

bruce2359 wrote:
psychodelic_drumboy wrote:
… this still results in files that cannot be reconstituted.

Please explain “reconstituted”. From what? To what?


We are sent, for example, three iBytes messages with names such as:
- "zip_file_28_04_2022_1_of_3.SPLIT"
- "zip_file_28_04_2022_2_of_3.SPLIT"
- "zip_file_28_04_2022_3_of_3.SPLIT"

These files, on their own, cannot be opened by tool like WinZip.

The utility simply combines the contents of these 3 x SPLIT files (I call that "reconstituting") into a single file and names that file .ZIP. I didn't write the utility, so note that this is only my interpretation based on the end results.

When the messages are downloaded via RfhUtil, with the "No Headers" option selected, the ZIP file created by the utility will open and unzip.

When the messages are downloaded via my code, the ZIP file created by the utility will not open and will not unzip.

There is still something unique to how RFHUtil works, that is different to my supplied code. I have no idea what it is.

I don't know why 47.6 MB is the maximum permitted size of a message in this environment. I also don't know if that is a limitation of MQ, or a constraint the supplier of the messages has mandated.

As you can tell - I don't know very much! I'm sorry about that.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2, 3  Next Page 1 of 3

MQSeries.net Forum IndexIBM MQ API SupportDoes RfhUtil Write No Headers exist in the XMS API?
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.