Implementing a COM Message Filter

Message filtering is a mechanism that allows server applications to decide if and when an incoming method call can be safely executed on one of their objects. COM generally does not know the reentrancy requirements of your application and consequently does not filter messages by default. Even though message filtering is not as significant as it was with 16-bit applications because the message queue size is now actually unlimited, you should still implement message filtering as a way to resolve blockages. COM will call your implementation of the IMessageFilter interface to find out if an application (a COM server) is blocking so that you can respond and handle the situation. For example, when accessing TwinCAT XAE via COM, the Visual Studio instance will reject further COM calls while still executing a previous COM call. As a result, the client application will issue an RPC_E_CALL_REJECTED error and, without further intervention, will not repeat the call. By writing a user-defined message filter, the programmer has the ability to retry the COM call when the client application receives notification of a denied COM call from the COM server.

The following screenshot shows a typical error output by the Visual Studio COM server when an instance is still busy executing a previous COM call.

Implementing a COM Message Filter 1:

To avoid this situation and implement a message filter that responds to this rejected COM call, the application engineer must implement the IMessageFilter interface. This interface consists of three methods:

HandleIncomingCall(): Provides a single entry point for incoming calls
MessagePending(): Indicates that a message has been received while COM is waiting for a remote call to be answered.
RetryRejectedCall(): Provides the ability to respond to a rejected COM call.

Note that message filters can only be applied to STA threads and that only one filter can be applied to each thread. Multithreaded apartments, e.g. console applications, cannot have message filters. These applications must run in an STA thread to apply message filtering. See the appendix of this documentation for more information on COM threading.

The following code snippet shows a sample of how the IMessageFilter interface can be used in C#. Note that this code is also used in many samples in our Samples section, and is also available as a separate sample download.

[ComImport(), Guid("00000016-0000-0000-C000-000000000046"), InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)] 
interface IOleMessageFilter 
{ 

[PreserveSig] 
int HandleInComingCall(int dwCallType, IntPtr hTaskCaller, int dwTickCount, IntPtr lpInterfaceInfo); 


[PreserveSig] 
int RetryRejectedCall(IntPtr hTaskCallee, int dwTickCount, int dwRejectType); 


[PreserveSig] 
int MessagePending(IntPtr hTaskCallee, int dwTickCount, int dwPendingType); 
}

The following class implements this interface and adds two more methods: Register() and Revoke().

public class MessageFilter : IOleMessageFilter 
{ 
 public static void Register() 
 { 
 IOleMessageFilter newFilter = newMessageFilter(); 
 IOleMessageFilter oldFilter = null; 
 int test = CoRegisterMessageFilter(newFilter, out oldFilter); 

 if (test != 0) 
 { 
 Console.WriteLine(string.Format("CoRegisterMessageFilter failed with error : {0}", test)); 
 } 
 } 


 public static void Revoke() 
 { 
 IOleMessageFilter oldFilter = null; 
 int test = CoRegisterMessageFilter(null, out oldFilter); 
 } 


 int IOleMessageFilter.HandleInComingCall(int dwCallType, System.IntPtr hTaskCaller, int dwTickCount, System.IntPtr lpInterfaceInfo) 
 { 
 //returns the flag SERVERCALL_ISHANDLED. 
 return 0; 
 } 


 int IOleMessageFilter.RetryRejectedCall(System.IntPtr hTaskCallee, int dwTickCount, int dwRejectType) 
 { 
 // Thread call was refused, try again. 
 if (dwRejectType == 2) 
 // flag = SERVERCALL_RETRYLATER. 
 { 
 // retry thread call at once, if return value >=0 & 
 // <100. 
 return 99; 
 } 
 return -1; 
 } 


 int IOleMessageFilter.MessagePending(System.IntPtr hTaskCallee, int dwTickCount, int dwPendingType) 
 { 
 //return flag PENDINGMSG_WAITDEFPROCESS. 
 return 2; 
 } 

 // implement IOleMessageFilter interface. 
 [DllImport("Ole32.dll")] 
 private static extern int CoRegisterMessageFilter(IOleMessageFilter newFilter, outIOleMessageFilter oldFilter);

An application engineer now only needs to call the Register() and Revoke() methods from another class to initialize and discard the MessageFilter. The result is that rejected COM calls are repeated as specified in the RetryRejectedCall() method.

The following code snippet shows how to call these methods in a console application written in C#. As mentioned above, console applications run in an MTA thread by default. For this reason, the Main() method must be configured to run in an STA apartment so that the message filter can be applied.

[STAThread] 
static void Main(string[] args) 
{ 
 MessageFilter.Register(); 

 /* ============================================================= 
 * place COM calls for the  Automation Interface here 
 * ... 
 * ... 
 * ============================================================= */ 

 MessageFilter.Revoke(); 
}

If you try to apply a message filter to an application running in the MTA apartment, the following error is issued when trying to execute the CoRegisterMessageFilter() method during runtime:

Implementing a COM Message Filter 2:

For more information about the different COM threading models, see the MSDN article Understanding and Using COM Threading models. For more detailed information about the IMessageFilter interface, see the MSDN documentation regarding IMessageFilter.

The following code snippet shows how to implement a COM MessageFilter for Windows Powershell by including it as a .NET type (C#) in PowerShell.

Code snippet (Powershell):

AddMessageFilterClass('') # Call function
[EnvDteUtils.MessageFilter]::Register() # Call static Register Filter Method

$dte = New-Object -COMObject TcXaeShell.DTE.15.0
$dte.SuppressUI = $false
$dte.MainWindow.Visible = $true
$solution = $dte.Solution
# do stuff
$dte.Quit()

[EnvDTEUtils.MessageFilter]::Revoke()


function AddMessageFilterClass 
{ 
 $source = @‘ 
 namespace EnvDteUtils
 {
 using System; 
 using System.Runtime.InteropServices; 
 
 public class MessageFilter : IOleMessageFilter 
 { 
 public static void Register() 
 { 
 IOleMessageFilter newFilter = new MessageFilter(); 
 IOleMessageFilter oldFilter = null; 
 CoRegisterMessageFilter(newFilter, out oldFilter); 
 } 

 public static void Revoke() 
 { 
 IOleMessageFilter oldFilter = null; 
 CoRegisterMessageFilter(null, out oldFilter); 
 } 
 
 int IOleMessageFilter.HandleInComingCall(int dwCallType, System.IntPtr hTaskCaller, int dwTickCount, System.IntPtr lpInterfaceInfo)
 { 
 return 0; 
 } 

 int IOleMessageFilter.RetryRejectedCall(System.IntPtr hTaskCallee, int dwTickCount, int dwRejectType) 
 { 
 if (dwRejectType == 2) 
 { 
 return 99; 
 } 
 return -1; 
 } 

 int IOleMessageFilter.MessagePending(System.IntPtr hTaskCallee, int dwTickCount, int dwPendingType) 
 { 
 return 2; 
 } 

 [DllImport("Ole32.dll")] 
 private static extern int CoRegisterMessageFilter(IOleMessageFilter newFilter, out IOleMessageFilter oldFilter); 
 } 

 [ComImport(), Guid("00000016-0000-0000-C000-000000000046"), InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)] 
 interface IOleMessageFilter 
 { 
 [PreserveSig] 
 int HandleInComingCall(int dwCallType, IntPtr hTaskCaller, int dwTickCount, IntPtr lpInterfaceInfo);

 [PreserveSig]
 int RetryRejectedCall(IntPtr hTaskCallee, int dwTickCount, int dwRejectType);

 [PreserveSig]
 int MessagePending(IntPtr hTaskCallee, int dwTickCount, int dwPendingType);
 }
 }
 ‘@
 Add-Type -TypeDefinition $source
}