2009-07Jul-16
by Christof Wollenhaupt, Foxpert
Microsoft is known to change technologies faster than the marketing department can print advertisement material. COM, however, is still the foundation for quite a few Microsoft technologies that are available today. A lot has been said about COM, even more complained, but hardly anyone really knows what COM is. This session attempts to cover COM not from a user perspective, but from its internals. What are interfaces? How does Windows load COM objects? What happens when you call a method? Why do so many things work, but other things don't? What are apartments and who cares about the threading model? What's the difference between thread and process, EXE, DLL and MTDLL, early-binding and late-binding, between VTable and IDispatch? This session requires basic knowledge of memory adresses, pointers and data storage.
An interface is not an object
Visual FoxPro developers make no differences between defining a class and implementing it. You define a class using the DEFINE CLASS statement and create procedures with code right away. One of the core principles of COM is to strictly separate the two.
An Interface in COM is kind of a contract between the caller and the callee. Unlike a Visual FoxPro class it doesn't contain any code nor data. It does, however, state in great detail how the object interacts: What methods are available? What parameters does each method expect. A world-wide unique2023 foxpert GmbH be used to uniquely identify each interface. Once you defined an interface, you cannot change it.
There's a language dedicated to define such an interface which is called IDL, short for Interface Description Language. Microsoft developed a compiler (MIDL) to compile these definitions into header files, type libraries and much more. Most development environments perform these steps automatically in the background.
Interface can derive from each other. Therefore COM is object oriented, even though it differs from implementations in C++ or VFP. The root of all interfaces is IUnknown that defines a few basic method that every COM object must support. IUnknown has got the methods QueryInterface, AddRef and Release. With SYS(3096), SYS(3097) and SYS(3098) you can call these methods in any COM object directly.
A COM object is a program that implements one or more interfaces. Implementing means that the program provides code for every single method in exactly the same way that the method has been defined. Because an interface is separated from the implementation, multiple developers and companies can provide different implementations for the same interface. Developers can decide which implementation they want to use. Typically, only simple interfaces are implemented by different companies.
For instance, there's no standard interface for a spell checker. If you want to add spell checking capabilities to your application, you have to make changes specifically for the spell checker that you want to implement. COM AddIns like those in Microsoft Office implement a specific COM interface defined by the vendor, Microsoft in the case of Office add-ons.
COM objects differ from VFP objects in that they can implement several different interfaces. Because interfaces are strictly separated from each other, it's possible to reuse the same method names in different interfaces. For every interface a COM object provides a separate pointer. To obtain such a pointer you call the QueryInterface() method.
Until VFP 6.0 you can only define one interface for every COM object. VFP 7.0 added the IMPLEMENTS keyword that allows us to implement multiple interface in a single COM object. The interface name precedes the method names. To quickly generate procedures stubs that implement the interface, you can drag an interface from the Object Browser and drop it into an editor window. Multiple COM objects can be stored in a COM server. A COM server is an .EXE or a .DLL file. A COM server is more than just a storage area for multiple COM objects. Additionally it defines the process space in which objects are executed.
An object reference in COM is a pointer to data block. The first entry in this block points to a list of addresses of methods. With this pointer and the number of a method, you can call a method directly. The remainder of the data block is the private storage area of a COM server and remains inaccessible from the outside.
The binary format of a COM object in memory is identical with C++ objects. Using COM directly in C++ is therefore not any slower than a regular method call in C++. Only when you need to use IDispatch or cross process boundaries COM becomes slower.
Loading COM objects
Loading a COM object is more complex than instantiating a Visual FoxPro object. To load the COM subsystem, Visual FoxPro calls the CoInitializeEx() function. When you create a COM object with CREATEOBJECT(), Windows first searches the registry for the CLSID of the object. The CLSID for the object is a globally unique identifier (GUID) that identifies a class. CoCreateInstance is next called to actually create the instance.
CoCreateInstance() delegates loading the physical file to the Service Control Manager (SCM) that is part of the RPCSS.EXE. The SCM uses the registry to determine the file that contains the COM object. If the object resides on a different machine, the SCM uses remote procedure calls (RPC) to contact the SCM of the other machine.
If it's a DLL, the file is loaded into the address space of the calling process. Every COM DLL must implement a number of plain procedures. One of them is DllGetClassObject() which is the one the SCM calls initially. This function creates a factory object and returns an interface pointer to the object. Such a factory object must implement the IClassFactory interface and usually implements the IClassFactory2 interface.
An EXE, on the other hand, is executed. An EXE that doesn't operate as a COM server by default usually receives some sort of command line parameter that is specified in the registry, as well. In case of a Visual FoxPro COM server the application is called with the parameter /automation. When you pass this parameter, Visual FoxPro calls the CoRegisterClassObject() API function and passes the interface pointer to the class factory object. This happens once for every COM server that is stored in the EXE. The SCM uses this list of registered objects to retrieve pointers to IClassFactory interfaces.
After obtaining the pointer to IClassFactory, CoCreateInstance() calls its CreateInstance() method. Now the COM server is in charge again. The code returns a physical copy of the data block that represents the object and returns its address. As IClassFactory isn't required anymore now the object is released by calling its Release method.
We now have an object but it's not initialized. All private data fields are at their default values. In Visual FoxPro the class factory automatically calls the Init method. However, as IClassFactory::CreateInstance() doesn't receive any parameters, you cannot pass a parameter to the Init event of a VFP COM server. Other products use different mechanisms to initialize objects. Frequently they implement one or more interfaces of the IPersis family of interfaces. These interfaces allow storing the state of an object outside the object and restore from there. VFP uses this mechanism with ActiveX controls. All property changes you make are stored in the OLE memo field of the VCX file.
As you can see the registry is only used for two pieces of information: It returns the CLSID for a more readable ProgID, and it tells you what file the COM server resides in. If you know the name of the DLL and the CLSID, what would you stop from loading the COM server directly?
DLL COM servers are actually procedural DLLs with just a few functions: DllRegisterServer, DllUnregisterServer, DllCanUnloadNow and DllGetClassObject. The first two functions are called when you use REGSVR32.EXE to register or unregister a COM server. DllCanUnloadNow is a generic mechanism to determine which DLLs can be released when you issue SYS(2339) or call the CoFreeUnusedLibraries API function directly. All the magic of creating objects is hidden in the last function.
DllGetClassObject receives the class ID of the object you want to create. The return value isn't the object itself, but the reference to a factory object that implements the well-known IClassFactory interface. The DllGetClassObject function returns such a reference not as a VFP object, but as an interface pointer. Such a pointer is a LONG value.
Fortunately, VFP supports converting such pointers to object references. SYS(3096) returns the reference for a given pointer. The return value of an API function can also be an OBJECT type. In both cases, the object pointer must be an IDispatch pointer. The following LoadCOM.prg put this all together.
*========================================================
* Loads a COM object that is not registered.
*========================================================
Procedure LoadCOM( tcCLSID, tcFile )
*------------------------------------------------------
* Declare API functions
*------------------------------------------------------
Declare Long DllGetClassObject in (m.tcFile) ;
String rclsid, ;
String riid, ;
Long @ppv
Declare Long CreateInstance ;
in IClassFactoryHelper.DLL ;
Long ptrCF
*------------------------------------------------------
* Determine all CLSIDs
*------------------------------------------------------
Local lcCLSID, lcIDispatch, lcIClassFactory
lcCLSID = StrToCLSID(m.tcCLSID)
lcIDispatch = StrtoCLSID( ;
"{00020400-0000-0000-C000-000000000046}" ;
)
lcIClassFactory = StrtoCLSID( ;
"{00000001-0000-0000-C000-000000000046}" ;
)
*------------------------------------------------------
* Load the class object
*------------------------------------------------------
Local lnPtr, lnOK
lnPtr = 0
lnOK = DllGetClassObject( ;
m.lcCLSID, m.lcIClassFactory, @lnPtr ;
)
If m.lnOK != 0
Return NULL
EndIf
*------------------------------------------------------
* Use the helper DLL to call the CreateInstance method.
* Then convert the IDispatch pointer into an object
* reference.
*------------------------------------------------------
Local lnPDisp, loObject
lnPDisp = CreateInstance( m.lnPtr )
If m.lnPDisp == 0
loObject = NULL
Else
loObject = Sys(3096,m.lnPDisp)
EndIf
Return m.loObject
*========================================================
* Converts a CLSID String into a CLSID
*========================================================
Procedure StrToCLSID( tcString )
Local lcCLSID
lcCLSID = Space(16)
Declare Long CLSIDFromString in Ole32.DLL ;
String lpsz, ;
String @pclsid
CLSIDFromString( Strconv(m.tcString,5), @lcCLSID )
Return m.lcCLSID
The code first calls the DllGetClassObject function in the DLL COM server requesting a class factory object that implements the IClassFactory interface. Microsoft also defined an IClassFactory2 interface, which has additional methods for dealing with licenses.
This pointer is passed to a DLL function named CreateInstance. The DLL function merely calls the method in the class factory object and returns the result. Because IClassFactory is an interface that directly inherits from IUnknown instead of IDispatch, we cannot call a method on this interface directly from VFP.
Hence, we use a small C DLL that calls the method. The entire Visual C++ project and the compiled DLL is available on the conference CD. The CreateInstance function consists of just a few lines C code
__declspec(dllexport) IDispatch *CreateInstance(
IClassFactory *ptrCF )
{
IDispatch* pDisp;
ptrCF->CreateInstance(
NULL,
IID_IDispatch,
(void**)&pDisp
);
ptrCF->Release();
return pDisp;
}
The return value is a pointer to an IDispatch interface. SYS(3096) converts into a COM object reference that we can use directly. A utility function converts class IDs from the common format with curly braces into the binary representation required by most API functions.
To call the LoadCOM() procedure you need two pieces of information: The class ID and the location of the DLL. If you have both, you can instantiate a COM server without it being registered at all. You call the function like this, of course, using the actual CLSID. As you can see, the return value is an object reference that you can use to execute any of its method. With a VFP COM server you even get IntelliSense since the type library is compiled into the DLL:
Local loRef
loRef = LoadCOM( ;
"{E394D13A-EC0D-4D28-8192-64DAE7494E16}", ;
"Sample.DLL" ;
)
loRef.Main
Marshalling, Proxis and Stubs
Only when the COM object resides in the same process space as the actual application, you can directly call the COM object using interface pointers. If you create a word instance in VFP, you end up with two different processes. In the task manager both, Word and VFP, are visible as separate threads. In old 16 bit operating systems like Windows 3.x all applications shared a common address space. In 32 and 64 bit operating systems - like Windows - processes run in isolated memory spaces each four GB in size. There is no way that one process can access memory of another process directly. Yet, it appears as if you can call Word directly, call methods and access properties.
COM accomplishes this with a number of tricks. When direct calls aren't working, COM switches over to using a proxy, a stub and interprocess communication (IPC). When VFP creates an instance of Word, Word isn't loaded into the same address space as VFP. Instead, VFP uses the Word interface definition to create a proxy object. The proxy object doesn't call Word directly. Instead they redirect every request to the COM subsystem. This proxy is created in the VFP address space. This way VFP has direct access to an object that has the same method which expects the same parameters as Word. Because COM strictly separates interface from implementation, it's easily possible to replace the entire implementation with a different one like that.
On the other side in Word's address space, there's another object that can be called. It has the same interface as Word and forwards every call to Word. This object on the receiving end is called a stub. The COM subsystem sits between proxy and stub objects. It relies, in turn, on the IPC services provided by Windows.
When a VFP application calls a Word method, it actually calls the proxy object. The proxy collects all parameters and packs them into a string. Then it calls the COM system with just this string parameter.
Whether the COM object sits beyond a process boundary (COM) or a computer boundary (DCOM) determines what mechanism COM chooses to forward this data package to the stub object. The stub unpacks all parameters and then calls the actual COM server. Return values take the reverse way. The stub creates a package that the proxy unpacks.
Packaging parameters is called marshaling in COM. The process of unpacking is called nmarshaling. In today's .NET world the same process is called serializing and deserializing. The difference is in the same and the format that is used for packing parameters. COM uses a compact, but proprietary binary format. .NET uses XML strings.
The author of a COM object should provide the proxy and stub for the COM object. This can be quite a bit of work and is prone to errors. To make the process easier, COM defines a number of known data types. The MIDL compiler creates the marshaler code automatically, when the IDL interface definition only uses these predefined data types.
Compared to direct method calls, marshaling is a very time consuming process. Performance suffers significantly when a COM call requires going from proxy to IPC to stub to COM server to sub to IPC to proxy. In VFP, for instance, calling a DLL object is twice as fast as calling an EXE object. The exact delay, though, depends on the amount of data that needs to be transferred and the speed of the connection between proxy and stub.
Programmers aren't forced to use this generated marshaling code. C++ programmer that want to optimize their COM object can create their own marshaler that might exploit known constellations to speed up the process of packaging paramters.
COM doesn't a format that is used for marshaling. This isn't really necessary, because only the proxy and the stub interpret the format. All intermediate layers like IPC, RPC only need to know how much data to pass on.
When the COM object can be used for automation (IDispatch interface), the number of variants is even further reduced. Therefore Windows contains a generic marshaler for IDispatch interfaces. It packs and unpacks method parameters dynamically. FoxPro uses this marshaler to make COM calls. It's also used to make FoxPro objects look like COM objects. For this reason, you can pass references to any FoxPro object to any COM server.
Apartments, Threads and Processes
COM has originally be developed for Windows 3.1, even though it might seem strange that we actually had COM that long ago. Windows 3.1 is a 16 bit version that uses cooperative multitasking. An application remains active until it calls a Windows API function that returns the next message in the message queue. If there is still time remaining on the process' time slice, Windows merely returns the next message. Otherwise the program is paused. The schedule passes control over to the next task in the queue. At any time there was only one process active and it never had more than one thread. The process was completely in control over when it handed the computer over to the next task.
COM calls between processes required marshaling. Due to the message queue based nature of the Windows task scheduling system, all calls where automatically serialized. At a time a process could only execute one method. Only when this method returned the next method could be called.
This concept became quickly outdated when Windows 95 was released. Suddenly a program could loose control at any time. Nothing prevented two applications from accessing the same object at the same time. Threads increased complexity even more. Threads running in one address space share most resources like memory. Yet, every thread can have its own Thread Local Storage (TLS). If a COM object stores data in the TLS, it can't see this information when it's called from a different thread. Even in calls between threads it might be necessary to use marshaling.
A process is very much like a virtual computer. All processes are strictly separated and can neither call each other nor exchange data. That's physically just impossible. Every communication between processes is managed by Windows. This is called Inter Process Communication. Threads, on the other hand, run within a process. They are executed independently of each other in parallel. But even though threads share memory and other resources, they can not call each other directly. For communicating they have to use messages and other inter-process communication means, too.
Windows 95 introduced the concept of apartments. Every COM object is assigned to one apartment. A single apartment might contain more than one COM server. Each apartment consists of one or multiple threads. Each process (EXE) can contain multiple Single Threaded Apartments (STA), but never more than one Multi Threaded Apartment (MTA). Within an apartment objects can be called directly. When calling other apartments or processes, COM uses the system of proxy and stub.
In total there three variants of threading in COM: single, apartment and free. The fourth variation is both which tells COM to pick either apartment threading or free threading, as the components supports both flavors.
Free threading runs in the MTA whereas the other two models use a STA. When a process initializes the very first apartment, COM creates a Single Threaded Apartment which is called the Main STA. All objects that demand single threading are executed in this apartment. These are basically COM server developed for Windows 3.x. Visual FoxPro 3.0 only supported the single threading model. Visual FoxPro 5.0, on the other hand, uses apartment threading.
When COM initializes an STA, COM creates a spezial window just fort his thread which bases on the OleMinThreadWndClass. This window is used to serialize method calls from other apartments. When a thread is idle it waits for message in the queue of this main window. If there are no more messages, the thread is paused until messages become available.
When an object in apartment A requests an interface pointer to an object in Apartment B, COM creates a proxy in apartment A. Instead of a pointer to the object, COM returns a pointer to this proxy. When calling a method on this proxy it marshals all parameters. Because directly calling the thread in Apartment B isn't possible, the proxy sends a message with all parameters to the OleMainThreadWnd window.
As soon as the object in apartment is ready for new job it will read the message from the queue. The stub of the object in apartment B unpacks the parameter and calls the method. The return value of the method call is packed and posted back as a reply using the message queue of the window in apartment A. There the proxy receives the reply, unpacks the result and returns it.
Always when a method in an STA is called from outside the apartment, the method is never called directly. Instead it is triggered indirectly by messages. Messages are always processes sequentially in first in first out order (FIFO queue). This guarantees that only one method at a time is executed and method calls are processed in calling order.
Multi Threaded Apartments are different. Within the MTA objects are free to make calls at any time. Therefore it can happen that the same method of the same object is executed for two different callers in two different threads at the same time. The object itself is responsible for avoiding conflicts, for example, when accessing a global variable or writing into some sort of property.
STA and MTA has nothing to do with the number of threads within a COM server. It only refers to instance of a single object. A single EXE might contain that require apartment threading (STA). There can be two instances of this object be loaded in two different STAs. In both STAs the same method can be executed at the same time.
IDispatch – OLE automation
In the beginning COM has been developed for development languages that are reasonable close to the machine, such as C++. Methods are called directly using pointers in a so-called v-table. The name origins from C++ where every class has got a list of methods. This list is used to implement virtual methods in objects. That's the origin for the name v-table for this list of method pointers.
In COM we still call this kind of method calls vtable or early-binding. The advantage of vtable binding is that method calls are as fast as method calls in C++. This isn't surprising because COM uses C++ objects internally. At the same time you gain the flexibility of COM. Objects can be located in different processes or even different machines without that you need to change the code. The disadvantage, of course, is that binding to the class happens at compile time. Instead of calling a method by name and signature, you call it by index. Additionally, a C++ programmer had to create rather complex proxy and stub code when they didn't limit themselves to just the basic COM data types. Much to the dislike of C++ programmers they don’t include structures and pointers.
Development languages that executed non-compiled code or didn't support the complex data types of C++ couldn't really make use of the vtable interface. Creating marshalling code or directly interfacing with the COM subsystem isn't everybody's favorite topic. That’s why Microsoft defined a further interface called IDispatch. IDispatch defines four methods: Invoke(), GetIDsOfNames(), GetTypeInfo() and GetTypeInfoCount().
The main method is Invoke. You pass several parameters. The two most important ones are an ID that identifies the method you want to call and an array of parameters. This array contains only variables of the VARIANT data type. Such a value is a variable that can hold different values just like in FoxPro. There is a strict definition for these data types which are limited to simple values such as numbers and strings. Complex data types cannot be passed.
To get the ID of a method you cal GetIDsOfName(). As the parameter you pass the name of a method, such as "Show". The method returns an internal number that represents this method. The number can match in index in the vtable, but it doesn't have to. With the GetTypeInfo() function you can obtain additional information about parameters. This way you can verify the call signature in the calling program before you actually attempt to call the COM server.
Invoke() is nothing but a huge DO CASE block that calls for every ID the actual method. When an application knows how to call Invoke, it can call any method of a COM server that supports IDispatch. An interface that inherits from IDispatch is called a dispinterface. Every COM object that is called by FoxPro applications must inherit from IDispatch.
The advantage of IDispatch is its flexibility. At the same time it slows down program execution and makes it really hard to use COM for C++ developers. That's why most COM interfaces are implemented as dual interface. Such an interface inherits from IDispatch. All methods that are available via Invoke are also added to the vtable. A program can decide whether it wants to úse early-binding or late-binding depending on what suits it better.
In the early time of VFP this was a huge problem. Many components have been marked as dual interface, but only implemented the IDispatch part. The classic Visual Basic didn't use the v-table interface and always used late-binding. That's why many components used to work in Visual Basic, but not in Visual FoxPro which attempted to use the faster v-table version.
Visual FoxPro and COM
COM servers developed in VFP are always running in a Single Threaded Apartment. This is true for classic DLLs as well as multi-threaded DLLs. The difference between the two types of DLLs is in the runtime library, not the COM threading model. The traditional runtime library (VFP9R.DLL) isn't thread safe. This means only one thread can use library. In other word, only one FoxPro application can run at any time. If the library is called from a different thread it crashes.
If you load two COM servers that are written in Visual FoxPro, this should cause a conflict. After all, both COM servers use the same runtime library. To avoid this the second DLL creates a copy of the runtime library and uses the copy instead. With two regular VFP DLL COM server the runtime library is loaded twice into physical memory. The multi-threaded version VFP7T.DLL, on the other hand, is thread safe. It can be called from multiple threads at the same time. Hence, when two DLLs that use the MTDLL runtime are loaded at the same time they use the same runtime DLL file.
Visual FoxPro 7.0 changed the way that Visual FoxPro locates the runtime library. Rather than in the System32 directory the runtime library is now installed to Common Files by default. The code that copies the runtime library in case of a conflict does not copy the resource file (VFP7RENU.DLL). This cause your DLL to crash when it’s the second one loaded into a process.
Despite its name, an MTDLL always loads objects into a single threaded apartment. You don’t gain any scalability by creating an MTDLL. To make use of multiple threads you need a client that creates multiple STAs. Neither Visual FoxPro, nor VB 6, nor the .NET runtime do this for COM servers.
If you write an application that creates multiple instances of an MTDLL object, VFP loads all these object into the same STA. Therefore they cannot be executed in parallel. To get real multithreading you have to create a new STA and then load the COM object into this STA. Microsoft Component Services or IIS do this automatically for you. In a VFP application you have to use a tool like VFPMTAPP from Remus Rusanu or my DMULT utility.
Download the companion files for this article.