A History of MS-RPC and open source equivalents

When people talk of Windows-Linux interoperability, just about everyone forgets the most important piece of technology that is the foundation for Windows-to-Windows interoperability- RPC. Remote Procedure Calls. In fact, if you ever get a hold of the first “Inside Windows NT” book by Helen Custer, Chuck Lenzmeier, one of the most influential engineers on the NT operating system makes the exact same assertion: “RPC is one of the most important pieces of technology in Windows NT.”

Microsoft RPC known as MSRPC is actually an implementation of the OSF DCE/RPC framework. If I recall correctly,  Microsoft licensed the DCE/RPC code base but then rewrote it substantially. In 1993, I was hired on  in Microsoft to work on the NT print spooler. The print spooler was one of the heavier users of RPC. The key developers in the RPC team were Bharat Shah (who wrote the runtime rpcrt4.dll) and Vibhas Chandorkar (who wrote the midl compiler). 

From Wikipedia

“DCE/RPC was commissioned by the Open Software Foundation in a “Request for Technology”. One of the key companies that contributed was Apollo Computer, who brought in NCA – “Network Computing Architecture” which became Network Computing System (NCS) and then a major part of DCE/RPC itself. The naming convention for transports that can be designed (as architectural plugins) and then made available to DCE/RPC echoes these origins, e.g. ncacn_np (SMB Named Pipes transport); ncacn_tcp (DCE/RPC over TCP/IP) and ncacn_http to name a small number.

DCE/RPC’s history is such that it’s sometimes cited as an example of design by committee. It is also frequently noted for its complexity, however this complexity is often a result of features that target large distributed systems and which are often unmatched by more modern RPC implementations such as SOAP.”

And since Microsoft hired Paul Leach, one of the founders of Apollo, I suspect Paul brought DCE/RPC in to Microsoft.

Recall that Microsoft really did not embrace IP protocols till NT 4.0/Win95 which means around circa 1996. The first thing Microsoft did with its RPC implementation was to retrofit it to run on named pipes. The protocol conventions in DCE/RPC were ncacn_ip_tcp and ncadg_ip_udp, but Microsoft added ncacn_np protocol – connection oriented semantics over SMB named pipes. Security for RPC over named pipes was done with named pipe transport security.

Named pipe transport security meant NTLM authentication which was soon roundly trashed in the industry as a weak security mechanism. And that was it – pretty any NTLM secured protocol aka SMB was also trashed and by extension MSRPC was also hammered. In general any traffic on port 139 or port 445 was immediately denounced as badness.

Later versions of MSRPC have full gss secured implementations – which means you can do kerberos and ntlm security over ncacn_ip_tcp.

In the meantime, HP and IBM both ported DCE/RPC on to their respective operating systems. HP-UX runs DCE/RPC and so does AIX atleast up to AIX 5.3. However nobody did an SMB stack so there is no ncacn_np support on any of these platforms. Sun did never support DCE/RPC – NFS uses Sun RPC so there was no DCE/RPC support on Solaris.

OSF in the meantime totally missed the boat and did not get DCE/RPC ported to Linux or BSD. It is only recently that they’ve open sourced the DCE environment and that thing has an absolutely horrific build environment. Its pretty tragic – there is some really good stuff out there, but except for the RPC framework, the rest of it – kerberos, a DFS, the directory service, the NTP server, there are better and more current alternatives available that use a sane build system. The LGPLed DCE environment’s build system is insane.

For a long time, there were no decent open source implementations of DCE/RPC. Early in 2007, I began looking at the state of affairs for DCE/RPC and I found three

While the proprietary OS vendors were incorporating non-interoperable versions of DCE/RPC (thanks to Microsoft’s ncacn_np), the Samba project was steadily working on a building an interoperable suite of the SMB protocols. In all of the versions of Samba upto 3.2, Samba was systematically hand marshalling DCE/RPC PDUs. It was only after a significant while did they realize that they were synthesizing DCE/RPC. In early 2003, Andrew Tridgell began work on Samba 4 and implemented a DCE/RPC idl compiler in Perl – pidl. Over time, Samba 4 synthesized all of the Windows RPC services idl files. Samba4 does have an RPC idl compiler and an runtime which is Windows compatible. However Samba 4 does not seem to have implemented a idl syntax which is fully compatible with DCE/RPC. In addition, core to the DCE/RPC framework is the RPC API and I’ve not yet been able to find a DCE/RPC compatible API as part of the Samba 4 suite. Since APIs are really syntactic sugar, one hopes that in time Samba 4 releases a DCE/RPC compatible API and idl compiler.

A surprising development was Novell releasing its DCE/RPC libraries under a BSD license early in 2007. Novell had acquired the rights to PADL Ltd XAD product line and had integrated it with eDirectory to build Domain Services for Windows. Domain Services for Windows was this AD proxy that was built on eDirectory. It allowed Windows clients to connect to eDirectory thinking that they were talking to an Active Directory Domain Controller. I’m speculating that PADL had possibly licensed the DCE/RPC libraries from OSF under a BSD license and used it for its XAD product line. At that time of its release, the Novell libraries were rather difficult to use. One of more challenging problems was that its threading libraries were non-portable and made use of glibc pthread internal data structures.

Thus by mid 2007, there were 3 possible choices for building DCE/RPC applications natively on Linux/UNIX platforms.

Around the same time, at Likewise we’d been discussing what it would take to build real Windows interoperability from the Linux side. And inarguably the conclusion was that without an API compatible DCE/RPC framework running natively on the Linux side, any interoperability effort would be futile. We selected the PADL libraries that Novell had open sourced for two reasons.

First and most important the lineage was from OSF which meant there were RPC APIs that were compatible with the MSRPC APIs and secondly the grammar that the IDL compiler parsed was fully compatible with midl (the Microsoft IDL compiler). This would mean that ISVs that had written RPC applications on Windows could cleanly port their infrastructure from Windows to Linux.

Second and less importantly, the licensing was a BSD style licensing. I am of the view that open source is all goodness, but ISVs still like to protect their IP. Inorder to evangelize and resurrect a technology, it would be important to get a lot of ISVs to write applications to this platform. So the ideal choice for us would be to have a LGPL or BSD style license for the libraries. We could have picked up the recently released OSF source code but the build system, as I noted, is insane. Any OSF developers/people out there, if you ever read, do yourselves a huge favor and fix your build system.

Once we picked this technology, we found out that the code simply did not work. After some really gnarly work and several changes, we got this libraries in a working state. The two most important pieces of work were redoing the threading library and adding client side named pipe support.

Remember that our goal was to build true Linux-Windows interoperability. The one sentence definition of Linux-Windows interoperability is to have the Windows Net and DSys APIs supported natively on Linux. And as I’ve noted earlier in this post, how pervasively RPC is the foundation for the Windows Net and DSys APIs, we began build equivalents of these APIs on Linux. In December of 2007, after a whirlwind 5 months of work, we had these APIs running natively on Linux. Since then, this work has expanded significantly in scope.

I want to conclude this piece with the following assertion.

“If seamless Windows-Linux interoperability is to be a reality and if we really want to see Linux systems as first class citizens in a Windows environment, then it is supremely important that an MS compatible DCE/RPC framework be a first class citizen on every flavor of Linux/UNIX/Mac”

The shunning of DCE/RPC on the Linux platform is probably one of the greater ironies in the evolution of Linux as a mainstream platform. Microsoft took an otherwise open programming platform and base all of its distributed systems technologies on it. The open source community and the industry around denounced RPC as a “Microsoft only” technology and considered insecure because of its association to a security mechanism that was not part of RPC. And this shunning resulted in an “open technology” becoming “proprietary” because the only mainstream implementation ended up being Microsoft.

Hopefully, we can fix this by making a mainstream open source Microsoft compatible DCE/RPC framework available to community at large.