Krishna Ganugapati’s Weblog

Likewise Open and Likewise Enterprise are now featured on RHX – RedHat Exchange. Check us out http://rhx.redhat.com/rhx/catalog/products.jspa?categoryId=1015

Our solutions Likewise Open and Likewise Enterprise are deployed in large enterprises – the Fortune 1000/Global 2000 represents our target audience.  Every now and then (more now than then ) I fly down to meet with the customers and articulate our product road map and  clarify specifics on the product.

Once in a while, I get the sense that the UNIX administrators may be concerned on whether they will be losing some degree of control once they join their systems into Active Directory.

The fact is that rather than lose control, the UNIX administrators get to be significantly more empowered than they were before.

An organization tends to consider consolidating its non-Windows systems into Active Directory only if it  already has Active Directory as it primary NOS authentication infrastructure. As a matter of fact, that’s part of our qualification process. We tend to engage with customers who already have a significant Active Directory deployment and broad expertise in Active Directory.

These organizations also have a sizeable Linux or UNIX environment. One interesting trend is that whereas you see a overwhelming number of Windows systems in the corporate intranet – make sense – Windows is currently the only corporate desktop (love to see more Macs and Linux desktops, but that’s another post),  its  not that simple in the corporate extranet. By my anecdotal measure, you can see almost a 50-50 split of Linux/UNIX vs Windows in the extranet. And that also makes sense.  Java, IBM Websphere, Apache, WebLogic, Oracle, DB2 are well established Web platforms.  This is where LAMP is big and for good reason. It is not unusual to see extranets with a couple of thousand non Windows servers.

Typically the Windows servers in the corporate extranet  will be joined to an Active Directory forest. The Linux/UNIX servers are usually configured in a pretty adhoc fashion. We’ve seen environments where customers maintained over thousand servers with local password databases. Many times we find the UNIX servers integrated into NIS or the Linux servers integrated into an OpenLDAP server. The end result is usually not pretty – an administrator can have several accounts on several machines and  account control is usually non-existent.

Whatever the reason, the powers-that-be in the organization decide that its time the non-Windows system into Active Directory. These days we rarely explain the value proposition to customers.We’re usually contacted with customers asking us very specific questions about our product. Consolidating non-Windows systems into Active Directory is mainstream demand fullfillment business (a good place to be) rather than a demand generation business (a bad place to be)

Back to our UNIX administrator. He’s rather concerned about whether this will restrict his ability to manage his systems. He’s seen the product work. He is intrigued by the ability to login with his Active Directory credentials – he no longer needs to handle password resets one of the more time consuming aspects of his job. He just needs to provision the UNIX attributes for his users. He also loves the fact that he can set up a sudoers file and store it as a group policy object and the Likewise group policy agent will just push the appropriate sudoers file to the right set of computers.

“Fine. This is all pretty neat. But this still means that I need a Windows computer to provision the UNIX attributes for my user. And I’m not a huge Windows fan.” says our sceptical UNIX administrator.

Well. Guess what! You don’t! In Likewise Enterprise 4.0, we built the Likewise Administrators Console. Its an analogue to the Windows MMC console with one huge difference. LAC will run natively on  Windows, Linux and the Mac. So instead of having to go to a Windows machine to set the attributes on your Active Directory user object, you can fire up LAC on your favorite desktop Linux and install the Active Directory Users and Computers plugin and manage any and all objects in the directory under your control.  You can also run the LAC Cell Manager plugin and manage any cells you might have created.

You, the UNIX administrator, have the same calibre of tools that the Windows system administrator has to manage his Windows servers. Rather than relegate the Linux/UNIX servers to a small minority, Likewise Enterprise and LAC helps you plan an expansion of your UNIX infrastructure in a AD environment.

Wouldn’t you say this is goodness?

When people talk of Windows-Linux interoperability, just about everyone forgets the most important piece of technology that is the foundation for Windows-to-Windows interoperability- RPC. Remote Procedure Calls. In fact, if you ever get a hold of the first “Inside Windows NT” book by Helen Custer, Chuck Lenzmeier, one of the most influential engineers on the NT operating system makes the exact same assertion: “RPC is one of the most important pieces of technology in Windows NT.”

Microsoft RPC known as MSRPC is actually an implementation of the OSF DCE/RPC framework. If I recall correctly,  Microsoft licensed the DCE/RPC code base but then rewrote it substantially. In 1993, I was hired on  in Microsoft to work on the NT print spooler. The print spooler was one of the heavier users of RPC. The key developers in the RPC team were Bharat Shah (who wrote the runtime rpcrt4.dll) and Vibhas Chandorkar (who wrote the midl compiler). 

From Wikipedia

“DCE/RPC was commissioned by the Open Software Foundation in a “Request for Technology”. One of the key companies that contributed was Apollo Computer, who brought in NCA – “Network Computing Architecture” which became Network Computing System (NCS) and then a major part of DCE/RPC itself. The naming convention for transports that can be designed (as architectural plugins) and then made available to DCE/RPC echoes these origins, e.g. ncacn_np (SMB Named Pipes transport); ncacn_tcp (DCE/RPC over TCP/IP) and ncacn_http to name a small number.

DCE/RPC’s history is such that it’s sometimes cited as an example of design by committee. It is also frequently noted for its complexity, however this complexity is often a result of features that target large distributed systems and which are often unmatched by more modern RPC implementations such as SOAP.”

And since Microsoft hired Paul Leach, one of the founders of Apollo, I suspect Paul brought DCE/RPC in to Microsoft.

Recall that Microsoft really did not embrace IP protocols till NT 4.0/Win95 which means around circa 1996. The first thing Microsoft did with its RPC implementation was to retrofit it to run on named pipes. The protocol conventions in DCE/RPC were ncacn_ip_tcp and ncadg_ip_udp, but Microsoft added ncacn_np protocol – connection oriented semantics over SMB named pipes. Security for RPC over named pipes was done with named pipe transport security.

Named pipe transport security meant NTLM authentication which was soon roundly trashed in the industry as a weak security mechanism. And that was it – pretty any NTLM secured protocol aka SMB was also trashed and by extension MSRPC was also hammered. In general any traffic on port 139 or port 445 was immediately denounced as badness.

Later versions of MSRPC have full gss secured implementations – which means you can do kerberos and ntlm security over ncacn_ip_tcp.

In the meantime, HP and IBM both ported DCE/RPC on to their respective operating systems. HP-UX runs DCE/RPC and so does AIX atleast up to AIX 5.3. However nobody did an SMB stack so there is no ncacn_np support on any of these platforms. Sun did never support DCE/RPC – NFS uses Sun RPC so there was no DCE/RPC support on Solaris.

OSF in the meantime totally missed the boat and did not get DCE/RPC ported to Linux or BSD. It is only recently that they’ve open sourced the DCE environment and that thing has an absolutely horrific build environment. Its pretty tragic – there is some really good stuff out there, but except for the RPC framework, the rest of it – kerberos, a DFS, the directory service, the NTP server, there are better and more current alternatives available that use a sane build system. The LGPLed DCE environment’s build system is insane.

For a long time, there were no decent open source implementations of DCE/RPC. Early in 2007, I began looking at the state of affairs for DCE/RPC and I found three

While the proprietary OS vendors were incorporating non-interoperable versions of DCE/RPC (thanks to Microsoft’s ncacn_np), the Samba project was steadily working on a building an interoperable suite of the SMB protocols. In all of the versions of Samba upto 3.2, Samba was systematically hand marshalling DCE/RPC PDUs. It was only after a significant while did they realize that they were synthesizing DCE/RPC. In early 2003, Andrew Tridgell began work on Samba 4 and implemented a DCE/RPC idl compiler in Perl – pidl. Over time, Samba 4 synthesized all of the Windows RPC services idl files. Samba4 does have an RPC idl compiler and an runtime which is Windows compatible. However Samba 4 does not seem to have implemented a idl syntax which is fully compatible with DCE/RPC. In addition, core to the DCE/RPC framework is the RPC API and I’ve not yet been able to find a DCE/RPC compatible API as part of the Samba 4 suite. Since APIs are really syntactic sugar, one hopes that in time Samba 4 releases a DCE/RPC compatible API and idl compiler.

A surprising development was Novell releasing its DCE/RPC libraries under a BSD license early in 2007. Novell had acquired the rights to PADL Ltd XAD product line and had integrated it with eDirectory to build Domain Services for Windows. Domain Services for Windows was this AD proxy that was built on eDirectory. It allowed Windows clients to connect to eDirectory thinking that they were talking to an Active Directory Domain Controller. I’m speculating that PADL had possibly licensed the DCE/RPC libraries from OSF under a BSD license and used it for its XAD product line. At that time of its release, the Novell libraries were rather difficult to use. One of more challenging problems was that its threading libraries were non-portable and made use of glibc pthread internal data structures.

Thus by mid 2007, there were 3 possible choices for building DCE/RPC applications natively on Linux/UNIX platforms.

Around the same time, at Likewise we’d been discussing what it would take to build real Windows interoperability from the Linux side. And inarguably the conclusion was that without an API compatible DCE/RPC framework running natively on the Linux side, any interoperability effort would be futile. We selected the PADL libraries that Novell had open sourced for two reasons.

First and most important the lineage was from OSF which meant there were RPC APIs that were compatible with the MSRPC APIs and secondly the grammar that the IDL compiler parsed was fully compatible with midl (the Microsoft IDL compiler). This would mean that ISVs that had written RPC applications on Windows could cleanly port their infrastructure from Windows to Linux.

Second and less importantly, the licensing was a BSD style licensing. I am of the view that open source is all goodness, but ISVs still like to protect their IP. Inorder to evangelize and resurrect a technology, it would be important to get a lot of ISVs to write applications to this platform. So the ideal choice for us would be to have a LGPL or BSD style license for the libraries. We could have picked up the recently released OSF source code but the build system, as I noted, is insane. Any OSF developers/people out there, if you ever read, do yourselves a huge favor and fix your build system.

Once we picked this technology, we found out that the code simply did not work. After some really gnarly work and several changes, we got this libraries in a working state. The two most important pieces of work were redoing the threading library and adding client side named pipe support.

Remember that our goal was to build true Linux-Windows interoperability. The one sentence definition of Linux-Windows interoperability is to have the Windows Net and DSys APIs supported natively on Linux. And as I’ve noted earlier in this post, how pervasively RPC is the foundation for the Windows Net and DSys APIs, we began build equivalents of these APIs on Linux. In December of 2007, after a whirlwind 5 months of work, we had these APIs running natively on Linux. Since then, this work has expanded significantly in scope.

I want to conclude this piece with the following assertion.

“If seamless Windows-Linux interoperability is to be a reality and if we really want to see Linux systems as first class citizens in a Windows environment, then it is supremely important that an MS compatible DCE/RPC framework be a first class citizen on every flavor of Linux/UNIX/Mac”

The shunning of DCE/RPC on the Linux platform is probably one of the greater ironies in the evolution of Linux as a mainstream platform. Microsoft took an otherwise open programming platform and base all of its distributed systems technologies on it. The open source community and the industry around denounced RPC as a “Microsoft only” technology and considered insecure because of its association to a security mechanism that was not part of RPC. And this shunning resulted in an “open technology” becoming “proprietary” because the only mainstream implementation ended up being Microsoft.

Hopefully, we can fix this by making a mainstream open source Microsoft compatible DCE/RPC framework available to community at large.

If you’re a Windows system administrator or a power user on Windows, you’ve visited the Window eventlog system at some point.  Every Windows operating system be it a desktop, workstation, server, domain or controller has an eventlog service running that stores system events as structured records. The eventlog service is accessible to all components of the operating system. Kernel mode drivers can write events to the eventlog. User mode daemons can write events to the eventlog. Authenticated clients can read and write records to the eventlog. Because the eventlog functions as an RPC server,  that means authenticated remote clients can access the eventlog.  In the days of Windows 2000. I remember Jim Allchin sending out directives to all developers in Windows to ensure that the quality of events that were written into the eventlog were system administrator relevant and more importantly actionable.  Developers with poor engineering hygiene were soundly chastised for misusing the eventlog.  The structured record storage model implied that you could do eventlog analysis in a deterministic fashion. Note however that the eventlog was not build on top of a database. The eventlog is still a flat file. Thus to do analysis, a client program has to sequentially scan the eventlog and filter events that match its requirements.

Even with this limitation, the deterministic and structured storage of the eventlog  in Windows allows for robust analysis, troubleshooting and problem diagnostics.

I’ve spent the last two years working on Linux and UNIX operating systems and I do miss structured system events which are deterministic. The logs are by nature text based and while I haven’t done much analysis myself, I wonder  whether text logs make it easy auto diagnostic systems to detect system anomalies. I also find it troubling when logs in general have developer debugging information. It adds to the noise.  Granted logs can be an invaluable aid to debugging, but I would be expect administrator logs to be different from developer logs. In any case, there are a host of companies that have built businesses around simplifying and reducing the noise-to-signal ratio UNIX log files.

Okay, so the title does sound provocative. But it really isn’t. If you’re starting a new company and you go the venture route, the first thing an investor should ask you is what is your unfair advantage? An unfair advantage is also known as a sustained competitive advantage. The phrase is self-explanatory and there are a zillion books on entrepreneurship that will tell you about SCA. Hopefully at the end of this piece, you’ll agree with me that Active Directory is a great SCA  to have.

In the mid 80’s Microsoft invested heavily in building the Windows desktop franchise. Yes late 80’s, think about it, by Windows 3.1 Microsoft had significantly dominated the desktop  OS business. By the early 90’s Apple made significant missteps with not having its next generation OS ready. Jobs had left to go build NeXT. Microsoft follows up in ‘95 with Windows 95 for the consumer and Dave Cutler’s NT team has released Windows NT 3.1 in ‘93 – by ‘96, there is Windows 95 for the consumer and Windows NT 4.0 for business now share the same desktop look and feel. In fact Windows NT 4.0 was internally referred to as SUR – the Shell Update Release. Some even know it as Tukwila – much closer to home (Redmond) than Cairo.

Point: by ‘96 – the desktop operating system is Microsoft’s SCA – its sustained competitive advantage. There is no other prevailing operating system. Microsoft owns 99% of the world-wide desktop operating system business.

1996 is a good point to mark time. A new battle is brewing. The war for directory services. Directory Services is not really a new idea – they’ve been directory services for a while. A lot of neat directory services were built in the ’80s, but what makes the ’90s the decade of directory services is that they are about the enter into mainstream (PC) computing consciousness.

In the ’80s through the early ’90s, Sun had already come out with yellow pages – what we all know as NIS today. Novell had dominated the ’80s with Netware. Netware 3.12’s bindery based architecture had users and groups and all that good stuff. Banyan comes out with StreetTalk – that was a real directory server. The OSF announces its  Cell Directory Service. Microsoft and IBM have released Windows for workgroups and LanManager. Windows NT 3.1 introduces domain controllers and a rudimentary replication model for them. Novell now releases Netware 5.0 and NDS – Novell Directory Services. Guess what we’re getting to closer to today – NDS is eDirectory. And in 1990, Microsoft begins its ill-fated Cairo adventure.  In 1997 Microsoft begins Windows 2000 development – it is probably the most influential version of Windows to have shipped.  The 1997 Professional Developers Conference has the tag line “The Rennaissance of Distributed Systems” – a slew of technologies that dominate todays corporate networks emerge – a distributed security infrastructure that  used kerberos for authentication, a Directory Service which served as the backbone for the KDC, a Distributed File System, a Radius Infrastructure, a  centralized policy distribution mechanism, a DNS service, a DHCP service, a certificate server infrastructure,. I haven’t even scratched the surface  and I’m still talking only about the core Windows server infrastructure.

Windows 2000 ships February 1999 and it immediately competes with a host of other directory products. Netscape has their Directory Server,  Novell continues with NDS, Sun has its own and then licenses Netscape’s Directory Server as iPlanet.

Fast forward 8 years – Windows Active Directory is the preeminent NOS directory for corporate intranets in 93% of the Fortune 1000. The numbers are the same for the Global 2000. The rest of the directory players have less than 7% market share.

So why is Active Directory such a superb “unfair” advantage for Microsoft? Let list how it affects the average corporate knowledge worker.

1) The average knowledge worker pretty much still goes to the office everyday. She still logs into her computer every day using her username and password. She is now authenticating to her company’s Active Directory domain controller

2) She then needs to access some files. Let’s say she still connects to a file share from her browser. She is issued a service ticket to that file server from her Active Directory domain controller. It happens  unobtrusively; she doesn’t have to enter her credentials again – she is granted access to the file server

3) Her company has a Sharepoint file server. She points her browser to the sharepoint server and magically she is granted access to content on the server. Again, under the covers her system has just talked to the DC requesting a ticket to talk to the Sharepoint server

4) And now the biggie. Exchange. Exchange and Active Directory are a matched pair. There are estimated around 250 million CALs of Exchange. Everyone of those clients is tightly integrated with Active Directory. First every Outlook client seamlessly authenticates to AD to talk to its Exchange server and every Exchange server’s routing information and users and groups and distribution lists are all stored in AD. In fact, there is no Exchange server deployment without an Active Directory backend,

5) But let’s get back to our user. She goes home and now wants to access her computer at work. She needs to VPN into the corporate network and guess what she has to enter her Active Directory credentials irrespective of whether she uses L2TP/IPSec or PPTP. Again she uses the same user name and password she has for logging into the corporate network when she is on the corporate premises.

6) Once she’s on the corporate network, she wants to use establish a remote desktop session. Again she has to enter her Active Directory credentials to get access to her desktop. Does it ever stop? no..

7) She now is back at the office and she is using a laptop. But because her company uses a secure wireless network, and her company uses WPA security, she will have to enter her AD credentials again before she can connect to the access point.

This is just the tip of the iceberg. This is the nature of a Windows centric corporate network in circa 2008. Every activity performed by a user is intrinisically controlled by Active Directory.
This should give you an idea of  how completely the everyday activities of a corporate knowledge worker are controlled by Active Directory. On my next posting, I’ll start looking at other non-Windows operating systems and show you how they have a hard time getting adopted within a corporation when they fail to seamlessly integrate in with Active Directory.

I like to distill technologies down to their essence.  I view most technologies with sceptism – do I really need this? is a question I ask myself all the time.  And I’m now in plumb in the center of interoperability business. After all, what could be more core to interoperability that making Windows and non-Windows systems speak to one another in a totally seamless fashion?

Interoperability today spans a wide universe of technologies. Authentication, databases, management infrastructure, web standards: the list goes on and on. Novell and Microsoft made an announcement last year that they would interoperate on directory standards, XML formats and virtualization efforts. All goodness.

But here’s my question:  I want the vanilla Windows client and server boxes to interoperate seamlessly with the Linux and UNIX client and server  boxes.  Let’s drill down even further.

I don’t want to buy a management solution to do this

I’d like to see the following

1) Linux clients can access files on Windows server seamlessly.  Windows clients can access files on Linux server seamlessly

2)Linux web browsers can browse Windows web server pages in a corporate intranet seamlessly. Likewise Windows web clients can browse Linux web servers in a corporate intranet seamlessly.

Today, you simply cannot do this. Let say you are a large F1000 company and you have several thousand Windows desktops and several thousand Windows servers, it is near impossible for you to want to introduce Linux or Mac desktops into this environment and  similarly it is near impossible for you to introduce Linux or Unix servers into this Windows centric environment.

Why is this so? This is because Windows environments are built around Active Directory. Every single Windows desktop or Windows server is “joined” to an Active Directory domain and the seamless highly integrated feel that you have when you’re on a Windows desktop to connect to resources is all because of this Active Directory integration. In my next post, I’ll talk about Active Directory being the new “unfair advantage” for Microsoft.