Krishna Ganugapati’s Weblog

One of the more painful things about using Linux as a client in a Windows corporate intranet is the clunkiness of integration. The most basic requirement is that you be able to bring up the Nautilus or your favorite graphical file browser and browse through files that live on a Windows (Samba) file server. I’ll reiterate: this is one of the most basic requirements.

Now most affficianados will be up in arms when I say this. Of course, Linux does this and Ubuntu does this out of the box. But here is the rub: Ubuntu’s Nautilus file browser will does this fine. But now I want to go to my bash command shell and I want it to also be able to browse to my Windows file shares. This does not happen out of the box.

Now even if you get all of the pieces to work, the credentials are messed up. The single-sign-on experience needs to immediately transfer itself to all client applications. Even though you can sign in with your AD credentials, if I have to re-enter my credentials, it adds to the clunkiness of the experience.

Introducing the SMB redirector in lwio.

Our SMB redirector driver provides all of the semantics necessary for a file client API. The obvious next step was to pick up FUSE and begin work on a FUSE driver that would call our lwio APIs – NtCreateFile, NtReadFile, NtWriteFile, NtClose. Once you provide a FUSE driver to our SMB client, automatically, a client can mount smb shares without reentering his/her credentials. Once the client mounts a share, he can browse the directory space, copy files, move files, create directories, delete files and perform every UNIX file system operation against the SMB volume. Even better, Nautilus and the Gnome shell work without any additional work.

We’ve started construction of this FUSE driver and its is amazing how fast we’re putting together the pieces. Our Posix Virtual File System (which is the most complete lwio driver) is proxying for the SMB Redirector File System (we’ve implemented enough for the named pipe interface and its missing some of the QueryInfo and SetInfo semantics)

Now here is the even more interesting part: FUSE driver run as a daemon themselves, so a client (like the shell) has to communicate to the FUSE daemon which in turn communicates to the lwio daemon. That’s a transition of 2 processes which means at least 4 context switches to service a call. But lwio can implement a FUSE driver as another lwio driver. Just like the SMB server driver calls the IO Manager to route requests to the Posix Virtual File System, the FUSE lwio driver calls the IO Manager to route requests to the SMB Redirector File System. This implies we reduce the number of context switches between processes to half what we would have if we ran FUSE independently as a separate daemon. This was Brian Koropoff’s epiphany over the weekend. We’re still going to do this first as a separate daemon, iron out the end-to-end scenario, but then placing it into the lwio kernel is a natural optimization.

Well, I thought we’d be doing three key scenarios by April 15th. It looks like we’re going to go for a fourth scenario.

Thanks for reading!

It has been a super productive week.

First the highlights. The lwio SMB File Server has hugely advanced. Most operations from the Windows XP command line work. Copying files, xcopying, making directories, net use, net use /del, single sign-on are all fully operational. In addition, the Windows explorer is almost completely functional. We’ve yet to support server-side file change notifications (i.e changes to the remote directory are not immediately visible in the Windows Explorer, the simple workaround is to hit the refresh button). You can view thumbnails of pictures and photographs, you can bring up notepad on a file directly. In all, we have a pretty useful file server.

Today we plan on starting internal dog-fooding. We will be hosting our own server and placing large size OS iso images which we use, so that everyone in the engineering team can upload/download files to the server. We’re putting all our VMs as well on the server and will subject the server to a significant amount of stress.

The day is not yet done. It is 8:38 hrs PST out here in Bellevue WA. We plan to have the file share management work done by end of day today. This will allow us to create file shares on the Likewise SMB File Server via MMC’s Add Share Wizard. Also, we plan on add a File Management plugin on the Likewise Administrators Console that will allow Linux Administrators to graphically manage their own Linux Likewise File Servers from Linux desktops.

Its been a good week!

Yesterday, we got Windows Explorer to work against the Likewise SMB File Server

Windows Explorer copying a file from the Likewise SMB File Server

As you may deduce, I’m a huge fan of unmanaged remote procedure call  systems.  They serve as the building blocks for most distributed systems problems. They provide an easy programming model – synchronous function calls.  They have a small memory footprint (unlike managed systems like Java and C#). DCE/RPC and Sun RPC are the two widespread models (one is the foundational technology for Windows operating systems) and the other atleast is the basis for NFS infrastructure.

DCE/RPC on UNIX platforms has been in wilderness for almost a decade. Hopefully Likewise’s rehabilitated DCE infrastructure can bring about some form of a renaissance of an unmanaged, easy to use, remote procedure call infrastructure.  But I digress. I wanted to talk about lwmsg.

In the Windows world of systems programming, user mode clients and servers are built around a particular paradigm.  Let say we are building the foo client -server system.  Foo will have a client dll – the fooclient.dll that exposes DLL entry points for a calling client application to link to. The calling client program has no 

When we started on lwis, we wanted to build that Windows style programming pattern – we could have used DCE/RPC using ncalrpc, but at the time, I was worried about the chicken-and-egg effect.  We wanted API  style communication to the lsassd daemon, but one of the primary interfaces would be our gss-ntlm client and our gss-ntlm server. The DCE/RPC system would use NTLM as one of its authentication mechanisms, so wouldn’t that mean we would be using DCE/RPC  to bootstrap itself? So we decided that we would need an effective authenticated  remote procedure call mechanism that would handle communication between the lsass client and the lsass server for each of the lsass server interfaces – the UNIX id mapper interface and the GSS NTLM interface.

Initially, we hand-marshalled these IPC constructs, but as the system grew more sophisticated, the number of hand-marshalled functions grew to the point, that adding a new function got to be tedious. So Brian Koropoff, with his extensive background in compiler and language design, decided to build a data-driven (think idl-like) runtime engine for client-server  systems. This is lwmsg.

lwmsg allows you to specify  function definitions as request-response messages.  A remote procedure call function comprises of a  request message and a pair of response messages (on SUCCESS and FAILURE). What is so compelling about lwmsg is that you can specify a extensive class of C datatypes to marshall across the wire. Pretty much the entire DCE/RPC idl can be represented using lwmsg specifications. A calling client-server program specifies its “idl” as a set of global arrays and the light-weight run-time now handles the entire marshalling and unmarshalling for a calling program. 

LWIS is using lwmsg in three subsystems – the lsass subsystem, the netlogon subsystem and our newest lsmb subsystem. lwmsg builds  very easily, has no dependencies and we’ve just released it on trunk. Its licensed under the LGPL so feel free to get it a whirl.

Our solutions Likewise Open and Likewise Enterprise are deployed in large enterprises – the Fortune 1000/Global 2000 represents our target audience.  Every now and then (more now than then ) I fly down to meet with the customers and articulate our product road map and  clarify specifics on the product.

Once in a while, I get the sense that the UNIX administrators may be concerned on whether they will be losing some degree of control once they join their systems into Active Directory.

The fact is that rather than lose control, the UNIX administrators get to be significantly more empowered than they were before.

An organization tends to consider consolidating its non-Windows systems into Active Directory only if it  already has Active Directory as it primary NOS authentication infrastructure. As a matter of fact, that’s part of our qualification process. We tend to engage with customers who already have a significant Active Directory deployment and broad expertise in Active Directory.

These organizations also have a sizeable Linux or UNIX environment. One interesting trend is that whereas you see a overwhelming number of Windows systems in the corporate intranet – make sense – Windows is currently the only corporate desktop (love to see more Macs and Linux desktops, but that’s another post),  its  not that simple in the corporate extranet. By my anecdotal measure, you can see almost a 50-50 split of Linux/UNIX vs Windows in the extranet. And that also makes sense.  Java, IBM Websphere, Apache, WebLogic, Oracle, DB2 are well established Web platforms.  This is where LAMP is big and for good reason. It is not unusual to see extranets with a couple of thousand non Windows servers.

Typically the Windows servers in the corporate extranet  will be joined to an Active Directory forest. The Linux/UNIX servers are usually configured in a pretty adhoc fashion. We’ve seen environments where customers maintained over thousand servers with local password databases. Many times we find the UNIX servers integrated into NIS or the Linux servers integrated into an OpenLDAP server. The end result is usually not pretty – an administrator can have several accounts on several machines and  account control is usually non-existent.

Whatever the reason, the powers-that-be in the organization decide that its time the non-Windows system into Active Directory. These days we rarely explain the value proposition to customers.We’re usually contacted with customers asking us very specific questions about our product. Consolidating non-Windows systems into Active Directory is mainstream demand fullfillment business (a good place to be) rather than a demand generation business (a bad place to be)

Back to our UNIX administrator. He’s rather concerned about whether this will restrict his ability to manage his systems. He’s seen the product work. He is intrigued by the ability to login with his Active Directory credentials – he no longer needs to handle password resets one of the more time consuming aspects of his job. He just needs to provision the UNIX attributes for his users. He also loves the fact that he can set up a sudoers file and store it as a group policy object and the Likewise group policy agent will just push the appropriate sudoers file to the right set of computers.

“Fine. This is all pretty neat. But this still means that I need a Windows computer to provision the UNIX attributes for my user. And I’m not a huge Windows fan.” says our sceptical UNIX administrator.

Well. Guess what! You don’t! In Likewise Enterprise 4.0, we built the Likewise Administrators Console. Its an analogue to the Windows MMC console with one huge difference. LAC will run natively on  Windows, Linux and the Mac. So instead of having to go to a Windows machine to set the attributes on your Active Directory user object, you can fire up LAC on your favorite desktop Linux and install the Active Directory Users and Computers plugin and manage any and all objects in the directory under your control.  You can also run the LAC Cell Manager plugin and manage any cells you might have created.

You, the UNIX administrator, have the same calibre of tools that the Windows system administrator has to manage his Windows servers. Rather than relegate the Linux/UNIX servers to a small minority, Likewise Enterprise and LAC helps you plan an expansion of your UNIX infrastructure in a AD environment.

Wouldn’t you say this is goodness?

When people talk of Windows-Linux interoperability, just about everyone forgets the most important piece of technology that is the foundation for Windows-to-Windows interoperability- RPC. Remote Procedure Calls. In fact, if you ever get a hold of the first “Inside Windows NT” book by Helen Custer, Chuck Lenzmeier, one of the most influential engineers on the NT operating system makes the exact same assertion: “RPC is one of the most important pieces of technology in Windows NT.”

Microsoft RPC known as MSRPC is actually an implementation of the OSF DCE/RPC framework. If I recall correctly,  Microsoft licensed the DCE/RPC code base but then rewrote it substantially. In 1993, I was hired on  in Microsoft to work on the NT print spooler. The print spooler was one of the heavier users of RPC. The key developers in the RPC team were Bharat Shah (who wrote the runtime rpcrt4.dll) and Vibhas Chandorkar (who wrote the midl compiler). 

From Wikipedia

“DCE/RPC was commissioned by the Open Software Foundation in a “Request for Technology”. One of the key companies that contributed was Apollo Computer, who brought in NCA – “Network Computing Architecture” which became Network Computing System (NCS) and then a major part of DCE/RPC itself. The naming convention for transports that can be designed (as architectural plugins) and then made available to DCE/RPC echoes these origins, e.g. ncacn_np (SMB Named Pipes transport); ncacn_tcp (DCE/RPC over TCP/IP) and ncacn_http to name a small number.

DCE/RPC’s history is such that it’s sometimes cited as an example of design by committee. It is also frequently noted for its complexity, however this complexity is often a result of features that target large distributed systems and which are often unmatched by more modern RPC implementations such as SOAP.”

And since Microsoft hired Paul Leach, one of the founders of Apollo, I suspect Paul brought DCE/RPC in to Microsoft.

Recall that Microsoft really did not embrace IP protocols till NT 4.0/Win95 which means around circa 1996. The first thing Microsoft did with its RPC implementation was to retrofit it to run on named pipes. The protocol conventions in DCE/RPC were ncacn_ip_tcp and ncadg_ip_udp, but Microsoft added ncacn_np protocol – connection oriented semantics over SMB named pipes. Security for RPC over named pipes was done with named pipe transport security.

Named pipe transport security meant NTLM authentication which was soon roundly trashed in the industry as a weak security mechanism. And that was it – pretty any NTLM secured protocol aka SMB was also trashed and by extension MSRPC was also hammered. In general any traffic on port 139 or port 445 was immediately denounced as badness.

Later versions of MSRPC have full gss secured implementations – which means you can do kerberos and ntlm security over ncacn_ip_tcp.

In the meantime, HP and IBM both ported DCE/RPC on to their respective operating systems. HP-UX runs DCE/RPC and so does AIX atleast up to AIX 5.3. However nobody did an SMB stack so there is no ncacn_np support on any of these platforms. Sun did never support DCE/RPC – NFS uses Sun RPC so there was no DCE/RPC support on Solaris.

OSF in the meantime totally missed the boat and did not get DCE/RPC ported to Linux or BSD. It is only recently that they’ve open sourced the DCE environment and that thing has an absolutely horrific build environment. Its pretty tragic – there is some really good stuff out there, but except for the RPC framework, the rest of it – kerberos, a DFS, the directory service, the NTP server, there are better and more current alternatives available that use a sane build system. The LGPLed DCE environment’s build system is insane.

For a long time, there were no decent open source implementations of DCE/RPC. Early in 2007, I began looking at the state of affairs for DCE/RPC and I found three

While the proprietary OS vendors were incorporating non-interoperable versions of DCE/RPC (thanks to Microsoft’s ncacn_np), the Samba project was steadily working on a building an interoperable suite of the SMB protocols. In all of the versions of Samba upto 3.2, Samba was systematically hand marshalling DCE/RPC PDUs. It was only after a significant while did they realize that they were synthesizing DCE/RPC. In early 2003, Andrew Tridgell began work on Samba 4 and implemented a DCE/RPC idl compiler in Perl – pidl. Over time, Samba 4 synthesized all of the Windows RPC services idl files. Samba4 does have an RPC idl compiler and an runtime which is Windows compatible. However Samba 4 does not seem to have implemented a idl syntax which is fully compatible with DCE/RPC. In addition, core to the DCE/RPC framework is the RPC API and I’ve not yet been able to find a DCE/RPC compatible API as part of the Samba 4 suite. Since APIs are really syntactic sugar, one hopes that in time Samba 4 releases a DCE/RPC compatible API and idl compiler.

A surprising development was Novell releasing its DCE/RPC libraries under a BSD license early in 2007. Novell had acquired the rights to PADL Ltd XAD product line and had integrated it with eDirectory to build Domain Services for Windows. Domain Services for Windows was this AD proxy that was built on eDirectory. It allowed Windows clients to connect to eDirectory thinking that they were talking to an Active Directory Domain Controller. I’m speculating that PADL had possibly licensed the DCE/RPC libraries from OSF under a BSD license and used it for its XAD product line. At that time of its release, the Novell libraries were rather difficult to use. One of more challenging problems was that its threading libraries were non-portable and made use of glibc pthread internal data structures.

Thus by mid 2007, there were 3 possible choices for building DCE/RPC applications natively on Linux/UNIX platforms.

Around the same time, at Likewise we’d been discussing what it would take to build real Windows interoperability from the Linux side. And inarguably the conclusion was that without an API compatible DCE/RPC framework running natively on the Linux side, any interoperability effort would be futile. We selected the PADL libraries that Novell had open sourced for two reasons.

First and most important the lineage was from OSF which meant there were RPC APIs that were compatible with the MSRPC APIs and secondly the grammar that the IDL compiler parsed was fully compatible with midl (the Microsoft IDL compiler). This would mean that ISVs that had written RPC applications on Windows could cleanly port their infrastructure from Windows to Linux.

Second and less importantly, the licensing was a BSD style licensing. I am of the view that open source is all goodness, but ISVs still like to protect their IP. Inorder to evangelize and resurrect a technology, it would be important to get a lot of ISVs to write applications to this platform. So the ideal choice for us would be to have a LGPL or BSD style license for the libraries. We could have picked up the recently released OSF source code but the build system, as I noted, is insane. Any OSF developers/people out there, if you ever read, do yourselves a huge favor and fix your build system.

Once we picked this technology, we found out that the code simply did not work. After some really gnarly work and several changes, we got this libraries in a working state. The two most important pieces of work were redoing the threading library and adding client side named pipe support.

Remember that our goal was to build true Linux-Windows interoperability. The one sentence definition of Linux-Windows interoperability is to have the Windows Net and DSys APIs supported natively on Linux. And as I’ve noted earlier in this post, how pervasively RPC is the foundation for the Windows Net and DSys APIs, we began build equivalents of these APIs on Linux. In December of 2007, after a whirlwind 5 months of work, we had these APIs running natively on Linux. Since then, this work has expanded significantly in scope.

I want to conclude this piece with the following assertion.

“If seamless Windows-Linux interoperability is to be a reality and if we really want to see Linux systems as first class citizens in a Windows environment, then it is supremely important that an MS compatible DCE/RPC framework be a first class citizen on every flavor of Linux/UNIX/Mac”

The shunning of DCE/RPC on the Linux platform is probably one of the greater ironies in the evolution of Linux as a mainstream platform. Microsoft took an otherwise open programming platform and base all of its distributed systems technologies on it. The open source community and the industry around denounced RPC as a “Microsoft only” technology and considered insecure because of its association to a security mechanism that was not part of RPC. And this shunning resulted in an “open technology” becoming “proprietary” because the only mainstream implementation ended up being Microsoft.

Hopefully, we can fix this by making a mainstream open source Microsoft compatible DCE/RPC framework available to community at large.

I like to distill technologies down to their essence.  I view most technologies with sceptism – do I really need this? is a question I ask myself all the time.  And I’m now in plumb in the center of interoperability business. After all, what could be more core to interoperability that making Windows and non-Windows systems speak to one another in a totally seamless fashion?

Interoperability today spans a wide universe of technologies. Authentication, databases, management infrastructure, web standards: the list goes on and on. Novell and Microsoft made an announcement last year that they would interoperate on directory standards, XML formats and virtualization efforts. All goodness.

But here’s my question:  I want the vanilla Windows client and server boxes to interoperate seamlessly with the Linux and UNIX client and server  boxes.  Let’s drill down even further.

I don’t want to buy a management solution to do this

I’d like to see the following

1) Linux clients can access files on Windows server seamlessly.  Windows clients can access files on Linux server seamlessly

2)Linux web browsers can browse Windows web server pages in a corporate intranet seamlessly. Likewise Windows web clients can browse Linux web servers in a corporate intranet seamlessly.

Today, you simply cannot do this. Let say you are a large F1000 company and you have several thousand Windows desktops and several thousand Windows servers, it is near impossible for you to want to introduce Linux or Mac desktops into this environment and  similarly it is near impossible for you to introduce Linux or Unix servers into this Windows centric environment.

Why is this so? This is because Windows environments are built around Active Directory. Every single Windows desktop or Windows server is “joined” to an Active Directory domain and the seamless highly integrated feel that you have when you’re on a Windows desktop to connect to resources is all because of this Active Directory integration. In my next post, I’ll talk about Active Directory being the new “unfair advantage” for Microsoft.

We released our Likewise Open Spring Edition to great acclaim. If you’re running Ubuntu you can install it in minutes like this happy user http://www.wantmoore.com/archives/2008/05/06/ubuntu-804-ad-via-likewise-open/. Also if you’re running any of over 110+ platforms, you can just download your bits and get up and running in minutes. Check out the likewise-open mailing lists – we’re on there all the time and can answer any questions you may have.

We had a terrific Samba XP. We did two presentations and both of them were really well received.

Our first talk was an overview of all our forthcoming technologies. We demonstrated the latest version of Likewise RPC infrastructure and showcased our NetAPIs running natively on Linux. Our Likewise Administrator’s Console was a huge hit. We showed plugins that demoed “Local users and Groups”, a combination ADSIEDIT/ADUC plugin, and our latest “EventLog” plugin.

The highlight of this talk was Andrew Tridgell asking us if we would release the framework to the community.  We’re working on this so stay tuned. I’ll post a few screenshots of LAC and its plugins working.

Brian Koropoff and Rafal Sczczeniak did our second talk. They described the changes we made to adapt DCE/RPC to be Windows compatible. Great job Brian and Rafal.