Krishna Ganugapati’s Weblog

Well, its 5 more days until 2009. It has been a very very eventful 2008. I’m not one who dwells on the past a whole lot. I’m much more a look to the future kind of person. But I thinks this year warrants some look back for me.

From a technology perspective, it has been a really rewarding year. Just one year ago, Sriram Nambakam and I started working on a design for a new authentication engine. The thesis for me was that to build an authentication engine that “talked” to Windows authentication infrastructure, then it would be important to have the same design artifacts that were available in Windows. In the past one year, we’ve rehabilitated DCE/RPC, made it Windows compatible, built the LanMan APIs native on Linux, built an eventlog system, built a modern management console infrastructure, gssified NTLM, built full Samba and Apache interoperability and last but not least of all released a new SMB framework. Not bad work at all.

LWIS has definitely left the building!

Funny thing is that when I tell people about the breadth and scope of the work we’ve pulled off, the tendency is to not believe me

All our developers deserve a huge pat on the back. One of the toughest challenges in the technology business is forging a team of great people. The Likewise development team is probably the best I’ve ever worked with. I’m obviously biased because I’ve recruited all of them

Likewise Open continues to grow in leaps and bounds. The number of downloads is staggering. Sometimes, I catch myself thinking we just might have done our job a little too well. As I said, I catch myself .

On the personal front, the year started out really great but has ended rather flat. It is perhaps partly to the fact that we’ve done something hugely challenging and accomplished it. The reward for me always seems to be in the challenge – I live for the climb. Its is the anticipation of close success that is addictive to me. Once we’ve taken the peak, there is a sense of emptiness.

On the home front, life could not have gone better. We welcomed our daughter Maya and she is a beauty!

Well here’s looking for new challenges in 2009!

Luke Howard just forwarded me a link to the kerberos consortium website. The roadmap says that krb 5 1.7 is in code freeze by the end of January 2009 and will ship April 2009. That’s a big deal for us – all of the necessary kerberos changes we need for dce/rpc and gss-ntlm will be upstream which means we can get out of maintaining our own patches in krb 5 1.6. Which would be a relief to me.

I read today on MSNBC that President Obama’s cabinet consists of a large number of pragmatic thinkers. This post has got nothing to do with Obama’s cabinet picks. You can read the analysis here: http://www.msnbc.msn.com/id/28323330/. It’s just that I was struck by one of the quotes that I thought with minor modifications is apt for how startup and companies in general should function.

The quote is “Pragmatism has its place, but there are limits, as well,” said  Peter Wehner, now a senior fellow at the Ethics and Public Policy Center. “If you aren’t anchored to a political philosophy, you get blown about, and government becomes ad hoc and you make it up as you go — and if you’re not careful, you begin to go in circles”

Now if you replace “political philosophy” with corporate vision/world view, you will see that this quote is pretty powerful advice for a startup. By the way, the lazy reader might skim through and think that the quote suggests that pragmatism and a political philosophy/corporate vision are mutually exclusive. That is not the case. Great visions are almost always pragmatic and tangible.

A core vision when building a company and while the vision needs to be tweaked and modified responsibly, lack of vision results in huge amounts of reactivity. The vision should be a unifying theory on how to exponentially (that’s how you get to maximizing shareholder value) solve a class of problems that vex a large number of customers. The vision is the company’s value proposition.

In the absence of a thoughtful vision, the company will be buffetted by every possible gale. A vision anchors the company to examine every opportunity and analyze against the parameters that the vision sets. Is this a business we should be in? Is this a project that makes sense? Are these customer requirements appropriate to our core vision? Is this a customer problem that we solve or some other problem? If the vision has been market validated and passes muster, then decision making within the organization becomes much easier

Easier said than done. Every startup talks about their vision. But the real test of that vision is executing on it. Temptation abounds to throw the vision out the door and seek out tactical returns. I’ll touch on two: processing customer feedback and strategic partnerships.

Customer feedback. All startups live or die by how they treat their customer feedback. Customer feedback represents the pinnacle of market validation of your value proposition.Let’s say customer A comes to you with a proposal. Build feature X and we will consider giving you an order. If the feature fits into your value proposition, the answer is easy.  But what if the feature has nothing to do with your value proposition.  Should you still pursue this?  Now come back to our original thesis. You need a vision. Atleast with a vision you have a proactive basis to evaluate whether or not, you should pursue this opportunity. In the absence of a core stabilizing vision, you “get blown about and decision making becomes ad hoc at best”

 Strategic partnerships are the same.  Suppose Company Z comes to you and says you need to integrate in with our system and when you do so, great things will happen to you. Now if integrating with company Z’s product line is in your core strategic interest, then its a no brainer. What if its not? Again your corporate vision dictates whether or not you should engage in the opportunity.

My friend Dan Rosen has this great saying  ”Startups die more often out of indigestion than starvation”.  In the absence of a vision, a company will reach out and tackle every kind of opportunity. Soon enough, indigestion of too many opportunities will ensue. The trick is to eat sparingly and eat only what is good and relevant for you.

Remember when I said that we’ve released the srvsvcd daemon as well.  The srvsvcd daemon currently has a fully implemented client and the server side contains RPC server stubs. With the new lsmb architecture, the srvsvcd daemon runs as a separate daemon in and of itself. It now calls the lsmb client library (which is really our  ”system call” dispatcher) and sends  ”IOCTL” requests (sort of DeviceIo requests) down to the SMB server driver. The  IOCTLs are simply AddShare, DeleteShare, EnumShares, GetShareInfo and SetShareInfo. The SMB server driver now processes these “IOCTLs” and writes them to a persisted database (our share database). When the SMB server loads up, it reads the persisted database and maintains an internal list of its shares. Thus when a new share is created, it is stored in the internal list of shares before being persisted to store.

Our lsmb name is really a misnomer – the lsmb server is really much more than a smbserver – it’s actually a user-mode minimal kernel with four drivers and an io manager/executive – the rdr file system driver, the local file system driver, the named pipe file system driver and the smb server driver.

Danilo Almeida deserves singular credit for this.

Today, we integrated lsmb into trunk. Which means that Jerry’s automatic scripts should pick up lsmb, lwmsg and the new retrofitted dce/rpc and push it out to our public facing git repositories.

This is a fantastic achievement for the team. This release pushes out the beginnings of our smb infrastructure. lsmb ships with a named pipe client and smb client.  Also our smb design is beginning to mature nicely.  A few thoughts about it..  We broke the system into a client shared object library that communicates with the lsmbd server. the lsmb server has a top level router that can route to several file system drivers. We will ship three such file system drivers – a “rdr” – an smb client file system, a local (posix) file system and named pipe file system. Adjacent to this multi-provider file system architecture, we will provide an smb server “driver” which will pass “file system requests to the file system interface. The elegant thing about this design is that if customers want to run Samba on this machine, they can disable the lsmb server driver and lsmb  will just handle local system named pipe calls. That means our dce/rpc infrastructure can still work and all the rpc interfaces will continue to function as is. Customers can use Samba’s smb server to service file server requests. Thus lsmb can coexist with smb.

One way of looking at this is that everything in the lsmb server is in the “kernel” and the client side library is our user mode library. An ipc call represents a user-to-kernel transition.

Feel free to download the source as it gets into trunk.

Perhaps I’m being hugely naive, but now that we’ve gotten a named pipe client and by extension most of a file client done as well, I’ve been looking at what it takes to build a file server as well. It doesn’t seem to be that complicated. I’m going to explore this further and see what it would take. I’ve structured the transport interface to be able to support SMBv1 and SMBv2 and defined a NT Virtual File System (NTVFS) provider architecture. The  system has a two defined providers – a  Posix provider and a Named Pipe provider

At Likewise, we’ll need the Named pipe provider because it completes the server side Win32 API interface. The Posix provider is  just for completeness. Let’s see how things play out.  This may turn out to be a non-starter but its looking rather promising.

As you may deduce, I’m a huge fan of unmanaged remote procedure call  systems.  They serve as the building blocks for most distributed systems problems. They provide an easy programming model – synchronous function calls.  They have a small memory footprint (unlike managed systems like Java and C#). DCE/RPC and Sun RPC are the two widespread models (one is the foundational technology for Windows operating systems) and the other atleast is the basis for NFS infrastructure.

DCE/RPC on UNIX platforms has been in wilderness for almost a decade. Hopefully Likewise’s rehabilitated DCE infrastructure can bring about some form of a renaissance of an unmanaged, easy to use, remote procedure call infrastructure.  But I digress. I wanted to talk about lwmsg.

In the Windows world of systems programming, user mode clients and servers are built around a particular paradigm.  Let say we are building the foo client -server system.  Foo will have a client dll – the fooclient.dll that exposes DLL entry points for a calling client application to link to. The calling client program has no 

When we started on lwis, we wanted to build that Windows style programming pattern – we could have used DCE/RPC using ncalrpc, but at the time, I was worried about the chicken-and-egg effect.  We wanted API  style communication to the lsassd daemon, but one of the primary interfaces would be our gss-ntlm client and our gss-ntlm server. The DCE/RPC system would use NTLM as one of its authentication mechanisms, so wouldn’t that mean we would be using DCE/RPC  to bootstrap itself? So we decided that we would need an effective authenticated  remote procedure call mechanism that would handle communication between the lsass client and the lsass server for each of the lsass server interfaces – the UNIX id mapper interface and the GSS NTLM interface.

Initially, we hand-marshalled these IPC constructs, but as the system grew more sophisticated, the number of hand-marshalled functions grew to the point, that adding a new function got to be tedious. So Brian Koropoff, with his extensive background in compiler and language design, decided to build a data-driven (think idl-like) runtime engine for client-server  systems. This is lwmsg.

lwmsg allows you to specify  function definitions as request-response messages.  A remote procedure call function comprises of a  request message and a pair of response messages (on SUCCESS and FAILURE). What is so compelling about lwmsg is that you can specify a extensive class of C datatypes to marshall across the wire. Pretty much the entire DCE/RPC idl can be represented using lwmsg specifications. A calling client-server program specifies its “idl” as a set of global arrays and the light-weight run-time now handles the entire marshalling and unmarshalling for a calling program. 

LWIS is using lwmsg in three subsystems – the lsass subsystem, the netlogon subsystem and our newest lsmb subsystem. lwmsg builds  very easily, has no dependencies and we’ve just released it on trunk. Its licensed under the LGPL so feel free to get it a whirl.

I’ve begun prototyping the NTVfs layer for a proposed smb server stack. Its pretty straightforward really. Every NTvfs “plugin”/”driver” manages a file system type. Currently, I’ve designed two drivers – the first is a posix driver and the second is the named pipe file system driver. The architecture is simple – When a request comes in to open a file system, the equivalent of the TCON request, each NTVfs driver is queried. When a driver notes that it can handle a particular “share” path, it returns back a TREE_OBJECT handle. All file system request operations are now performed on this TREE_OBJECT handle.

Here is the nice thing about this model. One of the biggest sore points  when mapping NT semantics to UNIX semantics is ACL mapping. Today’s UNIX file systems support Posix ACLs, though more recent operating systems are supporting NT style ACLs.  The entire ACL management problem is delegated to the VFS driver. If the underlying file system has a decent ACL system, then the design of VFs driver will be relatively straightforward. If not, then it will be more complex.

My current interest is in the Named Pipe semantics. Since named pipes are ephemeral fifo message queues, the named pipe VFS actually implements NT security descriptors to control access to Named Pipe file object.

Its been just 3 months since Likewise 5.0 was announced and the uptake has been phenomenal. But we have several new technologies that will be out very shortly. The most important of these is our new SMB architecture.

As you know we rehabilitated DCE/RPC and made it Microsoft extension compatible and added a new portable threading model.  We had to add named pipe support and to do this, we created a client side named pipe architecture on top of  Samba’s libsmbclient. This worked quite well, but we figured we would actually want native named pipe support.

lsmb is our effort in that direction.  lsmb consists of a multithreaded daemon that has the following components.

- A clean native named pipe API that includes the Windows named pipe APIs layered on top of our native API infrastructure. The named pipe API supports both client and server side named pipes.

- A dce/rpc systems whose ncacn_np infrastructure is built on top of this named pipe system. Basically we will be releasing both client and server side DCE/RPC support over named pipes in the next few months

- Because the named pipe APIs are the same as the standard remote file system APIs, that means we will be providing programmatic support for anyone who wants to read and write from and to a Windows file server

In addition, we’ve just released on trunk our srvsvcd daemon and client API. This contains the Windows File share APIs running natively on Linux

Stay tuned – its going to be a very compelling New Year 2009 for the Likewise Open project!