From Sriram today…

________________________________________
From: Sriram Nambakam
Sent: Tuesday, February 17, 2009 7:09 PM
To: Gerald (Jerry) Carter; Krishna Ganugapati
Subject: dir

C:\Documents and Settings\snambakam>net use
New connections will be remembered.

There are no entries in the list.

C:\Documents and Settings\snambakam>net use * \\sn-ubuntu-t\C$
Drive Z: is now connected to \\sn-ubuntu-t\C$.

The command completed successfully.

C:\Documents and Settings\snambakam>dir z:
Volume in drive Z has no label.
Volume Serial Number is DEAD-BEEF

Directory of Z:\

02/17/2009 03:00 PM 15 test2.dat
02/17/2009 03:00 PM 19 test4.dat
02/17/2009 03:00 PM 15 test1.dat
02/17/2009 03:00 PM 15 test3
4 File(s) 64 bytes
0 Dir(s) 0 bytes free

C:\Documents and Settings\snambakam>z:

Z:\>dir
Volume in drive Z has no label.
Volume Serial Number is DEAD-BEEF

Directory of Z:\

02/17/2009 03:00 PM 15 test2.dat
02/17/2009 03:00 PM 19 test4.dat
02/17/2009 03:00 PM 15 test1.dat
02/17/2009 03:00 PM 15 test3
4 File(s) 64 bytes
0 Dir(s) 0 bytes free

Z:\>c:

C:\Documents and Settings\snambakam>net use /d z:
z: was deleted successfully.

C:\Documents and Settings\snambakam>

lwio watch: Week 6

Monday – Sriram and Jerry demonstrated copy file from a Linux server to a Windows client. We’ve officially signed off on Likewise 5.1 release. You should see it up on our website by end of the week.
Tuesday – Sriram and Jerry have most of dir up and running. It looks like that we will have full dir functionality in the next day or two
We’re planning on getting the srvsvcd service up and running by end of this week. This will allow administrators to create and delete file shares on a Linux server remotely using the “net share add” command and the “net share delete” functions. We’re two weeks away from code complete. Our plan is to spend the month of March working on stability, robustness and performance.

lwio: The Three Key Scenarios for April 2009

My wife read my blog the other day and her comment was “its pretty technical”. Polite speak for saying “Does anybody really care about what you’re writing?” Well, here’s is a short note on what we’re going to accomplish by middle of April.

We have three key end user scenarios that we’re shooting for. Recall that our objective is to make Linux,UNIX and Mac systems first class citizens in a Windows centric corporate intranet. Here are the scenarios.

The first scenario lets an system administrator log into his Windows XP desktop. He logs in using his Active Directory credentials. He then launches the Windows MMC console and loads up the “Local Users and Groups” snapin. when asked to select a machine to manage, he enters his Ubuntu Linux server machine that has also been joined to Active Directory using our Likewise authentication agent. The MMC “Local Users and Groups” snapin allows the administrator to create a “local user” on the Ubuntu server, set a password for this user.

The administrator can then launch putty on his Windows desktop and open up and ssh session to the Ubuntu server. When prompted for credentials, he can use the “local user” that he just created from his Windows MMC session.

The second scenario
For the second scenario, the administrator can reopen his MMC “Local Users and Groups” snapin, and now add “local user” to the “Administrators” pre-defined group on the Ubuntu server. He can now add a new snapin – the Share Management snapin to his existing MMC session. When prompted to enter the name of server where he would like to manage shares, the administrator enters the same Ubuntu server running the Likewise SMB server. When asked for administrator credentials, the administrator can enter the “local user” and the “local user’s” password. Bingo, MMC will now be able to add, delete, manage SMB shares on a Ubuntu server running lwio. The administrator now creates a share assigns a local directory on the Ubuntu box to be managed.

The third scenario
The third scenario starts right up where the second scenario leaves off. The administrator now fires up a command shell on his Windows XP client and enters the following commands.

net use Z: \\UbuntuServer\Shared Pictures /u:”local user” password
Z:
dir – lets him see a listing of all the files in the share
copy z:\*.* c: – lets him copy his pictures on the Ubuntu share to his Windows client
copy c:\NewPictures\*.* z: – lets him copy his new pictures on his Windows client to his Ubuntu share

Summing it all up, lwio is our next set of distributed systems infrastructure that takes Window interoperability with Linux,UNIX and Mac to never before seen heights. Oh, and did I mention that this functionality will be available on all 130+ flavors of UNIX, Linux, Mac. lwio is our programmable SMB, named pipes and DCE/RPC infrastructure that allows us to build a host of Windows interoperable technologies on non-Windows systems. lwio comes with a new integrated SMB server, an integrated SMB client, a named pipe interface and a DCE/RPC runtime retroffited with named pipe support.

Please write and let me know if this works as a decent elevator pitch of the lwio end user scenarios.

Thanks for reading!

LWIO Watch: Weeks 4 and 5 ending February 13, 2009

I’m sorry that I missed Week 4. It was a phenomenal week where we’ve made huge breakthroughs.

First, Sriram (our SMB server guy) and Jerry (our Posix virtual file system guy) got to make the copy file scenario work. What this means is that you can login to your Windows XP client machine, net use to your favorite lwio Linux file server connect to it and copy files from the server or to the server.

The named pipe file system is fully functional (I’m the owner of the named pipe file system driver ). We’re shipping a named pipe client and a named pipe server and you can fire them up and send a bunch of traffic to the server which then echoes the traffic back to you. We support a single pipe, single instance server, a single pipe multiple instance server and a multiple pipe, multiple instance server as well.

Based on the named pipe client and server code, Sriram took the named pipe client code and ported it to Windows. We can now run the named pipe client on a Windows box that talks to our named pipe server running on the Linux server through SMB traffic. Enough said!

But that wasn’t good enough for us! Brian Koropoff retrofitted DCE/RPC to support client and server side named pipes to work. The first scenario we got running was the venerable DCE/RPC echo program working both client and server running on Linux on the same box.

Next he changed the client piece to go over our SMB redirector file system. So we had a Linux client talking named pipes over our SMB client stack to our SMB server which passed it to our named pipe file system which in turn passed it back to our DCE/RPC server runtime. RPC over named pipes fully working!!

Still not impressed! Brian then ported the client RPC named pipes echo program on to Windows and yesterday we were able to successfully demonstrate an MSRPC named pipe client program talking to a DCE/RPC named pipe echo server running on our Linux server

We’re almost there! We started working on this SMB infrastructure on January 2nd – after the holidays. Its February 13th today and we’ve made some pretty great progress

By the way, we’ve finished branching our source trees last week. We now have 5.0 in archive mode, 5.1 which will be our current stable production release and our latest bleeding edge functionality is in trunk -5.2! Stay tuned lwio is now 5.2 and we’re going as fast as we can to get this puppy out before summer.

Thanks for reading!

lwio: Week 3 ending January 30th, 2009

1) Sriram is probably one of our most prolific developers. The SMB server is moving at light-speed. As of yesterday, we can actually do a “net use * \\LinuxServer\share” command on a Windows XP client and connect to a linux server running our server. Sriram has finished NTCREATE, READ, WRITE and CLOSE semantics and has wired them to the Posix virtual file system, which means we are very close to actually being able to copy files from/to Linux server to/from a Windows client. This is our February milestone that we want to reach. Sriram is now on a minor detour where he wants to actually be able to do a “dir” on a Windows client after the “net use * \\LinuxServer/share

2) The Named Pipe file system supports CreateNamedPipe, ConnectNamedPipe, ReadFile, WriteFile and CloseHandle. We tested it yesterday with a named pipe server and a named pipe client with a single pipe, single instance. Next week, we’ll test single pipe, multiple instances and multiple pipe, multiple instances. At this point, the code is stable enough for Brian to retrofit the DCE/RPC server side run-time with named pipes. One of the challenges we’ll face soon enough is that ConnectNamedPipe is a synchronous blocking call. I’m toying with the idea of implementing a sockets-style API for named pipes. It will work better with DCE/RPC and would be an interim step before we incorporate full asynchronous semantics both in the kernel and in the user mode client. Next week should be super interesting.

3) DCE/RPC framework – Brian Koropoff has retrofitted the dce/rpc server side stack to call the named pipe server side infrastructure. We haven’t yet begun testing, but next week should

4) SMB redirector – The redirector has full big-endian support checked in. As a result all of our Solaris, HP-UX PA-RISC and AIX platforms are fully operational.

6) Jerry has gotten a file copy using the local client API running to and from the pvfs driver. The file copies in both directions are identical. In his words “The security and varied NtCreateFile() options is really going to be the lion’s share of the work.”

Next week, we will be branching trees. The 3xxx series (our current QFE release) will be tagged LW5.0-qfe. The 5xxx series (our current trunk) will be tagged LW5.1-qfe and the lwio branch will become our new trunk!! This is huge because once a branch is pushed upstream as trunk, it is now officially tested on a daily basis. In addition, trunk builds will be available on our website. So you can start playing with the new iomgr in a week.

Thanks for reading!

LWIO: Week 2 ending January 23rd 2009

1) The smb server has had the most code go in. Sriram has finished NEGOTIATE, SESSION_SETUP and TCONs
2) We have manually populated the share database so that you can actually connect to shares configured in the database
3) The redirector has been fixed to support Mac OS X x86 support and by end of next week we should have full big endian suppport
4) The named pipe file system driver is almost code complete
5) Rafal has begun checking in the beginnings of the Lsa RPC server

LWIO Watch: Week 1

I’ve gotten a lot of requests from people asking that they be updated on the status of lwio. We’re working on weekly milestones and I’ll update you on our progress on a week by week basis.

This week – ending tomorrow Friday January 16th, 2009

1) The Nt APIs have been wired to the IO Manager. This means that you can do a NtCreateFile, NtReadFile, NtWriteFile, etc from a user mode client to the lwiod – we’re routing requests to the smb redirector and the posix subsystem.
2) We’ve just enabled the posix subsystem, the redirector and the server driver – the IoMgr will load these drivers in.
3) We’ve nuked the old iomgr which used the old smb client model.
4) The srvsvcd daemon is up and running and will provider server side support for NetShareEnum, NetShareAdd, NetShareDelete, NetShareGetInfo and NetShareSetInfo. The srvsvcd daemon is critical because it represents the end to end server-side RPC pipe making device i/o calls into the server driver.
5) Sriram has pretty much gotten SMB session setup working and will start adding support for tree connects, followed by opens, reads, writes and closes. These five calls are basically what we’ll need for the named pipe server to work.
6) Last and the best, we’ve wired DCE/RPC to call the actual NtCreateFile calls to the redirector for client side named pipes to work.

The plan will be to self-host the lsass subsystem against lwio by the end of the month. That is also our timeline to push this branch of code into trunk where it will start undergoing our rigorous battery of daily bvt, stress and performance tests.

Thanks for reading!

Update Jan 16, 2009: Brian just informed me that lwio is now self-hosting the lsassd daemon over the new smb redirector driver. That means our end-to-end iomgr is fully operational!

A New Year and a new mission – the Likewise I/O Manager lwio

Happy New Year everyone!

I’ve some really great news to report. We’re full steam ahead on our new architecture for an SMB framework and hopefully we’ll show case this work at Samba XP 2009.

If you’ve looked at our trunk source tree, you will see that we have the new lsmb server. lsmb hosts an smb client and is itself a service that other clients can connect to. The client interface for lsmb manifests itself as the quasi Win32 API. Think CreateFile, ReadFile, WriteFile and CloseHandle. Remember that these functions are used to access named pipes on a Windows server. We retrofitted DCE/RPC to talk to this client library which in turn talked to the lsmb server which hosted an smb client. We now have a client named pipe API and we can do RPC over named pipes, a key requirement if you want to do any kind of MSRPC protocol.

We’ve checked this into trunk and it’s close to being declared stable. I suspect by the end of the month, we’ll declare this stable.

Remember that the goal for lsmb is that we could support server side named pipes as well. This is a much trickier problem. The problem here is that you need an SMB server – at least a minimal SMB server and you need to support the notion of fifo queues. The “server” client also needs to use the named pipe API, but this case, the server uses a special API – CreateNamedPipe which returns a handle and then subsequently uses that handle to do ReadFile, WriteFile and CloseHandle operations. We’d then need to retrofit the server side interface into the DCE/RPC library so that we could function as an RPC server over named pipes.

Our first cut was to divide lsmb into two halves. The first half was the existing SMB client piece and the second half would contain the server piece. I thought that we would implement a server side virtual file system layer where there would be an SMB server piece which would fork requests off to the named pipe file system and to test this we would put together a posix virtual file system. Let’s face it – I’ve spent the last three years with a lot of Linux and Samba developers and this architecture is very familiar to them. Let’s say I was inspired by them. But this architecture has a problem. The named pipe file system now has two interfaces – the server end of it routes to the VFS layer, but remember we still need to indicate data back to the calling client – the named pipe server. So the upper edge Win32 style API has to route down to the named pipe VFS provider so that we can indicate data. Its a kludgy design – doesn’t work!

When you rethink this – you will see that we really have three “file system” drivers – the smb redirector file system (which was the client side silo in lsmb), the named pipe file system (which actually only indicates data to calling “server” clients), the posix virtual file system (which receives data for writes and indicates data back for reads) and an SMB server driver that serves as a client (both for reading and writing data to the 2 of the file systems – the posix virtual file system and the named pipe file system. Now if you look a little more carefully, we now have four different “drivers”. All of these drivers really should have a common interface and this is the beginning of the Likewise lwio daemon. ln future versions of the product, the lsmb daemon will morph into lwio daemon.

lwio consist of a client side piece that IPCs file system requests to the protected server. We needed an client side API and here our design was inspired. Recall that our goal is to implement a minimal SMB server which is necessary to do named pipe server io which is necessary to do server side DCE/RPC over named pipes. The SMB architecture really is the NT NtCreateFile, NtReadFile, NtWriteFile, NtClose, NtQueryInformation and a NtSetInformation extended to the network. Thus the interface that the I/O Mgr exposes is an analogue to these system calls. The client side exposes these APIs as interfaces and they are ipced over to the server using lwmsg. On the server side the client API is represented similar to what it would be like in the NT kernel. We have IoCreateFile, IoReadFile, IoWriteFile, IoClose etc. This is where the “io” in lwiod resides. The lwiod daemon is the “kernel” and the client to lwiod represents the “user mode” side of the lwiod kernel. We called it io because we really don’t have a kernel and the common runtime routines that the drivers use are all io routines.

The IoMgr is responsible for routing requests to shared object libraries that represent “drivers”. Every driver implements a DriverEntry function and returns back a set of dispatch functions (not a dispatch table). Drivers create device objects which allow the IoMgr to route requests to the a specific driver. Thus when a client calls NtCreateFile, the Create dispatch function in the driver is called. Prior to calling the Create dispatch, the IoMgr creates a file object (represents the file handle) and passes the file object to the Create function. The driver then does the appropriate “create” stuff – in the case of the posix file system, it will call the Posix open function store the handle in a context and attach the context to the file object and return back. You get the rest of it. Similarly when the smb redirector file system gets a Create it does the SMB NEGOTIATE, SMB SESSION_SETUP, SMB_TREE_CONNECT, SMB_NTCREATE and stores the corresponding tree object with the Tid and the file object with the Fid . Understanding the driver architecture is pretty straightforward when you’re mapping this to a file system.

The SMB server driver is a different beast. It doesn’t implement any of the Create, Read, Write, Close dispatch functions. It does implement the DeviceIo dispatch function. Its a pretty empty driver from the file system call perspective. What you can do is send the driver control codes to do stuff. Its the quissential Toaster driver. But when it starts up, it starts listening on port 445. And it processes SMB requests and then translates them into file system requests and hands them to you guessed it. the IoCreateFile, IoReadFile, IoWriteFile, IoClose file functions which the IoMgr then routes to the posix virtual file system or the named pipe file system. The server driver keeps track of the TIDs (a list of tree objects) and the FIDs (the file ids).

Hope you find this as exciting as I do. Stay tuned. I plan to describe end user scenarios as they unfold over the next few months.

Once again, happy new year!

Waiting for 2009

Well, its 5 more days until 2009. It has been a very very eventful 2008. I’m not one who dwells on the past a whole lot. I’m much more a look to the future kind of person. But I thinks this year warrants some look back for me.

From a technology perspective, it has been a really rewarding year. Just one year ago, Sriram Nambakam and I started working on a design for a new authentication engine. The thesis for me was that to build an authentication engine that “talked” to Windows authentication infrastructure, then it would be important to have the same design artifacts that were available in Windows. In the past one year, we’ve rehabilitated DCE/RPC, made it Windows compatible, built the LanMan APIs native on Linux, built an eventlog system, built a modern management console infrastructure, gssified NTLM, built full Samba and Apache interoperability and last but not least of all released a new SMB framework. Not bad work at all.

LWIS has definitely left the building!

Funny thing is that when I tell people about the breadth and scope of the work we’ve pulled off, the tendency is to not believe me

All our developers deserve a huge pat on the back. One of the toughest challenges in the technology business is forging a team of great people. The Likewise development team is probably the best I’ve ever worked with. I’m obviously biased because I’ve recruited all of them

Likewise Open continues to grow in leaps and bounds. The number of downloads is staggering. Sometimes, I catch myself thinking we just might have done our job a little too well. As I said, I catch myself .

On the personal front, the year started out really great but has ended rather flat. It is perhaps partly to the fact that we’ve done something hugely challenging and accomplished it. The reward for me always seems to be in the challenge – I live for the climb. Its is the anticipation of close success that is addictive to me. Once we’ve taken the peak, there is a sense of emptiness.

On the home front, life could not have gone better. We welcomed our daughter Maya and she is a beauty!

Well here’s looking for new challenges in 2009!

krb 5 1.7 to be code complete January 2009 and ship April 2009

Luke Howard just forwarded me a link to the kerberos consortium website. The roadmap says that krb 5 1.7 is in code freeze by the end of January 2009 and will ship April 2009. That’s a big deal for us – all of the necessary kerberos changes we need for dce/rpc and gss-ntlm will be upstream which means we can get out of maintaining our own patches in krb 5 1.6. Which would be a relief to me.