Week 23 – sorry I haven’t posted in 12 weeks

My last week-by-week post was 12 weeks ago! Here is a summary of what we’ve accomplished

The lwio manager – The lwio kernel is fully asynchronous. IRP pending works like a charm.

The Posix Virtual File System -  the pvfs system has a complete lock manager implemented. As as result, we have full byte-range locking semantics. We actually pass the smb torture tests for byte range locks. For oplocks and BRLs, the  PVFS driver cleanly handle asynchronous semantics.

The lwmsg system  – our internal ipc mechanism support asynchronous semantics as well. Thus there are no blocking threads for calls on the server. Even when the client process makes synchronous calls, we translate them to async calls on the lwio kernel.

The smb server has been fully refactored so that the transports, and the smb1 and smb2 protocols are independently developed. See the last post  – we have almost completely finished the smb2 wire protocol engine.  Remember that our protocol engines are independent of the underlying file system, so when I say that the wire protocol engine is complete that means you have a complete smb2 file server.  We just wire the protocol engine to the underlying file system which was already in place for smb1.

lsass has gone through a significant upgrade. We have closed several large OEM deals so for each of them we were doing bits of fit and finish work.  The major improvements are the fully operational samdb backend as the local authentication provider, significant reduction in the number of shared object libraries and a much more simpler source code layout.

Coming soon… We will shortly be releasing our registry subsystem – all lsassd, eventlogd, lwiod infrastructure will store all its configuration information in the registry. At install time, we will provide utilities that export the text base configuration information for a subsystem into the registry. We will be providing, registry import-export tools, a registry shell and a graphical LAC plugin to graphically view, edit and manage the registry.  This is really important because it lets us manage upgrades smoothly, allow us to introduce new configuration parameters without having to write tons of parser and lexical analysis code.

Releases – We’re going to be releasing 5.2 very very soon .. so stay tuned.

Thanks for reading.

A month after Samba XP; onward to SNIA

I’ve gotten quite a few queries as to why I haven’t posted for the past six weeks. Sorry! Been awfully busy.
Well its been a month since Samba XP. But we continue to make huge strides in new features and functionality.

We gave a formal preview of the Likewise LWIO infrastructure at Samba XP. The numbers were: 4 demos, 5 developers, 60 slides and 60 minutes. We probably broke all of the standard presentation rules. But we made quite an impact.

We demonstrated the full lwio kernel architecture

We demonstrated the SMB redirector integrated with a FUSE client

We demonstrated the entire SMB server. Jerry Carter did a fantastic demo.

I’ll add more notes to this as I go.

lwio watch: Weeks 9, 10, 11 ending April 3rd , 2009

It has been a hectic three weeks and I haven’t been able to get to write a whole lot. I have been on the road quite a bit meeting with customers and I’d forgotten how busy things get on the road.

Anyway we’ve made huge strides ..

First, we’re dogfooding the SMB server on all our internal servers. We now have full security descriptor support in the Posix file system driver. Our first attempt was to write clean mapping functions from NT Security descriptors to Posix ACLs and vice versa, but we’ve settled to store native NT security descriptors as Posix file EAs. This allows us to quickly get the Windows MMC pieces to allow us to edit security descriptors.

Second, we now have a uniform access token model across the entire lwio “kernel”. What this means is that similiar to Windows ConvertAuthDataToToken function, we can pass in authorization data from an underlying security mechanism (in the case of the SMB server at session setup time) and create a generic lwio token that now holds the authentication state for the connection. This is now one of the parameters passed to IoCreateFile to the posix driver when creating/opening a handle to a(n) new/existing file managed by the posix driver. This access token model is a generic model common across all Likewise protected subsystems.

Brian Koropoff has finished asynchronous semantics in the lwmsg layer. What this means is that from “user space” to “kernel space”, we can support system calls with overlapped I/O semantics. Thus an NtCreateFile or a NtConnectNamedPipe call can immediately return back to the calling process, and when the “kernel” has data to indicate we can signal an event in the “user-mode” calling process. This was necessary because without this asynchronous behavior, each named pipe RPC server would be plumbing a ConnectNamedPipe call which would be consuming one blocking thread. This way, all named pipe RPC servers will be able to support asynchronous semantics. Huge piece of work.

The biggest piece of work left and we’ve made great progress on is the refactoring of our local authentication database in preparation of our lsarpc, netlogon, samr RPC server. I’m going to write a separate post on that. This is our SAM for Linux strategy.

The posix virtual file system has support for alternate data streams. I should clarify: the smb server will pass requests to the posix virtual file system driver which can implement them in a variety of ways: we can create a hidden directory where the additional data streams are stored as separate files. Alternatively, if the underlying file system has native support for alternate data streams, the posix virtual file system driver can support that.

Two weeks from now is Samba XP: I’m looking forward to this trip to Gottingen. It has become a yearly event and a greate time to meet with people.

Thanks for reading!

http://www.HowSoftwareisBuilt.com

Short post: I was recently interviewed by http://www.HowSoftwareisBuilt.com.

lwio watch: Week 8 ending March 13, 2009

This one is another short post. Here are our highlights

1) Share management is fully operational: This means you can launch MMC from your Windows client; load up the share management snapin and point to Linux box running the lwio SMB file server and srvsvc daemon. The result is that you can browse shares from a Windows client; you can create a new share on your Linux server as well and do all of the file share operations. We have yet to complete the “open files” and “open sessions” functionality. But we’re so pleased with seamless management that we’ve punted on this for now.
2) Our Likewise FuSE driver is fully operational: This means that a Linux client joined to Active Directory can easily mount SMB shares on a Windows/Samba/Likewise SMB file server. Here is the cool part: you login on your Linux machine using your Active Directory credentials and single sign-on mount your SMB file share. If your machine is multi-user and simultaneously your buddy logs on to the same machine and mounts his SMB file share using our FuSE driver; he connects seamlessly with his credentials to the SMB server. This is a really elegant piece of work. Brian tells me that OpenOffice running on a Linux client can save files to a Likewise/Samba/Windows SMB File Server through our SMB redirector. As you can notice, I am getting a kick out of saying Likewise/Samba/Windows SMB File Server. The cool thing here is not the file server, it is our FuSe driver that calls our SMB redirector (file client) within the LWIO Framework.

3) A ton of other coolness; I thought I’d just copy Jerry’s mail to me

“What works:

* User profiles load from the network
* Home directories mount properly at login time.
* My Documents redirection to %HOMESHARE%\%HOMEPATH%\Documents
works fine.

Also ran some tests with iTunes and was able to successfully import and existing library XML file and 4k mp3s from a network share to my current iTunes library. So now all my music is being served (and playing 24×7) from our server. Also editing mp3 song properties (artist, genre, etc…) works properly.

What is not working:

* Take ownership of files fails
* Roaming profiles are not saving

I believe that both of these are due to the lack of properly support partial security descriptor set operations (based on the SECURITY_INFORMATION flags) which I will implement today.”

I’ll post some screenshots once I get some cycles…

Thanks for reading!

lwio watch: Week 7 ending March 6th, 2009

This is a really short post, but I hope people get a kick out of it. Here is the current set of applications that the SMB server supports. This means an Windows XP client can run the applications below, read files from and write files to a file share on Linux machine running the Likewise SMB Server!

1) Word
2) Excel
3) Powerpoint
4) Explorer
5) IE
6) cmd shell
7) Tortoise SVN for checkouts
Visio
9) Project
10) Wireshark
11) Notepad
12) Outlook
13) Firefox
14) Safari
15) iTunes
16) Adobe Reader
17) Publisher
18) OpenOffice.org
19) QuickTime
20) Software Extraction and Installation/.MSI files all work
21) Image Previews and Thumbnails

Thanks for reading!

From Sriram today…

________________________________________
From: Sriram Nambakam
Sent: Tuesday, February 17, 2009 7:09 PM
To: Gerald (Jerry) Carter; Krishna Ganugapati
Subject: dir

C:\Documents and Settings\snambakam>net use
New connections will be remembered.

There are no entries in the list.

C:\Documents and Settings\snambakam>net use * \\sn-ubuntu-t\C$
Drive Z: is now connected to \\sn-ubuntu-t\C$.

The command completed successfully.

C:\Documents and Settings\snambakam>dir z:
Volume in drive Z has no label.
Volume Serial Number is DEAD-BEEF

Directory of Z:\

02/17/2009 03:00 PM 15 test2.dat
02/17/2009 03:00 PM 19 test4.dat
02/17/2009 03:00 PM 15 test1.dat
02/17/2009 03:00 PM 15 test3
4 File(s) 64 bytes
0 Dir(s) 0 bytes free

C:\Documents and Settings\snambakam>z:

Z:\>dir
Volume in drive Z has no label.
Volume Serial Number is DEAD-BEEF

Directory of Z:\

02/17/2009 03:00 PM 15 test2.dat
02/17/2009 03:00 PM 19 test4.dat
02/17/2009 03:00 PM 15 test1.dat
02/17/2009 03:00 PM 15 test3
4 File(s) 64 bytes
0 Dir(s) 0 bytes free

Z:\>c:

C:\Documents and Settings\snambakam>net use /d z:
z: was deleted successfully.

C:\Documents and Settings\snambakam>

lwio watch: Week 6

Monday – Sriram and Jerry demonstrated copy file from a Linux server to a Windows client. We’ve officially signed off on Likewise 5.1 release. You should see it up on our website by end of the week.
Tuesday – Sriram and Jerry have most of dir up and running. It looks like that we will have full dir functionality in the next day or two
We’re planning on getting the srvsvcd service up and running by end of this week. This will allow administrators to create and delete file shares on a Linux server remotely using the “net share add” command and the “net share delete” functions. We’re two weeks away from code complete. Our plan is to spend the month of March working on stability, robustness and performance.

lwio: The Three Key Scenarios for April 2009

My wife read my blog the other day and her comment was “its pretty technical”. Polite speak for saying “Does anybody really care about what you’re writing?” Well, here’s is a short note on what we’re going to accomplish by middle of April.

We have three key end user scenarios that we’re shooting for. Recall that our objective is to make Linux,UNIX and Mac systems first class citizens in a Windows centric corporate intranet. Here are the scenarios.

The first scenario lets an system administrator log into his Windows XP desktop. He logs in using his Active Directory credentials. He then launches the Windows MMC console and loads up the “Local Users and Groups” snapin. when asked to select a machine to manage, he enters his Ubuntu Linux server machine that has also been joined to Active Directory using our Likewise authentication agent. The MMC “Local Users and Groups” snapin allows the administrator to create a “local user” on the Ubuntu server, set a password for this user.

The administrator can then launch putty on his Windows desktop and open up and ssh session to the Ubuntu server. When prompted for credentials, he can use the “local user” that he just created from his Windows MMC session.

The second scenario
For the second scenario, the administrator can reopen his MMC “Local Users and Groups” snapin, and now add “local user” to the “Administrators” pre-defined group on the Ubuntu server. He can now add a new snapin – the Share Management snapin to his existing MMC session. When prompted to enter the name of server where he would like to manage shares, the administrator enters the same Ubuntu server running the Likewise SMB server. When asked for administrator credentials, the administrator can enter the “local user” and the “local user’s” password. Bingo, MMC will now be able to add, delete, manage SMB shares on a Ubuntu server running lwio. The administrator now creates a share assigns a local directory on the Ubuntu box to be managed.

The third scenario
The third scenario starts right up where the second scenario leaves off. The administrator now fires up a command shell on his Windows XP client and enters the following commands.

net use Z: \\UbuntuServer\Shared Pictures /u:”local user” password
Z:
dir – lets him see a listing of all the files in the share
copy z:\*.* c: – lets him copy his pictures on the Ubuntu share to his Windows client
copy c:\NewPictures\*.* z: – lets him copy his new pictures on his Windows client to his Ubuntu share

Summing it all up, lwio is our next set of distributed systems infrastructure that takes Window interoperability with Linux,UNIX and Mac to never before seen heights. Oh, and did I mention that this functionality will be available on all 130+ flavors of UNIX, Linux, Mac. lwio is our programmable SMB, named pipes and DCE/RPC infrastructure that allows us to build a host of Windows interoperable technologies on non-Windows systems. lwio comes with a new integrated SMB server, an integrated SMB client, a named pipe interface and a DCE/RPC runtime retroffited with named pipe support.

Please write and let me know if this works as a decent elevator pitch of the lwio end user scenarios.

Thanks for reading!

LWIO Watch: Weeks 4 and 5 ending February 13, 2009

I’m sorry that I missed Week 4. It was a phenomenal week where we’ve made huge breakthroughs.

First, Sriram (our SMB server guy) and Jerry (our Posix virtual file system guy) got to make the copy file scenario work. What this means is that you can login to your Windows XP client machine, net use to your favorite lwio Linux file server connect to it and copy files from the server or to the server.

The named pipe file system is fully functional (I’m the owner of the named pipe file system driver ). We’re shipping a named pipe client and a named pipe server and you can fire them up and send a bunch of traffic to the server which then echoes the traffic back to you. We support a single pipe, single instance server, a single pipe multiple instance server and a multiple pipe, multiple instance server as well.

Based on the named pipe client and server code, Sriram took the named pipe client code and ported it to Windows. We can now run the named pipe client on a Windows box that talks to our named pipe server running on the Linux server through SMB traffic. Enough said!

But that wasn’t good enough for us! Brian Koropoff retrofitted DCE/RPC to support client and server side named pipes to work. The first scenario we got running was the venerable DCE/RPC echo program working both client and server running on Linux on the same box.

Next he changed the client piece to go over our SMB redirector file system. So we had a Linux client talking named pipes over our SMB client stack to our SMB server which passed it to our named pipe file system which in turn passed it back to our DCE/RPC server runtime. RPC over named pipes fully working!!

Still not impressed! Brian then ported the client RPC named pipes echo program on to Windows and yesterday we were able to successfully demonstrate an MSRPC named pipe client program talking to a DCE/RPC named pipe echo server running on our Linux server

We’re almost there! We started working on this SMB infrastructure on January 2nd – after the holidays. Its February 13th today and we’ve made some pretty great progress

By the way, we’ve finished branching our source trees last week. We now have 5.0 in archive mode, 5.1 which will be our current stable production release and our latest bleeding edge functionality is in trunk -5.2! Stay tuned lwio is now 5.2 and we’re going as fast as we can to get this puppy out before summer.

Thanks for reading!