Five years later .. time to write again

Its been five years since I blogged. So much to write about. Last week VMware open sourced two projects that I incubated. The first is Project Photon. Photon is VMware’s lightweight container host runtime Linux distribution. The second is Project Lightwave. Lightwave is an cloud scale identity platform.  It comprises a massively scalable, multi-tenanted, multi-master directory service, a Kerberos KDC, a certificate services platform and a cloud authentication stack with support for WSTrust, SAML and OAuth2.0/OpenID Connect.

In the past five years, VMware has a pretty decent size presence in the Seattle area. From 1 to 6 to over 150 people in downtown Bellevue, the VMware Bellevue R&D Center has grown in leaps and bounds.

Its been awesome and we’re only just getting started


Moving on..

As many of you already know, I’ve left Likewise last week (April 15th). It has been an exhilarating four and a half years  doing some pretty great things with some really terrific people.  We built  what  I am convinced will be the definitive SMB/CIFS stack for the non-Windows world in a record  15 months . What an achievement! I will miss working with my Likewise team and I wish them the best of luck and good fortune as they move forward.

So what’s next for an old hand at distributed systems? Hopefully, reach for the “clouds” … I’m usually pretty cautious/skeptical when it comes to analyzing new trends, but this one seems  pretty obvious. To say that the cloud promises to be disruptive would be banal – it is the new “New World” for distributed computing. I think success will theirs who discover the most optimal route to this New World.  And I want to  be part of that discovery process.

So I’m going to take a few weeks off,  but get busy figuring out how best I can get involved in the new cloud world.

To all of you who read my blog, thank you very much. Hopefully, I will have interesting things to talk about in the future.

Coming soon: the Likewise Open Platform SDK

One of things I’ve been lax on is creating a Platform SDK for the Likewise Open Project. I’m happy to inform you that we are in the midst of a build restructure that should wrap up by the end of the week. The result of this restructure will be that you can now install devel-packages for Likewise Open.  All headers and libraries will be installed in the /opt/likewise/[include|lib].

Following which you can write your own DCE/RPC client and server programs. You can write LWIO drivers for the LWIO platform – you can create an NFS server or a WebDAV server or an FTP server. You can write your own backend to leverage the LWIO CIFS stack – say you want to write an LWIO driver for the  ZFS file system. All you would need to do is install the Likewise Open Platform SDK – install the devel packages and start coding.

Another great advantage is we’ve significantly reduce build times of the Likewise Open Platform.  Jerry reports with our first wave of componentization, the entire Platform builds in 22 minutes on a quad-core  4GB machine. It used to take around 30 minutes. We think that we can bring this down another 25% – so stay tuned.

My own particular  requirement was that OEMs should be able to pull down the Likewise Open tree and easily get up and running. I think we’re getting there.

This effort should wrap by the end of this week.

Rethinking the domain join utility.

One of the more thorny problems in building Linux-Windows interoperability technology has been designing the domain join process. The problem has more to do with how should you marry two different operating systems – the Linux world and the Windows AD world?  The first problem was always how do you get the machine name from the Linux machine to pass to the Active Directory domain controller.

In earlier versions,  we would grab this information from the Linux/UNIX/Mac machine and pass it to our authentication engine. The problem is that getting this information is very painful when you need to do this across disparate UNIX operating systems.

As the Likewise Open architecture has evolved it is now a full-fledged development and programming environment to the point where an end user ought to be able to specify a machine name directly to the authentication engine which then stores that information in the Likewise Registry. Thus the  authentication engine’s domain join process is rightly decoupled entirely from the  Linux/UNIX/Mac  hostname files and other  configuration files.

My development station could be, but I could join my machine to the Active Directory domain as  In the real world,  I would join my machine  as, but the decoupling of the UNIX system name from the  Active Directory FQDN and NETBIOS Name, provides a useful architectural separation/”join” point between the two worlds.

It is something we’ll explore further as the system evolves.

I guess this is a long roundabout way arguing for modular and simple architecture in our system. As the amount of code in the Likewise Open project grows, we will need to be mindful that the system is easily maintainable and manageable. I’m going to continue this theme on another post…

And the magic 10K!

  ’nuff said!

—–Original Message—–
From: Gerald (Jerry) Carter
Sent: Wednesday, March 03, 2010 6:03 AM
To: lwio-core
Subject: Re: Morning connection testing 🙂

And the magic 10k….

23904 39.1 00:12:19 1557200 916072 /opt/likewise/sbin/lwiod –syslog

23946  8.1 00:02:34 591192  6252 /opt/likewise/sbin/lsassd –syslog

23849 31.3 00:09:53 395068  2528 /opt/likewise/sbin/lwregd

24009  0.0 00:00:00 316600  2072 /opt/likewise/sbin/srvsvcd –syslog 23920  6.8 00:02:09 244968  2780 /opt/likewise/sbin/netlogond –syslog

23892  0.0 00:00:00 210440  1696 /opt/likewise/sbin/dcerpcd -f

23847  0.0 00:00:00 178104   752 /opt/likewise/sbin/lwsmd –start-as-daemon

Server statistics [level 0]:

Number of connections:           [10002]

Maximum Number of connections:   [10002]

Number of sessions:              [10002]

Maximum Number of sessions:      [10002]

Number of tree connects:         [10004]

Maximum Number of tree connects: [10004]

Number of open files:            [9956]

Maximum Number of open files:    [10005]

Gerald Carter wrote:

> Fyi…


> If we could figure out the memory usage in lwio, this would be so

> beautiful.


> $ (ps -eo pid,%cpu,cputime,vsz,rss,args | grep likewise.*sbin |\

>   grep -v grep | sort -r -k4 -n) && /opt/likewise/bin/lwio-cli

> –get-stats



> 23904 39.6 00:09:10 1491668 766024 /opt/likewise/sbin/lwiod –syslog

> 23946  9.2 00:02:07 591192  6648   /opt/likewise/sbin/lsassd –syslog

> 23849 35.5 00:08:12 395068  2772   /opt/likewise/sbin/lwregd

> 24009  0.0 00:00:00 316600  2212   /opt/likewise/sbin/srvsvcd –syslog

> 23920  7.7 00:01:47 244968  2756   /opt/likewise/sbin/netlogond –syslog

> 23892  0.0 00:00:00 210440  1708   /opt/likewise/sbin/dcerpcd -f

> 23847  0.0 00:00:00 178104   860   /opt/likewise/sbin/lwsmd –start-as-daemon


> Server statistics [level 0]:


> Number of connections:           [6656]

> Maximum Number of connections:   [10001]

> Number of sessions:              [6650]

> Maximum Number of sessions:      [10001]

> Number of tree connects:         [6650]

> Maximum Number of tree connects: [10003]

> Number of open files:            [6696]

> Maximum Number of open files:    [10000]




Senior Software Developer                               Likewise-CIFS

“What man is a man who does not make the world better?”      –Balian

Coming soon – net utilities and smbshell

Wei is back from China after a well deserved vacation. She (as always) is rareing to get started on a new project, so I’ve asked her to take point on our net utilities.  Stay tuned for the well-known familiar net utility on Linux. We’re not going to create a whole bunch of strange commands, we will make available the well known one’s that Windows users are familiar with – net use, net view, net localgroup, net user, net share, net time.

Adam Bernstein is going to work on smbshell. This utility will be similar to his super popular regshell utility that lets you browse the Likewise Registry. The best parts of regshell were the tab completion support. smbshell will be similar – it will allow you to interactively walk an smbshare and copy files. It is an absolute nightmare to copy files from the commandline in a Linux world and we hope to make this drop dead simple.

We’ll push these out in the 5.4 release which by the way was pushed out to the public git tree – we will need to release binaries quickly.

Thanks for reading

Blasting past 8000 concurrent SMB connections!

This morning we blasted past 8K concurrent connections to our SMB stack. We were originally trying to get FSCT running and we’re pretty close – we’ve run into client side configuration issues and controller side issues and we’re working with Microsoft to get these resolved. I’m probably going to head up there in the next couple of weeks and figure out how we can collaborate on making sure that fsct is tested against non-Windows SMB servers

In the interim, Brian fixed up our redirector to force it to open multiple connections out to the Likewise CIFS stack.  I’m sketchy on details – the idea is to force a single connection on a per user. So 10K users will translate to 10K connections on the server.

Once we did this, Jerry ran the test this morning and we blitzed past 8K SMB connections.  We hit this limit because server was hard-wired to do a max of 8K connections. Once we turned this off, we’ve seen this go way past 8K. Our belief is that we will scale to the resource limits on the server.

Note that the test shows that the stack can create the necessary data structures for 8K SMB sessions (or greater).  We still need to test i/o throughput numbers.  We want to ensure that when you have 10K or greater SMB sessions, given that the server will statisically multiplex (some connections will be doing active I/Os while other connections are idle), the active sessions can deliver reasonable throughput guarantees.

My own system service design schooling was always thread-based systems. And at Microsoft, we always wrote thread based  single process systems.  So it was surprising to hear from several people in the open source world that threads are bad and evil and you can’t get decent performance guarantees using threads.  lwio is our single process multi-threaded architecture and so far we seem to be doing okay. Perhaps we’ll get tripped up somewhere, but right now  its looking pretty good.