So what’s next for an old hand at distributed systems? Hopefully, reach for the “clouds” … I’m usually pretty cautious/skeptical when it comes to analyzing new trends, but this one seems  pretty obvious. To say that the cloud promises to be disruptive would be banal - it is the new “New World” for distributed computing. I think success will theirs who discover the most optimal route to this New World.  And I want to  be part of that discovery process.

So I’m going to take a few weeks off,  but get busy figuring out how best I can get involved in the new cloud world.

To all of you who read my blog, thank you very much. Hopefully, I will have interesting things to talk about in the future.

Following which you can write your own DCE/RPC client and server programs. You can write LWIO drivers for the LWIO platform – you can create an NFS server or a WebDAV server or an FTP server. You can write your own backend to leverage the LWIO CIFS stack – say you want to write an LWIO driver for the  ZFS file system. All you would need to do is install the Likewise Open Platform SDK – install the devel packages and start coding.

Another great advantage is we’ve significantly reduce build times of the Likewise Open Platform.  Jerry reports with our first wave of componentization, the entire Platform builds in 22 minutes on a quad-core  4GB machine. It used to take around 30 minutes. We think that we can bring this down another 25% – so stay tuned.

My own particular  requirement was that OEMs should be able to pull down the Likewise Open tree and easily get up and running. I think we’re getting there.

This effort should wrap by the end of this week.

In earlier versions,  we would grab this information from the Linux/UNIX/Mac machine and pass it to our authentication engine. The problem is that getting this information is very painful when you need to do this across disparate UNIX operating systems.

As the Likewise Open architecture has evolved it is now a full-fledged development and programming environment to the point where an end user ought to be able to specify a machine name directly to the authentication engine which then stores that information in the Likewise Registry. Thus the  authentication engine’s domain join process is rightly decoupled entirely from the  Linux/UNIX/Mac  hostname files and other  configuration files.

My development station could be krishnag-ubuntu.likewise.com, but I could join my machine to the Active Directory domain as foo.likewise.com.  In the real world,  I would join my machine  as krishnag-ubuntu.likewise.com, but the decoupling of the UNIX system name from the  Active Directory FQDN and NETBIOS Name, provides a useful architectural separation/”join” point between the two worlds.

It is something we’ll explore further as the system evolves.

I guess this is a long roundabout way arguing for modular and simple architecture in our system. As the amount of code in the Likewise Open project grows, we will need to be mindful that the system is easily maintainable and manageable. I’m going to continue this theme on another post…

—–Original Message—–
From: Gerald (Jerry) Carter
Sent: Wednesday, March 03, 2010 6:03 AM
To: lwio-core
Subject: Re: Morning connection testing

And the magic 10k….

23904 39.1 00:12:19 1557200 916072 /opt/likewise/sbin/lwiod –syslog

23946  8.1 00:02:34 591192  6252 /opt/likewise/sbin/lsassd –syslog

23849 31.3 00:09:53 395068  2528 /opt/likewise/sbin/lwregd

24009  0.0 00:00:00 316600  2072 /opt/likewise/sbin/srvsvcd –syslog 23920  6.8 00:02:09 244968  2780 /opt/likewise/sbin/netlogond –syslog

23892  0.0 00:00:00 210440  1696 /opt/likewise/sbin/dcerpcd -f

23847  0.0 00:00:00 178104   752 /opt/likewise/sbin/lwsmd –start-as-daemon

Server statistics [level 0]:

Number of connections:           [10002]

Maximum Number of connections:   [10002]

Number of sessions:              [10002]

Maximum Number of sessions:      [10002]

Number of tree connects:         [10004]

Maximum Number of tree connects: [10004]

Number of open files:            [9956]

Maximum Number of open files:    [10005]

Gerald Carter wrote:

> Fyi…

>

> If we could figure out the memory usage in lwio, this would be so

> beautiful.

>

> $ (ps -eo pid,%cpu,cputime,vsz,rss,args | grep likewise.*sbin |\

>   grep -v grep | sort -r -k4 -n) && /opt/likewise/bin/lwio-cli

> –get-stats

>

>   PID %CPU     TIME    VSZ   RSS   COMMAND

> 23904 39.6 00:09:10 1491668 766024 /opt/likewise/sbin/lwiod –syslog

> 23946  9.2 00:02:07 591192  6648   /opt/likewise/sbin/lsassd –syslog

> 23849 35.5 00:08:12 395068  2772   /opt/likewise/sbin/lwregd

> 24009  0.0 00:00:00 316600  2212   /opt/likewise/sbin/srvsvcd –syslog

> 23920  7.7 00:01:47 244968  2756   /opt/likewise/sbin/netlogond –syslog

> 23892  0.0 00:00:00 210440  1708   /opt/likewise/sbin/dcerpcd -f

> 23847  0.0 00:00:00 178104   860   /opt/likewise/sbin/lwsmd –start-as-daemon

>

> Server statistics [level 0]:

>

> Number of connections:           [6656]

> Maximum Number of connections:   [10001]

> Number of sessions:              [6650]

> Maximum Number of sessions:      [10001]

> Number of tree connects:         [6650]

> Maximum Number of tree connects: [10003]

> Number of open files:            [6696]

> Maximum Number of open files:    [10000]

>

>

–

=====================================================================

Senior Software Developer                               Likewise-CIFS

http://www.likewiseopen.org/

“What man is a man who does not make the world better?”      –Balian

Adam Bernstein is going to work on smbshell. This utility will be similar to his super popular regshell utility that lets you browse the Likewise Registry. The best parts of regshell were the tab completion support. smbshell will be similar – it will allow you to interactively walk an smbshare and copy files. It is an absolute nightmare to copy files from the commandline in a Linux world and we hope to make this drop dead simple.

We’ll push these out in the 5.4 release which by the way was pushed out to the public git tree – we will need to release binaries quickly.

Thanks for reading

In the interim, Brian fixed up our redirector to force it to open multiple connections out to the Likewise CIFS stack.  I’m sketchy on details – the idea is to force a single connection on a per user. So 10K users will translate to 10K connections on the server.

Once we did this, Jerry ran the test this morning and we blitzed past 8K SMB connections.  We hit this limit because server was hard-wired to do a max of 8K connections. Once we turned this off, we’ve seen this go way past 8K. Our belief is that we will scale to the resource limits on the server.

Note that the test shows that the stack can create the necessary data structures for 8K SMB sessions (or greater).  We still need to test i/o throughput numbers.  We want to ensure that when you have 10K or greater SMB sessions, given that the server will statisically multiplex (some connections will be doing active I/Os while other connections are idle), the active sessions can deliver reasonable throughput guarantees.

My own system service design schooling was always thread-based systems. And at Microsoft, we always wrote thread based  single process systems.  So it was surprising to hear from several people in the open source world that threads are bad and evil and you can’t get decent performance guarantees using threads.  lwio is our single process multi-threaded architecture and so far we seem to be doing okay. Perhaps we’ll get tripped up somewhere, but right now  its looking pretty good.

All 25 of our tier 1 applications pass. See my earlier post on the list of applications that we test against. All these mainstream applications pass.

Sriram figured out how to compute file ids – it turns out that they are good old fashioned GUIDs – a portion of the GUID is the persistent part of the file id and the other portion is the ephemeral part.

We still need to implement SMB2.1 semantics especially the Windows 7 oplock package, but that is a Windows 7 artifact..

So for now, we’re going to enjoy the view from up here.

BTW, feel free to build the lwio stack and enable smb2 support. If you find errors or bugs send them our way Our model is to continually focus on making things better, so all feedback is super appreciated.

The first reason is the cross protocol issues. Traditional NFS servers are  in the kernel, but CIFS (Likewise or Samba) has been in user space.  The challenge is  now reconciling access to the same file via separate protocols. With one server in the kernel and the other in user space, the left hand has no idea what the right hand does and vice versa.

The second is security controls.  CIFS is intimately tied with Kerberos authentication and I’m pleased to see (its been a while since I’ve looked at NFS) is Kerberized NFS as well.  I see a simplified multi-protocol stack where the authentication is Kerberos and the access control lists are NT security descriptors (or something like them).  Servers should be associated with unified authentication stores – so when you’re talking many many users, I expect to see a domain controller.

Finally, manageability. I like simple management semantics – I’m always forgetting what the NFS mount options are and frankly, I’d like to set up NFS exports the same way as I would an SMB share. I’d like to see an MMC snapin which can RPC to the server and set up the export.  So we’d actually use DCE/RPC and provide a new set of APIs – think NetExportAdd, NetExportDelete and NetExportGetInfo and NetExportSetInfo to create, delete and manage NFS exports – neat yes? I’d like a simple command-line interface that allows me to create an export something like -

%net export add file-export /usr/share/export ; creates the file-export

or

%net export delete file-export  ; removes the file-export export

I think we’ll have the NFS management daemon run over tcp. So it will permit an administrator to set up a NFS only file server and use dce/rpc for the management API interface.

The more exciting opportunity is that Likewise will be able to provide a multi-protocol file server infrastructure.

Like all other work we do, this work has to fit a key customer requirement. In this case, the requirement is that we scale to over 10K concurrent connections to the SMB server.

Brian has put together the thread pool mechanism and wired it to epoll. On a single Linux box, we’ve demonstrated I/O between  5k pairs of socket connections do reads and writes. So we can do 10K connections. We’ve also tested this code against our Netbench runs and we’ve not regressed.

So next week, we should have our first major fsct runs. fsct should help us simulate a demonstration of 10,000 socket connections to our SMB server. It should be pretty amazing if we can demonstrate minimal degradation in I/O performance by a server with 10,000 live connections. The goal is to ensure that on connections where active I/Os are performed the system will perform more than tolerably.

So I can’t wait and please keep reading. I should have more numbers later next week.

Evolution does support the Exchange WebDAV interface and more recently the OpenChange project has provided an native MAPI plugin for Evolution. But it seems that Evolution has generally petered out as a mainstream open source email client. Thunderbird and Lightning are available on Windows, Linux and the Mac but again MAPI support doesn’t exist yet.

I agree with Dirk completely. But I’d like to suggest that the problem is broader than just mail clients. I want to make two points: first corporate messaging infrastructure needs a complete overhaul and second that the debate on email has gotten so fragmented because new constituencies are always pushing their specific strategy as the panacea for all that ails email.

 Here is a sampling of the constituencies and their positions

 i) The free email server/ open source/standards based constituency

ii) The “email in the cloud” constituency

iii) The FAT clients are irrelevant,email delivery should be entirely browser-based constituency

While this debate is raging, the reality is somewhere along the following lines.  Exchange accounts for over 65% of installed corporate messaging infrastructure. In  the healthcare vertical, they dominate to around 75% of the market, in the telco vertical they dominate to 95% of the market. In a 2005 Radicati study, the numbers for 2009 were along the lines of total installed base of 500 million seats. Just under 200 million seats were Exchange seats, but they accounted for over 60% of the revenue.  Free and open source alternative accounted for around 40% of the installed base, but less than 10% of the total revenue. Note that the Radicati study was in 2005. The first set of numbers reflects what today really is. If you extrapolate the numbers that the Radicati report put out, that means Exchange should be around 300 million paid seats.  In effect, Exchange has done significantly better than predicted in 2005.  By the way, this is not hosted email services in the cloud. These numbers are about in-premise Exchange servers.

The free standards based open source email server systems constituency

First the standards – SMTP and IMAP.  When you install an email client and you have to connect to your email server, have you ever wondered why you have to provide an smtp server and an imap server  which invariably are the same. If you are like me, the answer is because email submission process whether you are a client or another mail server relay mail uses SMTP. So IMAP which is really the protocol that clients use to talk to their mail server is good to create folders, read messages but when it comes to message submission we need to use SMTP even though we’re a client. Check it out – if you want to use gmail through an email client, you specify your imap.gmail.com and your smtp.gmail.com Now calendaring is something entirely different. While iCal attachments are MIME attachments and can be sent to you as mail as they should be, calendaring typically uses another protocol as well – CalDAV. So a standards based email system (a groupware server) ends up have multiple disparate servers glued together

The challenge for the open source free email server is that it loyally wishes to build standards based systems. But the standards are insanely convoluted because they mirror the evolution of the standards. Email came first and then there was calendaring and that became groupware. So you have silos where people have designed the email protocols and different silos where people have designed the calendaring standards. Open source projects typically originate around implementing a standard not as implementing a product which may implement multiple standards  and therefore they become the lowest common denominator. Its not good enough to glue together a collection of  separately designed components. It is time that they are cleanly coalesced together and built as a unified cohesive stack.

In contrast, functionally MAPI is an elegant system. (I’m probably going to get killed for this, bear with me. Implementation wise, MAPI is super ugly. I’ll clarify that with one other statement. MAPI uses a COM based API that is not thread safe because of a global session context and uses traditionally MSRPC to marshall data across to the server before there was a remotable COM API (prior to DCOM). Even worse,  MAPI attempts to work as a transport over RPC, so there is just one function call EcDoRpc which is used to build a “protocol” instead writing individual functions in the interface description language. This is a mouthful – but it technically summarizes why MAPI is awful. The protocol is called OETP.) So why do I say that MAPI is  an functionally an elegant system. MAPI is functionally elegant because the MAPI API (the really interesting relevant tiny core without the  COM baggage) is a data driven API much like an LDAP API where the client can request data objects of different types – the same API handles  mail messages, folders, calendar objects, note object, public folder objects and a variety of other information. The MAPI server is therefore a hierarchical object system. Interpreting that an object is a  mail message is a client side detail. It is not a email MUA-MTA protocol, it is an integrated groupware client-server protocol.

The email in the cloud constituency

This constituency believes that all email should be hosted  in the cloud. Principally, I completely agree with this model. It mirrors Geoffrey Moore’s core versus context completely. Why on earth should Caterpillar host an in-premise email server for a couple of hundred thousand employees? They are not in the email business, they build heavy equipment.  The problem is that it is unlikely that everyone will move immediately to a hosted mail service in the cloud. The Radicati numbers reflect just that. There are over a half a billion seats in in-premise email server systems. There will be therefore a continuum of email services. Small businesses will move faster than anyone else to a hosted solution and over a period of time  larger organizations will move. The larger the company the more organizational inertia to move to a cloud based system.

The FAT clients are irrelevant and the email client should be browser-based constituency
Hosted cloud based providers of email evangelize this all the time. Yet there are over 300 million Outlook CALs out there.  Linux users use Evolution or Thunderbird and Mac users use Entourage or MacOS Mail/Calendar programs.  So it is hard to claim that the browser is the ubiquitous client for corporate email. Even hosted/third party mail providers all provide a MAPI plugin adaptor for Outlook for their email systems (Google, Zimbra, OpenXchange, Opengroupware, Lotus Domino and hundreds of others all provide MAPI plugins).  The previous statement results in two conclusions.  Corporate email users use Outlook as their default email client of choice. And secondly, it is almost unassailable barrier-to-entry if  you have to write a MAPI plugin connector that needs to be deployed on every desktop Outlook client. IT adminstrators detest having to glue on third-party plugin software on Windows desktops. Much better to just use Exchange. The Outlook-Exchange nexus is another superb competitive advantage for Microsoft.

Browser based approaches have another problem. How do you handle offline mode operations? The browser afficianados will tell you that RIAs are the way to go. RIAs now comprise of technologies which are largely new client-side programming platforms.  Google’s Chrome OS clearly captures this – The user experience for Chrome OS is the Chrome browser. The browser will support offline mode paradigms – viz Google Gears. Microsoft is following up with Silverlight – RIA apps can run in the browser and guess what run standalone. Silverlight  is the successor to Winforms, WPF, ASP.NET and is the next generation client programming platform. Guess what, by the time you are done, you have a rich FAT application – sort of like what you have today. Outlook rewritten in Silverlight is a next generation .NET desktop application.

The above represents the current state of affairs in corporate email. How does one effect a renaissance in corporate email servers?

First the servers – I think it is time for new development of  high-speed highly efficient cohesively integrated, light-weight group ware servers. Note I’m not saying email servers, I’m saying groupware servers. It will no longer to do to package  disparate open source servers together – this is a least common denominator approach.

Second, these servers should use a new Groupware client to Groupware server protocol. An enhanced IMAP protocol would be great. But as long as the protocol covers send and receiving all sorts of groupware objects to the groupware server.

Thirdly, for a period of time, existing email clients must be able to provide MAPI support so that there is an alternative to Outlook as a client. This is the point that Dirk makes

Fourthly, third party email servers ought to be able to support MAPI on the server side as well. This would allow third party email servers to work against Outlook natively without having to deploy a  third-party MAPI plugin into every single Outlook client.

Points three and four are tactical transition moves. No third party can afford to use MAPI/OETP  as their primary client-server protocol. It is a closed proprietary protocol and it would be unwise. Probably the most relevant near-term step is to build a new rich  lightweight cross platform (Mac, Windows, Linux)  groupware client  that provides a multiple groupware provider interface. There is a pressing need for  a next generation groupware RIA client that can serve as an alternative to Outlook.

As Dirk says, I wonder if someone is up for the challenge.