More on NFS..

If there is one project I want to get off the ground, it’s the NFS server project.  I think there is a good reason to move the NFS server to user space.

The first reason is the cross protocol issues. Traditional NFS servers are  in the kernel, but CIFS (Likewise or Samba) has been in user space.  The challenge is  now reconciling access to the same file via separate protocols. With one server in the kernel and the other in user space, the left hand has no idea what the right hand does and vice versa.

The second is security controls.  CIFS is intimately tied with Kerberos authentication and I’m pleased to see (its been a while since I’ve looked at NFS) is Kerberized NFS as well.  I see a simplified multi-protocol stack where the authentication is Kerberos and the access control lists are NT security descriptors (or something like them).  Servers should be associated with unified authentication stores – so when you’re talking many many users, I expect to see a domain controller.

Finally, manageability. I like simple management semantics – I’m always forgetting what the NFS mount options are and frankly, I’d like to set up NFS exports the same way as I would an SMB share. I’d like to see an MMC snapin which can RPC to the server and set up the export.  So we’d actually use DCE/RPC and provide a new set of APIs – think NetExportAdd, NetExportDelete and NetExportGetInfo and NetExportSetInfo to create, delete and manage NFS exports – neat yes? I’d like a simple command-line interface that allows me to create an export something like -

%net export add file-export /usr/share/export ; creates the file-export

or

%net export delete file-export  ; removes the file-export export

I think we’ll have the NFS management daemon run over tcp. So it will permit an administrator to set up a NFS only file server and use dce/rpc for the management API interface.

The more exciting opportunity is that Likewise will be able to provide a multi-protocol file server infrastructure.

FSCT Testing under way – 10K concurrent SMB connections!

Last week, we rewrote a threadpool engine for the lwio stack. The idea is we can build a pluggable threadpool engine that can work against multiple fd backends. Think select, epoll, kqueue, kevent.

Like all other work we do, this work has to fit a key customer requirement. In this case, the requirement is that we scale to over 10K concurrent connections to the SMB server.

Brian has put together the thread pool mechanism and wired it to epoll. On a single Linux box, we’ve demonstrated I/O between  5k pairs of socket connections do reads and writes. So we can do 10K connections. We’ve also tested this code against our Netbench runs and we’ve not regressed.

So next week, we should have our first major fsct runs. fsct should help us simulate a demonstration of 10,000 socket connections to our SMB server. It should be pretty amazing if we can demonstrate minimal degradation in I/O performance by a server with 10,000 live connections. The goal is to ensure that on connections where active I/Os are performed the system will perform more than tolerably.

So I can’t wait and please keep reading. I should have more numbers later next week.

Why Email needs a renaissance

Dirk Hohndel has a really neat blog. You can read it at http://www.hohndel.org. Dirk argues that the lack of decent MAPI support in Evolution and Thunderbird prevents broader adoption of  Linux and Macs in the corporate world.

Evolution does support the Exchange WebDAV interface and more recently the OpenChange project has provided an native MAPI plugin for Evolution. But it seems that Evolution has generally petered out as a mainstream open source email client. Thunderbird and Lightning are available on Windows, Linux and the Mac but again MAPI support doesn’t exist yet.

I agree with Dirk completely. But I’d like to suggest that the problem is broader than just mail clients. I want to make two points: first corporate messaging infrastructure needs a complete overhaul and second that the debate on email has gotten so fragmented because new constituencies (as always) is pushing a new strategy as the panacea for all that ails email.

 Here is a sampling of the constituencies and their positions

 i) The free email server/ open source/standards based

ii) The email in the cloud

iii) The FAT clients are irrelevant,email delivery should be entirely browser-based.

While this debate is raging, the reality is somewhere along the following lines.  Exchange accounts for over 65% of installed corporate messaging infrastructure. In  the healthcare vertical, they dominate to around 75% of the market, in the telco vertical they dominate to 95% of the market. In a 2005 Radicati study, the numbers for 2009 were along the lines of total installed base of 500 million seats. Just under 200 million seats were Exchange seats, but they accounted for over 60% of the revenue.  Free and open source alternative accounted for around 40% of the installed base, but less than 10% of the total revenue. Note that the Radicati study was in 2005. The first set of numbers reflects what today really is. If you extrapolate the numbers that the Radicati report put out, that means Exchange should be around 300 million paid seats.  In effect, Exchange has done significantly better than predicted in 2005.  By the way, this is not hosted email services in the cloud. These numbers are about in-premise Exchange servers.

The free standards based open source email server systems

First the standards – SMTP and IMAP.  When you install an email client and you have to connect to your email server, have you ever wondered why you have to provide an smtp server and an imap server  which invariably are the same. If you are like me, the answer is because email submission process whether you are a client or another mail server relay mail uses SMTP. So IMAP which is really the protocol that clients use to talk to their mail server is good to create folders, read messages but when it comes to message submission we need to use SMTP even though we’re a client. Check it out – if you want to use gmail through an email client, you specify your imap.gmail.com and your smtp.gmail.com Now calendaring is something entirely different. While iCal attachments are MIME attachments and can be sent to you as mail as they should be, calendaring typically uses another protocol as well – CalDAV. So a standards based email system (a groupware server) ends up have multiple disparate servers glued together

The challenge for the open source free email server is that it loyally wishes to build standards based systems. But the standards are insanely convoluted because they mirror the evolution of the standards. Email came first and then there was calendaring and that became groupware. So you have silos where people have designed the email protocols and different silos where people have designed the calendaring standards. Open source projects typically originate around implementing a standard not as implementing a product which may implement multiple standards  and therefore they become the lowest common denominator. Its not good enough to glue together a collection of  separately designed components. It is time that they are cleanly coalesced together and built as a unified cohesive stack.

In contrast, functionally MAPI is an elegant system. (I’m probably going to get killed for this, bear with me. Implementation wise, MAPI is super ugly. I’ll clarify that with one other statement. MAPI uses a COM based API that is not thread safe because of a global session context and uses traditionally MSRPC to marshall data across to the server before there was a remotable COM API (prior to DCOM). Even worse,  MAPI attempts to work as a transport over RPC, so there is just one function call EcDoRpc which is used to build a “protocol” instead writing individual functions in the interface description language. This is a mouthful – but it technically summarizes why MAPI is awful. The protocol is called OETP.) So why do I say that MAPI is  an functionally an elegant system. MAPI is functionally elegant because the MAPI API (the really interesting relevant tiny core without the  COM baggage) is a data driven API much like an LDAP API where the client can request data objects of different types – the same API handles  mail messages, folders, calendar objects, note object, public folder objects and a variety of other information. The MAPI server is therefore a hierarchical object system. Interpreting that an object is a  mail message is a client side detail. It is not a email MUA-MTA protocol, it is an integrated groupware client-server protocol.

The email in the cloud constituency

This constituency believes that all email should be hosted  in the cloud. Principally, I completely agree with this model. It mirrors Geoffrey Moore’s core versus context completely. Why on earth should Caterpillar host an in-premise email server for a couple of hundred thousand employees? They are not in the email business, they build heavy equipment.  The problem is that it is unlikely that everyone will move immediately to a hosted mail service in the cloud. The Radicati numbers reflect just that. There are over a half a billion seats in in-premise email server systems. There will be therefore a continuum of email services. Small businesses will move faster than anyone else to a hosted solution and over a period of time  larger organizations will move. The larger the company the more organizational inertia to move to a cloud based system.

The FAT clients are irrelevant and the email client should be browser-based
Hosted cloud based providers of email evangelize this all the time. Yet there are over 300 million Outlook CALs out there.  Linux users use Evolution or Thunderbird and Mac users use Entourage or MacOS Mail/Calendar programs.  So it is hard to claim that the browser is the ubiquitous client for corporate email. Even hosted/third party mail providers all provide a MAPI plugin adaptor for Outlook for their email systems (Google, Zimbra, OpenXchange, Opengroupware, Lotus Domino and hundreds of others all provide MAPI plugins).  The previous statement results in two conclusions.  Corporate email users use Outlook as their default email client of choice. And secondly, it is almost unassailable barrier-to-entry if  you have to write a MAPI plugin connector that needs to be deployed on every desktop Outlook client. IT adminstrators detest having to glue on third-party plugin software on Windows desktops. Much better to just use Exchange. The Outlook-Exchange nexus is another superb competitive advantage for Microsoft.

Browser based approaches have another problem. How do you handle offline mode operations? The browser afficianados will tell you that RIAs are the way to go. RIAs now comprise of technologies which are largely new client-side programming platforms.  Google’s Chrome OS clearly captures this – The user experience for Chrome OS is the Chrome browser. The browser will support offline mode paradigms – viz Google Gears. Microsoft is following up with Silverlight – RIA apps can run in the browser and guess what run standalone. Silverlight  is the successor to Winforms, WPF, ASP.NET and is the next generation client programming platform. Guess what, by the time you are done, you have a rich FAT application – sort of like what you have today. Outlook rewritten in Silverlight is a next generation .NET desktop application.

The above represents the current state of affairs in corporate email. How does one effect a renaissance in corporate email servers?

First the servers – I think it is time for new development of  high-speed highly efficient cohesively integrated, light-weight group ware servers. Note I’m not saying email servers, I’m saying groupware servers. It will no longer to do to package  disparate open source servers together – this is a least common denominator approach.

Second, these servers should use a new Groupware client to Groupware server protocol. An enhanced IMAP protocol would be great. But as long as the protocol covers send and receiving all sorts of groupware objects to the groupware server.

Thirdly, for a period of time, existing email clients must be able to provide MAPI support so that there is an alternative to Outlook as a client. This is the point that Dirk makes

Fourthly, third party email servers ought to be able to support MAPI on the server side as well. This would allow third party email servers to work against Outlook natively without having to deploy a  third-party MAPI plugin into every single Outlook client.

Points three and four are tactional transition moves. No third party can afford to use MAPI/OETP  as their primary client-server protocol. It is a closed proprietary protocol and it would be unwise. Probably the most relevant near-term step is to build a new rich  lightweight cross platform (Mac, Windows, Linux)  groupware client  that provides a multiple groupware provider interface. There is a pressing need for  a next generation groupware RIA client that can serve as an alternative to Outlook.

As Dirk says, I wonder if someone is up for the challenge.

I’m always looking for extraordinary systems programmers

If you worked on distributed systems, operating systems, networking as a software design engineer,  I’m always on the lookout for great and extraordinary engineers

This is how you would define yourself:
- You cut high calibre C code and a ton of it (really a lot of code, no kidding!)
- You are the kind of person who is super technical and is a force multiplier.  You’re generally delighted to work on hard technical problems.
- you are an outstanding systems programmer.
- You have superb analytical skills
- You have an high GPA at University. A high GPA in Computer Science tells me you are very smart and have had to work very hard. The course load and project load in most good CS schools is always crushing.
- You have strong formal Computer Science skills – Operating Systems, Compilers, Computer Architecture, Theory of Algorithms, Computability, Theory of Programming Languages, Advanced Operating Systems, Distributed Systems.
- You have a BS or MS in Computer Science
- You have written or contributed to the writing of one or more C based file servers, dns servers, web servers, ldap servers, certificate servers, radius servers etc etc
- You have a PhD in operating systems/distributed systems, but don’t want to be an academic or work for a research think tank
- You wanted to do a PhD, but finished a Masters and looked for a job
- You’re probably on an IETF or IEEE mailing list – I most definitely would like to talk to you
- You understand networking protocols intimately well – you’re at a place where you read IETF RFCs to do your job.

I value technical people at every level. I think the best product marketing people are those who have a rigorous background in the discipline they are trying to market. I think the best SEs have a solid grounding in the products and technologies they sell. I think the best test engineers are those who actually understand the internals of the technology they are testing. I need specialists, not generalists. I thinks specialists can work as generalists if they have to, but generalists can’t become specialists unless they are willing to roll up their sleeves, get in the muck and figure it out. A little anecdote below..

Recently, I was flying back from Boston and happened to sit next to a guy who was watching movies on his iPhone. We got talking and he told me he used to be the Northwest Account Manager for Cisco (i.e the sales rep). What he told me was astonishing. John Chambers is a extremely hard core driven sales CEO for Cisco. Yet the SEs (the Sales Engineers) are largely drawn from within the ranks of Cisco engineering. He mentioned that his SE was a developer for 6 years before he transitioned to the sales side of the business. This one SE managed the pre-sales process, the post sales process, the deployment and roll out for one of the largest cellular networks in the country – think of how many IOS routers this guy is deploying and setting up. That is impressive!

Anyway I hope to hear from you

Screenshots for Likewise 5.4

Likewise and Moblin

Arlene Berry got Likewise Open running on Moblin. We have a few hoops to jump through. The most
important being that Moblin does not provide a graphical login console – but there are packages available
that provide a gnome window manager login shell. Once you joined the Moblin netbook to Active Directory
you could enter your AD credentials and login with them.

Logging out is also a challenge. There is no application that lets you log out. But as far as Likewise is
concerned, you can install the Likewise Open bits (rpm ones for Fedora) and you’re good to go.

Getting AD authentication on netbooks is a huge deal. Given the fact that over 32% of netbooks are shipped with Linux,that means a Linux netbook with AD integration is a viable alternative to a netbook with Windows 7. Your Linux netbook can be used in corporate environments and can take advantage of the corporate network’s resources.

We’ve gotten Moblin up and running with very little work – hopefully we can do the same with Chrome OS and Android.

Thanks for reading.

Lots of cleanup in LAC and the Likewise Class Libraries

Wei Fu and Suneetha Gunasetti have been working steadily to clean up our LAC sources and make them ready for release. We’re almost there. Here are some of the highlights

1) First, we’ve separated our plugins into two directories – Open and Enterprise.

2) Build systems – we will make the open source LAC code base buildable on Visual Studio and on Monodevelop. This way if you have Visual Studio, you can build the source plugins, but if not, Monodevelop should work just fine.

3)We’re fixing up the licensing – LAC and all of its plugins will ship under the LGPL license. What this means is that you can write proprietary plugins if you want to. LAC then becomes a ready to release administration tool system for any product you’re shipping.

4) But the best thing of all is that we’ve refactored out the core class libraries. The Likewise Class Libraries will give you APIs for the NetAPIs, the Kerberos libraries, the Registry, the Eventlog, the Service Control Manager and a host of other classes that allow you the developer or ISV to build your own management applications. You’re not bound to LAC/LMC rather you have a powerful set of class libraries that let you exploit the power of the Likewise Distributed Systems Platform and build great applications in your favorite managed languages.

Thanks for reading and wait for the press release early November.

LAC now LMC coming soon in open source

One of the singularly important pieces of technology at Likewise has been the Likewise Administrators Console. Now renamed the Likewise Management Console, it is a multi-plugin graphical management console for administration of server infrastructure.

Very similar to those of you who have used the Microsoft Management Console, LMC is a significantly superior platform. It is written in C# and uses the .NET platform and is completely portable on Linux, Mac and Windows – The Linux and Mac ports are possible thanks to the magic of Miguel de Icaza and his Mono team. We love Mono out at Likewise and have been early adopters of the Mono platform for several years now.

LMC will ship with several plugins
- a local users and groups plugin -LUG allows you to point to a Linux machine and create and manage local users and groups
- an event viewer plugin – the EventViewer plugin allows you to examine the eventlog on a remote Linux machine
- a registry viewer plugin
- a file share management plugin
- and a active directory users and computers plugin

LMC is released under the LGPL license. This allows third parties and ISVs to write proprietary plugins if they want. So if you are an ISV or clustered NAS vendor, you would potentially ship LMC with your product line and write proprietary plugins that allow you to provide value-added management functionality to your products

We’re busy tidying up LMC. But expect to see it out in early-mid November.

Thanks for reading.

A NFS protocol head for LWIO

The number of requests we’re getting on an NFS4 protocol head for LWIO is steadily increasing to the point where we can’t ignore it any longer.
Most storage and clustered NAS players have challenges getting CIFS and NFS to work collaboratively without stepping on each other toes. The CIFS server is usually in user space and the NFS stack is usually in the kernel. The challenge is synchronizing file locks and generally making CIFS and NFS aware of each other.
lwio is a natural solution to this problem. The lwio architecture cleanly separates the file server protocol from the file system layer. Thus it is super easy for us to build an NFS protocol head driver that calls into our pvfs driver for file system service. Since PVFS manages oplocks and general file access, all file access is centralized in the PVFS driver.
Anybody out there have any thoughts on this. Feel free to drop me a line.
Thanks for reading

Service Control Manager, Registry, Spooler and Sample RPC Client-Server

October will be the month for finishing up 5.4.
5.4 will have every Likewise service use the Likewise registry. Current list of services that we will be converting over to use the registry are the following
a) lsassd
b) lwiod with all its drivers – specifically the share database will move into the registry
c) netlogond
d) eventlogd
e) gpagentd – all group policy configuration will move into the registry as well

Finally, Brian Koropoff has begun work on the lwscm – the Likewise Service Control Manager. And if all goes well, 5.4 will ship with the Service Control Manager. What does the lwscm do? Well it is responsible for forking off Likewise services. So instead of having multiple init scripts and having to deal with different configuration formats for different platforms, every Likewise service is stored in the registry. The Likewise Service Control Manager is responsible for identifying service dependencies and ensuring dependent services are launched prior to starting the target service.

While the first incarnation of this Likewise Service Control Manager is about separate services running in indepenedent processes, the next incarnation will be the ability to run multiple services in a single process. Subsequently we hope to provide a “svchost” model where we provide a service container which developers can leverage. There should be no need to have to write housekeeping code, but get developers to focus on their specific service offering.
Brian has just demoed some of this working and it is pretty impressive. The best part for me is to get away from maintaining multiple init scripts across multiple platforms.

Also, I’ve started the preliminary framework for a “spooler” architecture. We were flying back from the Linux Plumbers Conference and I began doodling on the flight and while we have no commercial reason at this time to be building a spooler, I’d like to get to a place where other developers on the team can jump in, add an RPC call if they want and get back to what their primary focus should be. I think it could be a fun distraction.
Lastly, I’m going to start getting dev packages ready, so that developers can install binaries and get the appropriate header files and starting building to the Likewise platform. One of the things I used to do at Microsoft was to design fully functioning template code which developers could take and then run a simple sed script which would replace all occurrences of the word Sample with their word fragment of choice. Within 10 minutes they would have an easy to read fully functioning RPC client, RPC server and in our platforms case integrated with the registry, the eventlog, the service control manager and our lwmsg minimal client-server framework (as opposed to RPC). We plan to release this to the public domain so you can build open source programs or proprietary programs but build on top of the Likewise platform.
Thanks for reading..