LINQ To LDAP: Dynamics!

Short post to be updated later. During my property mapping changes (added support for file time date mapping and enum mapping and cleaned up the structure to support directory modifications in the future), I was inspired in a massive way to add dynamic support for LINQ to LDAP. I'm happy to say that it works very well! You can't use expressions, but everything else works.

dynamic user = context.Query("CN=Users,CN=Employees,DC=Northwind,DC=local")
    .Select(new[]{"cn", "givenname", "sn", "telephonenumber"})
    .FilterWith("cn=Andrew Fuller")
    .FirstOrDefault();

Console.WriteLine(user.cn);
Console.WriteLine(user.givenname);
Console.WriteLine(user.sn);
Console.WriteLine(user.telephonenumber);

I added a Select extension that takes an array of strings so you still get projection support. FilterWith is the only way to query since everything else uses expressions (you can't use dynamic types in expressions). The only type support you get here is what can come back from the directory so that's string, string[], and byte[]. Also, property names will always be lower case so even though it may be givenName in the directory, use givenname to access it. That's just how DirectoryServices.Protocols rolls.

LINQ To LDAP 1.0

You can head over to codeplex and download the 1.0 version of LINQ to LDAP. Woohoo!

Download it now!

LINQ To LDAP: Connection Pooling

I got a question recently about conneciton pooling. S.DS.P is a low level API so it has no native support for connection pooling. For anyone familiar with S.DS, DirectoryEntry will reuse connections under certain circumstances. I took a stab at implementing connection pooling in LINQ to LDAP and so far so good.

I had to support the following capabilities:

• Maximum Pool Size: The maximum number of connections that can be created for the pool.
• Minimum Pool Size: The minimum number of connections to maintain in the pool.
• Connection cleanup: Cleaning up connections that have been unused for a period of time.

I created a new class called PooledConnectionFactory. You can access it from the LdapConfiguration class. There's a fluent interface for configuring the properties, but I'll only cover the key methods to support the features from above.

Getting Connections

LdapConnection IConnectionFactory.GetConnection()
{
    LdapConnection connection;
    lock (_connectionLockObject)
    {
        if (_isFirstRequest)
        {
            DateTime now = DateTime.Now;
            for (int i = 0; i < _minPoolSize; i++)
            {
                _availableConnections.Add(GetConnection(), now);
            }

            _isFirstRequest = false;
            _timer.Start();
        }

        var pair = _availableConnections.FirstOrDefault();

        if (Equals(pair, default(KeyValuePair<LdapConnection, DateTime>)))
        {
            if ((_inUseConnections.Count + _availableConnections.Count + 1) > _maxPoolSize)
                throw new InvalidOperationException(
                    string.Format("LdapConnection pool limit of {0} exceeded.", _maxPoolSize));

            connection = GetConnection();
            _inUseConnections.Add(connection);
        }
        else
        {
            _inUseConnections.Add(pair.Key);
            _availableConnections.Remove(pair.Key);
            connection = pair.Key;
        }
    }
    return connection;
}

private LdapConnection GetConnection()
{
    //code that constructs a connection from the fluent configuration
}

For all operations that interact with the pool, I have to first get a lock to ensure thread safety. I keep track of available connections via a dictionary that contains the connection and the last time it was used. Connections in use are tracked in a generic List. The first time a request for a connection comes in, I lazily initialize the pool if there's is a minimum pool size. I also start the timer for scavenging stale connections.

The first available connection is always used, and a new connection is created if one is not available. However, if the max pool size is exceeded, then I throw an exception. I've considered implementing a wait time for available connections, but I'll wait to see if it is a needed feature.

Releasing Connections

public void ReleaseConnection(LdapConnection connection)
{
    lock (_connectionLockObject)
    {
        if (_inUseConnections.Contains(connection))
        {
            _inUseConnections.Remove(connection);
            _availableConnections.Add(connection, DateTime.Now);
        }
        else
        {
            connection.Dispose();
        }
    }
}

There's not much going on here. If the connection is found in use, it's removed and returned as an available connection with an updated last used time. If it's not found then it's just disposed.

Stale Connections

private void TimerElapsed(object sender, ElapsedEventArgs e)
{
    lock (_connectionLockObject)
    {
        int amountToScavenge = _minPoolSize == 0
            ? _availableConnections.Count
            : (_availableConnections.Count - _minPoolSize);

        if (amountToScavenge <= 0) return;

        var expiredConnections = _availableConnections
            .Where(p => e.SignalTime.Subtract(p.Value).TotalMinutes > _connectionIdleTime)
            .Select(p => p.Key)
            .ToList();
                
        foreach (var expiredConnection in expiredConnections)
        {
            if (amountToScavenge == 0) break;

            _availableConnections.Remove(expiredConnection);
            expiredConnection.Dispose();
            amountToScavenge--;
        }
    }
}

First thing I do in this method is to determine if any connections actually need to be cleaned up. After that I look for stale connections. I iterate over those connections and dispose of them until I've satisfied the number to scavenge or until there are no more connections.

Benchmarking
I was curious what kind of benefits pooling would provide when hitting a directory server. My initial testing was against my local machine and I saw an average 10ms difference per query when using pooling. That's fine and dandy, but unrealistic.

So I found a public LDAP server hosted by verisign. So here's my test:

static void Main(string[] args)
{
    new LdapConfiguration()
        .MaxPageSizeIs(10)
        .ConfigurePooledFactory("directory.verisign.com")
        .AuthenticateBy(AuthType.Anonymous);

    var thread1 = new Thread(() => 
        DoVerisignWork(1000, "thread 1"))
                      {
                          IsBackground = true, Name = "thread 1"
                      };

    var thread2 = new Thread(() => 
        DoVerisignWork(500, "thread 2"))
                      {
                          IsBackground = true, Name = "thread 2"
                      };

    var thread3 = new Thread(() => 
        DoVerisignWork(1500, "thread 3"))
                      {
                          IsBackground = true, Name = "thread 3"
                      };

    var thread4 = new Thread(() => 
        DoVerisignWork(300, "thread 4"))
                      {
                          IsBackground = true, Name = "thread 4"
                      };

    var thread5 = new Thread(() => 
        DoVerisignWork(600, "thread 5"))
                      {
                          IsBackground = true, Name = "thread 5"
                      };

    var thread6 = new Thread(() =>
    {
        while (Watch.IsRunning)
        {

        }
        Thread.Sleep(1000);
        Console.WriteLine("Total iterations: " + TotalCount + Environment.NewLine +
                        "Operations over limit: " + OverCount + Environment.NewLine +
                        "Average operation time ms: " +
                        (TimeToExecute / TotalCount) + Environment.NewLine +
                        "Total time: " + Watch.ElapsedMilliseconds);
        Console.ReadLine();
    });

    Watch.Start();
    thread1.Start();
    thread2.Start();
    thread3.Start();
    thread4.Start();
    thread5.Start();

    thread6.Start();
}

I set up my threads to simulate a pretty light load. When everything is done thread 6 will display the results.

From what I can tell, the server only exposes the RootDSE. So I can only test the ListServerAttributes method, which is just a ListAttributes request with (objectClass=*) filter. Here's my work method:

private static void DoVerisignWork(int milliseconds, string threadName)
{
    for (int i = 0; i < 5; i++)
    {
        int count = 0;
        while (count < 10)
        {
            TotalCount++;
            var watch = new Stopwatch();
            watch.Start();
            using (var context = new DirectoryContext())
            {
                context.ListServerAttributes();
            }
            watch.Stop();
            TimeToExecute += watch.ElapsedMilliseconds;
            if (watch.ElapsedMilliseconds > 150)
            {
                OverCount++;
            }
            count++;

            Thread.Sleep(milliseconds);
        }
    }

    if (threadName == "thread 3")
    {
        Watch.Stop();
    }

    Console.WriteLine(threadName + " finished");
}

So here are my results for the default connection factory:

Total Iterations: 250
Operations Over 150 ms: 250
Average Operation Time: 272.42 ms
Total Time: 113.734 seconds

And for the pooled connection factory:

Total Iterations: 250
Operations Over 150 ms: 51
Average Operation Time: 149.58 ms
Total Time: 107.248 seconds

The pool created 5 connections for each thread since they are started in sequence. When I added a short sleep after the second thread was started, it was able to do the same work with 4 connections. I'm pretty happy with these results. ~120 ms improvement would definitely help with scalability. This has already been checked into the trunk.

You can download the sample code for the pool here:

upload files free