Tag Archives: csharp

MongoDb Ninjitsu: Using ObjectId as a Timestamp

Whilst reading around yesterday I stumbled upon this little gem of knowledge:

Mongo’s ObjectIds contains a Utc timestamp with 1 second resolution.

This means, that if we don’t need millisecond accuracy, we can drop all those “CreatedOn” fields from our schemas, and by doing so we win twice:

  • Storage – no need to store a seperate time stamp field means less to store on disk and less for the server to shunt around.
  • Free Index – assuming your ObjectId is your primary key, it already has an index on it by default, so not only are you saving the space of having to store a seperate time stamp, but it’s also indexed for free.

To make it a little easier, I created an extension class to convert DateTimes to and from ObjectIds. This makes it insanely simple to query for objects created before/after a certain date. Here’s the code:

/// Author	: Daniel Harman (http://www.danharman.net)
/// Date	: 26.Oct.2011
/// License : Public Domain with no warranty given. Please maintain author credit.
using System;
using System.Collections.Generic;
using System.Linq;
using MongoDB.Bson;

namespace Mmphs.Utils.MongoDb
{
	public static class DateTimeExtensions
	{
		/// <summary>
		/// Converts a DateTime to an ObjectId.
		/// n.b. missing values that would ensure uniqueness. This is only intended for time comparisons.
		/// </summary>
		/// <param name="dateTime"></param>
		/// <returns></returns>
		public static ObjectId ToObjectId(this DateTime dateTime)
		{
			var timestamp = (int)(dateTime - BsonConstants.UnixEpoch).TotalSeconds;
			return new ObjectId(timestamp, 0, 0, 0);
		}

		/// <summary>
		/// Convert an ObjectId to a DateTime.
		/// </summary>
		/// <param name="id"></param>
		/// <returns></returns>
		public static DateTime ToDateTime(this ObjectId id)
		{
			return id.CreationTime;
		}

		static readonly int DATETIME_TRUNCATE_FACTOR = 10000;

		/// <summary>
		/// Truncate the accuracy of a datetime to the same resolution as a mongo db datetime.
		/// </summary>
		/// <param name="dateTime"></param>
		/// <returns></returns>
		public static DateTime MongoTruncate(this DateTime dateTime)
		{
			long ticks = dateTime.Ticks / DATETIME_TRUNCATE_FACTOR;
			return new DateTime(ticks * DATETIME_TRUNCATE_FACTOR, dateTime.Kind);
		}
	}
}

I’ve thrown in a little bonus there too – a method to truncate a DateTime in the same way MongoDb does when it persists one. This is handy for unit tests where you want to compare an object you’ve created and persisted and then loaded back up e.g. when testing a query brings back the right record.

Here is an example of using the ObjectId to get items after a certain date:

		public IEnumerable<Drop> GetStreamActivities(ObjectId streamId, DateTime utcSince)
		{
			return _drops
				.AsQueryable()
				.Where(a => a.StreamId == streamId && a.Id >= utcSince.ToObjectId())
				.OrderBy(a => a.Id);
		}

n.b. I’m using FluentMongo, but you certainly don’t need to.

and here are the unit tests:

/// Author	: Daniel Harman (http://www.danharman.net)
/// Date	: 26.Oct.2011
/// License : Public Domain with no warranty given. Please maintain author credit.

using System;
using System.Collections.Generic;
using System.Linq;
using MbUnit.Framework;
using MongoDB.Bson;

namespace Mmphs.Utils.MongoDb.Test.DateTimeExtensions
{
	[TestFixture]
	public class DateTimeExtensionsTest
	{
		[Test]
		public void Can_Convert_DateTime_To_ObjectId()
		{
			// Arrange
			DateTime dateTime = DateTime.UtcNow;

			// Act
			ObjectId result = dateTime.ToObjectId();

			// Assert
			Assert.AreApproximatelyEqual(dateTime, result.CreationTime, TimeSpan.FromSeconds(1));
		}

		[Test]
		public void Can_Convert_ObjectId_To_DateTime()
		{
			// Arrange
			ObjectId objectId = ObjectId.GenerateNewId();
			DateTime dateTime = DateTime.UtcNow;

			// Act
			var result = objectId.ToDateTime();

			// Assert
			Assert.AreApproximatelyEqual(dateTime, result, TimeSpan.FromSeconds(1));
		}
	}
}
/// Author	: Daniel Harman (http://www.danharman.net)
/// Date	: 26.Oct.2011
/// License : Public Domain with no warranty given. Please maintain author credit.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using MbUnit.Framework;
using MongoDB.Bson;
using MongoDB.Bson.Serialization;

namespace Mmphs.Utils.MongoDb.Test.DateTimeExtensions
{
	[TestFixture]
	public class When_Truncating_DateTime
	{
		[Test]
		public void Should_Match_Mongo()
		{
			// Arrage
			DateTime dt = DateTime.UtcNow;
			var asJson = dt.ToJson();

			// Act
			var asTrunc = dt.MongoTruncate();
			
			// Assert
			var fromJson = BsonSerializer.Deserialize<DateTime>(asJson);
			Assert.AreEqual(fromJson, asTrunc);
		}
	}
}

MongoDB and C# Dictionary Serialisation, Part 1 – The Problem

I’m building a web site at the moment, and as consequence, have been doing a lot of work with MongoDB. One of the interesting things I’ve come across is the C# driver’s, and I don’t know if this applies to other languages, approach to serialising generic dictionaries. i.e. Dictionary<TKey,TValue>.

The driver has two methods depending on the type of the key. If TKey is an object then the serialiser will create a nested array e.g.:

class Thread
{
 // Id of the Thread.
 ObjectId Id;

 // Dictionary<MemberId, Message>
 Dictionary<ObjectId, string> Posts;
}

serialises to:

{ "_id" : ObjectId("4e519fe15fc9d4099c01733a"),
	"Posts" : [
		[ ObjectId("4e3de6255fc9d40fd437a5ac"), "Is anyone there?" ],
		[ ObjectId("4e3de6305fc9d40fd437a5ae"), "Nobody but us chickens."]
	]
}

Alternatively if, we store the member’s id as a string:

class Thread
{
 // Id of the Thread.
 ObjectId Id;

 // Dictionary<MemberName, Message>
 Dictionary<string, string> Posts;
}

we get:

{ "_id" : ObjectId("4e519fe15fc9d4099c01733a"),
	"Posts" : {
		"4e3de6255fc9d40fd437a5ac" : "Is anyone there?"
		"4e3de6305fc9d40fd437a5ae" : "Nobody but us chickens."
	}
}

This latter format is wonderfully compact and seems like a smart way of leveraging the map like properties of the json/bson collection type.

Unfortunately, I think there are some significant problems with both approaches, which I’ll outline below. If I’m wrong about these let me know as I’d gladly be corrected!

Ability to search by the Dictionary’s Key

With the nested dictionary, we can’t search for documents containing posts by a specific MemberId. This syntax, which one might think could do this:

var q = Query("Posts.0", "4e3de6255fc9d40fd437a5ac")

Will actually search for records where the first element in Posts equals "4e3de6255fc9d40fd437a5ac" – it’s not going to find any matches even if the TKey of the first element matches. This is because the first element is actually a nested array of two elements i.e.

[ ObjectId("4e3de6255fc9d40fd437a5ac"), "Is anyone there?" ]

So you can’t search by MemberId, you can only search if you know the complete contents of the dictionary entry, and only if you know its specific position in the dictionary. That’s not too useful when dealing with serialsed dictionaries!

Info

As a brief aside, there is an open ticket on Mongo’s Jira to add support for a syntax like this:

var q = Query.EQ("Posts.$.0", "4e3de6255fc9d40fd437a5ac")

This would be great as it would solve our problem. I’m not sure it’s on the cards to be implemented anytime soon though.

If instead, we now look at our example with the string key, we can search with the following:

var q = Query.Exists("Posts.4e3de6255fc9d40fd437a5ac", true)

I don’t know about you, but I find it a little weird that the search value has become part of the key and we are having to use ‘Exists’ rather than ‘EQ’. Still… at least we can search by our dictionary’s key with this schema.

Ability to Index by the Dictionary’s Key

We are stuffed here. We can’t create an index with the nested array because we can’t even index the fields with current mongo syntax. Nor can we create a dictionary of all the field names in a document. This means our searches for dictionary key’s or elements are going to be expensive…

Ability to Atomically Update a Value in the Dictionary

Working with a database like MongoDb, one of the most important tools in our arsenal is the ability to make atomic updates to documents without having to pull them back into memory. This side steps a lot of concurrency issues that are otherwise difficult to manage without native transactions and locking. With a simple document this is pretty easy. What about modifying values in our dictionaries?

Well, with the nested array, it is again a bit of a disaster. The only options is to load the whole document, edit in memory and Update. So long atomicity!

With the nested document, we are in better shape and can use the following:

threadsCollection
        .FindAndModify(
		Query.Exists("Posts.4e3de6255fc9d40fd437a5ac", true),
                null,
                Update.Set("Posts.4e3de6255fc9d40fd437a5ac",
                       "A new message replacing the old"));

Summary of Issues

The nested array is not a well supported structure in mongo, and using it for dictionary persistence is very limiting. We lose the ability to search, index and update atomically.

The array of documents is more successful in that we can search and update atomically. We do however lose indexing, which is fairly catastrophic if we actually want to search and update across any reasonably sized collections.

An Alternative

Fortunately, there is an alternate approach which resolves all these problem. All we need to do is create a hybrid of the two approaches – an array, but this time, of specifically formatted documents with known fields for the key and value:

{ "_id" : ObjectId("4e519fe15fc9d4099c01733a"),
	"Posts" : [
		{ "k" : ObjectId("4e3de6255fc9d40fd437a5ac"), "v" : "Is anyone there?" },
		{ "k" : ObjectId("4e3de6305fc9d40fd437a5ae"), "v" : "Nobody but us chickens." }
	]
}

By having well known fields we can now search by the key, the value, or if we want to store documents as the value, the fields in that document.

This search will find all the documents in the collection with a given key :

var q = Query.EQ("Posts.K", "4e3de6255fc9d40fd437a5ac")

To update the value:

threadsCollection
	.Update(
		Query.EQ("Posts.K", "4e3de6255fc9d40fd437a5ac"),
		Update.Set("Posts.$.v", "A new message replacing the old");

Its worth noting that if our TValue is a class containing further properties rather than a literal type, then the above syntax can be extended to index into it using the standard dot notation. e.g.

threadsCollection
	.Update(
		Query.EQ("Posts.K", "4e3de6255fc9d40fd437a5ac"),
		Update.Set("Posts.$.v.Status", "New Status");

And because we can directly index the key, or indeed any value, we can index them!

We can also do funky things like replicating the addToSet functionality with our dictionary. Imagine we have a second dictionary on our thread, this one indexed by member id, but containing the members name so that we can cache the user names of everyone who has posted in the thread:

class Thread
{
 // Id of the Thread.
 ObjectId Id;

 // Dictionary<MemberId, Message>
 Dictionary<ObjectId, string> Posts;

 // Dictionary<MemberId, Name>
 Dictionary<ObjectId, string> NameLookup;

}

We don’t however want to have someone recorded in this lookup table style dictionary more than once, so addToSet is exactly the approach we need. To do this, we first have to create the element we would insert as a BsonDocument:

class NameLookupEntry
{
	public ObjectId k { get; set; }
	public Name v { get; set}
}

var entry = new NameLookupEntry() { k = "4e3de6255fc9d40fd437a5ac", v = "Joe Blow" };

In this instance we are searching for a specific thread by its Id, as we want to add the guy’s name to that one thread, and only if it isn’t already there:

threadsCollection
	.Update(
		Query.AND(
			Query.EQ("Id", "4e519fe15fc9d4099c01733a"
			Query.NE("NameLookup.K", "4e3de6255fc9d40fd437a5ac"),
		Update.Set("NameLookup.$.v", entry.ToBsonDocument());

In part 2, I’ll show you the code to build a custom serialiser to support this alternate scheme.

Thanks to Robert Stam at 10gen for assistance and feedback on this topic.

SEO Slugification in Dotnet aka Unicode to Ascii aka Diacritic Stripping

This article looks at SEO using url slugs, and how we can generate them in dotnet/asp.

I’ve recently conducted quite a lot of research, looking for a class to create url slugs in C#. If you are not familiar with these, they are when sentences are converted to a url friendly format. Normally this involves converting spaces and punctuation to hyphens and removing accents from characters. e.g. “This is my resumé” becomes “this-is-my-resume”.

Slugification of urls to make them human readable, is considered good SEO strategy as it gets keywords into urls. Although how much weight engines put into it these days is a matter of debate as it has been abused by spammers. nonetheless I think humans are more likely to click on a human readable link if a search engine throws it up, so there is no downside.

Before digging into the implementation aspects, one thing worth mentioning, as it took me a while to realise, is that it’s not really worth trying to use the slug as the unique key to whatever resource you want to offer. Dealing with the problem of collisions where you have matching slugs is just not worth the pain. Especially if you want your slugs to contain something highly repeatable like names.

What you will see the experts do is include the id of the resource (e.g. a GUID) and the slug. You can see it on both Stack Overflow and Facebook:

http://stackoverflow.com/questions/3769457/how-can-i-remove-accents-on-a-string

http://ms-my.facebook.com/people/Joe-Bloggs/12343243267683877

For StackOverflow, the id is actually 3769457, and if you change the slug string it makes no odds to what you get back.

Facebook is exactly the same except they place the real id at the end instead of the middle. This seems a little smarter to me as google etc truncate very long URLs when they display them in search results, so by making sure the human readable part is at the front, you aren’t losing that whilst preserving a human meaningless id.

They also both do permanent redirects if you type in a random slug. This tackles the canonical problem of having the same content on multiple links if people mess up the slug, which is pure SEO poison if not tackled.

Right, to business – how do we do it? Well, the first part, punctuation removal and hyphenation, is trivial.

Removing accents however is rather more complex, and goes by many names including diacritic (accent) stripping and Unicode to ascii. My research took me to stack overflow where it became apparent that whilst there are libraries to help, C# coverage is very weak.

Looking at other language’s libraries, they tend to rely on lookup tables and regex. The tables are very difficult to make complete, and regex is quite an expensive operation.

There is also a more standards based way of doing this, which is known as Normalisation and is described in more detail on the Unicode website here. This describes the different forms of normalisation which I’m not going to go into here. What normalisation does, is split Unicode characters into the represented letter and a series of accents etc that follow (see the unicode link for more details).

Having got this string one then needs to remove the accent characters which should leave us with just the character and no accent. This is great in theory, unfortunately certain characters don’t map to a low Ascii character with normalise, so even with this approach one needs a lookup table for exceptions.

So the answer I’ve come to is based on two snippets which Jeff Atwood kindly shared on Stack Overflow. These are apparently the functions Stack Overflow uses for this very operation. You can find these here and here. Using these you have the warm glow of knowing they are production tested on a high volume site and performance is solid – note the lack of regex!

I have however, made a few changes:

  • The hyphens were appended in such a way that one could be added, and then need removing as it was the last character in the string. i.e. We never want “my-slug-“. This mean an extra string allocation. I’ve worked around this by delay-hyphening. If you compare my code to Jeff’s the logic for this is easy to follow.
  • His approach is purely lookup based and missed a lot of characters I found in examples whilst researching on stack overflow. To counter this, I first perform a normalisation pass, and then ignore any characters outside the acceptable ranges. This works most of the time…
  • …For when it doesn’t I’ve also had to add a lookup table. As mentioned above, some characters don’t map to a low ascii value when normalised. Rather than drop these I’ve got a manual list of exceptions that is doubtless full of holes, but better than nothing. The normalisation code was inspired by Jon Hanna’s great post here.
  • The case conversion is now also optional.

The upshot of all this, is that my version has better coverage than Jeff’s original and is a bit smarter with the hyphenation. I have a suspicion that the Microsoft implemented normalisation is likely to be slower than Jeff’s lookup table, so we are trading completeness for performance on that aspect. The hyphenation I would expect to be a bit faster, but not much as his extra string copy was only an edge case.

Anyway here’s the code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

	public static class Slug
	{
		public static string Create(bool toLower, params string[] values)
		{
			return Create(toLower, String.Join("-", values));
		}

		/// <summary>
		/// Creates a slug.
		/// Author: Daniel Harman, based on original code by Jeff Atwood
		/// References:
		/// http://www.unicode.org/reports/tr15/tr15-34.html
		/// http://meta.stackoverflow.com/questions/7435/non-us-ascii-characters-dropped-from-full-profile-url/7696#7696
		/// http://stackoverflow.com/questions/25259/how-do-you-include-a-webpage-title-as-part-of-a-webpage-url/25486#25486
		/// http://stackoverflow.com/questions/3769457/how-can-i-remove-accents-on-a-string
		/// </summary>
		/// <param name="toLower"></param>
		/// <param name="normalised"></param>
		/// <returns></returns>
		public static string Create(bool toLower, string value)
		{
			if (value == null) return "";

			var normalised = value.Normalize(NormalizationForm.FormKD);

			const int maxlen = 80;
			int len = normalised.Length;
			bool prevDash = false;
			var sb = new StringBuilder(len);
			char c;

			for (int i = 0; i < len; i++)
			{
				c = normalised[i];
				if ((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9'))
				{
					if (prevDash)
					{
						sb.Append('-');
						prevDash = false;
					}
					sb.Append(c);
				}
				else if (c >= 'A' && c <= 'Z')
				{
					if (prevDash)
					{
						sb.Append('-');
						prevDash = false;
					}
					// tricky way to convert to lowercase
					if (toLower)
						sb.Append((char)(c | 32));
					else
						sb.Append(c);
				}
				else if (c == ' ' || c == ',' || c == '.' || c == '/' || c == '\' || c == '-' || c == '_' || c == '=')
				{
					if (!prevDash && sb.Length > 0)
					{
						prevDash = true;
					}
				}
				else
				{
					string swap = ConvertEdgeCases(c, toLower);

					if (swap != null)
					{
						if (prevDash)
						{
							sb.Append('-');
							prevDash = false;
						}
						sb.Append(swap);
					}
				}

				if (sb.Length == maxlen) break;
			}

			return sb.ToString();
		}

		static string ConvertEdgeCases(char c, bool toLower)
		{
			string swap = null;
			switch (c)
			{
				case 'ı':
					swap = "i";
					break;
				case 'ł':
					swap = "l";
					break;
				case 'Ł':
					swap = toLower ? "l" : "L";
					break;
				case 'đ':
					swap = "d";
					break;
				case 'ß':
					swap = "ss";
					break;
				case 'ø':
					swap = "o";
					break;
				case 'Þ':
					swap = "th";
					break;
			}
			return swap;
		}
	}

and here are my mbunit tests:

	[TestFixture]
	public class When_Creating_Slug
	{
		[Test]
		[Row("ṃ,ỹ,ṛ,è,ş,ư,ḿ,ĕ", "m-y-r-e-s-u-m-e")]
		[Row("á-é-í-ó-ú", "a-e-i-o-u")]
		[Row("à,å,á,â,ä,ã,å,ą", "a-a-a-a-a-a-a-a")]
		[Row("è,é,ê,ë,ę", "e-e-e-e-e")]
		[Row("ì,í,î,ï,ı", "i-i-i-i-i")]
		[Row("ò,ó,ô,õ,ö,ø", "o-o-o-o-o-o")]
		[Row("ù,ú,û,ü", "u-u-u-u")]
		[Row("ç,ć,č", "c-c-c")]
		[Row("ż,ź,ž", "z-z-z")]
		[Row("ś,ş,š", "s-s-s")]
		[Row("ñ,ń", "n-n")]
		[Row("ý,Ÿ", "y-Y")]
		[Row("ł,Ł", "l-L")]
		[Row("đ", "d")]
		[Row("ß", "ss")]
		[Row("ğ", "g")]
		[Row("Þ", "th")]
		public void Should_Remove_Accents_Case_Invariant(string value, string expected)
		{
			var result = Slug.Create(false, value);
			
			Assert.AreEqual(expected, result);
		}

		[Test]
		[Row("ý,Ÿ", "y-y")]
		[Row("ł,Ł", "l-l")]
		public void Should_Remove_Accents_To_Lower(string value, string expected)
		{
			var result = Slug.Create(true, value);
			
			Assert.AreEqual(expected, result);
		}

		[Test]
		[Row("Slug Me ", "Slug-Me")]
		[Row("Slug Me,", "Slug-Me")]
		[Row("Slug Me.", "Slug-Me")]
		[Row("Slug Me/", "Slug-Me")]
		[Row("Slug Me\", "Slug-Me")]
		[Row("Slug Me-", "Slug-Me")]
		[Row("Slug Me_", "Slug-Me")]
		[Row("Slug Me=", "Slug-Me")]
		[Row("Slug Me--", "Slug-Me")]
		[Row("Slug Me---,", "Slug-Me")]
		public void Should_Remove_Trailing_Punctuation(string value, string expected)
		{
			var result = Slug.Create(false, value);

			Assert.AreEqual(expected, result);
		}
	}

After all this, I’ve now realised I don’t really need this code where I thought I did. My use case was to convert people’s names into slugs for a name directory, but having done all this work, I had a look at how facebook does it (after all no harm copying the industry leaders). Well I wish I’d done this first, as it turns out, they just leave the accents in the names when they display them in a directory! Oh well, I’m pretty sure they will normalise for searching to maximise matches, and this code is sound for article type slugification rather than names.

With thanks to Tom Chantler for the spotting the bug handling large whitespace strings.

Storing Custom Data in Forms Authentication Tickets

This article looks at storing custom data in asp.net forms authentication tickets. I recently updated the article to make the custom model binder generic, and add the necessary registration code which was missing from the first draft.

So you’ve decided to use FormsAuthentication, and perhaps enhanced it with your own custom providers. In your AccountController Login method you probably have a call along these lines:

FormsAuthentication.SetAuthCookie(account.Id.ToString(), model.RememberMe);

That all works great, but what if you need to store some extra data in the cookie. Perhaps the name you are passing into the AuthTicket isn’t actually the users name, but a GUID. Suddenly that built in ASP.Net login widget, in the top right of the page, doesn’t seem so great when it looks like this:

Hello 5D1D4743-9941-40B5-8931-6BC12617946C

What we need to do is store some extra data in that AuthTicket cookie right? That way we can keep the GUID as the authentication id, but still store things like the users first name in the cookie. Thus saving an expensive round trip to the db each time we render the widget.

Hmmm… whats this ‘UserData’ property we see on the AuthTicket? Perfect!

Erk… It’s read only?!?!?!

At least that’s how my thought process went.

So we need to make an authentication ticket ourselves:

var ticket = FormsAuthenticationTicket(int version, string name, DateTime issueDate,
	DateTime expiration, bool isPersistent, string userData, string cookiePath);

Unfortunately that’s quite a few more parameters than SetAuthCookie(…) required and they should be coming from the web.config rather than hard-coded.

On the plus side, there is access to the UserData!

To avoid losing the web.config driven settings, we can do a little trick and get FormsAuthentication to do the parsing for us. All we need to do is ask it for an AuthTicket and copy the settings from that into a new one we create.

To do this, a few steps are required. Firstly, after getting the ticket, we have to decrypt it, copy the data into a new ticket, and then make sure we encrypt that. Then we need to add it to the response.

Now before getting to the code, we should think about where it should live. It would seem logical to encapsulate this an extension method on FormsAuthentication, but being a static class we can’t. Instead we can attach it to HttpResponseBase which is not a bad home, especially as we have to add the cookie onto a response anyway. I’d recommend creating the following class in an ‘Infrastructure’ folder in your project:

	public static class HttpResponseBaseExtensions
	{
		public static int SetAuthCookie<T>(this HttpResponseBase responseBase, string name, bool rememberMe, T userData)
		{
			/// In order to pickup the settings from config, we create a default cookie and use its values to create a 
			/// new one.
			var cookie = FormsAuthentication.GetAuthCookie(name, rememberMe);
			var ticket = FormsAuthentication.Decrypt(cookie.Value);
			
			var newTicket = new FormsAuthenticationTicket(ticket.Version, ticket.Name, ticket.IssueDate, ticket.Expiration,
				ticket.IsPersistent, userData.ToJson(), ticket.CookiePath);
			var encTicket = FormsAuthentication.Encrypt(newTicket);

			/// Use existing cookie. Could create new one but would have to copy settings over...
			cookie.Value = encTicket;

			responseBase.Cookies.Add(cookie);

			return encTicket.Length;
		}
	}

There are a couple of things of note here, firstly we are accepting a generic type for the UserData, and secondly we are encoding it to Json!

Why? well lets think about the UserData field. Being on a cookie, this can only contain string data. Now we could do our own custom serialisation into this string, but my preference is to use JSON as it’s designed for the task. In this instance I’m using the serialiser from MongoDb as I happen to be using that in my project, but any Json serialiser will do. You might like to try the ServiceStack implementation for example.

I’m also returning the size of the cookie – cookies should never be longer than 4000 bytes as some browsers will just discard them. Its worth keeping an eye on this as it’s not just the size of your UserData but the other mandatory parts of the cookie too.

So let’s get this wired into our AccountController.

First we define a UserData class with a FirstName in it:

	public class UserData
	{
		public string FirstName { get; set; }

		public UserData()
		{
			FirstName = "Unknown";
		}
	}

Now here’s an example Login Action. There are some extras in here around validation, but you can use whatever approach here that fits your project.

[HttpPost]
		public ActionResult LogIn(AccountLoginVM model, string returnUrl)
		{
			try
			{
				if (ModelState.IsValid)
				{
					// Some code to validate and check authentication
					if (!Membership.ValidateUser(model.Email, model.Password))
						throw new RulesException("Incorrect username or password");

					Account account = _accounts.GetByEmail(model.Email);

					UserData userData = new UserData
					{
						FirstName = account.FirstName
					};

					Response.SetAuthCookie(account.Id.ToString(),
						model.RememberMe, userData);
				
					if (Url.IsLocalUrl(returnUrl))
					{
						return Redirect(returnUrl);
					}
					else
					{
						return RedirectToAction("Index", "Home");
					}
				}
			}
			catch (RulesException ex)
			{
				ex.CopyTo(ModelState);
			}

			model.Password = "";
			return View(model);
		}

That’s it. We’ve now got a cookie with our extra UserData in it.

Hang on… what about fixing that login widget in the top right?

One elegant way to crack this is to create a custom model binder, then if we swap the example widget from being a partial view to a partial action, all we need to do is demand a UserData object as an input param and the magic of binding will save us.

So, the custom model binder, again leveraging the MongoDb Json deserialiser:

	/// <summary>
	/// Binder to pull the UserData out for any actions that may want it.
	/// </summary>
	public class UserDataModelBinder<T> : IModelBinder
	{
		public object BindModel(ControllerContext controllerContext,
			ModelBindingContext bindingContext)
		{
			if (bindingContext.Model != null)
				throw new InvalidOperationException("Cannot update instances");
			if (controllerContext.RequestContext.HttpContext.Request.IsAuthenticated)
			{
				var cookie = controllerContext
					.RequestContext
					.HttpContext
					.Request
					.Cookies[FormsAuthentication.FormsCookieName];

				if (null == cookie)
					return null;

				var decrypted = FormsAuthentication.Decrypt(cookie.Value);

				if (!string.IsNullOrEmpty(decrypted.UserData))
					return BsonSerializer.Deserialize<T>(decrypted.UserData);
			}
			return null;
		}
	}

This is a generic so you can use whatever class suits to store the userdata. This then needs to be registered in Application_Start() in ‘Global.asax.cs’ :

ModelBinders.Binders.Add(typeof(UserData), new UserDataModelBinder<UserData>());

Now our login widget action, which passes a UserData object into our view (wrapped in a view model as we may not always want to pass all the UserData into the view).

		public ActionResult LoginWidget(UserData userData)
		{
			AccountLoginWidgetVM model = new AccountLoginWidgetVM();
			if (null != userData)
				model.UserData = userData;

			return PartialView(userData);
		}
@model TestProj.Web.Models.AccountLoginWidgetVM
         
@if(Request.IsAuthenticated) {
    <text>Welcome <b>@Model.UserData.FirstName</b>!
    [ @Html.ActionLink("Logout", "Logout", "Account") ]</text>
}
else {
...
}

We’ve covered quite a broad range of topics here, but hopefully its clear and of use. If you need any clarification leave a comment.

Next time… a change of tack. I’m going to look at how to get some performance out of a devexpress WPF grid.

Encrypting (Hashing) Passwords for your Website

In my previous blog entry which explains how to implement a custom MembershipProvider on an ASP.Net MVC web site, I mentioned the BCrypt library in passing. Today I’d like to dig into the library a little more and show you how and why you should consider using it.

Before we get into that though, what is what is ‘hashing’ and how is it different to ‘encryption’?

Encryption is part of a two way process in which data is encoded, but must be decryptable afterwards. For password storage, this decryption step is unnecessary and is actually a security vulnerability if someone gets hold of your keys.

In contrast, hashing is a one way transformation that is very hard to reverse. So to store passwords, a site would simply hash them before saving them to a database. Then when a user logs in, the password they supply is also hashed and compared to the hashed value in the database. If they typed in the same password the hashes will match and the authentication passes.

The advantage of this, is that the web site doesn’t need to decrypt a password at any point. All it ever does is hash them which is great for security.

There are various hashing algorithms out there, and many of them are wonderfully optimised and very fast. A typical example is SHA. An optimal hasher sounds great doesn’t it?

… well it isn’t. In fact it’s a weakness with the power of modern cpus and cloud computing resources. It means that its very possible and remarkably cheap to brute force SHA with a dictionary attack, simply because it is so fast.

This is where BCrypt comes in. It is intentionally slow, so should someone try to brute force it, its simply not viable. Whereas a hacker can generate 100s of millions of SHA hashes a second, they can’t with BCrypt, and as technology marches on, you can just increase the work factor on BCrypt to keep ahead of processors.

There are a few BCrypt.Net implementations floating around, and the one I’ve used I pulled down with nuget. To use it is as simple as:

private static readonly int BCRYPT_WORK_FACTOR = 10;
string hashedPassword = BCrypt.Net.BCrypt.HashPassword(account.HashedPassword, BCRYPT_WORK_FACTOR);
bool matched = BCrypt.Net.BCrypt.Verify(password, match.HashedPassword))

Every increment in work factor doubles the type the hash takes. Some of the .net implemtations struggle to get past 31 due to a bug, but you shouldn’t need a value anything like that high for a few years at least!

ASP.Net MVC 3 Custom Membership Provider with Repository Injection

In most serious ASP.NET MVC, or even legacy ASP.Net web sites, you are unlikely to want to use the default membership provider of ASP.Net. Its dependency on SQLServer and unhealthy predilection for littering databases with hundreds tables, just to support features you don’t care about, make it distinctly unattractive.

What we really want is to integrate our web site’s security with the project’s schema and bind directly to a table or repository encapsulating the users model for the site. The way to do this is through the implementation of a custom MembershipProvider.

This may seem a little daunting, but in practise is fairly simple. In fact, all we need do is override a pair of methods, on a couple of abstract classes, and all authentication and role checking will be routed to our code. Even better, by leveraging the well-tested and robust ASP.NET security facilities, we can still utilise the convenience and security of ASP.Net’s attribute based security to protect your controllers and controller methods. If you aren’t familiar with these, it’s as simple as attaching an Authorise tag, and optionally specifying a role the user must have to gain access e.g.

public class MemberController : Controller
{
	IAccountRepository repo;

	public MemberController(IAccountRepository accountRepository)
	{
		repo = accountRepository;
	}

	[Authorize]
	public ActionResult Index()
	{
...
	}

	[Authorize(Roles = "Admin")]
	public ActionResult Delete(int accountId)
	{
...
	}
}

Just being authorised means the user has been authenticated i.e. logged in. The role is supplementary to this and allows you finer grained access control. Easy huh?

So, firstly authentication. To take control of this, we have to create a class derived from the abstract class MembershipProvider.

Unfortunately, being a hangover from old-school ASP.net, wiring this in is a little more clumsy than one might expect. It predates the pluggable design pattern applied throughout the MVC platform. The upshot being, that if you want to use the repository pattern with it, you can’t pass a repository into the constructor as ASP.Net instantiates the class for you, and only knows how to do this through a default constructor.

One way to work around this is to make your repository a property on the class and update it after the framework has constructed it. In this example I’m going to use ninject as my DI framework, but you could use a different one, or in fact just set the property without using DI at all.

First lets start with an example membership provider. I’ve only bothered to override ValidateUser() as the other methods aren’t required to leverage ASP.NET’s integrated security features.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Security;
using Ninject;

namespace MyProj.Web.Infrastructure
{
	public class AccountMembershipProvider : MembershipProvider
	{
		[Inject]
		public IAccountRepository AccountRepository { get; set; }

		public override string ApplicationName
		{
			get
			{
				throw new NotImplementedException();
			}
			set
			{
				throw new NotImplementedException();
			}
		}

... lots of unimplemented overrides...

		public override void UpdateUser(MembershipUser user)
		{
			throw new NotImplementedException();
		}

		public override bool ValidateUser(string username, string password)
		{
			return AccountRepository.IsValidLogin(username, password);
		}
	}
}

You’ll notice there is an AccountRepository being used here to validate the user login. This is where your custom authorisation logic goes – as such I’m not going to provide an implementation here as it will depend on the specifics of your site.

Next, we need to ensure that the AccountRepository is injected into our MembershipProvider, and the place to do this is Application_Start() in Gloabl.asax.

	internal class MyNinjectModules : NinjectModule
	{
		public override void Load()
		{
			Bind<IAccountRepository>()
				.To<AccountRepository>();
		}
	}

	public class MvcApplication : System.Web.HttpApplication
	{
		private IKernel _kernel = new StandardKernel(new MyNinjectModules());

...code deleted....

		protected void Application_Start()
		{
			AreaRegistration.RegisterAllAreas();

			RegisterGlobalFilters(GlobalFilters.Filters);
			RegisterRoutes(RouteTable.Routes);

			// Inject account repository into our custom membership provider.
			_kernel.Inject(Membership.Provider);
		}
	}

You’ll notice that I’m grabbing the framework instantiated instance of our membership provider, and using ninject to set the repository property on our custom membership provider.

I’m also setting up a ninject controller factory which is a great class when you want to inject repositories into your controller’s constructors. This is based on code found in one of the best programming books I’ve ever read Pro ASP.NET MVC 2 Framework by Steven Sanderson. This book covers so much more than ASP.NET MVC, it teaches you how to design and build applications for testability, and is the most digestable explanation of modern test driven design I’ve ever come across. It’s worth a read even if you aren’t an MVC programmer! n.b. I believe a revised MVC3 edition is to be released shortly.

public class NinjectControllerFactory : DefaultControllerFactory
	{
		private IKernel _kernel;

		public NinjectControllerFactory(IKernel kernel)
		{
			_kernel = kernel;
		}

		protected override IController GetControllerInstance(System.Web.Routing.RequestContext requestContext, Type controllerType)
		{
			if (controllerType == null)
				return null;
		 
			return (IController)_kernel.Get(controllerType);
		}
	}

Now… what if we want to create a custom role provider too? Well that’s easy. Here is an exampe RoleProvider:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Security;
using Ninject;

namespace MyProj.Web.Infrastructure
{
	public class AccountRoleProvider : RoleProvider
	{
		[Inject]
		public IAccountRepository AccountRepository { get; set; }

		public override void AddUsersToRoles(string[] usernames, string[] roleNames)
		{
			throw new NotImplementedException();
		}

... lots of unimplemented overrides...

		public override string[] GetRolesForUser(string id)
		{
			return AccountRepository.GetRoles(id);
		}

... lots of unimplemented overrides...

		public override bool RoleExists(string roleName)
		{
			throw new NotImplementedException();
		}
	}
}

Again, you really don’t need to override much of the RoleProvider abstract class, simply implementing GetRolesForUser() and ensuring it returns a string array of the given users roles will suffice.

To inject the membership provider juat change the previous code version of Application_Start() to:

	public class MvcApplication : System.Web.HttpApplication
	{
		private IKernel _kernel = new StandardKernel(new MyNinjectModules());

...code deleted....

		protected void Application_Start()
		{
			AreaRegistration.RegisterAllAreas();

			RegisterGlobalFilters(GlobalFilters.Filters);
			RegisterRoutes(RouteTable.Routes);

			// Inject account repository into our custom membership & role providers.
			_kernel.Inject(Membership.Provider);
			_kernel.Inject(Roles.Provider);
		}
	}

Finally we need to register our providers in web.config:

<configuration>
...
  <system.web>
...
    <membership defaultProvider="AccountMembershipProvider">
      <providers>
        <clear/>
        <add name="AccountMembershipProvider"
             type="MyProj.Web.Infrastructure.AccountMembershipProvider" />
      </providers>
    </membership>

    <roleManager enabled="true" defaultProvider="AccountRoleProvider">
      <providers>
        <clear/>
        <add name="AccountRoleProvider"
             type="MyProj.Web.Infrastructure.AccountRoleProvider" />
      </providers>
    </roleManager>
...
  </system.web>
...  
</configuration>

It really us as simple as that!

One other note, if you are implementing your own password storage, make sure you hash them! (and I recommend you look at bcrypt for that).

ReactiveUI Property Name to Field Mapping Convention Override

In a previous post I described how you can use the ReactiveUI framework to bind properties to both INotifyPropertyChanged and make them Observable to Reactive Extensions (Rx).

The default binding behaviour of the RaiseAndSetIfChanged helper is to assume that a property named “MyProperty” has a backing field called “_MyProperty”. This is not a common convention, and whilst you can explicitly override this behaviour in RaiseAndSetIfChanged, it’s extra work.

Fortunately, the framework offers a way of modifying the default mapping behaviour by binding a different function to RxApp.GetFieldNameForPropertyNameFunc.

Here are a couple of example functions that implement alternative behaviours. The first of which maps to _propertyName and the latter, simply topropertyName.

                /// Maps property name MyProperty to field name _myProperty.
                Func<string, string> UnderscoreFirstToLowerBehaviour = x =>
                {
                        char[] arr = x.ToCharArray();
                        arr[0] = char.ToLower(arr[0]);
                        return '_' + new String(arr);
                };

                /// Maps property name MyProperty to field name myProperty.
                Func<string, string> FirstToLowerBehaviour = x =>
                {
                        char[] arr = x.ToCharArray();
                        arr[0] = char.ToLower(arr[0]);
                        return new String(arr);
                };

To use one of these behaviours, all you need do is call the following once during app startup:

RxApp.GetFieldNameForPropertyNameFunc = underscoreFirstToLowerBehaviour;

Given this is executed on each notification, I guess the second coding style is more optimal!

p.s. I’m not sure these are the fastest implementations of these conversions, and I’m not going to benchmark, however I did base the code on http://www.dotnetperls.com/uppercase-first-letter who did do some benchmarking on similar code.