Sql Server - Index on nvarchar field

You could use CHECKSUM command and put index on column with checksum.

--*** Add extra column to your table that will hold checksum
ALTER TABLE Production.Product
ADD cs_Pname AS CHECKSUM(Name);
GO

--*** Create index on new column
CREATE INDEX Pname_index ON Production.Product (cs_Pname);
GO

Then you can retrieve data fast using following query:

SELECT * 
FROM Production.Product
WHERE CHECKSUM(N'Bearing Ball') = cs_Pname
AND Name = N'Bearing Ball';

Here is the documentation: http://technet.microsoft.com/en-us/library/ms189788.aspx

You could create a hash code of the url and use this integer as a unique index on your db. Beware of converting all characters to lowercase first to ensure that all url are in the same format. Same url will generate equal hash code.

You can use a hash function (although theoretically it doesn't guarantee that two different titles will have different hashes, but should be good enough: MD5 Collisions) and then apply the index on that column.

MD5 in SQL Server