Efficiently finding the 1st order neighbors of 200k polygons
For a solution avoiding ArcGIS, use pysal. You could get the weights directly from shapefiles using:
w = pysal.rook_from_shapefile("../pysal/examples/columbus.shp")
or
w = pysal.queen_from_shapefile("../pysal/examples/columbus.shp")
Head for the docs for more info.
Just an update. After following Whuber's advice, I found that the Generate Spatial Weights Matrix simply uses Python loops and dictionaries to determine neighbors. I reproduced the process below.
The first part loops through every vertex of every block group. It creates a dictionary with vertex coordinates as the keys and a list of block group IDs that have a vertex at that coordinate as the value. Note that this requires a topologically neat dataset, as only perfect vertex/vertex overlap will register as a neighbor relationship. Fortunately the Census Bureau's TIGER block group shapefiles are OK in this regard.
The second part loops through every vertex of every block group again. It creates a dictionary with block group IDs as the keys and that block group's neighbor IDs as the values.
# Create dictionary of vertex coordinate : [...,IDs,...]
BlockGroupVertexDictionary = {}
BlockGroupCursor = arcpy.SearchCursor(BlockGroups.shp)
BlockGroupDescription = arcpy.Describe(BlockGroups.shp)
BlockGroupShapeFieldName = BlockGroupsDescription.ShapeFieldName
#For every block group...
for BlockGroupItem in BlockGroupCursor :
BlockGroupID = BlockGroupItem.getValue("BKGPIDFP00")
BlockGroupFeature = BlockGroupItem.getValue(BlockGroupShapeFieldName)
for BlockGroupPart in BlockGroupFeature:
#For every vertex...
for BlockGroupPoint in BlockGroupPart:
#If it exists (and isnt empty interior hole signifier)...
if BlockGroupPoint:
#Create string version of coordinate
PointText = str(BlockGroupPoint.X)+str(BlockGroupPoint.Y)
#If coordinate is already in dictionary, append this BG's ID
if PointText in BlockGroupVertexDictionary:
BlockGroupVertexDictionary[PointText].append(BlockGroupID)
#If coordinate is not already in dictionary, create new list with this BG's ID
else:
BlockGroupVertexDictionary[PointText] = [BlockGroupID]
del BlockGroupItem
del BlockGroupCursor
#Create dictionary of ID : [...,neighbors,...]
BlockGroupNeighborDictionary = {}
BlockGroupCursor = arcpy.SearchCursor(BlockGroups.shp)
BlockGroupDescription = arcpy.Describe(BlockGroups.shp)
BlockGroupShapeFieldName = BlockGroupDescription.ShapeFieldName
#For every block group
for BlockGroupItem in BlockGroupCursor:
ListOfBlockGroupNeighbors = []
BlockGroupID = BlockGroupItem.getValue("BKGPIDFP00")
BlockGroupFeature = BlockGroupItem.getValue(BlockGroupShapeFieldName)
for BlockGroupPart in BlockGroupFeature:
#For every vertex
for BlockGroupPoint in BlockGroupPart:
#If it exists (and isnt interior hole signifier)...
if BlockGroupPoint:
#Create string version of coordinate
PointText = str(BlockGroupPoint.X)+str(BlockGroupPoint.Y)
if PointText in BlockGroupVertexDictionary:
#Get list of block groups that have this point as a vertex
NeighborIDList = BlockGroupVertexDictionary[PointText]
for NeighborID in NeighborIDList:
#Don't add if this BG already in list of neighbors
if NeighborID in ListOfBGNeighbors:
pass
#Add to list of neighbors (as long as its not itself)
elif NeighborID != BlockGroupID:
ListOfBGNeighbors.append(NeighborID)
#Store list of neighbors in blockgroup object in dictionary
BlockGroupNeighborDictionary[BlockGroupID] = ListOfBGNeighbors
del BlockGroupItem
del BlockGroupCursor
del BlockGroupVertexDictionary
In hindsight I realize I could have used a different method for the second part that didn't require looping through the shapefile again. But this is what I used, and it works pretty well even for 1000s of block groups at a time. I haven't tried doing it with the whole USA, but it can execute for an entire state.
If you have access to ArcGIS 10.2 for Desktop, or possibly earlier, then I think the Polygon Neighbors (Analysis) tool which:
Creates a table with statistics based on polygon contiguity (overlaps, coincident edges, or nodes).
may make this task much easier now.