Multiple join conditions using the $lookup operator
Starting Mongo 4.4
, we can achieve this type of "join" with the new $unionWith
aggregation stage coupled with a classic $group
stage:
// > db.collection1.find()
// { "user1" : 1, "user2" : 2, "percent" : 0.56 }
// { "user1" : 4, "user2" : 3, "percent" : 0.14 }
// > db.collection2.find()
// { "user1" : 1, "user2" : 2, "percent" : 0.3 }
// { "user1" : 2, "user2" : 3, "percent" : 0.25 }
db.collection1.aggregate([
{ $set: { percent1: "$percent" } },
{ $unionWith: {
coll: "collection2",
pipeline: [{ $set: { percent2: "$percent" } }]
}},
{ $group: {
_id: { user1: "$user1", user2: "$user2" },
percents: { $mergeObjects: { percent1: "$percent1", percent2: "$percent2" } }
}}
])
// { _id: { user1: 1, user2: 2 }, percents: { percent1: 0.56, percent2: 0.3 } }
// { _id: { user1: 2, user2: 3 }, percents: { percent2: 0.25 } }
// { _id: { user1: 4, user2: 3 }, percents: { percent1: 0.14 } }
This:
Starts with a union of both collections into the pipeline via the new
$unionWith
stage:- We first rename
percent
fromcollection1
topercent1
(using a$set
stage) - Within the
$unionWith
stage, we specify apipeline
on thecollection2
in order to also renamepercent
this time topercent2
. - This way, we can differentiate the percentage field's origin.
- We first rename
Continues with a
$group
stage that:- Groups records based on
user1
anduser2
- Accumulate percentages via a
$mergeObjects
operation. Using$first: "$percent1"
and$first: "$percent2"
wouldn't work since this could potentially takenull
first (for elements from the other collection). Whereas$mergeObjects
discardsnull
values.
- Groups records based on
If you need a different output format, you can add a downstream $project
stage.
We can do multiple join conditions with the $lookup
aggregation pipeline operator in version 3.6 and newer.
We need to assign the fields's values to variable using the let
optional field; you then access those variables in the pipeline
field stages where you specify the pipeline to run on the collections.
Note that in the $match
stage, we use the $expr
evaluation query operator to compare the fields's value.
The last stage in the pipeline is the $replaceRoot
aggregation pipeline stage where we simply merge the $lookup
result with part of the $$ROOT
document using the $mergeObjects
operator.
db.collection2.aggregate([
{
$lookup: {
from: "collection1",
let: {
firstUser: "$user1",
secondUser: "$user2"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$user1",
"$$firstUser"
]
},
{
$eq: [
"$user2",
"$$secondUser"
]
}
]
}
}
}
],
as: "result"
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects:[
{
$arrayElemAt: [
"$result",
0
]
},
{
percent1: "$$ROOT.percent1"
}
]
}
}
}
]
)
This pipeline yields something that look like this:
{
"_id" : ObjectId("59e1ad7d36f42d8960c06022"),
"user1" : 1,
"user2" : 2,
"percent" : 0.3,
"percent1" : 0.56
}
If you are not on version 3.6+, you can first join using one of your field let say "user1" then from there you unwind the array of the matching document using the $unwind
aggregation pipeline operator. The next stage in the pipeline is the $redact
stage where you filter out those documents where the value of "user2" from the "joined" collection and the input document are not equal using the $$KEEP
and $$PRUNE
system variables. You can then reshape your document in $project
stage.
db.collection1.aggregate([
{ "$lookup": {
"from": "collection2",
"localField": "user1",
"foreignField": "user1",
"as": "collection2_doc"
}},
{ "$unwind": "$collection2_doc" },
{ "$redact": {
"$cond": [
{ "$eq": [ "$user2", "$collection2_doc.user2" ] },
"$$KEEP",
"$$PRUNE"
]
}},
{ "$project": {
"user1": 1,
"user2": 1,
"percent1": "$percent",
"percent2": "$collection2_doc.percent"
}}
])
which produces:
{
"_id" : ObjectId("572daa87cc52a841bb292beb"),
"user1" : 1,
"user2" : 2,
"percent1" : 0.56,
"percent2" : 0.3
}
If the documents in your collections have the same structure and you find yourself performing this operation often, then you should consider to merge the two collections into one or insert the documents in those collections into a new collection.
db.collection3.insertMany(
db.collection1.find({}, {"_id": 0})
.toArray()
.concat(db.collection2.find({}, {"_id": 0}).toArray())
)
Then $group
your documents by "user1" and "user2"
db.collection3.aggregate([
{ "$group": {
"_id": { "user1": "$user1", "user2": "$user2" },
"percent": { "$push": "$percent" }
}}
])
which yields:
{ "_id" : { "user1" : 1, "user2" : 2 }, "percent" : [ 0.56, 0.3 ] }
If you're trying to model your data, and came here to check if mongodb can perform joins on multiple fields before deciding to do so, please read on.
While MongoDB can perform joins, you also have the freedom to model data according to your application access pattern. If the data is as simple as presented in the question, we can simply maintain a single collection that looks like this:
{
user1: 1,
user2: 2,
percent1: 0.56,
percent2: 0.3
}
Now you can perform all the operations on this collection you would have performed by joining. Why are we trying to avoid joins? Because they are not supported by sharded collections (docs), which will stop you from scaling out when needed. Normalizing data (having separate tables/collections) works very well in SQL, but when it comes to Mongo, avoiding joins can offer advantages without consequences in most cases. Use normalization in MongoDB only when you have no other choice. From the docs:
In general, use normalized data models:
- when embedding would result in duplication of data but would not provide sufficient read performance advantages to outweigh the implications of the duplication.
- to represent more complex many-to-many relationships.
- to model large hierarchical data sets.
Check here to read more about embedding and why you would choose it over normalization.