Why does re.sub replace the entire pattern, not just a capturing group within it?

Because it's supposed to replace the whole occurrence of the pattern:

Return the string obtained by replacing the leftmost non-overlapping occurrences of the pattern in string by the replacement repl.

If it were to replace only some subgroup, then complex regexes with several groups wouldn't work. There are several possible solutions:

1. Specify pattern in full: re.sub('ab', 'ad', 'abc') - my favorite, as it's very readable and explicit.
2. Capture groups which you want to preserve and then refer to them in the pattern (note that it should be raw string to avoid escaping): re.sub('(a)b', r'\1d', 'abc')
3. Similar to previous option: provide a callback function as repl argument and make it process the Match object and return required result.
4. Use lookbehinds/lookaheds, which are not included in the match, but affect matching: re.sub('(?<=a)b', r'd', 'abxb') yields adxb. The ?<= in the beginning of the group says "it's a lookahead".

I'm aware that this is not strictly answering the OP question, but this question can be hard to google (flooded by \1 explanation ...)

for those who like me came here because they'd like to actually replace a capture group that is not the first one by a string, without special knowledge of the string nor of the regex :

#find offset [start, end] of a captured group within string
r = regex.search(oldText).span(groupNb)
#slice the old string and insert replacementText in the middle 
newText = oldText[:r[0]] + replacementText + oldText[r[1]:]

I know it's the wanted behavior, but I still do not understand why re.sub can't specify the actual capture group that it should substitute on...

Because that's exactly what re.sub() doc tells you it's supposed to do:

• the pattern 'a(b)' says "match 'a', with optional trailing 'b'". (It could match 'a' on its own, but there is no way it could ever match 'b' on its own as you seem to expect. If you meant that, use a non-greedy (a)??b).
• the replacement-string is 'd'
• hence on your string 'abc', it matches all of 'ab' and replaces it with 'd', thus result is 'dc'

If you want your desired output, you'd need a non-greedy match on the '(a)??':

>>> re.sub('(a)??b','d','abc')
'dc'