Newton's 3rd law... hitting drywall (which I break) vs hitting a brick (which breaks me)?

You've caught a non-intuitive part of Newton's 3rd law. It's actually applying in the case you mention, but because the objects involved are of dissimilar hardness it's easy to perceive the impact as a violation of the law.

Impacts are actually really complicated. Consider this slow motion video of a punch to the gut. We won't be able to cover all of the complexities we see here, but we can layer a few of them together to try to explain why the non-inutitive results you get are actually correct applications of Newton's 3rd law.

The key thing which makes impacts so complicated is that we have to pay attention to momentum. When you punch the brick wall or the drywall, your hand has quite a lot of momentum. When you punch the brick wall, that momentum has to be stopped. The only way to do this is through the reactionary force of the wall pushing back on your hand. The more momentum your hand has, the more reactionary force you deal with. In your brick example, that reactionary force is 50lbs, and the corresponding force of your hand on the wall is also 50lbs.

In the drywall case, we need to make a few adjustments. The first is to note that your hand goes through the drywall. It does not have to be stopped by the wall. This points out that the reactionary force will be less than it was in the brick wall case, because the brick wall had to stop the fist.

Well, almost. I cheated slightly, and that cheat may be a source of non-intuitive behavior. The more correct statement is that the brick wall had to impart more impulse to your hand, because it had to stop your hand. Impulse is force*time, and is a measure of change in momentum. The key detail here is that the distance can change. If you drop a superball on the ground, it rebounds almost back to where you threw it from. The impulse applied to the ball by the ground is very high. Contrast that with a steel ball bearing with the same mass as the superball, which does not rebound as much. The impulse applied to the bearing is lower. However, the superball deforms a great deal on impact, so it has a longer time to apply that impulse over. It is reasonable that superball could be subjected to less force than the ball bearing, and yet bounce higher because that lower force was applied for a longer distance.

In the case of the punch, we're lucky that 99% of the deformation in your punch occurs in your hand. Your skin and fat squish out of the way until your bones start to have to move. The shock in theory works its way all the way up your arm. However, we can ignore all of that for now, because we're just doing comparisons. It's the same hand in both the brick wall punch and the drywall punch, so it can be expected to deform in similar ways over similar distances and similar times. This is how we can claim that the brick wall punch must have a higher force. We know the impulse must be higher (because it stopped your hand, and the drywall punch didn't), and the times are the same for the reaction to both punches, so the brick wall punch must have more force.

Thus, the truth is that you did not punch the drywall with 50lbs. You actually supplied less force than that. In fact, you supplied just enough force to break the internal bonds that were keeping the drywall solid. Intuitively, we like to measure punches in forces (claiming a 50lb punch), but it's actually not possible to punch that hard unless the thing being punched is capable of providing a corresponding reactionary force! If you layered enough pieces of drywall to have the structural integrity to provide 50lbs of force, you would find that you don't break through, and it hurts almost as much as the bricks did (the first sheet of drywall will deform a little, so it wont hurt as much as the brick)

The issue of breaking through the wall is actually a very important thing for martial artists. Those who break boards or bricks in demonstrations all know that it hurts far more if you fail to break the board or brick. That's because the board stopped all of your forward momentum, meaning you had a lot of impulse over a short time, meaning a lot of force. If you break the brick, the reactionary forces don't stop your hand, so they are less. I would wager that the greatest challenge of breaking bricks with a karate chop is not breaking them, but in having conditioned your body and mind such that you can withstand the impulse when you fail to break them.

What makes you think that the maximum force you applied to the dry wall was anything like the maximum force you applied to the brick? It certainly wasn't. The dry wall gave way well before you were able to attain the same force as applied to the brick. Try punching the air and see how much force you are able to apply. The experimental evidence that the force you applied to the brick (and it applied to you) was higher than the force you applied to the dry wall was your injured hand.

There is no doubt the Newton's third law holds in this case. The source of confusion is the fact that you are neglecting the time interval of the collision as well as the momentum change the colliding body. As we shall see it is incorrect to assume you applied the same force in both cases just because you started with the same initial conditions, i.e. the same speed.

The best way to approach this problem is by considering the momentum principle. By Newton's second law we obtain the momentum change of the body after the collision, $$\Delta p=\int_{t_1}^{t_2} F\mathrm dt,$$ where $F$ is the force the wall does on the body and $\Delta t=t_2-t_1$ is the time interval of the collision.

Now let us assume you run with a given speed against a wall. Since the time interval of the collision is quite short we can approximate $$\Delta p= F_{\textrm{av}}\Delta t,$$ where $F_{\textrm{av}}$ is the average value of the force during the collision. Hence $$|F_{\textrm{av}}|=\frac{|\Delta p|}{\Delta t},$$ where we took the absolute value just for simplicity. What matters is that the greater the speed change the greater the force. The shorter the time of the collision the greater the force.

When hitting an unbreakable wall you at least stay at rest right after the collision so the momentum change is $\Delta p=0-mv_i$ where $v_i$ is the (initial) speed right before hitting the wall. The average force the wall does on you is $$|F_{\textrm{av}}|=\frac{mv_i}{\Delta t}.$$

If you run against a breakable wall instead your momentum change $\Delta p=mv_f-mv_i$, where $v_f>0$ is the (final) speed right after the collision, since you still go forwards right after you break the wall. The time interval of the collision is $\Delta t'$ which is greater than $\Delta t$, since you did not suddenly stop. Then the average force $F_{\textrm{av}}'$ the breaking wall does on the body is $$|F_{\textrm{av}}'|=\frac{m|v_i-v_f|}{\Delta t'}<\frac{mv_i}{\Delta t}=|F_{\textrm{av}}|.$$ Therefore even hitting the walls in the same way, the forces the wall applied on you are different. It is lesser in the second case. Notice that this is the same as saying the average force you applied to the breakable wall is lesser than the one you applied to the unbreakable wall.

It is also interesting to note that this question provides and example of the far more usefulness of the concept of linear momentum than of the Newton's third law. The latter is contained in the former but it can easily leads to misconceptions such as the one showed in this question or this one.