How is this bitwise AND operator masking the lower seven order bits of the number?

The number 0177 is an octal number representing the binary pattern below:

0000000001111111

When you AND it using the bitwise operation &, the result keeps the bits of the original only in the bits that are set to 1 in the "mask"; all other bits become zero. This is because "AND" follows this rule:

X & 0 -> 0 for any value of X
X & 1 -> X for any value of X

For example, if you AND 0177 and 0545454, you get

0000000001111111 -- 0000177
0101010101010101 -- 0545454
----------------    -------
0000000001010101 -- 0000154

In C an integer literal prefixed with 0 is an octal number so 0177 is an octal number.

Each octal digit (of value 0 to 7) is represented with 3 bits and 7 is the greatest value for each digit. So a value of 7 in octal means 3 bits set.

Since 0177 is an octal literal and each octal number is 3 three bits you have, the following binary equivalents:

7  = 111
1  = 001

Which means 0177 is 001111111 in binary.