x86 Assembly: INC and DEC instruction and overflow flag

Intel® 64 and IA-32 Architectures Software Developer's Manuals

Look at the appropriate manual Instruction Set Reference, A-M. Every instruction is precisely documented.

Here is the INC section on affected flags:

The CF flag is not affected. The OF, SZ, ZF, AZ, and PF flags are set according to the result.

The overflow flag is set when an operation would cause a sign change. Your code is very close. I was able to set the OF flag with the following (VC++) code:

char ovf = 0;

_asm {
    mov bh, 127
    inc bh
    seto ovf
}
cout << "ovf: " << int(ovf) << endl;

When BH is incremented the MSB changes from a 0 to a 1, causing the OF to be set.

This also sets the OF:

char ovf = 0;

_asm {
    mov bh, 128
    dec bh
    seto ovf
}
cout << "ovf: " << int(ovf) << endl;

Keep in mind that the processor does not distinguish between signed and unsigned numbers. When you use 2's complement arithmetic, you can have one set of instructions that handle both. If you want to test for unsigned overflow, you need to use the carry flag. Since INC/DEC don't affect the carry flag, you need to use ADD/SUB for that case.

try changing your test to pass in the number rather than hard code it, then have a loop that tries all 256 numbers to find the one if any that affects the flag. Or have the asm perform the loop and exit out when it hits the flag and or when it wraps around to the number it started with (start with something other than 0x00, 0x7f, 0x80, or 0xFF).

EDIT

.globl inc
inc:
    mov $33, %eax

top:
    inc %al
    jo done
    jmp top

done:
    ret

.globl dec
dec:
    mov $33, %eax

topx:
    dec %al
    jo donex
    jmp topx

donex:
    ret

Inc overflows when it goes from 0x7F to 0x80. dec overflows when it goes from 0x80 to 0x7F, I suspect the problem is in the way you are using inline assembler.

As many of the other answers have pointed out, INC and DEC do not affect the CF, whereas ADD and SUB do.

What has not been said yet, however, is that this might make a performance difference. Not that you'd usually be bothered by that unless you are trying to optimise the hell out of a routine, but essentially not setting the CF means that INC/DEC only write to part of the flags register, which can cause a partial flag register stall, see Intel 64 and IA-32 Architectures Optimization Reference Manual or Agner Fog's optimisation manuals.