Is adding to a "char *" pointer UB, when it doesn't actually point to a char array?

The addition is intended to be valid, but I do not believe the standard manages to say so clearly enough. Quoting N4140 (roughly C++14):

3.9 Types [basic.types]

2 For any object (other than a base-class subobject) of trivially copyable type T, whether or not the object holds a valid value of type T, the underlying bytes (1.7) making up the object can be copied into an array of char or unsigned char.⁴² [...]

^{42) By using, for example, the library functions (17.6.1.2) std::memcpy or std::memmove.}

It says "for example" because std::memcpy and std::memmove are not the only ways in which the underlying bytes are intended to be allowed to be copied. A simple for loop which copies byte by byte manually is supposed to be valid as well.

In order for that to work, addition has to be defined for pointers to the raw bytes that make up an object, and the way definedness of expressions works, the addition's definedness cannot depend on whether the addition's result will subsequently be used to copy the bytes into an array.

Whether that means those bytes form an array already or whether this is a special exception to the general rules for the + operator that is somehow omitted in the operator description, is not clear to me (I suspect the former), but either way would make the addition you're performing in your code valid.

Any interpretation that disallows the intended usage of offsetof must be wrong:

#include <assert.h>
#include <stddef.h>
struct S { float a, b, c; };

const size_t idx_S[] = {
    offsetof(struct S, a),
    offsetof(struct S, b),
    offsetof(struct S, c),
};

float read_S(struct S *sp, unsigned int idx)
{
    assert(idx < 3);
    return *(float *)(((char *)sp) + idx_S[idx]); // intended to be valid
}

However, any interpretation that allows one to step past the end of an explicitly-declared array must also be wrong:

#include <assert.h>
#include <stddef.h>
struct S { float a[2]; float b[2]; };

static_assert(offsetof(struct S, b) == sizeof(float)*2,
    "padding between S.a and S.b -- should be impossible");

float read_S(struct S *sp, unsigned int idx)
{
    assert(idx < 4);
    return sp->a[idx]; // undefined behavior if idx >= 2,
                       // reading past end of array
}

And we are now on the horns of a dilemma, because the wording in both the C and C++ standards, that was intended to disallow the second case, probably also disallows the first case.

This is commonly known as the "what is an object?" problem. People, including members of the C and C++ committees, have been arguing about this and related issues since the 1990s, and there have been multiple attempts to fix the wording, and to the best of my knowledge none has succeeded (in the sense that all existing "reasonable" code is rendered definitely conforming and all existing "reasonable" optimizations are still allowed).

(Note: All of the above code is written as it would be written in C to emphasize that the same problem exists in both languages, and can be encountered without the use of any C++ constructs.)