Optimising datastructure/word alignment padding in golang

It depends on type of application that you are developing and on usage of those structures. If application needs to meet some memory/performance criteria you definitely should care about memory alignment and paddings, but not only - there is nice article https://www.usenix.org/legacy/publications/library/proceedings/als00/2000papers/papers/full_papers/sears/sears_html/index.html that highlights theme of optimal CPU caches usage and correlation between struct layouts and performance. It highlights cache line alignment, false sharing, etc.

Also there is a nice golang tool https://github.com/1pkg/gopium that helps to automate those optimizations, check it out!

Or shouldn't I be worried about this at all?

Yes you should.
This is also called mechanical sympathy (see this Go Time podcast episode), so it also depends on the hardware architecture you are compiling for.

See as illustration:

• "The day byte alignment came back to bite me" (January 2014)
• "On the memory alignment of Go slice values" (July 2016)

The values in Go slices are 16-byte aligned. They are not 32 byte aligned.
Go pointers are byte-aligned.

Currently there's no compile-time optimisation; the values are padded to 8 bytes on x64.

You can manually arrange structs to optimally utilise space; typically by going from larger types to smaller; 8 consecutive byte fields for example, will only use 8 bytes, but a single byte would be padded to an 8 byte alignment, consider this: https://play.golang.org/p/0qsgpuAHHp

package main

import (
    "fmt"
    "unsafe"
)

type Compact struct {
    a, b                   uint64
    c, d, e, f, g, h, i, j byte
}

// Larger memory footprint than "Compact" - but less fields!
type Inefficient struct {
    a uint64
    b byte
    c uint64
    d byte
}

func main() {
    newCompact := new(Compact)
    fmt.Println(unsafe.Sizeof(*newCompact))
    newInefficient := new(Inefficient)
    fmt.Println(unsafe.Sizeof(*newInefficient))
}

If you take this into consideration; you can optimise the memory footprint of your structs.