Are there any modern CPUs where a cached byte store is actually slower than a word store?
My guess was wrong. Modern x86 microarchitectures really are different in this way from some (most?) other ISAs.
There can be a penalty for cached narrow stores even on high-performance non-x86 CPUs. The reduction in cache footprint can still make int8_t arrays worth using, though. (And on some ISAs like MIPS, not needing to scale an index for an addressing mode helps).
Merging / coalescing in the store buffer between byte stores instructions to the same word before actual commit to L1d can also reduce or remove the penalty. (x86 sometimes can't do as much of this because its strong memory model requires all stores to commit in program order.)
ARM's documentation for Cortex-A15 MPCore (from ~2012) says it uses 32-bit ECC granularity in L1d, and does in fact do a word-RMW for narrow stores to update the data.
The L1 data cache supports optional single bit correct and double bit detect error correction logic in both the tag and data arrays. The ECC granularity for the tag array is the tag for a single cache line and the ECC granularity for the data array is a 32-bit word.
Because of the ECC granularity in the data array, a write to the array cannot update a portion of a 4-byte aligned memory location because there is not enough information to calculate the new ECC value. This is the case for any store instruction that does not write one or more aligned 4-byte regions of memory. In this case, the L1 data memory system reads the existing data in the cache, merges in the modified bytes, and calculates the ECC from the merged value. The L1 memory system attempts to merge multiple stores together to meet the aligned 4-byte ECC granularity and to avoid the read-modify-write requirement.
(When they say "the L1 memory system", I think they mean the store buffer, if you have contiguous byte stores that haven't yet committed to L1d.)
Note that the RMW is atomic, and only involves the exclusively-owned cache line being modified. This is an implementation detail that doesn't affect the memory model. So my conclusion on Can modern x86 hardware not store a single byte to memory? is still (probably) correct that x86 can, and so can every other ISA that provides byte store instructions.
Cortex-A15 MPCore is a 3-way out-of-order execution CPU, so it's not a minimal power / simple ARM design, yet they chose to spend transistors on OoO exec but not efficient byte stores.
Presumably without the need to support efficient unaligned stores (which x86 software is more likely to assume / take advantage of), having slower byte stores was deemed worth it for the higher reliability of ECC for L1d without excessive overhead.
Cortex-A15 is probably not the only, and not the most recent, ARM core to work this way.
Other examples (found by @HadiBrais in comments):
Alpha 21264 (see Table 8-1 of Chapter 8 of this doc) has 8-byte ECC granularity for its L1d cache. Narrower stores (including 32-bit) result in a RMW when they commit to L1d, if they aren't merged in the store buffer first. The doc explains full details of what L1d can do per clock. And specifically documents that the store buffer does coalesce stores.
PowerPC RS64-II and RS64-III (see the section on errors in this doc). According to this abstract, L1 of the RS/6000 processor has 7 bits of ECC for each 32-bits of data.
Alpha was aggressively 64-bit from the ground up, so 8-byte granularity makes some sense, especially if the RMW cost can mostly be hidden / absorbed by the store buffer. (e.g. maybe the normal bottlenecks were elsewhere for most code on that CPU; its multi-ported cache could normally handle 2 operations per clock.)
POWER / PowerPC64 grew out of 32-bit PowerPC and probably cares about running 32-bit code with 32-bit integers and pointers. (So more likely to do non-contiguous 32-bit stores to data structures that couldn't be coalesced.) So 32-bit ECC granularity makes a lot of sense there.
cortex-m7 trm, cache ram section of the manual.
In an error-free system, the major performance impact is the cost of the read-modify-write scheme for non-full stores in the data side. If a store buffer slot does not contain at least a full 32-bit word, it must read the word to be able to compute the check bits. This can occur because software only writes to an area of memory with byte or halfword store instructions. The data can then be written in the RAM. This additional read can have a negative impact on performance because it prevents the slot from being used for another write.
.
The buffering and outstanding capabilities of the memory system mask part of the additional read, and it is negligible for most codes. However, ARM recommends that you use as few cacheable STRB and STRH instructions as possible to reduce the performance impact.
I have cortex-m7s but to date have not performed a test to demonstrate this.
What is meant by "read the word", it is a read of one storage location in an SRAM that is part of the data cache. It is not a high level system memory thing.
The guts of the cache is built of and around SRAM blocks that are the fast SRAM that makes a cache what it is, faster than system memory, fast to return answers back to the processor, etc. This read-modify-write (RMW) is not a high level write policy thing. What they are saying is if there is a hit and the write policy says to save the write in the cache then the byte or halfword needs to be written to one of these SRAMs. The width of the data cache data SRAM with ECC as shown in this document is 32+7 bits wide. 32 bits of data 7 bits of ECC check bits. You have to keep all 39 bits together for ECC to work. By definition you cant modify only some of the bits as that would result in an ECC fault.
Whenever any number of bits need to change in that 32 bit word stored in the data cache data SRAM, 8, 16, or 32 bits, the 7 check bits have to be recomputed and all 39 bits written at once. For an 8 or 16 bit, STRB or STRH write, the 32 data bits need to be read the 8 or 16 bits modified with the remaining data bits in that word unchanged, the 7 ECC check bits computed and the 39 bits written to the sram.
The computation of the check bits is ideally/likely within the same clock cycle that sets up the write, but the read and write are not in the same clock cycle so it should take at least two separate cycles to write data that arrived at the cache in one clock cycle. There are tricks to delay the write which sometimes can also hurt but usually moves it to a cycle that would have been unused and makes it free if you will. But it wont be the same clock cycle as the read.
They are saying if you hold your mouth right and manage to get enough smaller stores hit the cache fast enough they will stall the processor until they can catch up.
The document also describes the without ECC SRAM as being 32 bits wide, which implies this is also true when you compile the core without ECC support. I dont have access to the signals for this memory interface nor documentation so I cant say for sure but if it is implemented as a 32 bit wide interface without byte lane controls then you have the same issue, it can only write a whole 32 bit item to this SRAM and not fractions so to change 8 or 16 bits you have to RMW, down in the bowels of the cache.
The short answer to why not use narrower memory is, size of chip, with ECC the size doubles as there is a limit on how few check bits you can use even with the width getting smaller (7 bits for every 8 bits is a lot more bits to save than 7 bits for every 32). The narrower the memory you also have a lot more signals to route and cant pack the memory as densely. An apartment vs a bunch of individual houses to hold the same number of people. Roads and sidewalks to the front door instead of hallways.
And esp with a single core processor like this unless you intentionally try (which I will) it is unlikely you will accidentally hit this and why drive the cost of the product up on an: it-probably-wont-happen?
Note even with a multi-core processor you will see the memories built like this.
EDIT.
Okay got around to a test.
0800007c <lwtest>:
800007c: b430 push {r4, r5}
800007e: 6814 ldr r4, [r2, #0]
08000080 <lwloop>:
8000080: 6803 ldr r3, [r0, #0]
8000082: 6803 ldr r3, [r0, #0]
8000084: 6803 ldr r3, [r0, #0]
8000086: 6803 ldr r3, [r0, #0]
8000088: 6803 ldr r3, [r0, #0]
800008a: 6803 ldr r3, [r0, #0]
800008c: 6803 ldr r3, [r0, #0]
800008e: 6803 ldr r3, [r0, #0]
8000090: 6803 ldr r3, [r0, #0]
8000092: 6803 ldr r3, [r0, #0]
8000094: 6803 ldr r3, [r0, #0]
8000096: 6803 ldr r3, [r0, #0]
8000098: 6803 ldr r3, [r0, #0]
800009a: 6803 ldr r3, [r0, #0]
800009c: 6803 ldr r3, [r0, #0]
800009e: 6803 ldr r3, [r0, #0]
80000a0: 3901 subs r1, #1
80000a2: d1ed bne.n 8000080 <lwloop>
80000a4: 6815 ldr r5, [r2, #0]
80000a6: 1b60 subs r0, r4, r5
80000a8: bc30 pop {r4, r5}
80000aa: 4770 bx lr
there is a load word (ldr), load byte (ldrb), store word (str) and store byte (strb) versions of each, each are aligned on at least 16 byte boundaries as far as the top of loop address.
with icache and dcache enabled
ra=lwtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=lwtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=lbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=lbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=swtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=swtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=sbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=sbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
0001000B
00010007
0001000B
00010007
0001000C
00010007
0002FFFD
0002FFFD
the loads are on par with each other as expected, the stores though, when you bunch them up like this, a byte write is 3 times longer than a word write.
but if you dont hit the cache that hard
0800019c <nbtest>:
800019c: b430 push {r4, r5}
800019e: 6814 ldr r4, [r2, #0]
080001a0 <nbloop>:
80001a0: 7003 strb r3, [r0, #0]
80001a2: 46c0 nop ; (mov r8, r8)
80001a4: 46c0 nop ; (mov r8, r8)
80001a6: 46c0 nop ; (mov r8, r8)
80001a8: 7003 strb r3, [r0, #0]
80001aa: 46c0 nop ; (mov r8, r8)
80001ac: 46c0 nop ; (mov r8, r8)
80001ae: 46c0 nop ; (mov r8, r8)
80001b0: 7003 strb r3, [r0, #0]
80001b2: 46c0 nop ; (mov r8, r8)
80001b4: 46c0 nop ; (mov r8, r8)
80001b6: 46c0 nop ; (mov r8, r8)
80001b8: 7003 strb r3, [r0, #0]
80001ba: 46c0 nop ; (mov r8, r8)
80001bc: 46c0 nop ; (mov r8, r8)
80001be: 46c0 nop ; (mov r8, r8)
80001c0: 3901 subs r1, #1
80001c2: d1ed bne.n 80001a0 <nbloop>
80001c4: 6815 ldr r5, [r2, #0]
80001c6: 1b60 subs r0, r4, r5
80001c8: bc30 pop {r4, r5}
80001ca: 4770 bx lr
then the word and byte take the same amount of time
ra=nwtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=nwtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=nbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
ra=nbtest(0x20002000,0x1000,STK_CVR); hexstring(ra%0x00FFFFFF);
0000C00B
0000C007
0000C00B
0000C007
it still takes 4 times as long to do bytes vs words all other factors held constant, but that was the challenge to have bytes take more than 4 times as long.
so as I was describing before this question, that you will see the srams being an optimal width in the cache as well as other places and byte writes are going to suffer a read-modify-write. Now whether or not that is visible do to other overhead or optimizations or not is another story. ARM clearly stated it may be visible, and I feel that I have demonstrated this. This is not a negative to ARM's design in any way, in fact the other way around, RISC moves overhead in general as far as the instruction/execution side goes, it does take more instructions to do the same task. Efficiencies in the design allow for things like this to be visible. There are whole books written on how to make your x86 go faster, dont do 8 bit operations for this or that, or other instructions are preferred, etc. Which means you should be able to write a benchmark to demonstrate those performance hits. Just like this one, even if computing each byte in a string as you move it to memory this should be hidden, you need to write code like this and if you were going to do something like this you might consider burning the instructions combining the bytes into a word before doing the write, may or may not be faster...depends.
If I had halfword (strh) then no surprise, it also suffers the same read-modify-write as the ram is 32 bits wide (plus any ecc bits if any)
0001000C str
00010007 str
0002FFFD strh
0002FFFD strh
0002FFFD strb
0002FFFD strb
the loads take the same amount of time as the sram width is read as a whole and put on the bus, the processor extracts the byte lanes of interest from that, so there is no time/clock cost to doing that.