#pragma omp flush
A simple experiment to figure out how flush works

Lets look at the following source code. Compiled with flags -O3 -fopenmp

#include <stdio.h>

int main() {
    int i = 1;
    #pragma omp parallel
    {
        i++;
        #pragma omp flush
        i*=i;
        i--;
        printf("%d\n", i);
    }
    return 0;
}

The above experiment shows that #pragma omp flush causes two changes:

• before the flush, the variables modified in registers are moved back to the memory.
• after the flush, the variables are loaded back from memory to the register (if the subsequent instructions require that variable)

After the mov operation the cache coherence mechanisms of the CPU are responsible to make the value consistent among the different cores' cache and the memory. But, some optimizations in the microarchitecture come into play, and thus this is not quite true and we need memory barriers.

CPU will always see its own operations as if they happened in program order. The need for memory barriers arise in shared memory multiprocessor systems (SMP). SMPs have cache coherency protocols that maintain coherent view of memory among the processor caches, but as an optimization for those coherency protocols CPUs also have store queues and invalidate queues. It is really the store queues and invalidate queues that cause the need for memory barriers.

Stores are placed in the store queue if the cache block is either not in the cache or is in cache but in shared state. Store queue saves CPU from stalling for a store to complete. For subsequent loads the CPU forwards data from the store queue. So, the CPU still sees its operations in program order. But other CPUs can see the writes in different order. A write memory barrier ensures that all the writes before this point are completed before proceeding with new writes. And so other CPUs will see the writes preceeding the barrier before the writes that follow the barrier.

When a CPU needs to write to a block, it needs exclusive access. So, it sends invalidate message to all other CPUs and waits for acknowledgement before updating the value. The cache might be busy to handle the invalidate message right away, so the microarchitecture can have a invalidate queue as an optimization. Incoming invalidate messages are immediately acknowledged and placed in the queue but not invalidated in the cache. Thus the CPU can be oblivious to the fact that the load from the cache is actually invalid. A read memory barrier, ensures that all pending invalidates in the queue are processed before any subsequent load operations are allowed to complete. This guarantees that the CPU reads the most up-to-date data.

Take an example from Memory Barriers - A Hardware View for Software Hackers [pg. 8] with MESI protocol (Modified Exclusive Shared Invalid):

// Assume that initially b resides in CPU 0
// and a resides in CPU 1
void foo(void) { // Run by CPU 0
  a = 1;
  write_barrier(); // Otherwise b will be written while a remains in store queue
  b = 1;
}

void bar(void) { // Run by CPU 1
  while (b == 0) continue;
  // Otherwise invalidate for a might still be in invalidate queue
  // and the cache would have old value of a, still marked as valid
  read_barrier();
  assert(a == 1);
}