Memory Hierarchy - 存储器层次结构

计算机系统将存储器分成若干层级 (memory hierarchy) ，越靠近 CPU 的存储器容量越小但访问速度越快。

1. Memory Hierarchy (存储器层次结构)

Intel 北桥包含 2 个 channel，两组独立的线连接到各自的模块，每个 channel 包含 2 个DIMM。

Shared resources in multicore processors

DRAM system organization

Module architecture

Generalized memory structure

2. Channel (通道)

A memory channel is made up of ranks. Physically a memory channel with just one memory module might present itself as having one or more logical ranks.
memory channel 由 ranks 组成。在物理上，只有一个内存模块的 memory channel 可能会显示为具有一个或多个逻辑 ranks。

相同的 Rank 与 Chip Select (CS#) 连接在一起，当发出 CS# 信号时，相同 Rank 内所有的 IC 颗粒会同时被 Enable。2 个 Rank 表示能接受两条 CS# 的信号，它们的名称为 S0# 与 S1#。1 个 Rank 称为 1R，2 个 Bank 称为 2R。相同 Rank IC 的 CS# 全部与 DIMM 的 S0# 与 S1# 分别接在一起。

In the fields of digital electronics and computer hardware, multi-channel memory architecture is a technology that increases the data transfer rate between the DRAM memory and the memory controller by adding more channels of communication between them.
在数字电子和计算机硬件领域，多通道存储器技术 (multi-channel memory technology) 是一种提高数据传输速率的技术，通过在 DRAM 内存和内存控制器之间增加更多通信通道来实现。

2.1 Dual channel (interleaved) mode

The Dual Channel (Interleaved) mode provides the highest performance. It is activated whenever the particular total capacity of the built-in modules is identical in both channels. The slowest DIMM built into the system determines the pace of the bus clock and the memory access for the complete RAM. However, it is important that the total capacity of channel A and channel B is identical – a requirement that can be met with two, three or four DIMMs.
双通道 (交错) 模式提供最高性能。只要两个通道中内置模块的总容量相同，它就会被激活。系统内置的最慢 DIMM 决定了总线时钟的速度和整个 RAM 的内存访问。然而，重要的是通道 A 和通道 B 的总容量相同 - 两个、三个或四个 DIMM 可以满足这一要求。

Dual channel (interleaved) mode configuration with two DIMMs

Dual channel (interleaved) mode configuration with three DIMMs

Dual channel (interleaved) mode configuration with four DIMMs

2.2 Single channel (asymmetric) mode

The Single mode is activated when the particular total capacity of the built-in modules differs in both channels. The slowest DIMM built into the system determines the pace of the bus clock and the memory access for the complete RAM.
当内置模块的特定总容量在两个通道中不同时，将激活单一模式。系统内置的最慢 DIMM 决定了总线时钟的速度和整个 RAM 的内存访问。

Single channel (Asymmetric) mode configuration with one DIMM

Single channel (asymmetric) mode configuration with three DIMMs

2.3 Dual channel (flex mode)

This new, additional technology provides highest flexibility regarding the assembly of memory modules. It rather revokes all limitations and requirements for memory modules to date. The Dual Channel mode also works when using only two, three or four DIMMs with different total capacity in the memory channels.
这种新的附加技术为内存模块的组装提供了最高的灵活性。它取消了迄今为止对内存模块的所有限制和要求。双通道模式也适用于在内存通道中仅使用两个、三个或四个具有不同总容量的 DIMM。

Dual channel (flex mode) configuration with two DIMMs

2.4 DIMM per channel

2.5 T mode: T-topology

2.6 D mode: Daisy chain (菊花链)

daisy ['deɪzɪ]：n. 黛西，雏菊，延命菊，卓越的人或物 adj. 雏菊一样的，可爱的

Strict mapping of threads to ranks

Relaxed mapping of threads to ranks

3. Memory controller (内存控制器)

Memory controller

A conventional memory channel where a memory controller and two memory modules are connected through a shared bus. Each memory module is composed of one or more ranks, each rank with 8 or 16 DRAM chips.
一种传统的 memory channel (内存通道)，其中一个 memory controller (内存控制器) 和两个内存模块通过共享总线连接。每个内存模块由一个或多个 ranks 组成，每个 rank 有 8 或 16 个 DRAM 芯片。

A conventional memory channel which contains two DIMMs attached to a memory controller through a bus. Memory controllers have historically been placed outside of microprocessors (in a chip called the northbridge), but are more recently integrated.
一种传统的 memory channel (内存通道)，包含两个 DIMMs 通过总线连接到 memory controller (内存控制器)。内存控制器历来被放置在微处理器之外 (在称为北桥的芯片中)，但最近被集成。

memory controller，MC：内存控制器
northbridge：北桥，北桥芯片

Cache blocks map to separate memory controllers

DRAM structure

Memory parallelism from independent banks and channels

Channels

4. DDR SDRAM

Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM) is a double data rate (DDR) synchronous dynamic random-access memory (SDRAM) class of memory integrated circuits used in computers. DDR SDRAM, also retroactively called DDR1 SDRAM, has been superseded by DDR2 SDRAM, DDR3 SDRAM, DDR4 SDRAM and DDR5 SDRAM.
双倍数据率同步动态随机存取存储器 (Double Data Rate Synchronous Dynamic Random Access Memory，DDR SDRAM) 为具有双倍数据传输率的 SDRAM，其数据传输速度为系统时钟的两倍，在系统时钟的上升沿和下降沿都可以进行数据传输。

Comparison of DDR modules for desktop PCs (DIMM)

Comparison of memory modules for portable/mobile PCs (SO-DIMM)

random-access memory，RAM
static random-access memory，static RAM or SRAM
dynamic random-access memory，dynamic RAM or DRAM
synchronous dynamic random-access memory，synchronous dynamic RAM or SDRAM
double data rate synchronous dynamic random-access memory，DDR SDRAM
double data rate，DDR
single data rate，SDR
dual in-line memory module，DIMM：双列直插内存模块
retroactively：adv. 逆动地
supersede [ˌsuːpə(r)ˈsiːd]：v. 取代

5. Rank

A memory rank is a set of DRAM chips connected to the same chip select, which are therefore accessed simultaneously. In practice all DRAM chips share all of the other command and control signals, and only the chip select pins for each rank are separate (the data pins are shared across ranks).
memory rank 指的是一组连接到同一芯片选择 (chip select，CS) 的 DRAM 芯片，可以同时访问这些芯片。在实践中，所有 DRAM 芯片共享所有其它命令和控制信号，只有每个 rank 的 chip select 引脚是独立的 (数据引脚在 rank 之间共享)。

The term rank was created and defined by JEDEC, the memory industry standards group. On a DDR, DDR2, or DDR3 memory module, each rank has a 64-bits-wide data bus (72 bits wide on DIMMs that support ECC). The number of physical DRAMs depends on their individual widths. For example, a rank of ×8 (8-bits wide) DRAMs would consist of eight physical chips (nine if ECC is supported), but a rank of ×4 (4-bits wide) DRAMs would consist of 16 physical chips (18, if ECC is supported). Multiple ranks can coexist on a single DIMM, and modern DIMMs can consist of one rank (single rank), two ranks (dual rank), four ranks (quad rank), or eight ranks (octal rank).
rank 由内存行业标准组织 JEDEC 创建和定义。在 DDR, DDR2 or DDR3 内存模块上，每个 rank 都有一个 64 bits 宽的数据总线 (在支持 ECC 的 DIMM 上为 72 bits 宽)。物理 DRAM 的数量取决于它们各自的宽度。例如，一个 ×8 (8-bits wide) DRAM 将包含 8 个物理芯片 (如果支持 ECC，则为 9 个)，但 ×4 (4-bits wide) DRAM 将包含 16 个物理芯片 (如果支持 ECC，则为 18 个)。多个 ranks 可以在单个 DIMM 上共存，现代 DIMM 可以由 1 rank (single rank), 2 ranks (dual rank), 4 ranks (quad rank), or 8 ranks (octal rank)。

single rank memory, dual rank memory and quad rank memory

Generally Single Rank Memory is faster than Dual Rank Memory, in layman’s terms when a computer accesses Single Rank Memory it only has to go around the track once, where are Dual Rank it would have to go around the track twice.
一般来说单 rank 内存比双 rank 内存要快，通俗的讲，当计算机访问单 rank 内存时，它只需要绕轨道运行一次，而双列它必须绕轨道走两次。

single-sided module and double-sided module

single-sided single-rank, double-sided single-rank, double-sided dual-rank and double-sided quad-rank

x4 和 x8 代表每个 DRAM 芯片位宽。

双列直插内存模块 (dual in-line memory module，DIMM) 是指内存条印刷电路板正反面均有金手指与主板上的内存条槽接触，内存条称为 DIMM 条，主板上的内存槽称为 DIMM 槽。

single rank memory and dual rank memory

1 rank = 64 bit width (or 72 bit with ECC)
single rank = 64 bit
double rank = 64 bit + 64 bit
quad rank = 64 bit + 64 bit + 64 bit + 64 bit

• All chips comprising a rank are controlled at the same time (包含在一个 rank 的所有 chips 被同时控制)
• A DRAM module consists of one or more ranks
• If we have chips with 8-bit interface, to read 8 bytes in a single access, use 8 chips in a DIMM

quadrant [ˈkwɒdrənt]：n. 象限，象限仪，四分之一圆
octal ['ɒktl]：adj. 八的，八面的，八管脚的
layman ['leɪmən]：n. 外行，门外汉，非专业人员，在俗教徒 (非神职人员)
single-sided：单面
double-sided：双面

A 64 bit wide DIMM (one rank)

2 Independent channels - 2 Memory controllers

6. Bank - Column - Row

DRAM bank

A canonical representation of a DRAM chip with 8 banks. The movement of data during a typical memory transaction consists of an ACTIVATE command (1), READ or WRITE commands (2) and PRECHARGE command (3).
具有 8 个 banks 的 DRAM 芯片的规范表示。典型内存处理的数据移动包括 an ACTIVATE command (1), READ or WRITE commands (2) and PRECHARGE command (3)。

canonical [kəˈnɒnɪk(ə)l]：adj. 被收入真经篇目的，经典的，按照基督教教会法规的 n. (布道时应穿的) 法衣
transaction [træn'zækʃ(ə)n]：n. 处理，业务，办理
precharge：n. 预充电，预压力

DRAM operation

• A DRAM bank is a 2D array of cells: rows x columns，each address is a <row, column> pair.
• A DRAM row is also called a DRAM page
• sense amplifiers 也称为 row buffer

DRAM bank operation

128M x 8 bit DRAM chip

• Divide memory into M (= n + 1) banks and interleave addresses across them
  in bank (mod M)
at word (div M)
• Interleaved memory increases memory bandwidth without wider bus (交错内存增加内存带宽而无需更宽的总线)
• Use parallelism in memory banks to hide memory latency (使用 memory banks 中的并行性来隐藏内存延迟)
  

interleave [ˌɪntə(r)ˈliːv]：v. 夹进，交错，交织

7. Transferring a cache block (传输缓存块)

References

Advanced Computer Architecture I
https://pages.cs.wisc.edu/~sinclair/courses/cs752/fall2020/handouts/lecture/12-dram.pdf

Software-Defined Physical Memory
https://www.betriebssysteme.org/wp-content/uploads/2017/04/Hillenbrand.pdf

Improving System Energy Efficiency with Memory Rank Subsetting
http://csl.stanford.edu/~christos/publications/2012.mcdimm.taco.pdf

DRAM Memory Module Rank Calculation
https://www.digikey.com/en/pdf/v/viking-technology/dram-memory-module-rank-calculation

Memory rank
https://en.wikipedia.org/wiki/Memory_rank

Memory geometry
https://en.wikipedia.org/wiki/Memory_geometry

Single Rank Memory VS Dual Rank Memory VS Quad Rank Memory
https://www.oempcworld.com/support/singlevsdualram.html

RAM (Random Access Memory), SIMMs, DIMM’s, RIMM’s
https://sites.google.com/site/shaunaatvcc2/memory/ram-random-access-memory-simms-dimm-s-rimm-s

Mengenal Istilah Memory Rank Pada Memori (RAM)
https://catatanteknisi.com/mengenal-istilah-memory-rank-pada-memori-ram/

Enabling big memory with emerging technologies
https://www.cs.utah.edu/~rajeev/pubs/shevgoor-phd.pdf
https://slideplayer.com/slide/9254965/

Single Rank VS Dual Rank
https://www.pcgaming.tech/single-rank-vs-dual-rank.html

Dual channel mode for DDR, DDR2, DDR3 and DDR4
https://www.compuram.de/blog/en/single-dual-and-multi-channel-memory-modes/

Wiring method of motherboard memory
https://www.programmersought.com/article/36157239769/

DDR4
https://itgall.com/hardware/210621

DDR4 SDRAM - Understanding the Basics
https://www.systemverilog.io/ddr4-basics

2Rx8 and 1Rx8 DDR 的 Rank 与单面双面无关
https://www.strongpilab.com/ddr-2rx8-rank-not-side/