DDR3 8-DIMMs and Memory Mirroring on IMX6
matthewbeach , 10-27-2018, 12:11 PM
Hi everyone,
I'l getting ready to start a project in which I'm going to try and use 8 DDR3 DIMMS instead of the 4 used on the iMX6 Rex. I'm going to write this post on the perspective of what I, at the moment, think needs to be done to accommodate the extra memory, and would like feedback on anything I'm missing or will be doing wrong.
From looking into Jedec DDR3 modules, this looks like it is done by using DRAM_CS0/DRAM_CS1 pins [Y16/AD17], to select between DIMMS that share the same data bank. In addition to using both CS* pins, they must also be accompanied by the other CKE* and SDODT* pins, as well as the two differential clocks.
So for expanding the iMX6 schematics for the extra memory, I just plan on using the existing 4 DIMMS on the top layer using all of the data banks, and CS0, CKE0, SDOTD0, and SDCLK_0, and making a copy for the bottom layer that will use CS1, CKE1, SDOTD1, and SDCLK_1. All 8 DMMS will get the same A*, SDBA*, RAS, CAS, and WE pins.
For routing the board, I plan on using a Fly-By topology, and to make the layout easier, I was hoping to use Memory Mirroring like used in some Jedec DDR3 modules. From what I've looked at, it seems like memory mirroring is supported by the iMX6, but i'm not sure and would like to confirm before starting. I also have no idea how to tell the processor that is is dealing with mirrored DIMMS. Are their any extra steps in hardware to do the memory mirroring? What would the firmware steps be?
Thanks for the help!
Best,
Matthew Beach
I'l getting ready to start a project in which I'm going to try and use 8 DDR3 DIMMS instead of the 4 used on the iMX6 Rex. I'm going to write this post on the perspective of what I, at the moment, think needs to be done to accommodate the extra memory, and would like feedback on anything I'm missing or will be doing wrong.
From looking into Jedec DDR3 modules, this looks like it is done by using DRAM_CS0/DRAM_CS1 pins [Y16/AD17], to select between DIMMS that share the same data bank. In addition to using both CS* pins, they must also be accompanied by the other CKE* and SDODT* pins, as well as the two differential clocks.
So for expanding the iMX6 schematics for the extra memory, I just plan on using the existing 4 DIMMS on the top layer using all of the data banks, and CS0, CKE0, SDOTD0, and SDCLK_0, and making a copy for the bottom layer that will use CS1, CKE1, SDOTD1, and SDCLK_1. All 8 DMMS will get the same A*, SDBA*, RAS, CAS, and WE pins.
For routing the board, I plan on using a Fly-By topology, and to make the layout easier, I was hoping to use Memory Mirroring like used in some Jedec DDR3 modules. From what I've looked at, it seems like memory mirroring is supported by the iMX6, but i'm not sure and would like to confirm before starting. I also have no idea how to tell the processor that is is dealing with mirrored DIMMS. Are their any extra steps in hardware to do the memory mirroring? What would the firmware steps be?
Thanks for the help!
Best,
Matthew Beach
robertferanec , 10-31-2018, 07:18 AM
Maybe have a look at dual die memory chips, that could be something what could be interesting for you. However I am not sure how the price of 4 dual die memory chips will compare to 8 memory chips. Basically you would still solder down 4 chips, however each of these chips would have 2 memory chips inside. You still need to connect additional signals (CS1, CKE1, etc), but it's much easier than 8 memory chips.
If you would like to use 8 chips, ideally you would like to find imx6 design with second bank connected (that's what I would do). That could help you also to check memory controller settings and this way you would be also sure, that you would connect everything correctly and that it is going to work.
Possibly I would also do exactly same what you did - I would have a look at JEDEC reference design and I would double checked how they connected it.
PS: I really do not know how to setup memory mirroring in the iMX6. Once I tried to set the memory controller registers in iMX6 before, but even I thought I setup everything correctly, the settings from existing boards worked much more reliably than the settings what I tried. So, just be aware, setting up memory controller may take some time - ideally you would like to have a working example.
If you would like to use 8 chips, ideally you would like to find imx6 design with second bank connected (that's what I would do). That could help you also to check memory controller settings and this way you would be also sure, that you would connect everything correctly and that it is going to work.
Possibly I would also do exactly same what you did - I would have a look at JEDEC reference design and I would double checked how they connected it.
PS: I really do not know how to setup memory mirroring in the iMX6. Once I tried to set the memory controller registers in iMX6 before, but even I thought I setup everything correctly, the settings from existing boards worked much more reliably than the settings what I tried. So, just be aware, setting up memory controller may take some time - ideally you would like to have a working example.
matthewbeach , 11-01-2018, 12:09 AM
I've done a little more research and I think I might be on to something in regards to activating the memory mirroring.
I found an entry on memory mirroring in the iMX6 ​Applications Processor Reference Manual (table 44.4.4.4 on pg 3845) showing the signals that can be mirrored for each CS. It shows that the following address pairs can be mirrored for CS1 DIMMs; A3&A4, A5&A6, A7&A8, and BA0&BA1. This can also be found on Jedec Memory module schematics that feature the memory mirroring.
Looking further into the Jedec Memory modules, they all have something called an SPD (Serial Presence Detect) which is essentially an EEPROM. The EEPROM holds an SPD Table that has tons of critical information telling the system BIOS about the ddr3 being used. All of this is defined in the JEDEC Standard No. 21-C.
In the Jedec Standard document defining the SPD table, is shows that Byte 63 of the SPD defines if memory mirroring is used by setting it from 0x00 to 0x01 [JEDEC Standard No. 21-C Page 4.1.2.11 – 32]. Also in the section is another table confirming the signals that can be mirrored. I'm not sure that this is actually solving my issue as it may just be necessary for slotted DRAM modules, as modules may or may not be mirrored or just be different sizes and the CPU needs a way to know. Basically it seems the SPD is just a standardized bucket of info about the ddr3 card, not anything telling the iMX6 how to intemperate that bit of info and actually read the memory as mirrored.
Looking back into the Applications Processor Reference Manual, there is a Register Table on page 3875, showing that the register with address: 21B_0018 - bit 19 is the ADDR_MIRROR variable, and that by setting it to 1 you are activating the address mirroring described in the table found at the beginning of this post.
Is this the bit of info we need to enable the memory mirroring?
I'll do a bit more research to see if I can find any example of the iMX6 with the mirroring but so far i've only had luck with the jedec modules.
I found an entry on memory mirroring in the iMX6 ​Applications Processor Reference Manual (table 44.4.4.4 on pg 3845) showing the signals that can be mirrored for each CS. It shows that the following address pairs can be mirrored for CS1 DIMMs; A3&A4, A5&A6, A7&A8, and BA0&BA1. This can also be found on Jedec Memory module schematics that feature the memory mirroring.
Looking further into the Jedec Memory modules, they all have something called an SPD (Serial Presence Detect) which is essentially an EEPROM. The EEPROM holds an SPD Table that has tons of critical information telling the system BIOS about the ddr3 being used. All of this is defined in the JEDEC Standard No. 21-C.
In the Jedec Standard document defining the SPD table, is shows that Byte 63 of the SPD defines if memory mirroring is used by setting it from 0x00 to 0x01 [JEDEC Standard No. 21-C Page 4.1.2.11 – 32]. Also in the section is another table confirming the signals that can be mirrored. I'm not sure that this is actually solving my issue as it may just be necessary for slotted DRAM modules, as modules may or may not be mirrored or just be different sizes and the CPU needs a way to know. Basically it seems the SPD is just a standardized bucket of info about the ddr3 card, not anything telling the iMX6 how to intemperate that bit of info and actually read the memory as mirrored.
Looking back into the Applications Processor Reference Manual, there is a Register Table on page 3875, showing that the register with address: 21B_0018 - bit 19 is the ADDR_MIRROR variable, and that by setting it to 1 you are activating the address mirroring described in the table found at the beginning of this post.
Is this the bit of info we need to enable the memory mirroring?
I'll do a bit more research to see if I can find any example of the iMX6 with the mirroring but so far i've only had luck with the jedec modules.
matthewbeach , 11-03-2018, 08:32 PM
Also, if I were to route the board using the memory mirroring for the back-side chips and only populated the DIMMS on the front side that aren't mirrored, the board should still work as if it was designed with only one CS pin like the Rex and OpenRex projects? There would just be un-terminated stubs hanging off the data and address signals, which im not sure would be a problem or not. Or could all the DIMMs be populated if I need to used the ODT of those parts to help the signal quality, and then tell the memory controller to only access the non-mirrored DIMMs?
Just want to see how much risk there is to going ahead with the mirroring without finding a working example of it specificly with the iMX6.
-Matt
Just want to see how much risk there is to going ahead with the mirroring without finding a working example of it specificly with the iMX6.
-Matt
robertferanec , 11-05-2018, 03:07 AM
Also, if I were to route the board using the memory mirroring for the back-side chips and only populated the DIMMS on the front side that aren't mirrored, the board should still work as if it was designed with only one CS pin like the Rex and OpenRex projects? There would just be un-terminated stubs hanging off the data and address signals, which im not sure would be a problem or not. Or could all the DIMMs be populated if I need to used the ODT of those parts to help the signal quality, and then tell the memory controller to only access the non-mirrored DIMMs?
Use our interactive Discord forum to reply or ask new questions.