January 30, 2023

Kyu networking -- Study the NetBSD startup (part 2)

We started on this once, but got distracted. Two things are motivating this:

DCCISW

A quick note on DCCISW. This seems unique to the Cortex-A7 MPCore that is used in the Allwinner H3 chip (in the Orange Pi). Finding out just where and why it is used by NetBSD is likely to be a good clue. The instruction is: 
    mcr	p15, 0, r3, c7, c14, 2
It is described as "data cache clean and invalidate line by set/way". I have also seen it described as "write back and invalidate". The usual ARM v7 has one instruction for "clean" and another for "invalidate", but this seems to combine them both.

Some searching turns this up in archarm/arm/cpufunc_asm_armv7.S. It is used in one routine: 

armv7_dcache_wbinv_all().

There is a close brother operation DCCIMVAC that cleans and invalidates a line by MVA 

mcr     p15, 0, r0, c7, c14, 1
This is used (in the same file) in two routines: 
armv7_dcache_wbinv_range ( a, b )
armv7_idcache_wbinv_range ( a, b )

Now, let's try to track down where the first (wbinv_all) of those 3 routines are used. There is one call that looks to be of particular interest in armv6_start.S Other than that, a pointer to this function is placed into a "cpufunc" structure as cf_dcache_wbinv_all (see cpufunc.c), but I am going to ignore that for now.
the code in armv6_start.S that calls it looks like this: 

	mrc     p15, 0, r0, c1, c0, 0		// SCTRL
        tst     r0, #CPU_CONTROL_DC_ENABLE
        blne    armv7_dcache_wbinv_all
The bit being tested is 0x4, which is the bit to enable D and unified caches. So, if they are enabled, it is going to clean and invalidate them, otherwise they will be left alone.

So, how and where does NetBSD start up?

My build starts up at "generic_start", which is really the first place code might run in the file armv6_start.S. After doing just a bit of initialization, this calls generic_startv7() (in the same file). After doing what it does, this will branch to start(), which is in locore.S. This will ultimately call main().

There you have it in a nutshell.

It is in armv7_init() in armv6_start.S that we call armv7_dcache_wbinv_all(). It is worthy of note that we do this, then modify the SCTRL register, then call armv7_dcache_inv_all() in short order.

armv7_init() is essentially the first thing in generic_startv7(), but it is also called in cpu_mpstart() as part of firing up a second core.

If your interest wanders to cpu_mpstart(), take a look at these for the Orange Pi: 

arm/sunxi/sunxi_platform.c
arm/sunxi/sunxi_mc_mpstart.S
The platform file references this: arm_fdt_cpu_mpstart() However we will stop here with this multicore stuff.

Conclusions

I am strongly tempted to steal the file 
cpufunc_asm_armv7.S
Maybe I should say "copy verbatim", as I would certainly retain all license notices and copyrights and such. A big lump (532 lines) of ARM assembly with a lot of these wretched mrc and mcr instructions in it is nothing to sneeze at. Would anything be gained or learned by starting from scratch and reproducing it. Very little.

That being done, the thing is to study two files: 

armv6_start.S
locore.S
I am likely to take interest in some mmu/tlb related files as well. The file arm32/arm32_tlb.c has already caught my attention.
Likely hooks in armv6_start.S are: 
bl      arm_build_translation_table
b       armv7_mmuinit
Both are in armv6_start.S and could just be "copied verbatim" as well, along with other code in the same file.

Have any comments? Questions? Drop me a line!

Kyu / [email protected]