No man, you are confused. If the GPU works with graphics memory directly then it would have very slow access to memory as it is working. You would kill performance. Graphics cards typically have dedicated memory to allow fast access to that memory.

CPU memory and GPU memory is built to serve different needs. Typically CPU memory has low latency. GPUs have higher latency but more throughput. They are optimized differently.

With the M1 and M2 Apple managed to do both: low latency memory with high throughput. That is hard to achieve in a system where CPU and GPU are far apart.