That mostly sounds right to me, except that the attacker generally overwrites portions of memory with bytes that are in machine language (not C/C++), and then the machine ends up running that code.
Software that was originally written in C/C++ tends to be much more vulnerable to buffer overflows than Java, because Java always checks the length of an array whenever you attempt to assign to it, whereas C/C++ arrays are just pointers, and it can be easy to write data past the end of the array.