If it beeps with one 4GB DIMM installed, something isn't compatible about that DIMM type.
They should look exactly like the pic below. These exact DDR2-6400U 800mhz DIMMs have always worked for me. They must be 6400U, not just 6400.
As for your software, I've had no problems like that. Make SURE your USB flash drive with the OS and Ascent loaders on it is formatted as NTFS (Fam14 uses FAT32 formatted flash drives to load).
I have one 64GB flash drive that's NTFS formatted with the OS and Ascent loader, as well as my game loaders so I can easily clear, load the OS and Ascent files, and then games when booted with the diag so it's a bit easier not having to swap flash drives.