I am looking to upgrade my media server hardware. Im currently running an FM1 socket ECS board with an AMD Llano A8-3850 APU. When transcoding, this chip is pushed to the absolute max.
I dont want to spend more than $120 for the board and cpu (and RAM if other than DDR3). It must be at least a Quad core. The board needs to have 4+ SATA ports. ideally something where there is an M.2 SSD and 4 additional SATA ports.
I think you have too many stipulations to get what you're wanting:
The main culprit is your price limit - $120 is just too low to get a Quad core CPU with a mobo that has M.2, not to mention possibly DDR4 RAM.
To keep the DDR3, you'll most likely have to go with AMD or socket 1150 Intel (Haswell). In either case, you'll probably have to settle with SATA protocols and not NVMe, the main reason to get an M.2 so it's kind of self defeating.
To convert to an Intel 4 core CPU will probably exceed $120
I did find an AMD mobo with M.2 and a Quad core APU at $150, which is over your price cap and probably won't give you much better performance than the computer you have right now.
If you have time and you're patient, you might get lucky if you scour websites (eBay, Craigslist) and look for used components.