I am wondering if I was to play heavy graphic game or even video editing (which would make PC components hot), is it better to open the case door and leave it open while playing game(s)? Or, better to leave it closed to let the fans control the air flow?
Also, I have wonder why it has to be "bigger case" to have better fan cooling? If I was to get a smaller case (still ATX), would it still cool as well as bigger case?
This would apply to fan heatsink, but watercool as well?
The objective is to keep components as cool as possible to allow them to run at maximum performance. By nature, the case is an enclosed environment which traps heat. To get around this, people typically actively cool the inside with fans or apply waterblocks directly on the components to shift the heat to a radiator where it can be expelled from the case.
The most ideal situation would be to open up the case and have a 12" desk fan, or box fan, blow directly into you case. Obviously, this is noisy and can get in the way. Provided you keep fans on the front and rear of the case, the next best set up would be to open up the side of the case. Less obtrusive, but usually still noisy. Both of these options also tend to look pretty ugly and allow for dust to get into the case (and too much dust eventually becomes an insulator and traps heat). Leaving the side panel on, along with good airflow provides a balance between dust, noise, aesthetics and keeping your computer cool.
To your second question (would a smaller ATX case still cool as well as a bigger case?), it would depend. If the smaller case has more (or better) locations for fan placement, I would say it would. Generally speaking, if the setup were identical, a bigger case provides more room for air to circulate and heat to dissipate.