First the Dursley's since they are so obvious.
1. Harry sleeps under the stairs.
2. He is forced to use the stove at 9 and it sounds like this is just typical, so I'm assuming that he's been doing it for a couple of years.
3. He is locked in his place under the stairs and not given food for a while.
4. They let their son visibly beat up Harry and they say nothing about it.
I'm sure there are more things that I'm leaving out, but these were a few obvious ones. Umm... child abuse anyone? I know, I know. Its only a book and a fantasy one at that.
I have the same trouble with movies. We were watching one yesterday and the guy had his scarf trapped in the elevator. Why not just unwrap it? You didn't need those scissors that you just happened to have in your pocket. My husband tells me constantly, it's a movie. Not everything is completely reality. I can't take it seriously.
Putting that aside. Where he is sleeping is not proper, using the stove at 8-9 isn't all that safe and allowing your child to beat the snot out of your nephew isn't within your rights as a parent. Haven't any of the school teachers noticed anything? Even the clothes and broken glasses. Aren't any of the teachers concerned for his well-being? I know. I'm pushing things a little far.
Onto Dumbledore and McGonagall. They left Harry at a doorstep. Granted it was his Aunt and Uncle's, but they knew nothing about them. Were they safe? Would they take care of this infant that every single wizard loved (except of Voldomort's cronies of course) ... They knew NOTHING about these people. Except that they were muggles. Not much to go by in my book. I would almost have put him in an orphanage instead. In fact, why didn't the Dursley's? They wanted nothing to do with him. Why did they bother?
As you can imagine, things like this pop in my head waaaaay too much and I am consumed with it until some other thing pops into my head. You do not want to be in there. It's pretty crazy. :)