I'm sorry that your effort to honor and carry your father with you has been poorly received by some. These folks may be vets, the family members of vets, or just citizens who respect the military, so be polite.
Consider doing some research on what every patch is for, and maybe even talk with a friendly vet about what's showing. If nothing else, the guys at an Army recruitment center might help with that.
If you can look at your Dad's DD214 form, it will list the medals he earned.
It might help if you took a good photo of your Dad, especially one from the field or otherwise in uniform, and had it made into a big photo button to put on the jacket.
Lastly, out of respect, don't wear the jacket if out doing something questionable, like getting buzzed or raising hell. The guys who earned the patch can do that, but you shouldn't.