The wording of your question, "Why is it wrong," leads me to infer that you think atheists and agnostics should be include in group prayers.
I used to tell my guys before their first mission that if they were of a mind to pray, that "now would be as good a good time as any," and if they didn't know how to pray and wanted to learn, I'd set them up with the Chaplain after we got back. I had a few takers... usually when we got back. We didn't do the group thing because, when we went out our groups were pretty small.
Have you ever heard that there's no such thing as an atheist in a foxhole? It's pretty true. It's like here in the Veterans Home. There are guys who've never gone to church. Over the years they've developed a belief system that seems to work for them, but they've never adopted a formalize religion. Then one day they wake up to the fact that the days ahead are getting short, and suddenly they start going to the services here on campus... which is pretty non-denominational. Some head into town and to an even more formalized religion.
I'm not particularly religious. I'm one of those with the belief system that works for me. I absolutely detest ramming any religion down somebody's throat. But, a group prayer, if it bolsters the morale of the troops, even if they choose not to participate (some so choose), what the heck right does anybody think they have to denigrate it? Maybe the guys who don't actually pray get some comfort from the camaraderie. Maybe they pick up some of the calm prayer seems to have for those who do it. Maybe some join the group because they did it before and then made it back... kind of a lucky charm. Nobody's forced to participate.
However, if my inference is incorrect, and you think agnostics and atheists are being somehow religiously harassed, then I'm sorry to have to inform you, sir, that you have absolutely no idea what you're talking about. I'd surmise that you're a pontificating philosopher who's never seen the military... except maybe in the movies or on TV. You can't possibly have even the foggiest notion what it's like to be shot at, to see your buddy blown away buy a hail of bullets, a mortar or a landmine or IED. You've certainly never held your dying buddy in your arms and have him almost break your hand as he crushes it in pain... and there's not a blasted thing you can do to stop it. You can't ever have had to take a life. And, if Until you've been there and done that, kindly keep your philosophizing to the religion or philosophy sections where nobody else knows what they're talking about either... and where they too are all absolutely convinced that they do.
I have cousin who sent me a photo a few year ago. It was when there was that big push to get crosses removed from all government property because the atheists and agnostics felt they were being religiously harassed. It was a picture of Arlington National Cemetery.
ADDENDUM
If it is as you state, then I apologize. However, from your statement, I get an inkling that there might be something else going on besides infringement of your religious rights (a' la' Michael Newdow). Be that as it may, I would contend that your OIC or whoever required your participation was out of line. You owe it to your beliefs to sit down with a JAG officer and find out:
(1) If there's a UCMJ Article (I'm not certain there is one in this case) or one in your branch's Regs that refers to freedom of the personnel in that branch to practice their beliefs... find out if the order is, indeed, unlawful... and if it is, get Article, Section, Subsection, Paragraph, and then
(2) learn the proper way to decline to obey an unlawful order in your branch.
Then, with the courage of your convictions, don't do anything you're not required to do. Be careful. Religious beliefs or lack thereof only apply to matters of religion. For example, since you're atheist, you might expect to stand duty on religious holidays because the guys who believe in them also have the right to practice their beliefs.
And you can be assured that people will be looking for you to slip up... even the slightest bit... especially that OIC that gave you the hard time and who you had to disobey. You, of course, have the right to "find religion." But you can't flip-flop. If, for example, you get caught reading the Bible, you also better have close at hand some other religious textbooks to prove you're doing research... in the off chance you might one day find some kind of Deity in which you could believe.