The Secret Service has revoked the security clearances for the 11 agents accused of misconduct over a prostitution scandal in Colombia -- as the Pentagon looks at broadening its own investigation into the incident, which the top U.S. military official calls an embarrassment. 

The fallout from the alleged misconduct has spread across both the Secret Service and the U.S. military. On the Secret Service side, a senior law enforcement official confirmed to Fox News that all 11 who were recalled and placed on administrative leave have had their clearances revoke. The official added that among the 11 are two supervisors and three members of the counter-assault team, indicating that senior officials are involved. 

The incident late last week not only overshadowed President Obama's visit to Colombia but has raised troubling security questions. Some lawmakers have said agents could have been compromised, or opened themselves to blackmail, because of their actions. 

Obama said Sunday he would be "angry" if the allegations turn out to be true. Defense officials said Monday they, too, were "embarrassed" by the role of military personnel. 

"I can speak for myself and ... my fellow chiefs, we're embarrassed by what occurred in Colombia," Army Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff, said at a news conference. "Several of our members distracted the issue from what was a very important regional engagement for our president."

"We let the boss down," Dempsey said.

At least five U.S. service members may have been involved in "inappropriate conduct" while in Colombia in support of the Secret Service team, according to the military. The Pentagon so far, though, has not provided many specifics on what wrongdoing military personnel are accused of committing, other than violating curfew. They were staying in the same hotel where a dispute with prostitutes is alleged to have occurred.

"Whether our forces are in Colombia or any other country, or here in this country, we expect them to abide by the highest standards of behavior," Defense Secretary Leon Panetta said Monday. "I don't want to prejudge it, but obviously if violations are determined to have been the case, then these individuals will be held accountable."

The Pentagon also suggested it may be broadening its investigation into U.S. military members assigned to Colombia, with Pentagon spokesman George Little saying there "may be more than five" involved in the incident.

Little offered few details Monday, other than to say the U.S. Southern Command may be looking at more than one military service as potentially being involved. 

Obama addressed the incident for the first time on Sunday. He said Secret Service personnel, like the rest of any U.S. delegation abroad, must "observe the highest standards." 

"We're here on behalf of our people and that means that we conduct ourselves with the utmost dignity and probity. And obviously what's been reported doesn't match up with those standards," Obama said, on the closing day of his visit to Colombia. 

The president, though, said he would wait until the internal investigation is complete before rendering a judgment. He said he expects the probe to be "thorough" and "rigorous" -- and that if the allegations turn out to be true, "then of course I'll be angry." 

The president addressed the controversy during a joint press conference with Colombian President Juan Manuel Santos. Allegations that a Secret Service unit interacted with prostitutes in Colombia ahead of Obama's visit overshadowed a trip that was supposed to focus on trade and other pressing issues between the U.S. and its Latin American ally. 

Obama cast the incident as isolated, and praised the Secret Service as a whole. 

"These men and women perform extraordinary service on a day-to-day basis protecting me, my family, U.S. officials," he said. "They do very hard work under very stressful circumstances and almost invariably do an outstanding job." 

Details are still emerging, but at least one Secret Service agent in Colombia is said to have had a dispute with a prostitute at the hotel where agents were staying. Nearly a dozen agents were subsequently recalled and replaced with a new crew before Obama's arrival. 

Fox News' Ed Henry and Justin Fishel contributed to this report.