The swelling scandal over the alleged sexual indiscretions of nearly two dozen Secret Service agents and military personnel in Colombia has raised questions about the "secret" culture at the nation's elite protection agency -- and few are feeling the heat more intensely than its director, Mark Sullivan. 

So far, officials on both sides of the aisle are largely defending Sullivan's leadership. President Obama "has confidence" in the director, White House spokesman Jay Carney said Tuesday. Rep. Peter King, R-N.Y., chairman of the House Homeland Security Committee, and other lawmakers have vouched for Sullivan and praised his swift response to allegations of misconduct in Colombia last week.

But at the same time, lawmakers have questioned what the incident reveals about the "culture" at the Secret Service. Some doubt this is the first time, and warn that the agents' alleged actions could have put the U.S. at risk. 

At least one lawmaker, Rep. Randy Forbes, R-Va., says Sullivan should be replaced. "In baseball, you get three strikes and you're out. I don't know how many strikes you get in national security, but the strikes are mounting up," Forbes, a member of the House Armed Services Committee, told Fox News on Wednesday. "Now they can't even control the fact that their agents are busing in loads of prostitutes. I think it's about time that we say they're out." 

And Ronald Kessler, the author who broke the story on the prostitution scandal in coordination with The Washington Post, said repeatedly in interviews this week that Sullivan must go. 

"The answer is not congressional hearings and a lot of hot air out of Congress. The answer is replacing Mark Sullivan," Kessler told Fox News, accusing him of presiding over an agency marred by "corner-cutting" and "laxness." He cited, among other complaints, the 2009 incident in which aspiring reality-show couple the Salahis crashed a White House dinner. 

Sullivan, while pursuing an internal investigation, has requested an independent review from the agency's inspector general. 

For these steps, lawmakers have applauded him. The highly regarded director -- a Bush administration appointee retained by Obama -- is practically a lifelong Secret Service agent, having served nearly three decades at the agency. He started his career in the Detroit office in 1983, moving his way steadily up the ranks -- through the presidential protective division in the 1990s and ultimately to the position of director in 2006. 

But the alleged misconduct in Colombia is a stain on the agency he runs. Some lawmakers are dubious the incident is isolated, and warn that the security risks are far higher than is being portrayed. 

"It is hard for me to believe that this is the one and only time this has happened because there were so many Secret Service members involved," Sen. Susan Collins, R-Maine, told Fox News. After speaking with Sullivan, Collins claimed Monday he revealed up to 21 women were brought to the hotel where the agents were staying. The Department of Homeland Security later disputed that figure. Eleven Secret Service agents have nevertheless been implicated, along with 10 military members, according to sources. 

Collins said this indicates a "culture problem." 

"If it had been one or two, then you would think that it's just one bad apple. But for 11 or 12 to be involved is really alarming," Collins said. 

She added: "Who knows who (the women) were working for? So they may have been sent to lure these agents into a trap to kidnap them, to disable their guns, to plant eavesdropping devices. The implications are mind boggling." 

Rep. Darrell Issa, R-Calif., chairman of the House Oversight and Government Reform Committee, has raised similar concerns. 

But Dave Wilkinson, a retired agent who served for years with Sullivan, rejected the idea of a "culture" problem at the Service as well as Kessler's comments. 

"It's really just those few agents and/or military folks that made this bad decision," he said. "It certainly doesn't suggest that the culture of the Secret Service is in any way a part of this." 

He said reports of so-called "wheels-up" parties -- where agents would supposedly kick back upon the conclusion of an assignment -- have been misconstrued. He said the term typically refers to formal receptions held by host embassies and ambassadors after a delegation visit. 

In an interview with FoxNews.com, Wilkinson stood by Sullivan and said the incident should not cost him his job. 

"We came up through the Secret Service together," he said, recalling their time together during the Clinton administration. "You will not find a more dedicated, more conscientious, level-headed leader for the Secret Service." 

King earlier told Fox News that Sullivan took a "bad rap" for the Salahi incident. He blamed that breach on the White House social secretary, who later resigned, rather than the Service. 

"I have a great regard for Director Sullivan," King said. "This could not have been (investigated) more efficiently." 

As director, Sullivan is responsible for an agency with more than 150 offices around the world. The Service not only protects the president and his family, but also is tasked with protecting foreign delegations and other top U.S. officials as well as conducting criminal investigations. 

On Tuesday, Carney said Sullivan "acted swiftly in response to this incident." Obama earlier said he would be "angry" if the allegations against the agents turn out to be true, but voiced confidence in the Service itself. 

FoxNews.com's Judson Berger and Fox News' Catherine Herridge contributed to this report.