Washington â€"  Four Secret Service employees will fight their dismissals following the Colombian prostitution scandal -- arguing they were made scapegoats for behavior tolerated by the organization for years, The Washington Post reported late Tuesday.

Twelve agents were disciplined or lost their jobs following allegations that some brought back prostitutes to their Cartagena hotel rooms days before President Barack Obama's April 14 arrival in Colombia for a diplomatic summit.

Eight of the agents were forced out of the Secret Service in the wake of the scandal, while one had his security clearance revoked and was expected to leave the agency. The remaining three agents were cleared of serious wrongdoing.

According to the Post, some agents have told a very different story of what went on that night to what has been widely reported -- that the dozen men went on a group outing to seek out prostitutes.

Instead, they say the agents ventured out separately and met women at a variety of locations, with some encounters resulting in consensual sex that did not involve money changing hands.

One agent, described as a single 29-year-old assigned to the Washington office, told investigators during a polygraph test that he resigned after being threatened with being fired.

He brought two women back to his room and said he did not believe they were prostitutes. He told the women to leave when they asked for money for sex.

He is one of the four agents seeking to be reinstated and has withdrawn his resignation.

Another contesting his resignation is an agent who took a woman to a different hotel on another night during the trip and had a sexual encounter.

He came forward voluntarily to admit the encounter to his bosses and had not originally been under suspicion.

Current and former Secret Service employees told the newspaper that there is an unwritten code of "what happens on the road, stays on the road," and sexual encounters during official travel were condoned.

They said the organization is known as the "Secret Circus" -- describing what happens when large numbers of agents arrive in a city.

In addition to an internal probe by the Secret Service, the Department of Homeland Security Office of Inspector General has also opened its own inquiry into the scandal.

Secret Service Director Mark Sullivan and Charles K. Edwards, the acting inspector general of the Department of Homeland Security, will discuss the matter before a Senate committee Wednesday.