US Secret Service staff say three agents are leaving following allegations members of President Barack Obama's security staff were caught with prostitutes while preparing for his visit to Colombia.
US lawmakers said inviting the women to the secret service team's hotel in Cartagena ahead of the Americas Summit last weekend represented a security risk.
On Monday, the head of the US armed forces said the Secret Service and the military had "let the boss down".
Senator Susan Collins, briefed by the Secret Service director, said 20 women were found at the US hotel, the BBC reported. Some military personnel and 11 American agents were involved.
Secret Service Director Mark Sullivan ordered all the agents to return to Washington immediately, said Ms Collins, the top Republican on the Senate's Homeland Security and Governmental Affairs Committee.
During a phone briefing, she had asked Mr Sullivan a number of questions, including: "Who were these women? Could they have been members of groups hostile to the United States? Could they have planted bugs, disabled weapons, or... jeopardised [the] security of the president or our country?"
Speaking to reporters afterwards the senator said some agents "were uniformed personnel who are assigned to building security". Others were "these specialised agents who do security details", she said.
The group of military and Secret Service agents were reported to have been partying at Cartagena's Pley Club, which has been described as a high-end strip club in an industrial part of the port city.
Members of the Secret Service paid $US60 ($NZ73) each to the club's owners to bring women back to the Hotel Caribe, where they were staying, but the next morning one of the women demanded more money and a dispute ensued.