The Sea Shepherd Conservation Society anti-whaling organisation says it has ended its summer campaign against the Japanese fleet in the Southern Ocean.
Japan has been stepping up international pressure to try to rein in the group which has vowed to physically stop the slaughter of the whales.
Sea Shepherd said its ship, the Steve Irwin, which collided last week with a whaling vessel, was heading back to Australia with only four days of fuel reserves left.
The ship's Canadian captain, Paul Watson, said they have been successful in saving many whales but have decided to withdraw in the face of increasingly violent tactics from the Japanese whalers.
Japan has complained after activists hurled bottles of rancid butter at the whalers and tried to board the ship.
Sea Shepherd in turn accused Japan of crossing the line by deploying acoustic weapons, which send out high-frequency sound waves to disorient the activists.
The environmentalists said that use of the sonic weapons left three of their crew with injuries, with one man requiring five stitches above his left eye.
Glenn Inwood from Japan's Institute of Cetacean Research said the Steve Irwin had engaged in serious acts of violence that put lives at risk.
"The Netherlands and Australia need to examine closely their roles in allowing Watson to commit these maritime crimes," Inwood said.
Japan last week summoned the ambassador of The Netherlands, where the Steve Irwin is registered, to demand it take action against the environmentalists.
Japan kills whales using a loophole in a 1986 international moratorium on commercial whaling that allows "lethal research" on the mammals, and makes no secret of the fact that the animals' meat is then served as food.
Only Norway and Iceland defy the whaling moratorium altogether.
Japan, which says whaling is part of its culture, hunts up to 850 whales each year in the Antarctic Ocean despite strong objections from political allies Australia and New Zealand.
For the previous two seasons Japan's catch was curbed largely because of harassment by environmentalists