The party will also start "a disciplinary procedure that could lead to a permanent exclusion", and has announced that its president, former Prime Minister Édouard Philippe, will call the complainant Sandrine Josso (MoDem).
"If true, the facts mentioned are very serious (...) Horizons will never tolerate the slightest complacency towards sexual and gender-based violence," the party said in a statement.
Minister Christophe Béchu, secretary general of Horizons, said on Friday that the senator could not "remain within the party (...) if there was any doubt."
The senator was indicted and placed under judicial supervision on Friday evening.
The complainant reportedly felt unwell after having a drink on Tuesday night at the Paris home of the 66-year-old senator, with whom she did not have an intimate relationship, according to the Paris prosecutor's office.
Samples from Josso's body revealed the presence of ecstasy and searches were carried out at the senator's office and his home where investigators found ecstasy, according to the prosecutor's office.
Joël Guerriau denied the accusations during a confrontation.
According to Sandrine Josso's lawyer, Julia Minkowski, the deputy was "taken ill after drinking a glass of champagne", then saw the senator "grab a small plastic bag containing something white", and was able to "extricate himself in extremis from this ambush".
Joël Guerriau "will fight" to "demonstrate that he never wanted to administer a substance to his colleague and long-time friend to abuse her" and "will demonstrate that it was a handling error that caused the dramatic inconvenience," his lawyer Rémi-Pierre Drai told AFP.
According to sources familiar with the matter, Mr. Guerriau said he believed he had obtained a high, not ecstasy, from a member of the Senate for his own use in order to cope with what he described as personal hardships.
A banker by profession, elected to the upper house in 2011, Joël Guerriau is Secretary of the Senate and Vice-President of the Committee on Foreign Affairs, Defence and the Armed Forces.
Interviewed on Saturday on franceinfo, the head of the centrist Union group, Hervé Marseille, "thinks that there will be precautionary measures" against him in the upper house.
(With AFP)
This week's recapFrance 24 invites you to look back on the news that marked the week
Subscribe now
Take international news with you wherever you go! Download the France 24 app