Kazakhstan’s Rybakina, 25, said she would need “magic” to progress in Melbourne after needing treatment on her back during her third-round win over Ukraine’s Dayana Yastremska.
She had won her previous two matches against Keys, but dropped the first set with the American neutralising her big serve and taking control of the rallies.
Rybakina, a finalist in Melbourne in 2023, fought back midway through the encounter to win four straight games and secure the second set, but Keys would triumph in the decider.
At 3-3 she took control before eventually sealing her spot in the last eight with a searing cross-court winner on her second match point.
Keys will play Ukraine’s 28th seed Elina Svitolina in the quarter-finals after she beat Russian Veronika Kudermetova 6-4 6-1.
Svitolina, 30, recovered from a poor start to reach her 12th Grand Slam singles quarter-final, before dedicating the win to her country.
“This fighting spirit that I have is completely Ukrainian spirit that I try to show, that I try to represent, as well,” said Svitolina.
“For me to find a way to win matches, to find a way to bring a little light, a little win for Ukrainian people, is something that I feel I am responsible of. To bring the fight is the least that I can do.”