David Warner and Steve Smith both hit centuries as Australia reached 371-6 in their 50 overs at Kingsmead.
But an unbroken seventh-wicket stand of 107 by David Miller (118 not out) and Andile Phehlukwayo (42 not out) saw the Proteas home with four balls to spare.
South Africa now have an unassailable 3-0 lead in the five-match series.
Their 372-6 eclipsed India's score of 362-1 - also against Australia - in 2013.
The highest successful one-day international run chase was also made by South Africa, who hit 438-9 - against Australia - in Johannesburg in 2006.
Eight of the top 10 successful ODI run chases have now been achieved against Australia.