Fine kicking from Simione Kuruvoli put Fiji 12-8 up at half-time before Josua Tuisova crossed after the restart.
Lalakai Foketi went over late on to cut the deficit to 22-15 but Eddie Jones' side could not mount a comeback.
Fiji's first-ever World Cup win over the Wallabies moves them above their opponents into second in Pool C.
The pair are locked together on six points after two matches with Fiji ahead thanks to their head-to-head record.
With just the top two advancing to the quarter-finals, Fiji's match in Saint-Etienne was crucial.
The Flying Fijians had lost their opening pool game 32-26 against Wales and would have faced an uphill battle to reach the knockouts with a second consecutive defeat.
Victory appeared unlikely; they had last beaten Australia 69 years ago and had lost 17 of their past 18 Tests against the Wallabies.
Fiji's victory was their 15th over a tier one nation and their second in three weeks, having beaten England in a warm-up fixture in August.
Kuruvoli did miss a last-minute penalty that would have denied Australia a losing bonus point.
But the Fijians, who face Georgia next after a week off, are in a brilliant position to reach the knockouts of the World Cup for just a third time.