The firing of the director, comedian Kentaro Kobayashi, comes a little more than 24 hours before the scheduled start of the ceremony Friday.
Tokyo 2020 president Seiko Hashimoto said in a news conference that organizers are reviewing the entirety of the program for the opening ceremony in the wake of Kobayashi's dismissal. Hashimoto said they became aware of his past remarks about the Holocaust, which he made in 1998, on Wednesday morning.
Hashimoto said they "deeply regret" the situation.
Footage of Kentaro Kobayashi from the 1990s recently emerged in which he seems to be making jokes about the Holocaust.
Japan's Olympic chief Seiko Hashimoto called out the video saying it ridiculed "painful facts of history".
His dismissal is the latest in a string of scandals to hit the Games.
It comes days after a composer quit the team creating the ceremony after it emerged he had bullied classmates with disabilities at school.
The Simon Wiesenthal Center, a Jewish global human rights organization based in Los Angeles, condemned Kobayashi's remarks on Wednesday. The center, citing Japanese news reports, said Kobayashi "made malicious and anti-Semitic jokes including 'let's play Holocaust.' "
"Any person, no matter how creative, does not have the right to mock the victims of the Nazi genocide," said Abraham Cooper, the center's associate dean and global social action director.
Kobayashi's dismissal is the latest in a string of embarrassing moments for the organizing committee, and particularly its opening ceremony team.
Keigo Oyamada, a Japanese composer, resigned from the Tokyo 2020 creative team earlier this week amid backlash for bullying a classmate with disabilities during his childhood.
A previous creative director for the opening ceremony, Hiroshi Sasaki, resigned in March amid revelations that he had made sexist remarks about a well-known female entertainer in Japan, likening her in a brainstorming session to a pig.