The prime minister is expected to skip a visit to the Shinto shrine during the three-day festival, which runs through Monday.