“Women talk a lot,” the head of the Tokyo Olympics apologized for the statement