One should always keep their tefillin on at least until after the kaddish after ובא לציון. It is best to keep them on until after the kaddish after עלינו. One should not remove his tefillin while reciting אמן יהא שמיה רבא.