That ppa does not contain any 20.04 packages. You should purge that.
As to xfce sorry no idea.
Just to note. The Ubuntu Budgie team only supports Debian and Ubuntu Budgie. We have not tested any other combination nor other distros such as mint since we dont know how those distros differ from official Ubuntu.