Since the colors worked fine while being loggged in directly, I just un-uncommented the line force_color_prompt=yes in the file ~/.bashrc, that gave me colors over ssh, too:

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
force_color_prompt=yes

(Ubuntu 18.04 LTS)

https://askubuntu.com/questions/16336/how-to-get-coloured-terminal-over-ssh