Customize container bash

Does anyone know how to customize the bash interface of a container? I really only want to set -o vi so I can use vi style controls for the command line without having to remember to execute this command every time I start a new ssh session. I mean, I also wouldn’t mind being able to install oh-my-zsh, but that’s not strictly necessary.