What's the URL (the Youtube page) of the video you tried to play?
I've just tested with this video https://www.youtube.com/watch?v=ChOhcHD8fBA (extracting the 8K URL with youtube-dl) and it plays ok (or at least it tries, because my computer isn't fast enough to play 8K videos) with both mplayer and mpv.
About the subtitles, there is support for UTF-8 but you need to select it in preferences -> subtitles -> encoding.