Hi there people,
I just had an issue under Ubuntu Linux 17.10 (Gnome Wayland) with the video playing outside of the main smplayer window. I am with Version: 17.7.0 (Revision 8599) of SMPlayer.
While i found out that there is an option "Mplayer/mpv in own window" under program settings, it was already unchecked. But the video still played in an own window instance.
To fix this i changed General->Video->Output Driver->"x11 (slow)" in the user settings. Now the video plays in the main player window