I have had this same problem for many years through 6 different version of SMPlayer. I have never been able to get S/PDIF DTS or AC3 sound to work. I have other apps which work fine such as VLC Player. I have tried every combination of settings many times. This has been an issue on multiple computers for me, some running under Windows 7 64bit and now Windows 8 64bit.
I wish it would get resolved because I find SMPlayer better in some ways than VLC, but with this major FAIL I hardly ever use SMPlayer.