I'm not sure what you're trying to achieve by making GameState a typeclass. You might be too much in OOP mindset here — typeclasses are not OOP classes. A set of game states will probably be closed, so it might make more sense to just make it a single ADT.
Why does giving an explicit type when returning an element of a typeclass cause a compile error?
Look at your function's signature: GameState b => GameStateMonad a (Maybe b). Now remember that this implies forall b. GameState b => GameStateMonad a (Maybe b).
Implementation of the function cannot decide what b is — it must work for all of them (as long as they fit in constraints), because it's up for the caller to decide.
That's why return (Nothing :: Maybe Blank) is an error — it doesn't work for all types b— it only works for Blank. "Could not deduce (b ~ Blank)" means GHC can't prove type equality here. Same thing goes for Nothing :: Maybe Int. And that's also why adding type signature in the call site works.
I'll write something on ambiguities later maybe. I'm still pretty sure you're overengineering this code anyway, so the solution is to not do that.