x11vnc -create -forever -localhost
-create launches Xvfb (on DISPLAY=:20 or something like that)
-forever enables reconnection
-localhost only listens to local connections, so you'll have to forward the port (5900+nr) for instance over ssh.
Then one may need a windowmanager too, for instance
You may put fluxbox in the .xinitrc file too.