VirtualBox creates serial ports as unix domain sockets, which cannot be used with terminal emulators. Use socat to bind them to ptys, and Gnu Screen can deal with them just fine.
VBoxManage modifyvm <vmname> --uartmode1 server ~/.VirtualBox/serial/<vmname>-S0 # or 1 for uart 2 VBoxManage startvm <vmname> socat ~/.VirtualBox/serial/<vmname>-S0 pty,link=~/.VirtualBox/serial/<vmname>-S0.pty
In screen, attach to serial port using screen command entry mode
screen ~/.VirtualBox/serial/<vmname>-S0.pty