Correct me if I'm wrong, but I've tried pretty much every possible way to do this, and it hasn't worked. So I'd like vmware to simply put it in writing somewhere that the following isn't possible, so that other people don't waste as much time as I did. I've seen many posts about this but I've never seen solutions to those posts.
I have successfully been able to connect a kernel debugger from a windows host to a windows VM (debugee). However, I've not been able to connect a kernel debugger from another windows VM on either linux or windows host OS.
On windows, the virtual serial (named pipe) is named
.\pipe\com_2, yield CPU on poll is checked, and the debuggee is set as "This end is the server"/"The other end is an application". I boot the debugee with a boot.ini entry which includes /noexecute=optin /fastdetect /debug /debugport=com1 /baudrate=112500. Again, the host system can connect to this VM with "windbg.exe -k com:port=
.\pipe\com_2,baud=112500,pipe". But if I set up a debugger VM, with named pipe
.\pipe\com_2, "This end is the client"/"The other end is a virtual machine". Then I try to do many combinations of the windbg command from within the debugger VM but none work, despite the fact that I can see its virtual serial light blinking. Example commands are
windbg.exe -k com:port=com1,baud=112500,pipe
windbg.exe -k com:port=com2,baud=112500,pipe
windbg.exe -k com:port=
.\com1,baud=112500,pipe
windbg.exe -k com:port=
.\com2,baud=112500,pipe
for all of those, windbg will connect to the port and wait (I left out the ones I tried which it wouldn't even connec to), but windbg never receives the information to establish the connection. Is there any way to find out exactly which com port name the vmware virtual serial device gets named as? Because the only way I can see this working is if the virtual serial is getting named as something weird and if I just know that name, then maybe if I connect windbg to that name it would work.
As I said, I also did this with a linux host system. The only difference is that I named the named pipe (which isn't actually a unix-style named pipe) as /tmp/mysock.
Jack