As indicated by its name, “ResourceInUseException”, obviously the port/file are occupied.
Firstly we all thought that the target port 8307 is already listened by another process. But after some investigation it showed that the port aren’t bound by any process on any network interfaces.
This made the things become mysterious, and the log doesn’t help at all (by default only info level log are outputted :( )

Debugging

We tore the vmware-hostd.exe and found it was invoking the vmacore.dll internally. With the keyword “ResourceInUseException” we quickly located the RTTI structure, and thanks to the advanced IDA 7.0, the RTTI and Exception Record structures are parsed automatically. The real function were easily found with the help of xref.
After some debugging, it seems that the hostd was trying to bind the port on a same address for multiple times, which would definitely fail and trigger the “ResourceInUseException”.
But why?
Seems that the function was trying to bind on all endpoints returned by resolver. Then we noticed verbose logging codes, changed the config.xml to output the verbose log into console. And the log confirmed the debugging result.

2018-08-30T16:50:51.125+08:00 verbose hostd[18064] [Originator@6876 sub=IO] Resolve;
2018-08-30T16:50:52.339+08:00 verbose hostd[18064] [Originator@6876 sub=IO] set_option; >, ‘no_delay’
2018-08-30T16:50:52.340+08:00 verbose hostd[18064] [Originator@6876 sub=IO] set_option ‘v6_only’; >
2018-08-30T16:50:52.341+08:00 verbose hostd[18064] [Originator@6876 sub=IO] Opened; >
2018-08-30T16:50:52.342+08:00 verbose hostd[18064] [Originator@6876 sub=IO] BindAndListen; >
2018-08-30T16:51:31.417+08:00 verbose hostd[18064] [Originator@6876 sub=IO] set_option; >, ‘no_delay’
2018-08-30T16:51:31.435+08:00 verbose hostd[18064] [Originator@6876 sub=IO] set_option ‘v6_only’; >
2018-08-30T16:51:31.436+08:00 verbose hostd[18064] [Originator@6876 sub=IO] Opened; >
2018-08-30T16:51:31.436+08:00 verbose hostd[18064] [Originator@6876 sub=IO] BindAndListen; >

But doesn’t help much. We still didn’t know why it binds multiple times.
Accidentally we noticed the “Asio” in the logging outputs, which reminded us of boost::asio. Then we realized that the resolver was returning duplicated endpoints.
Digging deeper, we found that the getaddrinfo() itself is returning duplicated addrinfo, but we failed to find any useful information. We doubt that this will happen on Windows machine with multi network cards.

Solution

The solution is extremely simple. The “localhost” and “127.0.0.1” magically have the same length, so we changed the used “localhost” in vmware-hostd.exe to “127.0.0.1”

Works like a charm!