Trouble with Garmin G1000 Trainer, Dual Screen mode

[TangoWhiskey]

I'm having some trouble with the dual-screen trainer on ONE of my two Windows 7 x64 systems.... of course, it's my best, fastest, dual monitor system, and that's the one where I want to run this. Here's what I'm seeing, and what I've looked at or tried. I believe it's a UDP communication problem between the two CDUSIMv2 processes.

I'm launching them from Garmin's Start Dual Screen batch file, which executes these two commands:

start CDUSimv2.exe -pfd1
start CDUSimv2.exe -mfd

After powering up the PFD, we get this (exactly the same as on the other working system).

After a few moments, the Terrain functionality self-checks and we get SVT on the PFD. PFD is in automatic reversionary mode because the MFD is powered down:

As soon as I power on the MFD, and the second CDUSIMv2 process launches, the PFD loses all signals. (more on that later)

It's NORMAL for the PFD to flash red x's for a moment, but once the two simulators have their UDP connections (cross-talk) established, then the red x's go away on my good system.

Finally, the MFD comes up, but now it's in reversionary mode, because of the communication link failure between the two displays:

Here's how it SHOULD look after this step, and does look on my OTHER Windows 7 x64 machine:

Here are the two processes that get ran... they're running in 32 bit mode on both systems, so no differences there:

On both systems, the PFD always opens three UDP ports (always 5001 & 6549, plus one more in the 5xxxx range), and the MFD opens six UDP ports (always 6001, 6002, and 6539, plus three additional ports in the 5xxxx range, contiguous, such as 54241, 54242, and 54243).

netstat -ao (so I can see the PIDS or process id's attached to the ports) shows all ports created and open and listening across all adapters (0.0.0.0:6001, *:*, for example), on both systems.

I did some loopback packet tracing on both systems using rawcap and analyzing those with wireshark, and see no material differences between the working and non-working systems.

Both systems have Realtek hard-wired GigE adapters with latest firmware. I opened the adapter's advanced properties and found no differences in their settings. TCP v4 and v6 are enabled on both systems; binding order of adapters is set correctly, and the TCP v4 / v6 precedence is the same on both systems. Behavior is the same with all adapters disabled.

I also used msconfig to start in a more bare-bones (no startup programs) mode (won't run in safemode due to graphics card requirements), as well as cross-comparing services running on both machines to ensure there weren't any networking or DCOM services running on the working machine that were disabled or stopped on the non-working machine.

Here's what my networking settings look like on the non-working machine; they mirror (other than IP and hostname) the settings on the working host.

CDUSIMv2 has permissions for both UDP and TCP in Windows Firewall. Behavior is exactly the same, however, with Windows Firewall off.

Reversionary mode is not MANUALLY enabled on either screen:

Any ideas of other things to check or try? I can send the pcap dumps of the working and non-working systems to someone, if you're a network guru... I'm three days into trying to get this to work and am now banging my head against the wall.

Oh, one more thing: the SINGLE SCREEN mode works just fine... I can toggle between PFD, MFD, Reversionary mode, the menu-menu manual and tracking types work as they should. It clearly seems to be a crosstalk UDP communications issue between the two running instances.

P.S.--dual screen mode USED to work on this machine; not sure what changed that. I noticed it wasn't working with my v8.x sim, so ordered the v12.x latest version, and it's not working either, so it's not the software. It's something about this particular computer....

If it matters or helps, here's some system information from the non-working host:

------------------
System Information
------------------
Time of this report: 6/24/2012, 11:47:33
Operating System: Windows 7 Ultimate 64-bit (6.1, Build 7601) Service Pack 1 (7601.win7sp1_gdr.120503-2030)
Language: English (Regional Setting: English)
System Manufacturer: Gigabyte Technology Co., Ltd.
System Model: GA-MA790GP-DS4H
BIOS: Award Modular BIOS v6.00PG
Processor: AMD Phenom(tm) II X4 940 Processor (4 CPUs), ~3.0GHz
Memory: 8192MB RAM
Available OS Memory: 8190MB RAM
Page File: 2458MB used, 8802MB available
Windows Dir: C:\Windows
DirectX Version: DirectX 11
DX Setup Parameters: Not found
User DPI Setting: 96 DPI (100 percent)
System DPI Setting: 96 DPI (100 percent)
DWM DPI Scaling: Disabled
DxDiag Version: 6.01.7601.17514 32bit Unicode

---------------
Display Devices
---------------
Card name: NVIDIA GeForce 9500 GT
Manufacturer: NVIDIA
Chip type: GeForce 9500 GT
DAC type: Integrated RAMDAC
Device Key: Enum\PCI\VEN_10DE&DEV_0640&SUBSYS_00000000&REV_A1
Display Memory: 4069 MB
Dedicated Memory: 997 MB
Shared Memory: 3071 MB
Current Mode: 1280 x 1024 (32 bit) (60Hz)
Monitor Name: Generic PnP Monitor
Monitor Model: DELL 1905FP
Monitor Id: DEL400D
Native Mode: 1280 x 1024(p) (60.020Hz)
Output Type: DVI
Driver Name: nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
Driver File Version: 8.17.0013.0142 (English)
Driver Version: 8.17.13.142
DDI Version: 10
Driver Model: WDDM 1.1
Driver Attributes: Final Retail
Driver Date/Size: 5/15/2012 05:48:00, 18044224 bytes
WHQL Logo'd: Yes
WHQL Date Stamp:
Device Identifier: {D7B71E3E-4500-11CF-3E63-0D201FC2C535}
Vendor ID: 0x10DE
Device ID: 0x0640
SubSys ID: 0x00000000
Revision ID: 0x00A1

 

[TangoWhiskey]

Oh, and of course I ran latest version of MalwareBytes definitions in safe mode, and checked my hosts file... all is well there.

 

[Alexb2000]

If you believe this is a network problem, perhaps UDP port blocking. Have you tried just using a crossover cable to narrow it down? Also, disable the windows firewall, get back to a basic IP stack and work up from there?

Just Shootin' in the dark.

 

[TangoWhiskey]

If you believe this is a network problem, perhaps UDP port blocking. Have you tried just using a crossover cable to narrow it down? Also, disable the windows firewall, get back to a basic IP stack and work up from there?

Just Shootin' in the dark.

I did perform a netsh ip stack reset, too... not sure how a crossover cable would help, as this UDP communication is on the loopback adapter, which occurs even if the network adapter on the computer is disabled. Already tried with Windows Firewall disabled, as described above.

Thanks for the ideas!

 

[Alexb2000]

I did perform a netsh ip stack reset, too... not sure how a crossover cable would help, as this UDP communication is on the loopback adapter, which occurs even if the network adapter on the computer is disabled. Already tried with Windows Firewall disabled, as described above.

Thanks for the ideas!

OK never mind. I thought you were trying to put two systems together, one PFD one MFD. Now that I read it again, you are just having trouble with one system that drives two displays.

You put that in caps, don't know how I missed that.

 

[Alexb2000]

OK I'll try to offer a more intelligent somewhat relevant response. Have you looked at this:

http://support.microsoft.com/kb/979612

Possible latency issue?

 

[TangoWhiskey]

OK I'll try to offer a more intelligent somewhat relevant response. Have you looked at this:

http://support.microsoft.com/kb/979612

Possible latency issue?

I found that too--downloaded it (they send it to you via email) and tried to apply it--it said it didn't apply to my system, even though I picked the Win7 x64 version.

But... I think I'm onto something here with some more investigation. When the MFD is powered up, it gets chatty on its dynamically allocated UDP port with the PFDs static port 5001, every time. On the GOOD system, there's lots of chatter, and the first packet is always 77 bytes, then a varying packet size thereafter. See the packet sequence numbers, packet sizes, below. I've filtered the view on the good system's data here, from two different runs, to port 5001 as the UDP destination port:

GOOD SYSTEM TEST 1:

GOOD SYSTEM TEST 2:

Now, compare that with the system that doesn't work:

BAD SYSTEM TEST:

ALL of the packets are only 54 bytes long! There's only one request per second; repeated handshake attempts, I presume. When I use wireshark's "follow UDP stream" from the first packet to 5001, the good system has a conversation of 397kb; the bad system has a conversation of only 910 bytes (not kb). Both logs were stopped as soon as the MFD fully booted.

I did discover that the GOOD system had a default TTL of 128, and the non-working system had a default TTL of 64; I changed it to 128 and rebooted, and verified the latest capture shows an IP v4 TTL of 128 on the loopback, but the problem persists.

So either communication to 5001 is being blocked, or the packet is getting truncated from the expected 77 bytes to 54 bytes, and the G1000 PFD doesn't know what to do with the initial communication.

ALSO... I verified both systems and both displays (PFD/MFD) are all running the same plane (C172SP), so the communications patterns should be predictable and similar.

I should also say that not ALL UDP packets on the bad system are 54 bytes long--just all the packets to port 5001. There's definitely something to this thought, though... here's the packet size analysis between the GOOD system and the BAD system:

GOOD SYSTEM PACKET SIZES to ports 5001 and 6001 only:

BAD SYSTEM PACKET SIZES to ports 5001 and 6001 only:

P.S.--I only started using WireShark this year... what a great and useful program (and FREE!). I'd never get this level of granularity without it. It runs on Windows or Linux. On Windows systems, winpcap can't capture loopback traffic details, but a free download of rawcap can, and produces pcap files that wireshark can analyze. Should be in every IT engineer's toolset. There's a good 3-video primer on YouTube that starts here: http://www.youtube.com/watch?v=pk4OfsxxB4g

 

[Alexb2000]

I am thinking that since UDP is stateless, that the problem must be on the sender side vs. a conversational problem. No handshake in UDP that I am aware of?

Interesting, the time delay between packets is also really long on the bad system vs. good.

Also, have you compared the actual files that make up the IP stack for the same versions between machines? Checked for multiple versions, etc?

I'll keep thinking about it.

 

[TangoWhiskey]

I am thinking that since UDP is stateless, that the problem must be on the sender side vs. a conversational problem. No handshake in UDP that I am aware of?

Interesting, the time delay between packets is also really long on the bad system vs. good.

Also, have you compared the actual files that make up the IP stack for the same versions between machines? Checked for multiple versions, etc?

I'll keep thinking about it.

I'll try that IP stack component check. I just tried removing and rediscovering the network adapter in the machine (Windows re-installing drivers)--same behavior afterwards.

I've discovered that the time delay too which you refer is the same on the good system as the bad, for any communications between the PDF and port 6001 on the MFD, in dual mode, the PFD polls port 6001 via UDP once a second to see if the MFD is up and running yet. Once the PFD is up and running, it connects, then it gets chatty.

The "bad" system DOES connect from the PFD to the MFD's port 6001 once it starts; it's the MFD trying to connect back to the PFD's port 5001 that fails on the errant server.

 

[TangoWhiskey]

I am thinking that since UDP is stateless, that the problem must be on the sender side vs. a conversational problem. No handshake in UDP that I am aware of?

Only whatever the programmer wrote for the custom socket communication....

 

[TangoWhiskey]

Whoo-hoo! Problem solved. Easy solution, but before I give the answer for anybody else that runs into this issue, let's have a little fun with it.

I downloaded Microsoft's PortQry tool from here, and ran it first on the good system and then on the bad system, as I started up the PFD and MFD.

I'll give you one screenshot... from the bad computer, before either screen is powered on:

What's wrong? And no, there's not a orphaned MFD running somewhere.

 

[Alexb2000]

Well you're way ahead of me technically. I was going to go with a malware software or firewall disabling 5001 because of a possible denial of service type attack over that port.

However, you certainly tested that early in the process.

How about in the network card settings disabling the UDP checksum offload feature and letting the OS do it?

Or alternatively-

Disabling IPv6 because it can't handle a zero checksum and perhaps the application wasn't written with IPv6 considered.

Crappy guess I know, I'm no computer professional.

 

[TangoWhiskey]

Alex, the checksum offloading really doesn't come into play on a loopback port, because the computer knows the packets aren't leaving the computer, so it doesn't checksum them anyway.... but, that's a good point you raise, as in WireShark I was seeing errors about "checksum doesn't match", and that wasn't really a "problem". Since I have checksum generation offloaded to my GigE NIC, and these packets didn't travel out the NIC, they didn't HAVE checksums... those values were "0x0000" in the packets, and didn't match WireShark's computed Checksum, so it flagged most packets as having errors. Fortunately, in WireShark's preferences pages for Internet Protocol, you can tell it to 'don't compute checksums' and those "false errors" go away.

IPv6 wasn't in play here, either... although it's installed on the box, it's protocol #2 in the binding / preference order, and WireShark clearly showed this traffic was protocol v4.

So... the actual problem ended up being.... (the clue is in the last picture I posted, in the black command line box, and what is being shown for port 6001 on the MFD, which isn't even powered up yet).

 

[TMetzinger]

Sounds like you have a firewall or something else listening on that port.

 

[Alexb2000]

Alex, the checksum offloading really doesn't come into play on a loopback port, because the computer knows the packets aren't leaving the computer, so it doesn't checksum them anyway.... but, that's a good point you raise, as in WireShark I was seeing errors about "checksum doesn't match", and that wasn't really a "problem". Since I have checksum generation offloaded to my GigE NIC, and these packets didn't travel out the NIC, they didn't HAVE checksums... those values were "0x0000" in the packets, and didn't match WireShark's computed Checksum, so it flagged most packets as having errors. Fortunately, in WireShark's preferences pages for Internet Protocol, you can tell it to 'don't compute checksums' and those "false errors" go away.

IPv6 wasn't in play here, either... although it's installed on the box, it's protocol #2 in the binding / preference order, and WireShark clearly showed this traffic was protocol v4.

So... the actual problem ended up being.... (the clue is in the last picture I posted, in the black command line box, and what is being shown for port 6001 on the MFD, which isn't even powered up yet).

OK so I was looking at it backwards. In that I thought the 5001 port not listening was the issue, but it wouldn't be if neither PFD or MFD wasn't powered up yet. So that would mean that some other process is using 6001, which the MFD also wants to use, close?

 

[TangoWhiskey]

Sounds like you have a firewall or something else listening on that port.

Correct,Tim and Alex!

Back when I bought my Fujitsu duplex scanner (which is amazing, by the way, just ask Spike,I think he uses one too), it installed software for a hardware key... that software was called Sentinel, by Safe-Net. That software runs as a service and listens on port 6001 for authentication requests.

Just random luck that the Garmin G1000 sim uses the same port. I uninstalled the Sentinel software after proving that stopping their services fixed the red x's on the PFD. If I need the software for my scanning software to work, I can reinstall it--they have an XML file that will let you specify which port it should use.

Now, the funny thing is, I SHOULD HAVE CAUGHT THIS three days ago, when I was mapping process ids to ports. But, I was filtering my netstat output by PID, not by PORT, so I only saw six ports used by the MFD (including 6001). If I'd thought to ALSO try filtering the netstat output by PORT (6001), I would have SEEN that there were two different PIDs vying for its attention.

Alex, thanks for the help and ideas today. I appreciate it!

 

[Alexb2000]

Correct,Tim and Alex!

Back when I bought my Fujitsu duplex scanner (which is amazing, by the way, just ask Spike,I think he uses one too), it installed software for a hardware key... that software was called Sentinel, by Safe-Net. That software runs as a service and listens on port 6001 for authentication requests.

Just random luck that the Garmin G1000 sim uses the same port. I uninstalled the Sentinel software after proving that stopping their services fixed the red x's on the PFD. If I need the software for my scanning software to work, I can reinstall it--they have an XML file that will let you specify which port it should use.

Now, the funny thing is, I SHOULD HAVE CAUGHT THIS three days ago, when I was mapping process ids to ports. But, I was filtering my netstat output by PID, not by PORT, so I only saw six ports used by the MFD (including 6001). If I'd thought to ALSO try filtering the netstat output by PORT (6001), I would have SEEN that there were two different PIDs vying for its attention.

Alex, thanks for the help and ideas today. I appreciate it!

Fun problem. I learned something, thanks for taking the time to share it.

 

[TangoWhiskey]

You bet. I'm hoping it helps some other soul who runs into this... or myself, in the future, when it happens again and I wonder "how'd I fix that last time?!"

There was nothing online useful, searching on the G1000 sim's executable name...

 

[ChrisK]

netstat -ano show it? =)

 

[TangoWhiskey]

netstat -ano show it? =)

Yeah, I ran that, too. You need to run it with elevated privileges to see the executable name. I still would have missed the conflict, as I wasn't looking for all instances of 5001 or 6001 or the other ports those two apps use, I was copying the output to Notepad++ and filtering for lines with CDUSIMv2 in them... and thus not seeing the OTHER app that was ALSO using 6100.

 

[ChrisK]

Nice. Will keep that in mind! It would be nice if the software alerted you on startup that it could not listen on that port.

 

[TangoWhiskey]

LOL. It does. Big red x's!! ;-)

But I know what you mean... "in a more user-friendly manner!!

 

[Zbuilder]

I just discovered this Trainer and having the same problem on Windows 10. I don't know what the solution was. I didn't really understand much of what was said in this thread. I did try Run as administrator but no joy. :-( Any help would be appreciated.