Troubleshooting Reflection X 14.x XDMCP Connections to UNIX and Linux Hosts

This technical note describes how to troubleshoot problems connecting with XDMCP (X Display Manager Control Protocol), such as timeout or "cannot open display" errors. The troubleshooting options in this note apply to XDMCP connections made from Reflection X 14.x to the most common UNIX and Linux hosts.

For information specific to Red Hat Linux, see KB 7021775.

The XDMCP protocol is used to communicate between the X Display Manager (XDM) running on a host machine and the X server (Reflection X) running on your workstation (computer). The X Display Manager provides a user friendly interface for starting X clients and opening X windows.

Troubleshooting XDM Connection Problems

There are several reasons why an XDMCP connection might fail. The most common reasons are:

• The XDM daemon is not running on the host, or the host is unavailable.
• Port 177 (UDP) or port 6000 (TCP) is blocked.
• The workstation is not correctly configured in the network DNS table.

To determine the cause of the problem and resolve it, start with section A. XDMCP Error Codes, and proceed through each subsequent section as directed.

Note: The troubleshooting steps in this note help you ensure that the XDM daemon is running and that Reflection X can successfully connect and run sessions using this daemon. An alternate workaround for starting sessions is to manually start X sessions. This procedure is described below under "To manually start an X session without using the XDM Daemon."

A. XDMCP Error Codes

The most common error codes and their definitions are:

• RX2102 - "Your XDMCP connection timed out - make sure that the hosts on your network are running XDM programs. (RX2102)"

• RX2112 - "XDM session to host failed (Session XXX failed for display <fully qualified PC name>:0: Cannot open display). (RX2112)"

B. Examine the Network Environment

Before you begin the troubleshooting steps, be sure that your network environment is compatible with XDMCP. XDMCP does not work with, or needs to be specifically configured for, the following network environments:

• Secure Shell (SSH): XDMCP connections cannot be created through an SSH tunnel, because SSH does not support UDP transmissions. See KB 7021257 for more information.
• Network Address Translation (NAT): If you are using Reflection X in a network environment configured for NAT (sometimes referred to as IP masquerading, transparent proxying, or IP address overloading), read KB 7021672 before proceeding.
• Virtual Private Networks (VPN): If you are connecting to the network through a VPN tunnel, read KB 7021765 before proceeding.
• Some hosts include security features that prevent Reflection X (or other X servers) from establishing XDMCP connections. In some cases, this feature is enabled by default. Verify that your host is configured to permit XDMCP connectivity.

Once you determine that your environment is compatible with XDMCP, proceed to C. Increase the Connection Timeout.

C. Increase the Connection Timeout

Increasing the connection timeout value may resolve connection problems, particularly when working in a low bandwidth environment. The connection timeout defines the number of seconds that Reflection X will try to make an XDMCP connection. By default, the value is set to 15 seconds.

Follow these steps to configure the connection timeout value.

1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
2. Click Advanced to open the Advanced XDMCP Settings dialog box.
3. In the Connection timeout field, enter the number of seconds that Reflection X should wait before timing out, and then click OK.
4. Try to connect.

Did this work?

D. Make a Direct XDMCP Connection

By default, Reflection X makes XDMCP connections using broadcast UDP packets. Some routers and firewalls are configured to block UDP or broadcast packets. Follow these steps to make a non-broadcast XDMCP connection:

1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
2. In the Method drop-down list, select Direct.
3. In the Host name field, enter your IP address or fully-qualified host name (including the domain name). For example, unixhost.mycompany.com.
4. Click Connect.

Did this work?

1. Follow the steps provided in the previous section, C. Increase the Connection Timeout, and increase the timeout setting for your Direct XDMCP connection. Try the connection again. If it still fails, proceed to step b.
2. If connecting using the fully-qualified host name did not work, try using the host IP address. If the connection still fails, proceed to E. Ping the Host from the Workstation.

E. Ping the Host from the Workstation

Follow the steps below to determine if the host is available.

1. Click Start > Run.
2. In the Open field, enter command, and then click OK.
3. At the command prompt, enter:

Did the host respond?

F. Verify that Reflection X is Detecting the Correct IP Address

Reflection X sends the workstation's IP address to the XDM daemon on the host. To verify that the workstation IP address is correct, follow the steps below.

1. Click Start > Run.
2. In the Open field, enter command, and then click OK.
3. At the command prompt, enter the following command and note the IP address displayed:

4. Note the IP address of the interface you want to use.
5. In Reflection X Manager 14.1 SP1 or higher:
   1. Click Settings > View Settings.
   2. In the Search box, type “Detected IP”.
   3. Click the appropriate option for IPv4 or IPv6.
   4. Compare the IP address referenced under “Settings details” with the one you identified above.

1. Click Settings > Network.
2. Compare the IP address referenced under "Autodetect network interface" with the one you identified above.

Do the IP addresses match?

G. Ping the Workstation from the Host

Verify that the host can access the workstation and correctly resolve the workstation's fully-qualified name. Incorrect resolution of the workstation name to the workstation IP address may cause XDMCP connections to fail.

An example of a fully-qualified workstation name is "lindaj.mycompany.com" (not "Lindaj"). Depending on the configuration of your network, it may be possible to connect to your host using an unqualified host name; however, because this does not work in all network configurations, using a fully-qualified name is recommended.

To locate your workstation's fully-qualified name, follow the appropriate steps for your operating system:

Follow the steps below to ping your workstation from the host.

1. Make a Telnet (or SSH if required) connection to your host using Reflection for UNIX and OpenVMS, or Reflection Workspace if you are running Reflection X as part of Reflection X 2011.

1. Click Start > Programs > Attachmate Reflection > Host - UNIX and OpenVMS.
2. Click Connection > Connection Setup.
3. Select the protocol to use, such as TELNET or SECURE SHELL.
4. Enter Host name or IP address, and click Connect.
5. Enter your credentials to log into the host.

1. Click Start > Programs > Attachmate Reflection > Reflection Workspace.
2. Select VT Terminals, and click Create.
3. Under Connection, select the protocol to use, such as Telnet or Secure Shell, enter Host name / IP address and click OK.
4. Enter your credentials to log into the host.
2. At the command prompt, enter the appropriate command for your host type to ping your workstation:

Did this work?

1. Open the Local Area Connection Properties dialog box:

2. Select Internet Protocol (TCP/IP). If multiple protocol versions are present, select the appropriate protocol version (IPv4 or IPv6).
3. Click Advanced.
4. On the DNS tab, check "Use the connection's DNS suffix in DNS registration," click OK, and exit Local Area Connection Properties.
5. Re-try your host connection.

H. Confirm that the Host XDM Daemon is Running

Follow these steps to confirm that the XDM daemon is running on your host.

1. Make a connection to your host using either Reflection for UNIX and OpenVMS or Reflection Workspace, as in step G.
2. At the command prompt, enter the appropriate command for your host system:

Host	Command
UNIX host running CDE	ps -eaf | grep dtlogin
UNIX – Generic	ps -eaf | grep login
Linux host running KDM	ps -eaf | grep kdm
Linux host running GDM	ps -eaf | grep gdm
Linux host running XDM	ps -eaf | grep xdm

A typical UNIX host response looks similar to the following; the path will vary depending on the host environment.

In this example, the 'root' line confirms that the XDM daemon, entitled dtlogin, is running on the host, and is owned by the user 'root.'

A typical Linux host response looks similar to the following:

In this example, the 'root' line confirms that the XDM daemon, entitled kdm, is running on the host, and is owned by the user 'root.' (The '-nodaemon' parameter does not indicate that the daemon is unavailable.)

Note the following:

• The 'guest' line should be disregarded; it shows the status of the 'grep' command, not the status of the XDM daemon.
• If you do not get a response after entering the appropriate command for your host, the XDM daemon is not currently running. Is the Daemon running?

Note: Whether the XDM Daemon is running or not, you can follow the steps below to manually start your X sessions. This workaround may allow you to use Reflection X while you are troubleshooting the XDMCP connection problem.

To manually start an X session without using the XDM Daemon

You can use the following steps to start an X desktop session if the XDM Daemon is not running.

1. Start Reflection X (it can be minimized).
2. Connect to your host using Reflection for UNIX and OpenVMS or Reflection Workspace.
3. At the host command prompt, enter:

Note: When you manually start an X session, some windows may stay open when you exit. To close these windows, reset your Reflection X server from the Action menu in the Reflection X Manager.

I. Check Port 177

To make an XDMCP connection, the network must be configured to allow the host and X server to communicate using UDP packets over port 177.

To verify that port 177 is open, contact your network administrator for assistance or use the Microsoft Portqry.exe command-line utility to determine if port 177 is available. For information about this utility, see Microsoft article 310099 at http://support.microsoft.com/default.aspx?scid=kb;en-us;310099.

Is Port 177 open on the host?

J. Check Port 6000

X protocol typically uses port 6000 for communication between the host and Reflection X; therefore, if port 6000 is blocked between the host and the workstation, you will probably be unable to connect using Reflection X (one exception is when using Secure Shell). Follow the steps below to determine if port 6000 is available for the X protocol.

1. Start Reflection X and minimize the Reflection X Manager window. When Reflection X is started, it attempts to open port 6000 on the workstation.
2. Make a connection to the host using Reflection for UNIX and OpenVMS or Reflection Workspace.
3. At the command prompt, type the following:

Is port 6000 open?

• Ensure that there is not a personal firewall blocking port 6000. See K. Check the Windows Firewall.
• Some versions of the Cisco VPN Client contain a built-in firewall called Stateful Firewall. If this is running on your workstation, configure it to allow traffic on port 6000. For further information regarding this product, contact Cisco support or your company’s IT department.
• Some versions of the Checkpoint VPN client contain a built-in firewall. If this is running on your workstation, configure the client to allow traffic over port 6000. For further information regarding this product, contact Checkpoint support or your company's IT department.
• Talk to your network administrator to determine if a router or network firewall is blocking port 6000.

• If port 6000 is being used, you will see a line similar to the following:

• If port 6000 is being used, and Reflection X is not running, another application is using port 6000 (and possibly disallowing Reflection X access to the port). Determine what application is using the port and disable it.
• Depending on which operating system you are using, you may be able to use a freeware utility, such as Active Ports by SmartLine Inc, to determine what application is using this port. This utility can be obtained by doing a web search for "active ports smartline".

K. Check the Windows Firewall

Follow the steps below to view and modify the Windows Firewall settings

1. Click Start > Control Panel > Windows Firewall.

Is the firewall enabled?

1. Click Start > Control Panel > Windows Firewall.
2. From the left pane, click Allow a program or feature through Windows Firewall.
3. Click Allow another program.
4. Browse to select the Reflection X program file. (The default location is either C:\Program Files\Attachmate\Rx.exe, or C:\Program Files\Attachmate\R14\Rx.exe.)

1. Click Start > Control Panel > Windows Firewall.
2. Click Exceptions.
3. Click Add Program.
4. Browse to select the Reflection X program file. (The default location is either C:\Program Files\Attachmate\Rx.exe, or C:\Program Files\Attachmate\R14\Rx.exe.)

L. Record and Review a Network Trace

If you are familiar with network traces, take a network trace of your XDMCP connection attempt and refer to the following Reflection XDMCP connection details while reviewing the trace.

Note: If you do not have a network trace utility, you may want to try one of the free online utilities, such as Wireshark, a multi-platform network protocol analyzer available at http://www.wireshark.org/.

In the network trace, you should see something similar to the following, showing Reflection X and the host negotiating and maintaining the connection. (Exactly how the information is presented depends on the trace utility you are using.)

The first set of entries show Reflection X connecting to port 177 on the host using XDMCP protocol over UDP. Communication is established between a random port number on the PC and port 177 on the host. During this process Reflection X (on the PC) provides the host with information about which port number it is configured to use for X11 protocol. Typically, this is port 6000.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow UDP packets from the PC to the host on port 177.

Source	Destination	Protocol	Packets	Information
PC	Host	XDMCP	UDP	Query
Host	PC	XDMCP	UDP	Willing
PC	Host	XDMCP	UDP	Request
Host	PC	XDMCP	UDP	Accept
PC	Host	XDMCP	UDP	Manage

The second set of entries shows the host sending a SYN request to the PC on the port that Reflection X specified for X11 protocol (typically port 6000), using TCP. The following SYN and ACK communications are made between a random host port and the port Reflection X specified for X11 protocol.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow TCP packets from the host to the PC on port 6000 (or whichever port Reflection X is listening on for X11 traffic).

Source	Destination	Protocol	Packets	Information	Example
Host	PC	X11	TCP	specified host port number > X11 port number [SYN]	38197 > 6000 [SYN] or 38197 > X11 [SYN] Note: If the default X11 port is being used, 6000, "X11" may be displayed instead of 6000.
PC	Host	X11	TCP	X11 port number > specified host port number [SYN, ACK]	X11 > 38197 [SYN, ACK]
Host	PC	X11	TCP	specified host port number > X11 port number [ACK]	38197 > X11 [ACK]

Finally, the host and Reflection X begin communicating using X11 protocol over TCP on the ports established during the SYN-ACK process above. No additional ports need be opened in the firewall.

Source	Destination	Protocol	Packets	Information
Host	PC	X11	TCP	Initial connection request
PC	Host	X11	TCP	Initial connection reply
Host	PC	X11	TCP	Requests
PC	Host	X11	TCP	Responses
PC	Host	X11	TCP	Events
PC	Host	X11	TCP	Errors

Contact Attachmate

If you have worked through all of the troubleshooting suggestions in this document and are still having difficulties, contact Attachmate Technical Support, https://support.microfocus.com/contact/, for further assistance.

Legacy KB ID

This article was originally published as Attachmate Technical Note 1229.