Technical Notes

Troubleshooting XDMCP Connections to UNIX and Linux Hosts
Technical Note 1229
Last Reviewed 05-Dec-2008
Applies To
Reflection X version 12.0 through 14.x
Summary

This technical note describes how to troubleshoot problems connecting with XDMCP (X Display Manager Control Protocol), such as timeout or "cannot open display" errors. The troubleshooting options in this note apply to connections made to the most common UNIX and Linux hosts.

For information specific to Red Hat Linux, see Technical Notes 1523 and 1680.

XDMCP Connections

The XDMCP protocol is used to communicate between the X Display Manager (XDM) running on a host machine and the X server (Reflection X) running on your workstation (computer). The X Display Manager provides a user friendly interface for starting X clients and opening X windows.

Troubleshooting XDM Connection Problems

There are several reasons why an XDMCP connection might fail. The most common reasons are:

• The XDM daemon is not running on the host, or the host is unavailable.
• Port 177 (UDP) or port 6000 (TCP) is blocked.
• The workstation is not correctly configured in the network DNS table.

To determine the cause of the problem and resolve it, start with section A, XDMCP Error Codes, and proceed through each subsequent section as directed.

A. XDMCP Error Codes

The most common error codes and their definitions are:

• RX2102 - "Your XDMCP connection timed out - make sure that the hosts on your network are running XDM programs. (RX2102)"

• RX2112 - "XDM session to host failed (Session XXX failed for display <fully qualified PC name>:0: Cannot open display). (RX2112)"

B. Examine the Network Environment

Before you begin the troubleshooting steps, be sure that your network environment is compatible with XDMCP. XDMCP does not work with, or needs to be specifically configured for, the following network environments:

• Secure Shell (SSH): XDMCP connections cannot be created through an SSH tunnel, because SSH does not support UDP transmissions. See Technical Note 1818 for more information.
• Network Address Translation (NAT): If you are using Reflection X in a network environment configured for NAT (sometimes referred to as IP masquerading, transparent proxying, or IP address overloading), read Technical Note 1513 before proceeding.
• Virtual Private Networks (VPN): If you are connecting to the network through a VPN tunnel, read Technical Note 1580 before proceeding.
• Some hosts include security features that prevent Reflection X (or other X servers) from establishing XDMCP connections. In some cases, this feature is enabled by default. Verify that your host is configured to permit XDMCP connectivity.

Once you determine that your environment is compatible with XDMCP, proceed to C. Increase the Connection Timeout.

C. Increase the Connection Timeout

Increasing the connection timeout value may resolve connection problems, particularly when working in a low bandwidth environment. The connection timeout defines the number of seconds that Reflection X will try to make an XDMCP connection. By default, the value is set to 15 seconds.

Follow these steps to configure the connection timeout value.

1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
2. Click Advanced to open the Advanced XDMCP Settings dialog box.
3. In the Connection timeout field, enter the number of seconds that Reflection X should wait before timing out, and then click OK.
4. Try to connect.

Did this work?

D. Make a Direct XDMCP Connection

By default, Reflection X makes XDMCP connections using broadcast UDP packets. Some routers and firewalls are configured to block UDP or broadcast packets. Follow these steps to make a non-broadcast XDMCP connection:

1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
2. In the Method drop-down list, select Direct.
3. In the Host name field, enter your IP address or fully-qualified host name (including the domain name). For example, unixhost.mycompany.com.
4. Click Connect.

Did this work?

1. Follow the steps provided in the previous section, C. Increase the Connection Timeout, and increase the timeout setting for your Direct XDMCP connection. Try the connection again. If it still fails, proceed to step b.
2. If connecting using the fully-qualified host name did not work, try using the host IP address. If the connection still fails, proceed to E. Ping the Host from the Workstation.

E. Ping the Host from the Workstation

Follow the steps below to determine if the host is available.

1. Click Start > Run.
2. In the Open field, enter command, and then click OK.
3. At the command prompt, enter:

Did the host respond?

F. Verify that Reflection X is Detecting the Correct IP Address

Reflection X sends the workstation's IP address to the XDM daemon on the host. To verify that the workstation IP address is correct, follow the steps below.

1. Click Start > Run.
2. In the Open field, enter command, and then click OK.
3. At the command prompt, enter the following command and note the IP address displayed:

4. Note the IP address of the interface you want to use.
5. In the Reflection X Manager, click Settings > Network.

Do the IP addresses match?

G. Ping the Workstation from the Host

Verify that the host can access the workstation and correctly resolve the workstation's fully-qualified name. Incorrect resolution of the workstation name to the workstation IP address may cause XDMCP connections to fail.

An example of a fully-qualified workstation name is "lindaj.mycompany.com" (not "Lindaj"). Depending on the configuration of your network, it may be possible to connect to your host using an unqualified host name; however, because this does not work in all network configurations, using a fully-qualified name is recommended.

To locate your workstation's fully-qualified name, follow the appropriate steps for your operating system:

Follow the steps below to ping your workstation from the host.

1. Make a Telnet (or SSH if required) connection to your host using Reflection for UNIX and OpenVMS.
   1. Start Reflection for UNIX and OpenVMS, click Connection > Connect to Best Network.
   2. Enter your fully-qualified host name or IP address and click OK.
2. Enter your host logon name and password.
3. At the command prompt, enter the appropriate command for your host type:

Did this work?

H. Confirm that the Host XDM Daemon is Running

Follow these steps to confirm that the XDM daemon is running on your host.

1. Make a Telnet connection to your host and log on.
2. At the command prompt, enter the appropriate command for your host system:

Host	Command
UNIX host running CDE	ps -eaf | grep dtlogin
UNIX - Generic	ps -eaf | grep login
Linux host running KDM	ps -eaf | grep kdm
Linux host running GDM	ps -eaf | grep gdm
Linux host running XDM	ps -eaf | grep xdm

A typical UNIX host response looks similar to the following; the path will vary depending on the host environment.

In this example, the 'root' line confirms that the XDM daemon, entitled dtlogin, is running on the host, and is owned by the user 'root.'

A typical Linux host response looks similar to the following:

In this example, the 'root' line confirms that the XDM daemon, entitled kdm, is running on the host, and is owned by the user 'root.' (The '-nodaemon' parameter does not indicate that the daemon is unavailable.)

Note the following:

• The 'guest' line should be disregarded; it shows the status of the 'grep' command, not the status of the XDM daemon.
• If you do not get a response after entering the appropriate command for your host, the XDM daemon is not currently running. Is the Daemon running?

Note: Whether the XDM Daemon is running or not, you can follow the steps below to manually start your X sessions. This workaround may allow you to use Reflection X while you are troubleshooting the XDMCP connection problem.

To manually start an X session, follow the steps below:

1. Start Reflection X (it can be minimized).
2. Connect to your host using Reflection for UNIX and OpenVMS.
3. At the host command prompt, enter:

Note: When you manually start an X session, some windows may stay open when you exit. To close these windows, reset your Reflection X server from the Action menu in the Reflection X Manager.

I. Check Port 177

To make an XDMCP connection, the network must be configured to allow the host and X server to communicate using UDP packets over port 177.

To verify that port 177 is open, contact your network administrator for assistance.

Note: You may wish to use the Microsoft Portqry.exe command-line utility to determine if port 177 is available. For information about this utility, see Microsoft article 310099 at http://support.microsoft.com/default.aspx?scid=kb;en-us;310099.

Is Port 177 open on the host?

1. Click Start > Control Panel > Network and Internet Connections > Network Connections.
2. Right-click Local Area Connections > Properties.
3. Click Internet Protocol (TCP/IP) > Properties.
4. Click Advanced.
5. On the DNS tab, check "Use the connection's DNS suffix in DNS registration," click OK, and exit Local Area Connection Properties.
6. Re-try your host connection.

J. Check Port 6000

X protocol uses port 6000 for communication between the host and Reflection X; therefore, if port 6000 is blocked between the host and the workstation, you will be unable to connect using Reflection X. Follow the steps below to determine if port 6000 is available for the X protocol.

1. Start Reflection X and minimize the Reflection X Manager window. When Reflection X is started, it attempts to open port 6000 on the workstation.
2. Make a Telnet connection to the host using Reflection for UNIX and OpenVMS.
3. At the command prompt, type the following:

Is port 6000 open?

• Ensure that there is not a personal firewall blocking port 6000. If you are running Windows XP, see K. Check the Windows XP Firewall.
• Some versions of the Cisco VPN Client contain a built-in firewall called Stateful Firewall. If this is running on your workstation, configure it to allow traffic on port 6000. For further information regarding this product, contact Cisco support or your company’s IT department.
• Some versions of the Checkpoint VPN client contain a built-in firewall. If this is running on your workstation, configure the client to allow traffic over port 6000. For further information regarding this product, contact Checkpoint support or your company's IT department.
• Talk to your network administrator to determine if a router or network firewall is blocking port 6000.

• If port 6000 is being used, you will see a line similar to the following:

• If port 6000 is being used, and Reflection X is not running, another application is using port 6000 (and possibly disallowing Reflection X access to the port). Determine what application is using the port and disable it.
• Depending on which operating system you are using, you may be able to use a freeware utility, such as Active Ports by SmartLine Inc, to determine what application is using this port. This utility can be obtained by doing a web search for "active ports smartline".

K. Check the Windows XP Firewall

Windows XP includes the Internet Connection Firewall (ICF). Follow the steps below to determine if the Windows XP firewall is enabled on your system.

1. Click Start > Control Panel > Network and Internet Connections > Network Connections.
2. Right-click Local Area Connections > Properties, and then select the Advanced tab.

If “Protect my computer and network by limiting or preventing access to this computer from the Internet” is selected, the ICF is enabled.

Is the firewall enabled?

1. Click Start > Control Panel > Network and Internet Connections > Network Connections.
2. Right-click Local Area Connections > Properties. On the Advanced tab, click Settings.
3. On the Services tab, click Add, and enter the following values in the entry fields:

Field	Value
Description of service	Allow port 6000 for X11
Name or IP address...	<the workstation's IP address or name>
External port number for the service	6000
Internal Port number for the service	6000

4. Select TCP, and then click OK.
5. If Reflection X is configured for multiple Reflection X server instances, you will need to repeat steps 1 through 4 for each additional server port. For example, ports 6001, 6002, and 6003.

L. Record and Review a Network Trace

If you are familiar with network traces, take a network trace of your XDMCP connection attempt and refer to the following Reflection XDMCP connection details while reviewing the trace.

Note: If you do not have a network trace utility, you may want to try one of the free online utilities, such as Wireshark, a multi-platform network protocol analyzer available at http://www.wireshark.org/.

In the network trace, you should see something similar to the following, showing Reflection X and the host negotiating and maintaining the connection. (Exactly how the information is presented depends on the trace utility you are using.)

The first set of entries show Reflection X connecting to port 177 on the host using XDMCP protocol over UDP. Communication is established between a random port number on the PC and port 177 on the host. During this process Reflection X (on the PC) provides the host with information about which port number it is configured to use for X11 protocol. Typically, this is port 6000.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow UDP packets from the PC to the host on port 177.

Source	Destination	Protocol	Packets	Information
PC	Host	XDMCP	UDP	Query
Host	PC	XDMCP	UDP	Willing
PC	Host	XDMCP	UDP	Request
Host	PC	XDMCP	UDP	Accept
PC	Host	XDMCP	UDP	Manage

The second set of entries shows the host sending a SYN request to the PC on the port that Reflection X specified for X11 protocol (typically port 6000), using TCP. The following SYN and ACK communications are made between a random host port and the port Reflection X specified for X11 protocol.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow TCP packets from the host to the PC on port 6000 (or whichever port Reflection X is listening on for X11 traffic).

Source	Destination	Protocol	Packets	Information	Example
Host	PC	X11	TCP	specified host port number > X11 port number [SYN]	38197 > 6000 [SYN] or 38197 > X11 [SYN] Note: If the default X11 port is being used, 6000, "X11" may be displayed instead of 6000.
PC	Host	X11	TCP	X11 port number > specified host port number [SYN, ACK]	X11 > 38197 [SYN, ACK]
Host	PC	X11	TCP	specified host port number > X11 port number [ACK]	38197 > X11 [ACK]

Finally, the host and Reflection X begin communicating using X11 protocol over TCP on the ports established during the SYN-ACK process above. No additional ports need be opened in the firewall.

Source	Destination	Protocol	Packets	Information
Host	PC	X11	TCP	Initial connection request
PC	Host	X11	TCP	Initial connection reply
Host	PC	X11	TCP	Requests
PC	Host	X11	TCP	Responses
PC	Host	X11	TCP	Events
PC	Host	X11	TCP	Errors

Contact Attachmate

If you have worked through all of the troubleshooting suggestions in this document and are still having difficulties, contact Attachmate Technical Support for further technical assistance. For contact information, see http://support.attachmate.com/contact/.

Related Technical Notes

1513	Reflection X and Network Address Translation (NAT)
1523	XDMCP Connections to Red Hat Linux 6.2
1530	Startup Parameters for Reflection X
1580	Connecting through a VPN (Virtual Private Network) with Reflection X
1680	XDMCP Connections to Red Hat Linux 7.0 or Higher
9992	Reflection X Technical Notes