Using Remote Desktop Protocol (RDP) to Log Into Your Windows Server

Remote Desktop Protocol (RDP) is a proprietary Microsoft network protocol that enables encrypted, full graphical access to a remote Windows machine over TCP port 3389. It transmits display output from the server to the client and keyboard/mouse input from the client back to the server, effectively giving you a live interactive session on a machine you are not physically present at.

For server administrators, RDP is the primary method of managing a Windows Server instance — whether that server is a bare-metal machine in a data center, a VPS Hosting environment, or a Dedicated Server running Windows Server 2019, 2022, or later. This guide covers the complete workflow: enabling RDP, securing it against real-world attack vectors, connecting from Windows, macOS, and Linux, and diagnosing the failures that catch administrators off guard.

How RDP Works Under the Hood

Before touching a single configuration setting, understanding the protocol's architecture pays dividends when troubleshooting.

RDP operates over TCP (and optionally UDP for multimedia redirection) and uses TLS 1.2/1.3 for transport encryption in all modern Windows Server releases. The session stack consists of several virtual channels that carry distinct data types simultaneously:

• Graphics channel — compressed display updates using RemoteFX or GDI acceleration
• Input channel — keyboard and mouse events
• Clipboard channel — bidirectional clipboard sharing
• Drive/printer redirection channels — local resource mapping into the remote session
• Audio channel — remote audio playback and recording redirection

Each channel is multiplexed over a single TCP connection to port 3389 by default. The server-side component is TermService (Remote Desktop Services), and the listener is managed by RDPWinST.sys at the kernel level. When you change the listening port, you are modifying a registry value that this driver reads at service start.

Network Level Authentication (NLA) adds a pre-session authentication layer using CredSSP (Credential Security Support Provider). With NLA enabled, the client must authenticate with valid credentials before the server allocates a full desktop session, which dramatically reduces the attack surface for denial-of-service and brute-force attacks against the login screen.

Prerequisites Checklist

Confirm every item below before attempting a connection:

• A Windows Server instance (2016, 2019, or 2022) with RDP enabled
• The public IPv4 address or a resolvable hostname for the server
• A valid administrator-level account on the server
• An RDP client installed on your local machine
• Port 3389 (or your custom port) open in both the OS firewall and any upstream network firewall or security group
• NLA support on the client side (enabled by default on all modern Windows, macOS, and Linux RDP clients)

Step 1: Enable RDP on the Windows Server

RDP is disabled by default on fresh Windows Server installations. There are two reliable methods to enable it.

Method A: GUI (System Properties)

1. Open Server Manager or press Win + R, type sysdm.cpl, and press Enter.
2. Navigate to the Remote tab.
3. Under Remote Desktop, select Allow remote connections to this computer.
4. Ensure Allow connections only from computers running Remote Desktop with Network Level Authentication is checked.
5. Click OK.

Method B: PowerShell (Preferred for Automation)

For headless or scripted deployments, PowerShell is faster and scriptable:

# Enable RDP
Set-ItemProperty -Path 'HKLM:SystemCurrentControlSetControlTerminal Server' `
    -Name "fDenyTSConnections" -Value 0

# Enable NLA
Set-ItemProperty -Path 'HKLM:SystemCurrentControlSetControlTerminal ServerWinStationsRDP-Tcp' `
    -Name "UserAuthentication" -Value 1

# Allow RDP through Windows Firewall
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

# Confirm the service is running
Set-Service -Name TermService -StartupType Automatic
Start-Service TermService

Verifying the Firewall Rule

The built-in firewall rule group "Remote Desktop" covers the necessary inbound rules. Verify with:

Get-NetFirewallRule -DisplayGroup "Remote Desktop" | Select-Object DisplayName, Enabled, Direction

If you are behind a cloud provider's security group (common on VPS with cPanel or managed VPS environments), you must also open port 3389 in the provider's external firewall panel — the OS firewall alone is not sufficient.

Step 2: Locate the Server's Public IP Address

From the Server Console

Open an elevated Command Prompt and run:

ipconfig /all

Look for the IPv4 Address under the active network adapter. On a server with multiple NICs (common on dedicated hardware), identify the adapter connected to the public network by cross-referencing the default gateway.

For a cleaner output, use PowerShell:

Get-NetIPAddress -AddressFamily IPv4 | Where-Object { $_.IPAddress -notlike "127.*" } |
    Select-Object InterfaceAlias, IPAddress

From Your Hosting Control Panel

If you provisioned the server through a hosting provider, the public IP is listed in the control panel dashboard. For AlexHost Dedicated Servers, the IP is displayed immediately after provisioning in the client area.

Important edge case: If your server sits behind NAT (e.g., a private cloud or a hypervisor with internal networking), the IP shown by ipconfig will be a private RFC 1918 address. You will need to connect to the NAT gateway's public IP and configure port forwarding to the server's private IP on port 3389.

Step 3: Connect to the Server via RDP

From Windows

The built-in client is mstsc.exe (Microsoft Terminal Services Client). Launch it from Run (Win + R) or the Start menu:

mstsc /v:YOUR_SERVER_IP:3389

For a full-screen session with drive redirection disabled (a security-conscious default):

mstsc /v:YOUR_SERVER_IP /f /nodrives

In the GUI:

1. Enter the server IP in the Computer field.
2. Click Show Options to pre-fill the username, which avoids the extra credential prompt.
3. Under the Experience tab, select the appropriate connection speed to optimize rendering.
4. Click Connect, accept the certificate warning on first connection (verify the certificate thumbprint if security is critical), and enter your password.

From macOS

Microsoft's official Microsoft Remote Desktop app (available free on the Mac App Store) is the recommended client.

1. Open the app and click the + button, then Add PC.
2. Enter the server IP in PC Name.
3. Under User Account, click Add User Account and enter your credentials.
4. Optionally, configure Display settings and Devices & Audio redirection.
5. Double-click the saved connection to initiate the session.

From Linux

Two mature options exist on Linux:

Remmina (GTK-based, recommended for desktop environments):

sudo apt install remmina remmina-plugin-rdp   # Debian/Ubuntu
sudo dnf install remmina remmina-plugin-rdp   # RHEL/Fedora

Launch Remmina, create a new connection, select RDP as the protocol, enter the server IP, credentials, and connect.

FreeRDP (command-line, ideal for scripting or headless clients):

sudo apt install freerdp2-x11
xfreerdp /v:YOUR_SERVER_IP /u:Administrator /p:'YourPassword' /cert:ignore /dynamic-resolution

The /cert:ignore flag suppresses the certificate warning — acceptable in a controlled environment but should be replaced with proper certificate pinning in production.

RDP Client Comparison

Step 4: Hardening RDP Against Real-World Threats

RDP is one of the most actively targeted services on the internet. Shodan consistently indexes millions of exposed RDP endpoints, and automated brute-force campaigns run 24/7. The following measures are not optional for any production server.

Change the Default Listening Port

Changing from 3389 to a non-standard port eliminates the vast majority of automated scanners. Edit the registry:

$newPort = 54321  # Replace with your chosen port
Set-ItemProperty -Path 'HKLM:SystemCurrentControlSetControlTerminal ServerWinStationsRDP-Tcp' `
    -Name "PortNumber" -Value $newPort -Type DWord

# Update firewall rule
New-NetFirewallRule -DisplayName "RDP Custom Port" -Direction Inbound `
    -Protocol TCP -LocalPort $newPort -Action Allow

# Disable the default rule
Disable-NetFirewallRule -DisplayName "Remote Desktop - User Mode (TCP-In)"

Restart-Service TermService

Remember to update your cloud security group to allow the new port and block 3389.

Restrict Access by IP Address

If your administrative workstation has a static IP, restrict RDP to that IP only:

Set-NetFirewallRule -DisplayName "RDP Custom Port" `
    -RemoteAddress "YOUR_ADMIN_IP"

Enforce Account Lockout Policy

Prevent brute-force attacks by configuring an account lockout threshold. In Group Policy (gpedit.msc):

• Account lockout threshold: 5 invalid attempts
• Account lockout duration: 30 minutes
• Reset account lockout counter after: 15 minutes

Or via PowerShell:

net accounts /lockoutthreshold:5 /lockoutduration:30 /lockoutwindow:15

Deploy a VPN or SSH Tunnel as a Gateway

The most robust approach is to not expose RDP to the public internet at all. Place RDP behind a VPN (WireGuard or OpenVPN) or an SSH tunnel. Administrators connect to the VPN first, then RDP to the server's private IP. This eliminates the attack surface entirely.

Enable Windows Defender Credential Guard

On Windows Server 2016 and later, Credential Guard isolates credential hashes in a virtualization-based security enclave, preventing pass-the-hash attacks that can pivot from a compromised RDP session.

Audit and Log RDP Sessions

Enable auditing via Group Policy under Computer Configuration > Windows Settings > Security Settings > Advanced Audit Policy Configuration > Logon/Logoff. Log both success and failure events. Forward logs to a SIEM or at minimum review Event ID 4624 (successful logon) and Event ID 4625 (failed logon) in Event Viewer regularly.

Step 5: Troubleshooting Common RDP Connection Failures

Connection Refused (Error Code 0x204 / "Remote computer cannot be reached")

This is almost always a firewall or routing issue.

# Test from the client machine (replace with your server IP and port)
Test-NetConnection -ComputerName YOUR_SERVER_IP -Port 3389

If TcpTestSucceeded returns False, the port is blocked. Check:

1. The OS firewall on the server (Get-NetFirewallRule -DisplayGroup "Remote Desktop")
2. The cloud provider's security group or ACL
3. Any intermediate hardware firewall between client and server

Authentication Error ("An authentication error has occurred. The function requested is not supported.")

This typically means the client does not support the Credential Security Support Provider (CredSSP) version required by the server, often surfacing after a Windows security update (CVE-2018-0886 patch). Fix on the server:

# Temporary workaround — update the client instead when possible
Set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindowsCurrentVersionPoliciesSystemCredSSPParameters' `
    -Name "AllowEncryptionOracle" -Value 2 -Type DWord

The permanent fix is to update the RDP client to a patched version that supports the updated CredSSP protocol.

"The remote session was disconnected because there are no Remote Desktop License Servers available"

This occurs when the server's Remote Desktop Session Host (RDSH) role is installed but no RD Licensing server is configured. For simple administrative access (not multi-user RDSH deployments), remove the RDSH role or configure a 120-day grace period:

# Check licensing mode
(Get-WmiObject -Namespace root/CIMV2/TerminalServices -Class Win32_TerminalServiceSetting).LicensingType

A value of 2 means Per Device, 4 means Per User. For admin-only access, the value should be 5 (not configured / grace period).

Black Screen After Login

A black screen on RDP connection usually indicates one of three causes:

• Explorer.exe crash: Press Ctrl + Alt + End to open Task Manager, then run explorer.exe from File > Run New Task.
• Display driver conflict: Disable hardware acceleration in the RDP session via Group Policy.
• GPU remoting issue: On servers with dedicated GPUs (relevant for GPU Hosting workloads), ensure the RemoteFX vGPU adapter is properly configured.

Slow or Laggy RDP Session

Optimize performance by adjusting the connection experience settings in mstsc:

mstsc /v:YOUR_SERVER_IP /f

In the Experience tab, select LAN (10 Mbps or higher) and disable Desktop background, Font smoothing, and Desktop composition for maximum responsiveness over high-latency links.

RDP vs. Alternative Remote Access Protocols

For Windows Server administration, RDP remains the superior choice in terms of performance, native integration, and feature depth. SSH is the preferred protocol for Linux-based VPS Hosting environments.

Managing Multiple RDP Sessions and Saved Credentials

Saving RDP Connection Files

mstsc supports .rdp configuration files, which are plain-text files you can version-control:

full address:s:YOUR_SERVER_IP:3389
username:s:Administrator
prompt for credentials:i:1
audiomode:i:0
redirectdrives:i:0
redirectclipboard:i:1

Save as server.rdp and double-click to launch, or invoke via:

mstsc server.rdp

Credential Manager

On Windows, saved RDP credentials are stored in Credential Manager (control keymgr.dll). For automated scripts or CI/CD pipelines that need RDP access, use the cmdkey utility:

cmdkey /generic:YOUR_SERVER_IP /user:Administrator /pass:YourPassword
mstsc /v:YOUR_SERVER_IP
cmdkey /delete:YOUR_SERVER_IP

Delete credentials immediately after use to avoid persistent credential exposure.

Practical Key-Takeaway Checklist

Use this as a pre-deployment and post-incident verification matrix:

Initial Setup

• RDP enabled via fDenyTSConnections = 0 in the registry
• NLA enforced (UserAuthentication = 1)
• TermService set to Automatic and running
• Firewall rule active for the correct port

Network & Access Control

• Default port 3389 changed to a non-standard port
• Cloud security group updated to reflect the new port
• IP allowlist configured in the firewall rule
• RDP not directly exposed to the internet (VPN gateway preferred)

Authentication & Hardening

• Account lockout policy configured (5 attempts / 30-minute lockout)
• Strong, unique password for all accounts with RDP access
• Credential Guard enabled on Windows Server 2016+
• RDP access limited to the built-in Administrators group or a dedicated RDP group

Monitoring

• Audit policy enabled for Logon/Logoff events
• Event ID 4625 alerts configured for repeated failures
• Session logs reviewed periodically or forwarded to a SIEM

Client-Side

• RDP client updated to the latest version
• .rdp files stored securely and not committed to public repositories
• Saved credentials in Credential Manager reviewed and pruned regularly

Frequently Asked Questions

What port does RDP use, and can it be changed?

RDP listens on TCP port 3389 by default. You can change it by modifying the PortNumber DWORD value under HKLM:SystemCurrentControlSetControlTerminal ServerWinStationsRDP-Tcp and restarting the TermService service. Update your firewall rules to match.

What is Network Level Authentication (NLA) and should it always be enabled?

NLA requires the client to authenticate via CredSSP before the server creates a full desktop session. This prevents unauthenticated users from reaching the Windows login screen, significantly reducing exposure to credential-stuffing and denial-of-service attacks. It should always be enabled on production servers unless you have a specific legacy client compatibility requirement.

Why does my RDP connection drop after a period of inactivity?

Session timeouts are controlled by Group Policy under Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Session Time Limits. The relevant policies are Set time limit for disconnected sessions and Set time limit for active but idle Remote Desktop Services sessions. Set these to your operational requirements rather than leaving them at defaults.

Can multiple users connect to a Windows Server via RDP simultaneously?

A standard Windows Server installation supports two concurrent administrative RDP sessions. For more simultaneous users, you need the Remote Desktop Session Host (RDSH) role and valid Remote Desktop Services Client Access Licenses (RDS CALs). Without proper licensing, the server enters a 120-day grace period before refusing connections.

Is RDP safe to expose directly on the public internet?

No. Directly exposing RDP on port 3389 to the internet invites automated brute-force attacks, ransomware delivery campaigns (RDP is the leading initial access vector for ransomware), and exploitation of unpatched vulnerabilities. Always place RDP behind a VPN, restrict access by IP, change the default port, and enforce NLA and account lockout policies as a minimum baseline.