Citrix Remote Code Execution Vulnerability Exploited in the Wild

Dataprise Defense Digest

ID: D3-2023-0008-1

Severity: 9.5

Published: July 27, 2023

Executive Summary

On July 18, 2023 Citrix announced a critical Remote Code Execution (RCE) vulnerability in NetScaler ADC and NetScaler Gateway being exploited in the wild. This vulnerability with a CVSS score of 9.8 out of 10 is tracked as CVE-2023-3519. Citrix learned of the 0-day RCE vulnerability from a post in a hacker forum advertising an unauthenticated RCE for versions of Citrix ADC up to 13.1 build 48.47. Citrix made the announcement on the same day they made patches for all affected versions available, encouraging clients to upgrade all affected products immediately. As of July 22, 2023 Security researchers from the Shadowserver Foundation, a non-profit organization dedicated to enhancing Internet security have identified over 15,000 vulnerable Citrix applications exposed to the Internet.

Citrix did note that unpatched Netscaler appliances must be configured as a gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) or an authentication virtual server (the so-called AAA server) to be vulnerable to attacks.

Dataprise is aware of the critical nature of this vulnerability and is conducting a review of all available analyses of these vulnerabilities and the potential impact on our clients. This has been a major exercise as the investigation requires that a specific order of actions is taken to achieve the response objectives. Right now, our teams are working to confirm whether the recommended mitigation steps can be applied without causing any customer-facing service interruptions. If your organization’s Netscaler ADC is covered under a Dataprise Managed Service agreement, we will send a follow-up communication with details on our mitigation efforts.

Detailed Analysis

The Cybersecurity & Infrastructure Security Agency (CISA) detailed an exploit in their July 20, 2023 Cybersecurity Advisory, AA23-201A. As part of their initial exploit chain, the threat actors uploaded a TGZ file containing a generic web shell, discovery script , and setuid binary on the ADC appliance and conducted SMB scanning on the subnet. The actors used the web shell for AD enumeration and to exfiltrate AD data. Specifically, the actors:

• Viewed NetScaler configuration files /flash/nsconfig/keys/updated/* and /nsconfig/ns.conf. Note: These configuration files contain an encrypted password that can be decrypted by the key stored on the ADC appliance.
• Viewed the NetScaler decryption keys (to decrypt the AD credential from the configuration file).
• Used the decrypted AD credential to query the AD via ldapsearch. The actors queried for:
  • Users (objectClass=user) (objectcategory=person)
  • Computers (objectClass=computer)
  • Groups (objectClass=group)
  • Subnets (objectClass=subnet)
  • Organizational Units (objectClass=organizationalUnit)
  • Contacts (objectClass=contact)
  • Partitions (objectClass=partition)
  • Trusts (objectClass=trustedDomain)
• Used the following command to encrypt discovery data collected via openssl in “tar ball”: tar -czvf – /var/tmp/all.txt | openssl des3 -salt -k <> -out /var/tmp/test.tar.gz. (A “tar ball” is a compressed and zipped file used by threat actors for collection and exfiltration.)
• Exfiltrated collected data by uploading as an image file to a web-accessible path : cp /var/tmp/test.tar.gz /netscaler/ns_gui/vpn/medialogininit.png.

The actors’ other discovery activities were unsuccessful due to the critical infrastructure organization’s deployment of their NetScaler ADC appliance in a segmented environment. The actors attempted to:

• Execute a subnet-wide curl command to identify what was accessible from within the network as well as potential lateral movement targets.
• Verified outbound network connectivity with a ping command (ping -c 1 google.com).
• Executed host commands for a subnet-wide DNS lookup.

The actors also attempted to delete their artifacts. The actors deleted the authorization configuration file (/etc/auth.conf)—likely to prevent configured users (e.g., admin) from logging in remotely (e.g., CLI). To regain access to the ADC appliance, the organization would normally reboot into single-use mode, which may have deleted artifacts from the device; however, the victim had an SSH key readily available that allowed them into the appliance without rebooting it.

The actors’ post-exploitation lateral movement attempts were also blocked by network-segmentation controls. The actors implanted a second web shell on the victim that they later removed. This was likely a PHP shell with proxying capability. The actors likely used this to attempt proxying SMB traffic to the DC (the victim observed SMB connections where the actors attempted to use the previously decrypted AD credential to authenticate with the DC from the ADC via a virtual machine). Firewall and account restrictions (only certain internal accounts could authenticate to the DC) blocked this activity.

Detection Methods

As part of their Cybersecurity Advisory, CISA provided the following detection methods for exploit of CVE-2023-3519. Run the following victim-created checks on the ADC shell interface to check for signs of compromise:

1. Check for files newer than the last installation.
2. Modify the -newermt parameter with the date that corresponds to your last installation:
   • find /netscaler/ns_gui/ -type f -name *.php -newermt [YYYYMMDD] -exec ls -l {} \;
   • find /var/vpn/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;
   • find /var/netscaler/logon/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;
   • find /var/python/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;
3. Check http error logs for abnormalities that may be from initial exploit:
   • zgrep ‘\.sh’ /var/log/httperror.log*
   • zgrep ‘\.php’ /var/log/httperror.log*
4. Check shell logs for unusual post-ex commands, for example:
   • grep ‘/flash/nsconfig/keys’ /var/log/sh.log*
5. Look for setuid binaries dropped:
   • find /var -perm -4000 -user root -not -path “/var/nslog/*” -newermt [YYYYMMDD] -exec ls -l {} \;
6. Review network and firewall logs for subnet-wide scanning of HTTP/HTTPS/SMB (80/443/445) originating from the ADC.
7. Review DNS logs for unexpected spike in internal network computer name lookup originating from the ADC (this may indicate the threat actor resolving host post-AD enumeration of computer objects).
8. Review network/firewall logs for unexpected spikes in AD/LDAP/LDAPS traffic originating from the ADC (this may indicate AD/LDAP enumeration).
9. Review number of connections/sessions from NetScaler ADC per IP address for excessive connection attempts from a single IP (this may indicate the threat actor interacting with the web shell).
10. Pay attention to larger outbound transfers from the ADC over a short period of session time as it can be indicative of data exfiltration.
11. Review AD logs for logon activities originating from the ADC IP with the account configured for AD connection. 
12. If logon restriction is configured for the AD account, check event 4625 where the failure reason is “User not allowed to logon at this computer.”
13. Review NetScaler ADC internal logs (sh.log*, bash.log*) for traces of potential malicious activity (some example keywords for grep are provided below):
    • database.php
    • ns_gui/vpn
    • /flash/nsconfig/keys/updated 
    • LDAPTLS_REQCERT 
    • ldapsearch 
    • openssl + salt
14. Review NetScaler ADC internal access logs (httpaccess-vpn.log*) for 200 successful access of unknown web resources.

Affected Versions

• NetScaler ADC and NetScaler Gateway 13.1-49.13  and later releases
• NetScaler ADC and NetScaler Gateway 13.0-91.13  and later releases of 13.0 
• NetScaler ADC 13.1-FIPS 13.1-37.159 and later releases of 13.1-FIPS 
• NetScaler ADC 12.1-FIPS 12.1-65.36 and later releases of 12.1-FIPS 
• NetScaler ADC 12.1-NDcPP 12.1-65.36 and later releases of 12.1-NDcPP

Sources

• https://www.bleepingcomputer.com/news/security/over-15k-citrix-servers-vulnerable-to-cve-2023-3519-rce-attacks/
• https://www.bleepingcomputer.com/news/security/new-critical-citrix-adc-and-gateway-flaw-exploited-as-zero-day/
• https://support.citrix.com/article/CTX561482/citrix-adc-and-citrix-gateway-security-bulletin-for-cve20233519-cve20233466-cve20233467
• https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a

Contributing Authors

Stephen Jones, VP Cybersecurity

View all Dataprise Defense Digests here.