Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for new Windows PV tools #306

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Add documentation for new Windows PV tools #306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 72 additions & 28 deletions docs/troubleshooting/windows-pv-tools.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,6 @@
# Windows Agent / PV-Tools
# Windows PV Tools

Common issues with Windows PV tools.

## Windows Management Agent failed to install

### Cause
There was an issue with the installing of the drivers certificate, so the drivers did not load silently.

### Solution
Resolved with version 8.2.2.200-RC1 and newer.

***
Common issues and topics related to Windows PV tools.

## PV-Drivers missing in the Device Manager

Expand All @@ -21,19 +11,8 @@ If despite running the Windows tools installer, there's no devices visible in th
### Solutions

#### Leftovers from old Citrix XenServer Client Tools.
1. remove any xen*.* files from `C:\Windows\system32` like
* xenbus_coinst_7_2_0_51.dll
* xenvbd_coinst_7_2_0_40.dll
* xenbus_monitor_8_2_1_5.exe
* and similiar `xen*_coinst` and `xen*_monitor` files
2. remove any leftover `XenServer` devices from device manager, also display hidden `XenServer` devices and remove them!
* To show hidden devices in Device Manager: `View -> Show Hidden Devices`

#### There was an issue with the installing of the drivers certificate, so the drivers did not load silently

Resolved with version 8.2.2.200-RC1 and newer.

***
See the XenClean guides below for instructions.

## Network PV drivers aren't working.

Expand All @@ -43,12 +22,10 @@ If the tools are installed, while XCP-ng Center says that I/O is optimized, but

### Possible Solutions

* Resolved with version 8.2.2.200-RC1 and newer.

* Clean your system from `Citrix Client Tools` _AND_ `XCP-ng Client Tools` to create a clean state.
* Then install the Client Tools from scratch.

[This Guide](../../vms#upgrade-from-citrix-xenserver-client-tools) may help you through the process.
See the XenClean guides below for instructions.

## Not all PV drivers are correctly installed

Expand All @@ -60,4 +37,71 @@ It's possible that some antivirus blocks the end of the installation of the PV d

### Solution

Simply pausing the agent and doing a reboot will make the XenTools to install succesfully. After a succesfull installation, enabling the SentinelOne agent again is possible without any other issues regarding the tools or drivers.
Simply pausing the agent and doing a reboot will make the XenTools to install succesfully. After a succesfull installation, enabling the SentinelOne agent again is possible without any other issues regarding the tools or drivers.

## Completely removing existing Xen PV drivers with XenClean

XenClean is an utility for cleanly removing Xen PV drivers and management agents of the following products:

* XCP-ng Windows PV Tools, versions 8.2 and 9.1
* XenServer VM Tools for Windows, versions 9.3 and 9.4
* Other Xen drivers

It is included in the installation CD of XCP-ng Windows PV Tools 9.1 and above.

To use XenClean, simply run the `Invoke-XenClean.ps1` script **as Administrator**. Your system will automatically reboot.

**Note**: You should disable the "Manage Citrix PV drivers via Windows Update" option on your VM before running XenClean. Otherwise, Windows may reinstall PV drivers from Windows Update after rebooting.

**Note**: If you downloaded XenClean from the internet, you may need to unblock the script file before running it. This can be done by right-clicking the file, then choosing **Properties** - **Unblock** - **OK**.

**Tip**: XenClean leaves its log files at `%TEMP%\xenclean-<time>`. Please provide these logs in case of uninstallation failure.

## Windows fails to boot (hanging at boot or BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`)

In some situations (failed uninstallation, major Windows version upgrades), Xen PV drivers (whether Citrix or XCP-ng) may cause Windows to fail to start (hanging at boot, BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`).
The XenBootFix utility included with XCP-ng Windows PV Tools 9.1 and above helps you disable any active Xen PV drivers and get your system to a bootable state before running XenClean.

**Note**: The utility only runs in Windows Preinstallation Environment (PE) or Windows Recovery Environment (RE). It will not run from Safe Mode.

Below is a procedure for using XenBootFix to recover a non-booting VM:

1. Boot into Windows PE or Windows RE in command line-only mode. There are a few ways to accomplish this:
* If your Windows installation BSODs on boot multiple times, it will automatically boot into Windows RE. Choose **Troubleshoot** - **Command Prompt**.
* When running Windows Server, press **F8** before Windows starts, then choose **Repair Your Computer**. Choose **Troubleshoot** - **Command Prompt**.
* Boot your VM using a Windows Setup or Windows PE CD image. If you don't see a command line, press **Shift+F10**.
2. Identify your Windows installation drive letter.
* Use the `dir` command to list files in a given drive letter. For example: `dir C:\` (the backslash is required)
* In some cases, your Windows partition should already be mounted. Try the first few letters (`C:`, `D:`, `E:`).
* If you cannot find your Windows drive letter, you may need to assign a new drive letter with Diskpart.
* Type `diskpart` at the command line. The command prompt should change to `DISKPART>`
* Type `list vol` then make a note of your Windows partition and its drive letter (if any).
* If your Windows partition does not have a drive letter, type `sel vol N` where `N` is the volume number shown in Diskpart, then type `assign letter=W`. Your Windows partition will be assigned the drive letter `W:`.
* Finally, at the `DISKPART>` prompt, type `exit` to exit Diskpart.
3. Obtain XenBootFix.
* If you're using XCP-ng Windows PV Tools 9.1 or later, it is located at `W:\Program Files\XCP-ng\Windows PV Drivers\XenBootFix\XenBootFix.exe` where `W:` is your Windows drive letter.
* If you have PowerShell, run the following command: `Invoke-WebRequest https://<download path of XenBootFix.exe> -OutFile XenBootFix.exe`
* Failing all that, you can create a new ISO image containing XenBootFix using WinCDEmu, ImgBurn or a similar tool, then attach it to your VM.
* **Note**: If using Windows PE, do not remove its CD image when it's running. You may encounter unexpected errors otherwise.
4. Run the command `<path to XenBootFix.exe> W:\Windows` where `W:` is your Windows drive letter.
* **Note**: Make sure the drive letter belongs to your actual Windows installation and not Windows PE/RE. By default, Windows PE/RE use the drive letter **X:**.
5. Type `exit` to close Command Prompt. If using Windows RE, choose **Continue** to boot into Windows. Windows should now start normally.
6. **You must immediately run XenClean from within Windows to remove the remaining Xen drivers**. See instructions above.

## Connecting to guests using serial console

In some situations, you might want to expose a VM's serial console to network for troubleshooting purposes (e.g. Windows kernel debugging).
You can use the following procedure:

* Shut down the VM.
* On your host, run the following command: `xe vm-param-set uuid=<uuid> platform:hvm_serial=tcp::<port>,server,nodelay,nowait` where `<uuid>` is the VM's UUID and `<port>` is the TCP port on the host that exposes the VM's serial port, e.g. 7001.
* You need to open the selected port on the host's firewall: `/etc/xapi.d/plugins/firewall-port open 7001`
* Note that any host running the VM will need this command.
* Connecting to port 7001 on the host will connect to your VM's serial port. You can use Telnet, PuTTY or any similar tools.
* To configure kernel debugging on Windows, disable Secure Boot and BitLocker then run the following commands within the VM:
```
bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200
```
Connect using [WinDbg](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/) using the `Attach to kernel` option with a connection string: `com:ipport=7001,port=<host IP>`
* To undo the changes and remove the serial setting, use `xe vm-param-remove uuid=<uuid> param-name=platform param-key=hvm_serial`
95 changes: 10 additions & 85 deletions docs/vms/vms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,9 +370,7 @@ Citrix tools:
XCP-ng tools:
* :heavy_plus_sign: Fully open-source.
* :heavy_plus_sign: Maintained by the XCP-ng project.
* :heavy_minus_sign: The drivers are not signed with a key recognized by Microsoft, so Secure Boot can't be enabled with them.
* :heavy_minus_sign: The sources for the management agent are older than that of Citrix (they have stopped updating GitHub a while ago).
* :heavy_minus_sign: Have not been updated in a long time, due to difficulties finding skilled developers to work on them. Being actively worked on.
* :heavy_minus_sign: At the moment, only contains I/O device drivers and not the management agent.
* :heavy_minus_sign: Won't transparently replace existing Citrix tools. You need to remove Citrix tools first if they are present, in order to install XCP-ng tools.

It's now up to you to choose.
Expand Down Expand Up @@ -403,7 +401,7 @@ If you are using Xen Orchestra, you can switch the "Windows Update tools" advanc
If your VM already has XCP-ng tools and you wish to switch to Citrix tools, then you need to do the same kind of clean-up as described farther in this document for the opposite situation.

#### XCP-ng Windows Guest Tools
Drivers built by the XCP-ng community.
Drivers built and signed by Vates.

**Download**: [https://github.com/xcp-ng/win-pv-drivers/releases](https://github.com/xcp-ng/win-pv-drivers/releases)

Expand Down Expand Up @@ -445,95 +443,22 @@ xe vm-param-get param-name=has-vendor-device uuid={VM-UUID}
`True` means that it's active, `False` that it isn't. It needs to be `False`.

###### Install the XCP-ng drivers
0. snapshot before just in case
1. unpack the ZIP file
2. start setup.exe
3. follow the install wizard
0. Snapshot your VM before installing (just in case)
1. Unpack the ZIP file
2. Start the installation MSI
3. Follow the install wizard

**Note**: Restart can take a while if your windows is currently updating. Restart only occurs after windows has the updates finished.
##### Upgrade from XCP-ng Windows PV Tools (8.2 or earlier) or XenServer VM Tools

4. after restart one of two messages should pop up
* request for restart \<- just restart!
* Management Agent installed successfully \<- enjoy :-)
Our installer will block installation when these tools are already present. These tools must be cleanly uninstalled before continuing with the installation.

##### Upgrade from Citrix :registered: XenServer :registered: client tools

Our installer is not able currently to cleanly uninstall Citrix tools. Citrix tools' uninstaller itself isn't either: it leaves various things behind.

So we need to perform a complete manual clean-up of the tools:
* either entirely manually
* or using the experimental PowerShell script contributed by one of our users at [https://github.com/siodor/win-tools-cleanup](https://github.com/siodor/win-tools-cleanup)
:warning: If you're using XCP-ng Windows PV Tools 8.2 or earlier, it must be uninstalled using XenClean to avoid encountering BSODs. Follow the procedure in our [Troubleshooting Guide - Windows PV Tools](../troubleshooting/windows-pv-tools.md).

:warning: In any case, first disable "Windows Update tools" for the VM (Xen Orchestra, advanced tab) and reboot it.

Following is the manual process.

###### The confident option

You can try a simple process first with some chances of success.

0. Make a snapshot so you can rollback. Windows can get unstable/unbootable if things go wrong.
1. Uninstall Citrix :registered: XenServer :registered: Client Tools
2. Reboot
3. Uninstall `XenServer PV`-Drivers in Device Manager in following order (reboots may be needed):
* `XenServer PV Network Device` (one ore more Devices)
* `XenServer PV Storage Host Adapter`
* `XenServer PV Network Class`
* `XenServer Interface`
* `XenServer PV Bus (c000)` (if present)
* `XenServer PV Bus (0002)` or `XenServer PV Bus (0001)`
4. Reboot
5. Check that you see this unknown device in Device Manager:
* `SCSI-Controller` - PCI-Device ID `5853:0002`
6. Unpack ZIP file
7. Start setup.exe
8. Follow the install wizard

**Note**: Restart can take a while if your windows is currently updating. Restart only occurs after windows has the updates finished.

###### The nuclear option

If the *confident option* above didn't yield the expected results, then we switch to a more aggressive attitude towards the old tools.

:::tip
What follows works in many cases, but some users occasionally still meet the following issues: XCP-ng tools not installing (but Citrix tools install well, so that is a solution to have working tools), and occasional BSODs in some cases or versions of Windows.

Through many tests, a user came up with a similar yet slightly different procedure that allowed them to avoid Blue Screens Of Death in their situation: https://xcp-ng.org/forum/post/27602.

Help is welcome to help us reconcile both procedures into one.
:::

* Follow the steps 0 to 4 of the "confident option" above if not done yet.
* Follow this (ignore steps 6 and 7, do not try to install the tools yet) [https://support.citrix.com/article/CTX215427](https://support.citrix.com/article/CTX215427)
* Now open regedit and go to HKLM\SYSTEM\CurrentControlSet\Services and delete entries for all xen* services.
* In regedit, also go to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\DIFx\DriverStore and remove ONLY xennet* xenvif*
* Go to C:\Windows\System32 and remove: (you may not have all these)
* xenbus_coinst*.dll
* xenvbd_coinst*.dll
* liteagent.exe
* Now go to C:\Windows\System32\drivers and remove any files named xen*
* Go to C:\Windows\system32\DriverStore\FileRepository and remove xennet* and Xenvif* directories.
* Open the Device Manager and Click View --> Show Hidden Devices. Select Other Devices and Right click on XENBUS VIF and select uninstall. If it asks to delete the driver, check yes. Do this for any xen related thing you see in device manager. Also do the same for any unknown devices.
* Lastly, reboot the VM. You should now hopefully be able to install xen tools regularly.

**Note**: Also have a look at our [Troubleshooting Guide - Windows PV-Tools](../troubleshooting/windows-pv-tools.md).

##### VMs with INACCESSIBLE_BOOT_DEVICE error

You can try to manually inject the missing drivers in recovery mode.

* Get the "Drivers" folder from the XCP Tools installation path (C:\PROGRAM FILES...) - from another VM or install the tools somewhere else to get it.
* Create an ISO-Image containing the "Drivers" folder (see [http://imgburn.com](http://imgburn.com)) and mount that ISO-Image to your VM
* Boot to recovery mode and use the command line and the tool "dism" (see [Microsoft Docs](https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/add-and-remove-drivers-to-an-offline-windows-image)) to inject the drivers (specifically the xenbus and xenvbd drivers) - watch out for the drive letter of the Windows installation and the CD-Drive ('D' and 'E' in the following example):

````
dism /image:d:\ /add-driver /driver:e:\Drivers\xenbus\x64\xenbus.inf
dism /image:d:\ /add-driver /driver:e:\Drivers\xenvbd\x64\xenvbd.inf
````

#### Contributing
##### Linux / xBSD
If you would like to contribute improvements to the `install.sh` script so that it supports your distro, create a pull request against: https://github.com/xcp-ng/xe-guest-utilities/tree/master/mk. Relevant files are usually `xe-linux-distribution` and `install.sh`.

##### Windows
The XCP-ng team is looking for help in improving the guest tools installer, build process, and clean-up tools.
The XCP-ng team is looking for help in improving the guest tools installer, build process, and clean-up tools. For more details, see the [win-pv-drivers/xcp-ng-9.1](https://github.com/xcp-ng/win-pv-drivers/tree/xcp-ng-9.1) branch.