Troubleshooting Imaging

These pages provide various troubleshooting guides for FileWave IVS

Authentication Credentials Error

What

When deploying a Windows image and the IVS errors with a message:

“IVS request for URL: https:<your.IVS.IP.address:20044/imagingwindows/boot/get_image_info/ failed with code: 403 Authentication credentials were not provided.”

This will prevent the deployment of Windows images. FileWave will need to re-establish the secure connection between your IVS and server services.

IVSAuthCredsError.png

When/Why

This error comes up when the IVS loses its shared key, which causes the IVS to fail and connect with your FileWave services. The connection between FileWave’s IVS and server does have encryption and can be established again. To fix you may follow the steps below. In some cases, you will want to check and verify the shared key. Remember the last 4 of the shared key and once you have re-generated, be sure the last 4 have changed.

How

  1. Navigate to FileWave Central (native admin)

  2. Open the Imaging tab

  3. Highlight and double-click on your IVS

  4. Check the box to “Generate new key on Save”

  5. Press OK to save

  6. Click on the Monitor button still with the Imaging tab preferences

  7. Click on Verify to have the IVS check-in

After performing these steps, please try again to deploy your image. Be sure the image association is set to True before PXE booting the machine.

How to re-enroll an IVS

What

You may need to remove and re-enroll the IVS to troubleshoot. Instead of straightforward deleting and enrolling the IVS again, you will need to remove the client and admin IVS configurations, before removing. Once these configurations have been deleted you then may remove the IVS from FileWave Admin Central.

When/Why

When the IVS loses connection or stops imaging, troubleshooting may require to remove and re-enroll the IVS.

How

  1. Remove IVS Client configuration
    1. log into your IVS by ssh into the server and execute the commands below:

      $ sudo killall fwcld
      $ rm -rf /etc/xdg/FileWave/Client.conf

  2. Remove IVS Admin configuration
    1. Open a web browser and navigate to your IVS admin address, i.e. https://<IVS.IP.address>:20444

      IVSadminconfig1

    2. Click 'Sign in' and enter the username and password

      username: fwadmin
      password: filewave

    3. Once logged in, click Admin at the top, Preferences > Check the box to the left of Preference to check all boxes > Click the drop-down to the right of Action and select "Delete selected Preferences".

      IVSadminconfig2

    4. Click on 'Go' and confirm by clicking 'Yes' to remove. After completed the steps, you should see 0 Preferences.

      IVSadminconfig3

  3. Remove IVS from FileWave Admin Central
    1. Open FileWave Admin Central and navigate to Preferences > Imaging tab > click your IVS to highlight it > Click the minus sign to the bottom left and then hit OK to close preferences. Re-open Preferences > Imaging tab and make sure your IVS hasn't reappeared.IVSadminconfig4
  4. Restart the IVS. SSH into your IVS and run the command to reboot:

    $ sudo shutdown -r now

  5. Once the IVS has restarted, you may begin the enrollment process normally; Setting up the IVS (Imaging Virtual Server)

Image creation or deployment hangs on "calling subprocess.Popen"

Problem

When trying to create a Master image, or deploy a freshly captured image on a Windows device, the entire process will stall at the message "Calling subprocess.Popen with: parted -m /dev/sda print".

dirty_partition.png

Solution

The cause of this issue is a bad partition on the machine that results in an imaging creation or deployment stall. In order to resolve this issue, you will need to modify a file on the Imaging Virtual Server (IVS) and use a prompt on the device you are wanting to capture the image from / deploy to. 

The following steps will allow you to clear the error from the device.

  1. Make a note of the partition that seems to be stuck. From the screen shot it is "/dev/sda". Your drive may have a different name.

  2. Once you know the drive name, go ahead and turn off the machine that is stuck capturing the image.

  3. Connect to your IVS and run the below command.

    touch /etc/fw_master_debug
  4. PXE boot the machine giving the error again.

  5. The machine will go to a prompt where you are able to type the below command. For the example, "/dev/sda", but yours may be different.

    sgdisk --zap /dev/sda
  6. Shutdown the machine you are capturing the image from / deploying to.

  7. Run the below command on your IVS to delete the file you created.

     rm -rf /etc/fw_master_debug
  8. PXE boot the machine again to capture the image and it will no longer hang at the step.

Imaging Issue After Upgrading FileWave and Using Self-Signed SSL Certificate

What

You are experiencing difficulties imaging machines after upgrading your FileWave Server, IVS, and Clients while using a Self-Signed SSL Certificate.

When/Why

This step is necessary when using a Self-Signed SSL Certificate. Ensure to include this additional step in your IVS upgrade process if you are not using a Root Trusted SSL Certificate.

How

  1. Access the IVS via SSH or locally:
    • Connect to the IVS via SSH or access it locally. Use sudo -s to make sure you are using the root user.
  2. Edit the dnsmasq.lua file:
    • Use your preferred command-line editor (e.g., vi or nano) to edit the dnsmasq.lua file. For example: 
      • vi /imaging/scripts/bin/dnsmasq.lua
      • nano /imaging/scripts/bin/dnsmasq.lua
  1. Navigate to line 128:
    • Use the arrow keys or appropriate commands to navigate to line 128.
      Screenshot 2024-04-19 at 5.21.03 PM.png
  2. Switch to insert mode:
    • Press 'i' to switch to insert mode in vi. Or, if you're using nano, you can just use the arrow keys and edit.
  3. Add the following line:
    • req.tls = false
  4. Save and exit vi or nano:
    • Press the Esc key to exit insert mode.
    • Type ':wq' and press Enter to save and exit vi. Or, ctrl+x and save the file with nano.
  5. Verify functionality:
    • You should now be able to image machines successfully.

RAM listing 0-15 Error

What

Machines using the latest M.2 drives may run into an error listing RAM failures when deploying an image.

ImagingErrorRAM0-15.jpeg

When/Why

New machines with M.2 drives may have been set up with a pre-configuration of RAID within the machine’s BIOS. You will want to log into your machine’s BIOS and change the storage controller or SATA configuration from RAID to AHCI.

How

Depending on the manufacturer/brand of BIOS, be sure to review the options and verify the method of logging into the BIOS. Once logged in, perform the following steps:

  1. Search the BIOS for storage, controller, or SATA settings

  2. Change the storage/SATA mode from RAID to AHCI

  3. Confirm the changes and save

  4. Exit BIOS and restart the machine

  5. Prepare PXE boot to image deployment

After the storage/SATA setting has been changed and saved, please try again to deploy your image. Be sure the image association is set to True before PXE booting the machine.

Third Party Vendors

Each Brand/Manufacturer has their own options to enter BIOS. Below are a few examples to search for:

Sysprep not able to validate Windows installation

Sysprep is mandatory for FileWave Windows disk imaging. Possible consequences of not sysprepping are outlined by Microsoft here. It accomplishes the following goals to prepare your reference system for capturing the master image.

  1. Removes computer-specific info from a Windows installation by doing the following items below - You may find that some Windows functionality no longer works correctly when computer specific info is duplicated between multiple PCs.
    • Generates a new computer SID
    • Sets a new computer name
    • Clears out event logs
    • Runs mini setup to deal with hardware differences
  2. Performs a full Windows shutdown when the "/shutdown" switch is specified, which is required on Windows 8 and 10 - Starting with Windows 8, Microsoft added a fast startup feature that helps your PC start up faster after shutdown, even faster than hibernate. Windows does this by saving an image of the Windows kernel and loaded drivers to C:\hiberfil.sys upon shutdown so when you start your PC again, Windows simply loads the C:\hiberfil.sys file into memory to load Windows instead of starting from scratch. When it does this, Windows leaves the main partition hosting Windows in a state that prevents FileWave from properly capturing it. When you sysprep with the "/shutdown" parameter, it performs a full shutdown without generating a hiberfil.sys file and leaves the partition hosting Windows in a state that allows FileWave to capture it.

Sysprep can occasionally fail with a validation error due to a provisioned Microsoft Store Appx app being updated automatically by Windows 10.

SysprepError.png

Sysprep has an additional provider in Windows 8 and 10 to clean Microsoft Store Appx packages and generalize the image. This provider will fail if an all-user package is updated for one of the users on this reference computer, which Windows will do automatically if it is connected to the internet long enough. To minimize the the chances of this happening on the reference system, keep it disconnected from the internet as much as possible.

The error message you'll see in %WINDIR%\System32\Sysprep\Panther\setupact.log, and more importantly in setuperr.log, when sysprep fails under these circumstances is that "an app was installed for a user, but not provisioned for all users".

<Date> <Time>, Error SYSPRP Package <PackageFullName> was installed for a user, but not provisioned for all users. This package will not function properly in the sysprep image.
<Date> <Time>, Error SYSPRP Failed to remove apps for the current user: 0x80073cf2.
<Date> <Time>, Error SYSPRP Exit code of RemoveAllApps thread was 0x3cf2.
<Date> <Time>, Error [0x0f0082] SYSPRP ActionPlatform::LaunchModule: Failure occurred while executing 'SysprepGeneralize' from C:\Windows\System32\AppxSysprep.dll; dwRet = 0x3cf2
<Date> <Time>, Error SYSPRP ActionPlatform::ExecuteAction: Error in executing action; dwRet = 0x3cf2
<Date> <Time>, Error SYSPRP ActionPlatform::ExecuteActionList: Error in execute actions; dwRet = 0x3cf2
<Date> <Time>, Error SYSPRP SysprepSession::Execute: Error in executing actions from C:\Windows\System32\Sysprep\ActionFiles\Generalize.xml; dwRet = 0x3cf2
<Date> <Time>, Error SYSPRP RunPlatformActions:Failed while executing SysprepSession actions; dwRet = 0x3cf2
<Date> <Time>, Error [0x0f0070] SYSPRP RunExternalDlls:An error occurred while running registry sysprep DLLs, halting sysprep execution. dwRet = 0x3cf2
<Date> <Time>, Error [0x0f00a8] SYSPRP WinMain:Hit failure while processing sysprep generalize internal providers; hr = 0x80073cf2

Follow the steps below to remove the offending apps causing sysprep to fail before sysprepping again.

  1. Check %WINDIR%\System32\Sysprep\Panther\setuperr.log for errors like the ones above and note the "<PackageFullName>" of the app, e.g. "9E2F88E3.Twitter_5.4.1.0_x86_wgeqdkkx372wm".

  2. Launch a PowerShell session with admin privileges and run the following command to remove the Microsoft Store Appx app in question, where "<PackageName>" is "Twitter" in this example.

    Remove-AppxPackage *<PackageName>*
  3. If sysprep continues to fail because of the same app, it means the app is installed for another user on the system. Log into this other user account and repeat step 2 to remove the app for that user.

  4. Sysprep again.

  5. Repeat steps 1-4 until sysprep is successful.

Windows Imaging in FileWave: Secure Imaging Option (15.5+)

What

In FileWave version 15.5.0, significant changes have been made to the Windows Imaging process using the Imaging Virtual Server (IVS). Previously, when imaging or capturing a Windows system, the device would mount NFS (Network File System) volumes directly over TCP/UDP port 2049. Starting with FileWave 15.5, the imaging process has been enhanced for security and reliability by allowing the creation of a VPN tunnel over TCP/UDP port 20490. Over this secure VPN tunnel, the system accesses the NFS mounts, providing a more secure and efficient imaging environment. This secure functionality was initially enabled by default, but from 16.2.0 onward it is disabled by default and can be enabled or disabled via a command.

When/Why

When to Use

Secure imaging is something you want to consider if you frequently capture images of devices that have user data on them. Secure imaging will prevent someone from grabbing an image from the IVS server. If you don't typically do this, and typically use the IVS to simply capture base images and deploy them then there is better performance if Secure Imaging is disabled. If you setup your IVS on version 16.2.0 then it will be disabled by default. If you were running an older IVS you may see it enabled but can easily toggle it off or on in 16.2.0.

How

Enabling Secure Imaging

You can enable it with this command on FileWave 16.2.0 or beyond:

sudo imaging-control enable secure-mount
sudo reboot

Disabling Secure Imaging

You can disable it with this command on FileWave 16.2.0 or beyond:

sudo imaging-control disable secure-mount
sudo reboot

Important Considerations

Firewall Configuration: Make sure that your network’s firewalls allow traffic over the necessary ports:

Digging Deeper

Secure Imaging Flag File

On FileWave versions older than 16.2.0, you can delete the flag file to enable Secure Imaging:

sudo rm /etc/fw_insecure_nfs_mount
sudo reboot

On FileWave versions older than 16.2.0, you can create the flag file to disable Secure Imaging:

sudo touch /etc/fw_insecure_nfs_mount
sudo ufw allow 2049/tcp
sudo ufw allow 2049/udp
sudo reboot

Modifying IVS Init.gz for testing purposes

What

With the IVS, target devices uses init.gz as the boot image over the network. For troubleshooting purposes, you may want/need to make a change to this image (for instance to change a driver file or to make some workaround). 

When/Why

This is not an activity most FileWave administrators will do but is being documented for cases where it is needed. Usually you will do this with support to test something.

How

  1. SSH into the IVS

  2. Backup the original init.gz so you can restore it if need be (original is in /imaging/netboot/kernel/init.gz)

  3. Create a temp working directory like /tmp/working, and make sure you cd into that temp directory

  4. Expand the init.gz boot image, as: zcat /imaging/netboot/kernel/init.gz | cpio -i -d

  5. You should now find that the boot image is expanded in you working directory

  6. Make your changes - such as edit filewave/bootup.py for instance, replace a driver file, add another utility, or modify a version of partclone

  7. Make sure everything is owned by root before the next step. When I do my changes on macOS the directory is owned by my username so I have to chown -R root to the temp folder before the next step or some things like SSH won't work. 
  8. Once changes are made, now we’ll want to rebuild init.gz incorporating our changes (again make sure you are in your working directory): find . | cpio -o -H newc | gzip -9 > /imaging/netboot/kernel/init.gz

  9. Enable an imaging association and PXE boot a system and it will use this updated filesystem when it boots

Troubleshooting BitLocker Activation Issues on Windows 11 Post-Imaging

Overview

This article outlines a known issue encountered with Windows 11 deployments where BitLocker encryption fails to initialize properly after imaging. The failure presents as a specific Boot Configuration Data (BCD) error and prevents the successful activation of BitLocker. A resolution is included in this article, along with an explanation of the root cause and steps to remediate the issue.

Issue Description

After deploying a Windows 11 image to devices, attempts to enable BitLocker fail with the following error:

"The path specified in the Boot Configuration Data (BCD) for a BitLocker Drive Encryption integrity-protected application is incorrect. Please verify and correct your BCD settings and try again."

This problem was observed across multiple devices imaged with a FileWave-managed deployment, suggesting a systemic issue with the imaging or BCD configuration process.

Initial Troubleshooting Attempts

Unattend File Adjustments

One of the first suspected causes was the Windows unattend.xml file used during deployment. Specifically, we considered that the partitioning and wiping directives in the answer file conflicted with FileWave’s imaging and partitioning steps.

To test this theory:

Result: This change did not resolve the BitLocker error.

Manual BCD Edits

We experimented with manual edits to the BCD store using bcdedit, in an attempt to update or repair paths that might be misconfigured post-image. However, these attempts did not lead to a consistent fix.

Resolution

A working solution was identified via a community-sourced thread on Reddit (source).

The issue appears to be related to incorrect device and osdevice settings within the BCD store. BitLocker can initialize successfully by explicitly setting these values to point to the system partition.

Required Commands

Execute the following commands in an elevated Command Prompt:

bcdedit -set {current} osdevice partition=C:
bcdedit -set {current} device partition=C:
bcdedit -set {memdiag} device partition=\Device\HarddiskVolume1

Optional: Batch File Version

You may also save the above commands to a .bat file for repeated use. Below is the complete content of the file:

@echo off
bcdedit -set {current} osdevice partition=C:
bcdedit -set {current} device partition=C:
bcdedit -set {memdiag} device partition=\Device\HarddiskVolume1
echo Edit complete.
pause

Post-Fix Behavior

After running the commands (or executing the batch script) and rebooting the device:

This fix has been validated across multiple test devices and resolves the issue consistently. Below is a PowerShell script that may be deployed.

##
##.SYNOPSIS Fixes BCD configuration to resolve BitLocker activation issues on Windows 11.
##
##.DESCRIPTION
##  This script sets the correct BCD partition values for osdevice, device, and memdiag using bcdedit.
## Intended for deployment through FileWave as a Fileset or custom script.
##

# Requires elevation
if (-not ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(`
    [Security.Principal.WindowsBuiltInRole] "Administrator")) {
    Write-Host "This script must be run as Administrator."
    exit 1
}

# Define target values
$osDevice = "partition=C:"
$device = "partition=C:"
$memdiagDevice = "\Device\HarddiskVolume1"

try {
    Write-Host "Applying BCD changes..."

    # Set the current OS device and boot device
    bcdedit /set {current} osdevice $osDevice
    bcdedit /set {current} device $device
    bcdedit /set {memdiag} device $memdiagDevice

    Write-Host "BCD changes applied successfully."

    # Optional: Trigger reboot after applying fix
    # Restart-Computer -Force

} catch {
    Write-Error "An error occurred while editing BCD: $_"
    exit 2
}

exit 0

Optional verification/detection script:

$bcdOutput = bcdedit /enum {current}
if ($bcdOutput -match "osdevice.*partition=C:" -and $bcdOutput -match "device.*partition=C:") {
    Write-Host "BCD is already configured correctly."
    exit 0
} else {
    Write-Host "BCD configuration needs to be fixed."
    exit 1
}

Conclusion

The root cause appears to be incorrect or incomplete BCD configuration following image deployment. This is a result of how the imaging process or unattend file interacts with the BCD setup on Windows 11 systems.

If BitLocker activation issues are encountered, we recommend incorporating the BCD fix as a post-deployment step. This can be integrated into your provisioning workflow until a more permanent fix is identified at the image or unattended setup level.


References

Imaging Speed Test for IVS Performance Verification

What

When imaging Windows devices from a FileWave Image Virtualization Server (IVS), it’s important to ensure that data transfer speeds are optimal. Bottlenecks can occur if the IVS and client devices are on different subnets, or if network links between them are not operating at full capacity.

Two tools—iftop and iperf3—can help diagnose and measure network performance for imaging.

Note that if stunnel is enabled then there is some additional bandwidth used for the encrypted tunnel. You can disable stunnel for better performance. 

When/Why

You might use these tests when:

How

1. Measuring Real-Time Traffic with iftop

iftop can show live network traffic on the IVS.

Install and run it on the IVS:

Note that in FileWave 16.2.0+ this is already installed on the IVS so you will not need to do the apt install command below.

sudo apt install iftop
sudo iftop

image.png

The first column will be the server you are running iftop on. The second column will show you the devices connected so if I was imaging 152.32.183.31 then that would be the last entry on the screen and the first line is traffic TO that host and the second is traffic FROM that host as indicated by the arrows. 

Understanding the three traffic columns in iftop:

You can use this to see per-client speeds during imaging and spot if a client is receiving data more slowly than others.


2. Load Testing with iperf3

iperf3 simulates network traffic to measure throughput between the IVS and a client device.

Install on the IVS:

Note that in FileWave 16.2.0 the IVS already installs iperf3 so you may not need to do the install and systemctl commands below.

The second line below is only needed if you picked "No" to running iperf as a service and will set it to be a service.

sudo apt install iperf3
sudo systemctl enable --now iperf3
sudo ufw allow 5201/tcp

Note that iperf3 listens on TCP 5201 so make sure that from your clients you can reach the IVS on that port and you don't have a firewall blocking that port.

On the client side (macOS, Windows, or Debian):

As of FileWave 16.2.0 the IVS has iperf3 already included in the OS that boots on a client when PXE booting. To test you just need to put the IVS in debug mode and then boot a client to be imaged. In debug mode that client will boot to a unix shell instead of imaging or capturing. From there you can use the following command to test;

1. Enable debug on IVS;

sudo touch /etc/fw_master_debug
sudo reboot

2. Command to type on the PXE booted client when in debug;
Note that in tbe below command the 5 is the number of connections and 30 is the duration to test. This would simulate 5 clients for 30 seconds. You can change the number and duration to do more extensive testing.

network-test 5 30

3. Disable debug on IVS once you are done with testing;

sudo rm -f /etc/fw_master_debug
sudo reboot

Other than testing via a client in debug mode you would need to first install iperf3. This can be found for Windows here: https://iperf.fr/iperf-download.php and for Debian can be installed with "sudo apt install iperf3" and for macOS can be installed via Homebrew with "brew install iperf3". On macOS many admins will already have Homebrew installed, but it can take a few minutes to install it if not.

iperf3 -c <IVS_IP> -P <parallel_streams> -t <seconds>

Example:

iperf3 -c 192.168.1.50 -P 10 -t 30

Tests 10 parallel streams for 30 seconds.


3. Bandwidth → MB/s Cheat Sheet

(Real-world speeds will be ~10–15% less due to protocol overhead, duplex negotiation, etc.)

Network Speed

Theoretical MB/s

Real-World MB/s (approx.)

1 Gb 125 MB/s 105-115 MB/s
2.5 Gb 312.5 MB/s 265-285 MB/s
5 Gb 625 MB/s 525-565 MB/s
10 Gb 1250 MB/s 1050-1150 MB/s

4. Using Results to Predict Imaging Times

Once you know your average throughput from iperf3, you can estimate imaging time.

Example 1: 1 Gb Network Connection

Calculation:

13 GB ÷ 110 MB/s ≈ 118 seconds (~2 minutes) per machine.
For 10 machines (unicast), ≈ 20 minutes (plus minor overhead).

Example 2: 10 Gb Network Connection

Calculation:

13 GB ÷ 1100 MB/s ≈ 12 seconds per machine.
For 10 machines (unicast), ≈ 2 minutes total.


Expected Imaging Times by Network Speed

(Assumes 13 GB total image + drivers per machine, real-world throughput values, unicast imaging)

Network Speed

Real-World Throughput

1 Machine Time

10 Machines Time

20 Machines Time

1 Gb ~110 MB/s ~2 min ~20 min ~40 min
2.5 Gb ~275 MB/s ~47 sec ~8 min ~16 min
5 Gb ~550 MB/s ~24 sec ~4 min ~8 min
10 Gb ~1100 MB/s ~12 sec ~2 min ~4 min

Additional Notes:

Digging Deeper

UC9WJYYfW87SU3LE-drawing-6-1755179672.png