Running large batches of files through ssh node

This script works for upto 13 files in n8n and the objective of the flow is achieved as seen in the image below


but if I use 14 or more files, the n8n workflow goes indefinitely


Is there an internal timeout with the SSH node? Because the PowerShell script is working perfectly using the remote desktop I’ve connected it to.

Information on your n8n setup

  • n8n version: Version 1.104.0
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via Docker
  • Operating system: Windows 10

Im not sure if you can timeout the ssh node, but you can timeout your workflow in its settings. Click the 3 dots menu in the top right and click Settings then enable timeout

It could also be that the ssh node is not retuning anything or errors silently on a specific command making it seem like the workflow is waiting forever, so try and enable this setting on all of them and see if the workflow continues

I did modify the ssh node as per your requirements, but the workflow began to run indefinitely and eventually I had to stop the execution after 31 minutes, Although the purpose of PowerShell script was achieved, the workflow kept moving on forever and wouldn’t move to the next ssh node as you can see here,

I can show you examples of where it worked, and as per behavior, I notice it works for a specific amount of files as you can see the output in the image below. This workflow execution fully completed in 5 minutes, 33 seconds.

At this point, when 13 files are being used as input files by the PowerShell script, there seems to be a cutoff in the output text, comparing it to the previous input. This workflow execution fully completed in 6 minutes, 32 seconds.

And if I increase the number of input files, although the script is executed successfully. the workflow does not end.

On paper everything looks fine to me. Its really difficult to understand the issue without being able to debug it myself

Could there be an issue with the PowerShell script I’m using?

Add-Type -AssemblyName "System.Windows.Forms"

# Define folder paths
$inputFolder = "C:\Users\Zigron\OneDrive\Desktop\Terrasmart\OUTOFSPECFILES\GroupedBoxes"
$screwPointsFolder = "C:\Users\Zigron\OneDrive\Desktop\Terrasmart\OUTOFSPECFILES\SCREWPOINTS"
$screwPointDocsFolder = "C:\Users\Zigron\OneDrive\Desktop\Terrasmart\OUTOFSPECFILES\SCREWPOINTDOCS"
$excelFolder = "C:\Users\Zigron\OneDrive\Desktop\Terrasmart\ALGO"

# Function to count files in each folder
function Get-FileCount {
    param($folderPath, $extension)
    if (Test-Path $folderPath) {
        return (Get-ChildItem "$folderPath\*.$extension" -ErrorAction SilentlyContinue | Measure-Object).Count
    }
    return 0
}

# PRE-PROCESSING CLEANUP
Write-Host "Starting pre-processing cleanup..."

# Kill any existing Excel processes
Write-Host "Terminating existing Excel processes..."
Get-Process -Name "EXCEL" -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 3

# Remove any temp Excel files (copies created due to locking)
Write-Host "Cleaning up temporary Excel files..."
Get-ChildItem "$excelFolder\*DESKTOP-BTKVU4M*" -ErrorAction SilentlyContinue | Remove-Item -Force -ErrorAction SilentlyContinue
Get-ChildItem "$excelFolder\*DESKTOP-*" -ErrorAction SilentlyContinue | Remove-Item -Force -ErrorAction SilentlyContinue

# Clear any Excel temp files in system temp
$tempFolder = $env:TEMP
Get-ChildItem "$tempFolder\*TRACKER*" -ErrorAction SilentlyContinue | Remove-Item -Force -ErrorAction SilentlyContinue

# Start timing and get initial counts
$startTime = Get-Date
$inputFileCount = Get-FileCount $inputFolder "csv"
Write-Host "Starting batch processing at $startTime"
Write-Host "Input files to process: $inputFileCount"

if ($inputFileCount -eq 0) {
    Write-Host "No CSV files found in input folder. Exiting."
    exit 0
}

# Clear SCREWPOINTDOCS folder before processing
if (Test-Path $screwPointDocsFolder) {
    Remove-Item "$screwPointDocsFolder\*" -Recurse -Force -ErrorAction SilentlyContinue
    Write-Host "Cleared SCREWPOINTDOCS folder"
}

try {
    # Launch Excel with additional safety measures
    Write-Host "Launching Excel application..."
    $excel = New-Object -ComObject Excel.Application
    $excel.Visible = $false
    $excel.DisplayAlerts = $false
    $excel.ScreenUpdating = $false
    $excel.EnableEvents = $false
    $excel.AskToUpdateLinks = $false
    $excel.AlertBeforeOverwriting = $false
    
    # Open workbook with error handling
    Write-Host "Opening Excel workbook..."
    $workbookPath = "C:\Users\Zigron\OneDrive\Desktop\Terrasmart\ALGO\TRACKER ALGORITHM BETA.xlsb"
    
    # Verify the main file exists and is accessible
    if (-not (Test-Path $workbookPath)) {
        throw "Main Excel file not found: $workbookPath"
    }
    
    $workbook = $excel.Workbooks.Open($workbookPath, $false, $false)
    Write-Host "Excel workbook opened successfully: $($workbook.Name)"
    
    # Verify we opened the correct file (not a copy)
    if ($workbook.Name -ne "TRACKER ALGORITHM BETA.xlsb") {
        Write-Warning "Opened file name: $($workbook.Name) - may be a copy"
    }
    
    Write-Host "Starting macro execution..."
    $excel.Run("CREATEROWS")
    Write-Host "Macro execution completed"
    
    $workbook.Save()
    Write-Host "Workbook saved"
    
    $workbook.Close($false)
    Write-Host "Workbook closed"
}
catch {
    Write-Error "Error during Excel processing: $($_.Exception.Message)"
    
    # Emergency cleanup on error
    if ($workbook) {
        try { $workbook.Close($false) } catch { }
    }
    if ($excel) {
        try { $excel.Quit() } catch { }
    }
    
    # Kill any remaining Excel processes
    Get-Process -Name "EXCEL" -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
    
    exit 1
}
finally {
    # Enhanced cleanup
    Write-Host "Starting Excel cleanup..."
    
    if ($excel) {
        try {
            $excel.Quit()
            [System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel) | Out-Null
        }
        catch {
            Write-Warning "Error during Excel COM cleanup: $($_.Exception.Message)"
        }
    }
    
    # Force garbage collection
    [System.GC]::Collect()
    [System.GC]::WaitForPendingFinalizers()
    [System.GC]::Collect()
    
    # Kill any lingering Excel processes
    Start-Sleep -Seconds 2
    Get-Process -Name "EXCEL" -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
    
    Write-Host "Excel cleanup completed"
}

# POST-PROCESSING CLEANUP
Write-Host "Performing post-processing cleanup..."

# Remove any additional temp files that may have been created
Get-ChildItem "$excelFolder\*DESKTOP-*" -ErrorAction SilentlyContinue | Remove-Item -Force -ErrorAction SilentlyContinue

# Wait and verify completion by checking output file counts
$maxWaitTime = 30  # Reduced to 30 seconds since files should be ready
$checkInterval = 2  # Check every 2 seconds
$elapsedTime = 0

Write-Host "Verifying output file generation..."

do {
    Start-Sleep -Seconds $checkInterval
    $elapsedTime += $checkInterval
    
    $csvOutputCount = Get-FileCount $screwPointsFolder "csv"
    $docOutputCount = Get-FileCount $screwPointDocsFolder "docx"
    
    Write-Host "Progress check - Input: $inputFileCount, CSV Output: $csvOutputCount, Doc Output: $docOutputCount"
    
    # Check if all files are generated
    if ($csvOutputCount -eq $inputFileCount -and $docOutputCount -eq $inputFileCount) {
        $endTime = Get-Date
        $duration = $endTime - $startTime
        
        Write-Host "SUCCESS: All files processed successfully!"
        Write-Host "Input files: $inputFileCount"
        Write-Host "CSV files generated: $csvOutputCount"
        Write-Host "Word documents generated: $docOutputCount"
        Write-Host "Total processing time: $($duration.TotalMinutes.ToString('F2')) minutes"
        Write-Host "PROCESSING COMPLETE - ALL OUTPUTS MATCH INPUT COUNT"
        
        exit 0
    }
    
} while ($elapsedTime -lt $maxWaitTime)

# If we reach here, do a final check but don't fail the process
$csvOutputCount = Get-FileCount $screwPointsFolder "csv"
$docOutputCount = Get-FileCount $screwPointDocsFolder "docx"

Write-Host "Final verification - Input: $inputFileCount, CSV Output: $csvOutputCount, Doc Output: $docOutputCount"

if ($csvOutputCount -eq $inputFileCount -and $docOutputCount -eq $inputFileCount) {
    Write-Host "SUCCESS: All files processed (verification took longer than expected)"
    Write-Host "PROCESSING COMPLETE - ALL OUTPUTS MATCH INPUT COUNT"
    exit 0
} else {
    # Don't fail - just report and continue (files might still be writing)
    Write-Warning "WARNING: File count mismatch after $maxWaitTime seconds"
    Write-Host "This may be normal for large batches - files might still be writing"
    Write-Host "Expected: $inputFileCount, CSV: $csvOutputCount, DOCS: $docOutputCount"
    Write-Host "PROCESSING COMPLETE - CHECK OUTPUT FOLDERS MANUALLY"
    exit 0  # Exit successfully to allow workflow to continue
}

The resulting output in command terminal is as follows:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.