The File Transfer Bottleneck in Modern Networks

Pushing a 2GB firmware image to 120 Arista switches shouldn’t feel like playing Russian roulette with your SSH session. Yet, most network engineers still babysit brittle Python scripts, praying a minor EOS or Junos update doesn’t break the regex parsing their pre-transfer disk space checks.

Consider a typical maintenance window: you need to deploy an emergency patch across a multi-vendor spine-leaf fabric. Your automation script breaks at device 47 because an EOS update altered the dir command’s output format, causing your pre-transfer free-space check to fail. The next morning, your Cisco ASR edge routers need a network configuration backup, but the TFTP timeout handling differs from your Juniper QFX datacenter switches. Each vendor demands bespoke expect scripts, and every OS update becomes a maintenance window risk.

This brittleness stems from hardcoded CLI scraping and vendor-specific quirks masquerading as automation. When your file transfer logic is buried deep inside imperative Python scripts or complex Ansible playbooks maintained by a single “automation wizard,” the rest of the operations team is left in the dark.

The Solution: Operator-Owned YAML Workflows

Netpicker shifts this paradigm by putting control back into the hands of network operators through declarative YAML workflows. Instead of writing complex exception-handling code for SCP, SFTP, or TFTP, you define the intent of the file transfer. Netpicker handles the underlying transport mechanics, state verification, and vendor-specific CLI variations.

By separating the execution logic from the device-specific commands, you get a clean, readable workflow that any network engineer can audit, modify, and run.

Here is how a production-grade firmware distribution device-specific commands file in Netpicker (Source: https://github.com/netpicker/restore/blob/main/cisco_ios.yml):

.transfer-prompts: &prompts
  "Destination file": ""
  "Destination filename": ""
  "file already existing": ""
  "Do you want to overwrite": "n"
  "Do you want to over\\s?write": "n"

verify:
  - command: "verify /md5 {dst_file}"
    method: send_command
    expect_string: "([0-9a-fA-F]{32})"
    kwargs:
      read_timeout: 300
    prompts:
      '= {md5}':
        exc: TransferOK
      '{md5}':
        exc: TransferOK
      'Invalid input':
        msg: 'md5 verify command failed'
      'No such file':
        msg: 'file not found after transfer'

transfers:
  scp:
    - command: "copy scp://{username}:{password}@{src_ip}/{src_file} {dst_file}"
      method: send_command_timing
      prompts:
        <<: *prompts
        assword:
          command: "{password}"
      kwargs:
        read_timeout: 1800
        last_read: 60

  tftp:
    - command: "copy tftp://{src_ip}/{src_file} {dst_file}"
      method: send_command_timing
      prompts:
        <<: *prompts
      kwargs:
        read_timeout: 1800
        last_read: 60

errors:
  kibbitzer.exceptions.FileTransferError:
    - No such file or directory
    - Error opening
    - Transfer failed
    - Permission denied

  kibbitzer.exceptions.InvalidCommand:
    - "^% (.*)"

Why This Approach Works

  1. Structured Data Validation: Instead of parsing raw CLI text with fragile regular expressions, the workflow uses structured JSON output from the device to verify disk space before initiating the transfer.
  2. Protocol Abstraction: The netpicker.file.transfer module manages the SCP session lifecycle. If the connection drops mid-transfer, Netpicker handles the cleanup and reports the failure cleanly, rather than leaving a hung SSH process.
  3. Idempotency and Verification: The workflow doesn’t just copy the file; it verifies the MD5\SHA512 checksum on the device before marking the task as successful. If the transfer corrupted the file, the workflow halts before you trigger a reload.

Implementing File Transfers in Your Environment

To run these workflows, Netpicker uses a lightweight execution engine that communicates directly with your inventory. You don’t need to install agents on your switches or configure complex control nodes.

For configuration backups, the workflow reverses the direction, pulling the running configuration via SCP and saving it directly to your version-controlled repository:

.transfer-prompts: &prompts
  "Address or name of remote host ": ""
  "Destination username ": ""
  "Destination file": ""

protocols:
  default:
    initial_context:
      src_file: startup-config

    initialize:
      - command: copy running-config {src_file}
        prompts:
           <<: *prompts

    transfers:
      tftp:
        - command: "copy {src_file} tftp://{dst_ip}/{dst_file}"
          prompts:
            <<: *prompts

      ftp:
        - command: "copy {src_file} ftp://{username}:{password}@{dst_ip}:/{dst_file}"
          prompts:
            <<: *prompts
      scp:
        - command: "copy {src_file} scp://{username}:{password}@{dst_ip}:/{dst_file}"
          prompts:
            <<: *prompts
          kwargs:
            read_timeout: 120
            last_read: 10

      screen:
        - show-run: show running-config
          kwargs:
            read_timeout: 30
          ignore_patterns:
          - Building configuration

    errors:
      kibbitzer.exceptions.FileTransferError:
        - No such file or directory
        - Error opening
        - Transfer failed
        - Permission denied

      kibbitzer.exceptions.InvalidCommand:
        - "^% (.*)"

    diff-ignore:
      - Last configuration change

This approach ensures that your configuration backups are handled out-of-band, reducing the CPU load on your control plane compared to running heavy SNMP or API-based serialization tasks.

Take Control of Your Network Deployments

Stop babysitting manual file transfers and debugging legacy scripts during your maintenance windows.

To get started, clone the Netpicker community repository, launch the local environment using Docker Compose, and run your first file transfer workflow against your lab devices:

# Clone the repository
git clone https://github.com/netpicker/netpicker-community.git
cd netpicker-community

# Start the Netpicker container stack
docker-compose up -d