SSH and SSH Applications from a Security Gateway Perspective

Part 4: SFTP from a Gateway Perspective

In the previous article, we saw that even if you're using tools labeled scp, you're likely transferring files over SFTP, not SCP. In this post, we’ll dive deeper into the SFTP protocol and what a security gateway must do to properly filter files transferred this way.

Protocol Standardization Challenges

Unlike SCP, which was never standardized, SFTP (SSH File Transfer Protocol) had an opportunity to follow an IETF standards process. Initially proposed outside the IETF in 1997, it was taken up by the SSH Working Group, resulting in a series of Internet Drafts published between 2001 and 2006.

These drafts defined various versions of the protocol:

Internet Draft	SFTP version	Notes
draft-ietf-secsh-filexfer-00	v3	Initial IETF draft (2001)
draft-ietf-secsh-filexfer-01	v3	Documented changes to v0, v1 and v2
draft-ietf-secsh-filexfer-02	v3	Final draft of SFTP v3
draft-ietf-secsh-filexfer-03	v4	New features introduced
draft-ietf-secsh-filexfer-06	v6	First appearance of v6
draft-ietf-secsh-filexfer-13	v6	Final SFTP draft (2006)

However, none of these drafts were ever finalized into an RFC, and the drafts have since expired. A later attempt to publish at least an Informational RFC to describe the implementations also failed due to lack of consensus - mainly over which version to standardize: While many products implemented SFTP version 6, the most widely used implementations (first and foremost OpenSSH) decided to stick with version 3.

As a result, SFTP has two competing versions in the field - v3 and v6 - and no formal RFC standard.
A security gateway must support at least v3 and ideally v6 to ensure compatibility without forcing version downgrades for newer implementations.

Naming Confusion

Part of the confusion around SFTP starts with the tools themselves. Many client tools (like scp) now use SFTP under the hood despite their names suggesting otherwise.

The name SFTP stands for "Secure File Transfer Protocol" or "SSH File Transfer Protocol" and while it replaces SCP and FTP, it's more than just a File Transfer Protocol.
SFTP is essentially a protocol to remotely interact with a file system. This richer feature set brings flexibility - but also complexity and new security implications.

Some in the community argue that SFTP tries to do too much, which has contributed to the lack of consensus in the standardization process. Regardless, it has become the de-facto method for secure file transfers between clients and servers.

Starting an SFTP Channel

From the SSH protocol perspective, launching SFTP differs from SCP in one key step:

• With SCP, the SSH channel is opened with an exec request (like scp -f /tmp).
• With SFTP, the channel uses a subsystem request with system name sftp.

The difference is subtle but significant:

• exec launches a command via the remote shell and depends on shell configuration (paths, environment, etc.).
• subsystem invokes a named service defined in the SSH server configuration - no shell expansion or parameters involved.

After the subsystem channel has been established, client and server exchange SSH_MSG_CHANNEL_DATA messages containing SFTP protocol messages. The first messages exchanged are:

1. SSH_FXP_INIT (sent by the client, specifying the latest SFTP version it supports).
2. SSH_FXP_VERSION (sent by the server, confirming or reducing the version number, and announcing server capabilities).
3. Optional SSH_FXP_EXTENDED messages, for example to query server limits such as the number of open files or read/write message lenghts.

While SFTP messages can span across SSH packets or be grouped together, early in the session the exchange is typically 1:1 - each SSH data message carries a single SFTP message.

This layered protocol design offers strong benefits (dual-authentication and encryption without reinventing the wheel), but comes with startup overhead: Just opening an SFTP session will usually require ~11 TCP round trips (including TCP handshake, SSH negotiation, and SFTP handshake)!
This adds latency in workflows where files are transferred one per connection. But for longer-lived sessions with many file operations, this startup cost becomes negligible.

Uploading and Downloading Files with SFTP

Unlike SCP, SFTP is stateful and based on file handles. Clients interact with remote filesystems by opening file handles, reading/writing at offsets, and closing them. Reading from a remote file will result in a download and writing to a remote file will be an upload.

Let's walk through the simple file upload example, we introduced in the last article:
The user issues the command scp testfile.txt user@example.org:/tmp (this time without the -O option!)

1. The client starts with an SSH_FXP_STAT message to query about the target directory (“/tmp/”) in order to ensure that this is indeed an existing directory that it can write to.
2. The server responds with an SSH_FXP_ATTRS message listing all the file properties. (Actually, depending on the SFTP protocol version, the payload of the SSH_FXP_ATTRS message will have a different format and content!)
3. With that confirmation the client sends an SSH_FXP_OPEN message to open the file (“/tmp/testfile.txt”) and requests write access to that file.
4. If permitted, the server will respond with an SSH_FXP_HANDLE message that contains a binary string value that the client shall use in subsequent calls to reference to that file handle - important for parallel operations on multiple open files. (In our example the file handle is simply “XY”).
5. The client sends an SSH_FXP_WRITE message, specifying the file handle (“XY”) and the offset (0) at which the write shall occur together with the data to be written (“abc”). If the file is larger than the limit for write messages that was given at session start, the client will send multiple SSH_FXP_WRITE messages.
6. Without waiting on a write confirmation, the client sends an SSH_FXP_FSETSTAT message which sets the size of the file (3 in our example).
7. The server acknowledges the write command of step 5 by sending an SSH_FXP_STATUS message.
8. The server also acknowledges the message of step 6.
9. The client sends an SSH_FXP_CLOSE message to free up the open file handle. At a next open command, the server may recycle that handle and provide the same identifier back to the client for the new file.
10. The server finally also acknowledges the close command.

As shown in the example, the client does not need to wait for each acknowledgement before issuing the next file operation. Each message (except SSH_FXP_INIT) sent by the client contains an incrementing request ID and the server will refer to that request ID in its response. This allows for parallel file handling as well as for command pipeling.

Unlike SCP, where client and server swap roles and with that reverse the data flow between upload/download, SFTP is always client-driven.

In contrast to the upload example above, in a download scenario:

1. The client sends an SSH_FXP_OPEN message with read access request.
2. The client sends one or more SSH_FXP_READ messages with offset and length to read.
3. The server replies with SSH_FXP_DATA messages containing the file content.

This unified direction makes it easier to manage parallel operations and specifically allows uploads and downloads within the same SFTP session.

What a Security Gateway must do for SFTP

Our SSH Gateway abstracts SCP and SFTP operations under a unified file inspection policy. Admins define rules using policy events like "filestart" and "filedata", independent of the SSH application; this avoids policy code duplication and reduces complexity.

The customer policy is given all level of freedom and can freely decide at each data chunk when data shall be forwarded to the other side or shall first be held back, if the file transfer shall be blocked or the data shall be modified.

SFTP gives more control to the gateway to react on those kinds of policy decisions. For example, if a file has already been (partly) transferred before a block verdict got issued, the gateway can end-of-file the transmission early and can then also issue an SSH_FXP_REMOVE message to delete the file from the server again.

But this comes to the price of a lot of housekeeping on the gateway, especially when many files are transferred in parallel. One particular aspect occurs if the client does not read/write data in sequential order or only parts of the file but the policy requires to see the full file. The gateway can partly mitigate by issuing different read requests to the server.

Those message flow modifications result in a different amount of SFTP messages being exchanged between client and gateway vs. messages being exchanged between gateway and server. Hence, the gateway must manage different request ID series for the upstream messages and map them back to the original IDs of the client.

In essence, SFTP filtering is possible—but non-trivial!

Looking Ahead

All SSH-based file transfer protocols present different challenges:
SCP is simple but limited. SFTP is powerful but complex.

And then there's Git over SSH - a completely different beast - which we’ll explore in the next article.

The articles in this series

1. The SSH Core Protocol and the SSH Remote Command
2. SSH Remote Shell from a Gateway Perspective
3. SCP from a Gateway Perspective
4. SFTP from a Gateway Perspective [this article]
5. Git via SSH from a Gateway Perspective
6. TCP Tunnels via SSH from a Gateway Perspective