[Proftpd-user] Dismal write speeds to NFS homedirs over sftp

[Cal Sawyer]

I didn't include possibly the most important info, did i? Sheesh. Issue
also occurs under 1.3.5a, btw

Version: 1.3.5b (maint)
Platform: LINUX [Linux 2.6.32-573.12.1.el6.x86_64 x86_64]
Built: Mon Jun 6 2016 12:46:29 BST
Built With:
configure '--prefix=/usr' '--sysconfdir=/etc'
'--with-modules=mod_sftp'

CFLAGS: -O2 -Wall
LDFLAGS: -L$(top_srcdir)/lib
LIBS: -lssl -lcrypto -L$(top_srcdir)/lib/libcap -lcap -lpam
-lcrypto -lz -lsupp -lcrypt -ldl

Files:
Configuration File:
/etc/proftpd.conf
Pid File:
/usr/var/proftpd.pid
Scoreboard File:
/usr/var/proftpd.scoreboard

Features:
- Autoshadow support
- Controls support
+ curses support
- Developer support
- DSO support
+ IPv6 support
+ Largefile support
- Lastlog support
- Memcache support
+ ncurses support
- NLS support
+ OpenSSL support (FIPS enabled)
- PCRE support
- POSIX ACL support
+ Shadow file support
+ Sendfile support
+ Trace support

Tunable Options:
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 30
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10


thx again

c sawyer

Hi
I had users point me to the fact that they were seeing incredibly slow
upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw
throughput between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same server
as proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good
In all of these cases, throughput was limited pretty much only by disk
read/write speeds or, in the last test, the size of the internet pipe
and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd at
debug level 10 shows nothing of interest, aside from a lot of the
usual chat during the upload
Flummoxed! Hope this twigs with someone
many thanks
- c sawyer

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[Matus UHLAR - fantomas]

I had users point me to the fact that they were seeing incredibly slow
upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw throughput
between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same server as
proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good

scp and sftp use different protocol (if they use proftpd at all, usually
they go through sshd)
how does the directory listing work in PASSIVE and PORT mode ?

In all of these cases, throughput was limited pretty much only by disk
read/write speeds or, in the last test, the size of the internet pipe
and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd at
debug level 10 shows nothing of interest, aside from a lot of the usual
chat during the upload

--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[Cal Sawyer]

Hi

Jumping the gun before my daily digest gets here because i just saw this
reply on the listsite

I had users point me to the fact that they were seeing incredibly slow
upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw

throughput

between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same

server as

proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good

scp and sftp use different protocol (if they use proftpd at all, usually
they go through sshd)
how does the directory listing work in PASSIVE and PORT mode ?

Thanks, Matus

Having mod_sftp compiled into proftpd does use SSH protocol, as does
scp. Of course scp doesn't use proftpd (or vice versa), but i think the
sftp/scp comparision it helps highlight a big difference in up/.downoads
speeds via scp vs extremely imbalanced up/download speeds via
sftp/proftpd Seems from this we know from that it's not system sshd
causing the problem since the same system libraries are being used for
both sshd and compiled proftpd.

I neglected to add that sftp to the filesystem local to the sftp server
is also fast in both directions. NFS seems implicated here but i can't
put my finger on where exactly. I've tried a few different wsize/rsize
options on the mount with no luck

PASV and PORT commands aren't applicable when using sftp

In all of these cases, throughput was limited pretty much only by disk
read/write speeds or, in the last test, the size of the internet pipe
and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd at
debug level 10 shows nothing of interest, aside from a lot of the

usual

chat during the upload

--
Matus UHLAR - fantomas, ***@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.

I didn't include possibly the most important info, did i? Sheesh.
Issue also occurs under 1.3.5a, btw
Version: 1.3.5b (maint)
Platform: LINUX [Linux 2.6.32-573.12.1.el6.x86_64 x86_64]
Built: Mon Jun 6 2016 12:46:29 BST
configure '--prefix=/usr' '--sysconfdir=/etc'
'--with-modules=mod_sftp'
CFLAGS: -O2 -Wall
LDFLAGS: -L$(top_srcdir)/lib
LIBS: -lssl -lcrypto -L$(top_srcdir)/lib/libcap -lcap -lpam
-lcrypto -lz -lsupp -lcrypt -ldl
/etc/proftpd.conf
/usr/var/proftpd.pid
/usr/var/proftpd.scoreboard
- Autoshadow support
- Controls support
+ curses support
- Developer support
- DSO support
+ IPv6 support
+ Largefile support
- Lastlog support
- Memcache support
+ ncurses support
- NLS support
+ OpenSSL support (FIPS enabled)
- PCRE support
- POSIX ACL support
+ Shadow file support
+ Sendfile support
+ Trace support
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 30
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
thx again
c sawyer

Hi
I had users point me to the fact that they were seeing incredibly
slow upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw
throughput between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same server
as proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good
In all of these cases, throughput was limited pretty much only by
disk read/write speeds or, in the last test, the size of the internet
pipe and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd at
debug level 10 shows nothing of interest, aside from a lot of the
usual chat during the upload
Flummoxed! Hope this twigs with someone
many thanks
- c sawyer

[Bob Friesenhahn]

I neglected to add that sftp to the filesystem local to the sftp server is
also fast in both directions. NFS seems implicated here but i can't put my
finger on where exactly. I've tried a few different wsize/rsize options on
the mount with no luck

NFS normally uses synchronous writes so the time that the NFS server
takes to confirm each write becomes the bottleneck to write
performance. Perhaps the sftp is producing many tiny writes.
Potential solutions are for sftp to do more local buffering in RAM, or
to write to a local temporary file first and then use ideally-sized
write requests to copy the file to the final location.

Another solution is (depending on the NFS server) to add a
non-volatile write buffer in the server such as a dedicated SSD so
that the latency of each write is reduced. By adding risk of data
loss and problems if the server should spontaneously reboot, the NFS
server could be configured to pretend that the writes took place even
while they are still in progress.

What operating system does the NFS server use? What NFS protocol
version is being used?

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[Matus UHLAR - fantomas]

Post by Cal Sawyer

We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw throughput
between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s

Having mod_sftp compiled into proftpd does use SSH protocol, as does
scp. Of course scp doesn't use proftpd (or vice versa), but i think
the sftp/scp comparision it helps highlight a big difference in
up/.downoads speeds via scp vs extremely imbalanced up/download
speeds via sftp/proftpd Seems from this we know from that it's not
system sshd causing the problem since the same system libraries are
being used for both sshd and compiled proftpd.
I neglected to add that sftp to the filesystem local to the sftp
server is also fast in both directions. NFS seems implicated here
but i can't put my finger on where exactly. I've tried a few
different wsize/rsize options on the mount with no luck
PASV and PORT commands aren't applicable when using sftp

In all of these cases, throughput was limited pretty much only by disk
read/write speeds or, in the last test, the size of the internet pipe
and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd at
debug level 10 shows nothing of interest, aside from a lot of the

usual

chat during the upload

--
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.

I didn't include possibly the most important info, did i? Sheesh.
Issue also occurs under 1.3.5a, btw
Version: 1.3.5b (maint)
Platform: LINUX [Linux 2.6.32-573.12.1.el6.x86_64 x86_64]
Built: Mon Jun 6 2016 12:46:29 BST
configure '--prefix=/usr' '--sysconfdir=/etc'
'--with-modules=mod_sftp'
CFLAGS: -O2 -Wall
LDFLAGS: -L$(top_srcdir)/lib
LIBS: -lssl -lcrypto -L$(top_srcdir)/lib/libcap -lcap -lpam
-lcrypto -lz -lsupp -lcrypt -ldl
/etc/proftpd.conf
/usr/var/proftpd.pid
/usr/var/proftpd.scoreboard
- Autoshadow support
- Controls support
+ curses support
- Developer support
- DSO support
+ IPv6 support
+ Largefile support
- Lastlog support
- Memcache support
+ ncurses support
- NLS support
+ OpenSSL support (FIPS enabled)
- PCRE support
- POSIX ACL support
+ Shadow file support
+ Sendfile support
+ Trace support
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 30
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
thx again
c sawyer

Hi
I had users point me to the fact that they were seeing incredibly
slow upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's
NFS backend are connected via 10Gb. iperf shows over 9GB/sec raw
throughput between these systems
From my tests, run from inside our network with the client and
server located on the same network segment and subnet, only
proftpd sftp PUT alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same
server as proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good
In all of these cases, throughput was limited pretty much only by
disk read/write speeds or, in the last test, the size of the
internet pipe and even that beats local PUT
I tried disabling Sendfile with no improvement. Running proftpd
at debug level 10 shows nothing of interest, aside from a lot of
the usual chat during the upload
Flummoxed! Hope this twigs with someone
many thanks
- c sawyer

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Windows found: (R)emove, (E)rase, (D)elete

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[Matus UHLAR - fantomas]

sorry for empty post...

Post by Matus UHLAR - fantomas

I had users point me to the fact that they were seeing incredibly slow
upload speeds to our proftpd (sftp) server.
We home FTP users on an NFS share. The proftpd server and it's NFS
backend are connected via 10Gb. iperf shows over 9GB/sec raw throughput
between these systems
From my tests, run from inside our network with the client and server
located on the same network segment and subnet, only proftpd sftp PUT
alone is affected. I'm getting at best 4MB/s
- sftp GET is fine
- scp transfers to/'from the proftpd server are fine
- plain old FTP to a test vsftpd instance (running on the same server as
proftpd, using same NFS homedirs) is fine.
- sftp PUT from a remote location is good

scp and sftp use different protocol (if they use proftpd at all, usually
they go through sshd)
how does the directory listing work in PASSIVE and PORT mode ?
Having mod_sftp compiled into proftpd does use SSH protocol, as does
scp. Of course scp doesn't use proftpd (or vice versa), but i think
the sftp/scp comparision it helps highlight a big difference in
up/.downoads speeds via scp vs extremely imbalanced up/download
speeds via sftp/proftpd Seems from this we know from that it's not
system sshd causing the problem since the same system libraries are
being used for both sshd and compiled proftpd.

I'm afraid you did confuse me even more. What's slow exactly?
- writing to NFS or writing to local disk on remote host?
- writing single files or writing multiple files?
- writing through scp+sftp/ssh or scp+sftp/mod_sftp ?
- writing through FTP protocol to vsftpd or proftpd?

If TJ's recommendation did not help you, can you please make proper comparisons?

Post by Matus UHLAR - fantomas
PASV and PORT commands aren't applicable when using sftp

but they are when using FTP.
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Christian Science Programming: "Let God Debug It!".

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[TJ Saunders]

Post by Cal Sawyer
I didn't include possibly the most important info, did i? Sheesh. Issue
also occurs under 1.3.5a, btw
Version: 1.3.5b (maint)

Oops; just saw this *grin*.

If possible, you might try proftpd-1.3.6rc2; there were some mod_sftp
changes in the 1.3.6 branch which attempted to improve performance of
SFTP transfers on NFS directories:

https://github.com/proftpd/proftpd/commit/e91edc53325cd6e9dabce62647741dbb71023e4d
https://github.com/proftpd/proftpd/commit/159aa730db32a2acedc948fd39ffd9c4ab8eab5c

Cheers,
TJ

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[TJ Saunders]

I had users point me to the fact that they were seeing incredibly slow
upload speeds to our proftpd (sftp) server.

Which version of ProFTPD/mod_sftp are you running?

Cheers,
TJ

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

[Cal Sawyer]

Hi, all

Switched out of digest mode and haven;t received the most recent so i'm
reconstructing from listsite again ...

SUCCESS, basically, and proftpd-1.3.6rc2 was the key. IO over sftp is
now symmetrical and more importantly, fast at 54-55MB/sec, probably
limited by backend storage performance over NFS

I'd normally be loathe to use an rc in production, but this is great and
i'm sticking with it. Definitely will be following updates as 1.3.6
goes to release and testing new rc's when and as can. Changes done in
mod_sftp are surely going in the right direction

Many thanks to TJ for the suggestion - made my day.

bests,

- c sawyer

Re: [Proftpd-user] Dismal write speeds to NFS homedirs over sftp

I didn't include possibly the most important info, did i? Sheesh.

Issue

also occurs under 1.3.5a, btw
Version: 1.3.5b (maint)

Oops; just saw this *grin*.

If possible, you might try proftpd-1.3.6rc2; there were some mod_sftp
changes in the 1.3.6 branch which attempted to improve performance of
SFTP transfers on NFS directories:

https://github.com/proftpd/proftpd/commit/e91edc53325cd6e9dabce62647741dbb71023e4d
https://github.com/proftpd/proftpd/commit/159aa730db32a2acedc948fd39ffd9c4ab8eab5c

Cheers,
TJ