[Proftpd-user] ftpd big memory usage

Post by Matus UHLAR - fantomas
I'm running proftpd 1.5 under debian jessie.
23211 username 20 0 7299676 4.717g 4116 S 0.0 61.7 8:01.00 proftpd
the client is active, 7G of virtual memory is just too much.
This COULD be a problem reported in bug 4247, provided the memroy is not
released after listing...
drwx--x--x 6 username users 4096 Apr 20 2016 .
drwx------ 2 username username 4096 Jun 19 2008 ./.ssh
drwxr-xr-x 2 username users 479232 Nov 10 11:58 ./dokumenty
drwxr-xr-x 2 username users 90112 Nov 10 13:25 ./normy
drwxr-xr-x 2 username users 2244608 Nov 10 13:51 ./protokoly
any idea how to track memory usage in proftpd?

Just to add - other memory leaks I found in logs are sftp-related,
this client is FTP-only, no SSL.

after watching the tshark's outout, seems that process size increases any
time MSLD is issued.
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Windows found: (R)emove, (E)rase, (D)elete

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

TJ Saunders

2016-11-13 01:01:44 UTC

One way in which I start tracking down this sort of thing is use
'ftpdctl debug memory', assuming that proftpd has been compiled with
--enable-devel and the mod_ctrls_admin module; see:

http://www.proftpd.org/docs/contrib/mod_ctrls_admin.html#debug

This will cause the process to give a debug dump of the various memory
pools ProFTPD is using. This method, however, is usually best done when
the issue can be reproduced locally, using a special build of proftpd,
rather than used on a running production system.

Post by Matus UHLAR - fantomas
after watching the tshark's outout, seems that process size increases any
time MSLD is issued.

How large (i.e. how many files/entries) are the directories being listed
via MLSD? On the order of hundreds/thousands?

What tools/commands are you using to measure memory usage? top? ps?

Cheers,
TJ

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

Matus UHLAR - fantomas

2016-11-13 11:29:16 UTC

Post by Matus UHLAR - fantomas
I'm running proftpd 1.5 under debian jessie.
23211 username 20 0 7299676 4.717g 4116 S 0.0 61.7 8:01.00 proftpd

[...]

Post by Matus UHLAR - fantomas
drwxr-xr-x 2 username users 2244608 Nov 10 13:51 ./protokoly
any idea how to track memory usage in proftpd?

One way in which I start tracking down this sort of thing is use
'ftpdctl debug memory', assuming that proftpd has been compiled with
http://www.proftpd.org/docs/contrib/mod_ctrls_admin.html#debug
This will cause the process to give a debug dump of the various memory
pools ProFTPD is using. This method, however, is usually best done when
the issue can be reproduced locally, using a special build of proftpd,
rather than used on a running production system.

Post by Matus UHLAR - fantomas
after watching the tshark's outout, seems that process size increases any
time MSLD is issued.

How large (i.e. how many files/entries) are the directories being listed
via MLSD? On the order of hundreds/thousands?

it was the "protokoly" directory. client was uploading and downloading
files, running mlsd once in a while. after 3 hours the memory usage grown to
the one shown above.

during each mlsd the memory usage increased...

Post by TJ Saunders
What tools/commands are you using to measure memory usage? top? ps?

the output above is from top. I got to this issue because monitoring
notified us with too huge memory usage.
--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Silvester Stallone: Father of the RISC concept.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

TJ Saunders

2016-11-13 21:36:51 UTC

Post by TJ Saunders
How large (i.e. how many files/entries) are the directories being listed
via MLSD? On the order of hundreds/thousands?

it was the "protokoly" directory.

Which has approximately how many files/entries in it?

Post by Matus UHLAR - fantomas
client was uploading and downloading
files, running mlsd once in a while. after 3 hours the memory usage grown
to the one shown above.
during each mlsd the memory usage increased...

Could you provide the debug logging for that session, to see the full
commands/responses used, so that we can better try to recreate the issue
locally?

Cheers,
TJ

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

TJ Saunders

2016-11-13 21:56:07 UTC

Post by TJ Saunders
How large (i.e. how many files/entries) are the directories being listed
via MLSD? On the order of hundreds/thousands?

it was the "protokoly" directory.

Which has approximately how many files/entries in it?

Could you provide the debug logging for that session, to see the full
commands/responses used, so that we can better try to recreate the issue
locally?

Could you also provide the output of `proftpd -V`, and the proftpd.conf
you're using please, so that I can make sure to test with a proftpd
built and configured the same way?

Thanks,
TJ

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
ProFTPD Users List <proftpd-***@proftpd.org>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html

Matus UHLAR - fantomas

2016-11-16 14:08:44 UTC

Post by TJ Saunders
How large (i.e. how many files/entries) are the directories being listed
via MLSD? On the order of hundreds/thousands?

it was the "protokoly" directory.

Which has approximately how many files/entries in it?

65968 files currently. I can try to reproduce the issue with the same
account, if needed.

Could you provide the debug logging for that session, to see the full
commands/responses used, so that we can better try to recreate the issue
locally?

I have turned on extended logging
("ExtendedLog /var/log/proftpd/extended ALL full")
after I noticed the issue
(the commands I saw were captured by wireshark).

Post by TJ Saunders
Could you also provide the output of `proftpd -V`, and the proftpd.conf
you're using please, so that I can make sure to test with a proftpd
built and configured the same way?

standard debian 8/amd64 proftpd
disabled radius, quotatab, memcache modules, enabled and configured tls

--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"They say when you play that M$ CD backward you can hear satanic messages."
"That's nothing. If you play it forward it will install Windows."

TJ Saunders

2016-11-16 22:54:31 UTC

Post by TJ Saunders
Which has approximately how many files/entries in it?

65968 files currently. I can try to reproduce the issue with the same
account, if needed.

OK. That helps.

When handling the MSLD command, the mod_facts module would use the
command's temporary memory pool for the handling of ALL entries in the
directory being listed. This means that for a wide directory, that
temporary memory pool could grow quite large, before the handling of
that command is completed. Once completed, that memory pool is "freed"
-- which, in ProFTPD's case, simply means that the pool is made
available for re-use later. It is not necessary released back to the
operating system.

When looking into this issue last week, I noticed the above behavior,
and changed the mod_facts behavior (in the master branch on GitHub) so
that a new temporary memory pool is allocated *per directory entry*,
then freed up after that entry. This should help keep the memory usage
lower for wide directories. See:

https://github.com/proftpd/proftpd/commit/25dffc6dea2b0c478d539e5fa9b46bd20f965d25

Thus if you can, you might try getting the proftpd source from master,
recompiling, and testing/verifying.

Hope this helps,
TJ

Matus UHLAR - fantomas

2016-12-02 12:11:33 UTC

Hello,

Post by TJ Saunders
When handling the MSLD command, the mod_facts module would use the
command's temporary memory pool for the handling of ALL entries in the
directory being listed. This means that for a wide directory, that
temporary memory pool could grow quite large, before the handling of
that command is completed. Once completed, that memory pool is "freed"
-- which, in ProFTPD's case, simply means that the pool is made
available for re-use later. It is not necessary released back to the
operating system.
When looking into this issue last week, I noticed the above behavior,
and changed the mod_facts behavior (in the master branch on GitHub) so
that a new temporary memory pool is allocated *per directory entry*,
then freed up after that entry. This should help keep the memory usage
https://github.com/proftpd/proftpd/commit/25dffc6dea2b0c478d539e5fa9b46bd20f965d25
Thus if you can, you might try getting the proftpd source from master,
recompiling, and testing/verifying.

I have applied this patch to proftpd-1.3.5 (a few rejects applied manually),
but after runnind "mlsd" multiple times (luckily tnftp supports this
command) I still have the same problem.

after further diggging (should have done this previously, just needed to get
user's pasword), seems that this bug appears when mod_facl is loaded.
Even turning "FaclEngine off" did not help.

--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
BSE = Mad Cow Desease ... BSA = Mad Software Producents Desease

TJ Saunders

2016-12-03 19:38:34 UTC

Post by Matus UHLAR - fantomas
I have applied this patch to proftpd-1.3.5 (a few rejects applied manually),
but after runnind "mlsd" multiple times (luckily tnftp supports this
command) I still have the same problem.
after further diggging (should have done this previously, just needed to
get user's pasword), seems that this bug appears when mod_facl is loaded.
Even turning "FaclEngine off" did not help.

Ah, that's very helpful. Thank you!

Based on that pointer, I've dug into the mod_facl code more closely, and
indeed saw the memory leaks. I've created a pull request which fixes
the issue:

https://github.com/proftpd/proftpd/pull/357

And I've verified the fix using Valgrind. While there, I also fixed the
handling of the FACLEngine directive to do what you'd want/expect here.

I'll be merging this PR shortly, and backporting it to the 1.3.5 branch.

Cheers,
TJ

Matus UHLAR - fantomas

2016-12-12 16:39:31 UTC

Ah, that's very helpful. Thank you!
Based on that pointer, I've dug into the mod_facl code more closely, and
indeed saw the memory leaks. I've created a pull request which fixes
https://github.com/proftpd/proftpd/pull/357
And I've verified the fix using Valgrind. While there, I also fixed the
handling of the FACLEngine directive to do what you'd want/expect here.
I'll be merging this PR shortly, and backporting it to the 1.3.5 branch.

ugh, I've had to apply manually (unless you have backported it already

this seems to have fixed at least the memory leak. HAven't tried
disabling facls.

--
Matus UHLAR - fantomas, ***@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
REALITY.SYS corrupted. Press any key to reboot Universe.

Thomas Spicer

2016-12-12 18:48:29 UTC

Noticed a behavior I was not expecting where the path logging of ftp and
sftp differ. When using "%f" I would expect to see the full path of the
file. I do for FTP but not for SFTP. The path for SFTP is relative to the
user, not the system path (like FTP)

ftp: /mnt/ebs/ftpd/tester/172.22.0.2-8-customers.csv
sftp: /172.22.0.2-8-customers.csv

SQLLog AUTH,ABOR,DELE,MKD,RETR,RMD,RNFR,RNTO,ERR_STOR,STOR,STOU,APPE
accesslog IGNORE_ERRORS
SQLNamedQuery accesslog FREEFORM "INSERT INTO ftpdaccesslogs (`userid`,
`server_ip`, `time`, `operation`, `protocol`, `client_ip`, `transfer_time`,
`bytes_transfer`, `bucket`, `file_path`, `transfer_status`) VALUES ('%u',
'%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{env:BUCKET}', '%f',
'%{transfer-status}')â logging

Using the latest compiled form source

What would cause this to behave differently? Any pointers would be
appreciated.

-Tom

TJ Saunders

2016-12-17 15:05:25 UTC