Bug 7677 - (int-152501) Copying files over USB is very slow when using CFQ I/O scheduler
(int-152501)
: Copying files over USB is very slow when using CFQ I/O scheduler
Status: NEW
Product: Core
general
: 5.0/(1.2009.42-11)
: N900 Maemo
: Low normal with 6 votes (vote)
: ---
Assigned To: unassigned
: core-general-bugs
:
: performance
:
:
  Show dependency tree
 
Reported: 2010-01-05 17:45 UTC by Paul Hartman
Modified: 2010-05-10 18:51 UTC (History)
2 users (show)

See Also:


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description Paul Hartman (reporter) 2010-01-05 17:45:53 UTC
SOFTWARE VERSION:
1.2009.42-11.002

EXACT STEPS LEADING TO PROBLEM: 
When N900 is connected in mass storage mode to a Linux PC, copying more than 1
file to the N900 is very slow if the CFQ I/O scheduler is in use on the PC (CFQ
is default in most modern Linux kernels, I think). Changing scheduler to
deadline improves speeds back to normal (17M/s). Other USB mass storage devices
don't suffer from this symptom when using CFQ in my tests.

Also to note that, while slow, the speed in kernel 2.6.32 is MUCH better than
in 2.6.31 and previous (because pdflush has been replaced). Copying 1 gigabyte
with CFQ took about 20 minutes in 2.6.31 kernel, reduced to about 5 minutes in
2.6.32 (compared to about 1 minute if speed was proper).

I am using iotop to monitor the I/O in general, plus I performed the following
test. file1 and file2 are each 700M and both housed on a ramdrive for this
test. They were deleted from the destination between runs.

# one file at a time with sync in-between, fast speeds:
$ sync; time sh -c "cp file1 /mnt/usb; sync; cp file2 /mnt/usb; sync"

real    1m25.697s
user    0m0.005s
sys     0m2.509s

# copy two files in a row, then sync, speed is bad:
$ sync; time sh -c "cp file1 file2 /mnt/usb; sync"

real    6m51.439s
user    0m0.007s
sys     0m2.615s


Using all I/O schedulers, the speed of the first test was the same. So it's
only related to writing more than 1 file to the N900. The timing results for
the second test ended up as such:

cfq: 6m51.439s
noop: 3m0.733s
anticipatory: 1m44.348s
deadline: 1m36.804s

I don't know if this is fault of N900, vfat, I/O scheduler, USB/SCSI generic
driver in linux, etc. but I post it here for your consideration and in case
others search for the same issue.
Comment 1 Eero Tamminen nokia 2010-01-11 17:29:11 UTC
> cfq: 6m51.439s
> noop: 3m0.733s
> anticipatory: 1m44.348s
> deadline: 1m36.804s

I wonder whether N900 itself using noop scheduler could affect this...
Comment 2 Paul Hartman (reporter) 2010-01-12 08:03:13 UTC
(In reply to comment #1)
> > cfq: 6m51.439s
> > noop: 3m0.733s
> > anticipatory: 1m44.348s
> > deadline: 1m36.804s
> 
> I wonder whether N900 itself using noop scheduler could affect this...

I just tried, it seems setting to noop iosched on N900 results in even longer
write time:

real    8m37.795s
Comment 3 Andre Klapper maemo.org 2010-02-01 13:56:08 UTC
Could you try using memory as backend and ruling eMMC out of the problem ?

Follow these:
1. boot device to initial shell
2. mount -t tmpfs -o size=35M none /tmp
3. dd if=/dev/zero of=/tmp/file.bin bs=1M count=33
4. modprobe g_file_storage file=/tmp/file.bin stall=0 removable
5. connect usb cable
6. run your tests

See if you can note the same problem when writing to memory. Note that you only
have about 33MB storage backend, so use smaller files ;-)
Comment 4 Paul Hartman (reporter) 2010-02-03 04:13:47 UTC
(In reply to comment #3)
> Could you try using memory as backend and ruling eMMC out of the problem ?
> 
> Follow these:
> 1. boot device to initial shell
> 2. mount -t tmpfs -o size=35M none /tmp
> 3. dd if=/dev/zero of=/tmp/file.bin bs=1M count=33
> 4. modprobe g_file_storage file=/tmp/file.bin stall=0 removable
> 5. connect usb cable
> 6. run your tests
> 
> See if you can note the same problem when writing to memory. Note that you only
> have about 33MB storage backend, so use smaller files ;-)
> 

I just tried this (nice trick about g_file_storage using custom file :), I
formatted it to vfat and copied two 16M files, repeating my original tests. All
operations seem to be the basically same speed (around 1.4 seconds) for every
combination. I don't know if this is a large enough amount of data to really
see if the problem occurs. Please let me know if you want me to try any other
tests, thanks!
Comment 5 Philipp Zabel 2010-02-03 17:45:13 UTC
*** This bug has been confirmed by popular vote. ***
Comment 6 Paul Hartman (reporter) 2010-05-06 02:17:23 UTC
Since originally creating this bug report, I've seen this behavior reported by
users of various flash-based devices (both the difference in schedulers and
differences when tweaking dirty_bytes). I've even seen a kernel patch to
default I/O sched to deadline for certain flash memory devices. So I think it's
probably a bug (or deficiency) in the linux I/O handling (on the USB host) and
not a bug in the N900. 

If Nokians agree, you can close this bug as INVALID.
Comment 7 Eero Tamminen nokia 2010-05-06 10:35:30 UTC
(In reply to comment #0)
> I am using iotop to monitor the I/O in general, plus I performed the following
> test. file1 and file2 are each 700M and both housed on a ramdrive for this
> test. They were deleted from the destination between runs.

What if you copy same amount of data, but split to many smaller files, of e.g.
<10MB size?

I mean does this issue happen only for huge files, or also for smaller ones?

(I'm thinking about kernel buffer/cache sizes etc.)
Comment 8 Andre Klapper maemo.org 2010-05-07 12:56:55 UTC
Internal comment:

"I've performed some tests and collected following statistics (average from 5
tryouts) of time required to copy two files 700 Mb each to the N900:
  = host PC under Ubuntu, kernel ver. 2.6.27.15-generic
1) copy with "sync" between, cfq scheduler on host PC - 92.94 sec
2) copy "in a row", cfq scheduler - 93.97 sec
3) copy "in a row", deadline scheduler - 93.93 sec
4) copy "in a row", noop scheduler - 98.44 sec
5) copy "in a row", anticipatory scheduler - 93.08 sec

  = host PC under Win XP
6) copy "in a row" - 114 sec

  = host PC under Gentoo, kernel ver. 2.6.32.8
7) copy "in a row", cfq scheduler - 93.67 sec


according to these results there is no slowing down observed for cfq scheduler,
so it seems that this bug is WORKSFORME candidate"
Comment 9 Paul Hartman (reporter) 2010-05-10 18:51:30 UTC
(In reply to comment #8)
> Internal comment:
> 
> "I've performed some tests and collected following statistics (average from 5
> tryouts) of time required to copy two files 700 Mb each to the N900:
>   = host PC under Ubuntu, kernel ver. 2.6.27.15-generic
> 1) copy with "sync" between, cfq scheduler on host PC - 92.94 sec
> 2) copy "in a row", cfq scheduler - 93.97 sec
> 3) copy "in a row", deadline scheduler - 93.93 sec
> 4) copy "in a row", noop scheduler - 98.44 sec
> 5) copy "in a row", anticipatory scheduler - 93.08 sec
> 
>   = host PC under Win XP
> 6) copy "in a row" - 114 sec
> 
>   = host PC under Gentoo, kernel ver. 2.6.32.8
> 7) copy "in a row", cfq scheduler - 93.67 sec
> 
> 
> according to these results there is no slowing down observed for cfq scheduler,
> so it seems that this bug is WORKSFORME candidate"
> 

Based on the last few months Googling and reading other bugreports about the
same kind of problem (not N900-specific, but some also with N900 have this same
problem), I think it may be exposed on systems with a large amount of RAM. I
don't know about your internal test machines, but in my case I tested on two
different machines and both have the same "slowness problem", both are 64-bit
linux and the first had 8 gigabytes of RAM while the second had 12 gigabytes.

But, either way, I still think it's a bug/side-effect of linux kernel on PC and
nothing to do specifically with the N900 itself, other than some characteristic
of N900 that exposes this problem for some people.