Bug 3354 - Publishing packages to garage.maemo.org oftens results in "lost connection" and "permission denied" from scp
: Publishing packages to garage.maemo.org oftens results in "lost connection" a...
Status: RESOLVED WORKSFORME
Product: maemo.org Website
Repositories
: 4.0
: All Linux
: High major with 5 votes (vote)
: ---
Assigned To: Ferenc Szekely
: repositories@maemo.bugs
:
:
:
:
  Show dependency tree
 
Reported: 2008-06-30 21:49 UTC by Andrew Flegg
Modified: 2010-06-20 16:31 UTC (History)
12 users (show)

See Also:


Attachments
Scp log resulting in permission denied (7.61 KB, text/plain)
2008-07-10 21:07 UTC, Daniel Martin Yerga
Details
Scp log with success (9.54 KB, text/plain)
2008-07-16 18:46 UTC, Eduardo Lima (Etrunko)
Details


Note

You need to log in before you can comment on or make changes to this bug.


Description Andrew Flegg (reporter) maemo.org 2008-06-30 21:49:27 UTC
SOFTWARE VERSION:
Ubuntu 8.04 x86_64, OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007

STEPS TO REPRODUCE THE PROBLEM:
* Try and push packages to the extras repository. For example:
   scp netcat_*.tar.gz vnetcat_*.dsc 
jaffa@garage.maemo.org:/var/www/extras-devel/incoming-builder/diablo/

EXPECTED OUTCOME:
Packages copied over scp.

ACTUAL OUTCOME:
No files copied:
    Permission denied (publickey,keyboard-interactive).
    lost connection

REPRODUCIBILITY:
sometimes

OTHER COMMENTS:
This is an intermittent error. Occurred yesterday (Sunday, 29th June) for at
least 90 minutes. And again now (Monday, 30th June):

    http://www.gossamer-threads.com/lists/maemo/developers/38238#38238

Other users (e.g. Graham Cobb) report also seeing this behaviour (in that
thread).

Intermittent problems like this just cause frustration when developers are
trying to publish packages and will just push them into hosting their own
repos.

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.9)
Gecko/2008061017 Firefox/3.0
Comment 1 Ferenc Szekely maemo.org 2008-07-02 22:42:29 UTC
This could have happened when I was struggling with some script kiddies who
kept on bombing garage thru ssh. I myself could not get a shell. I applied some
iptables rules and changed the ssh configuration slightly. After that I had to
restart ssh a couple of times. I guess this might have caused the connection
breaks. I just did not have time to mail the list that 'hey guys, i am
restarting ssh, so your connections might drop etc etc.' It was more urgent to
make garage accessible for us, admins. Sorry for this.

It looks like the kids have gone away, for a while, and garage seem to be
responsive again, at least ssh and scp works fine for my test user. I have to
mark this bug WORKSFORME, but it does not mean that I am ignoring the bug or
you, so don't get me wrong.

Please re-open it as soon as you see this happening again. I would also
appreciate -if possible- starting the scp with -vvv to get debug messages. If
you use dput for package upload then you might try to configure scp verbosity
elsewhere, like in your ~/.ssh/config file by adding  LogLevel DEBUG3 to the
'garage' section.
Comment 2 Niels Breet maemo.org 2008-07-04 12:44:39 UTC
This is still happening. I have seen > 30 connection problems in the logs for
yesterday.
Comment 3 Thomas Schmidt 2008-07-04 21:57:59 UTC
This happens to me also, i just keep retrying and sometime later, sometimes
only a few seconds, sometimes much more, it is working again - without any
change on my side.
Comment 4 Daniel Martin Yerga 2008-07-10 21:07:34 UTC
Created an attachment (id=815) [details]
Scp log resulting in permission denied
Comment 5 Daniel Martin Yerga 2008-07-10 21:08:38 UTC
I have added the following line to the .ssh/config file: "LogLevel DEBUG3" and
have attached the log obtained with scp (well it was with dput, but dput uses
scp). I don't know if the log is useful, but here we go.
Comment 6 Eduardo Lima (Etrunko) maemo.org 2008-07-16 18:46:11 UTC
Created an attachment (id=826) [details]
Scp log with success

After several attempts, I've been able to upload the packages. The attached
file shows the scp log.
Comment 7 Ferenc Szekely maemo.org 2008-07-22 21:03:14 UTC
I can not reproduce this problem. My uploads always complete, never got a
single access denied from ssh.
Comment 8 Andrew Flegg (reporter) maemo.org 2008-07-22 21:17:14 UTC
> I can not reproduce this problem. My uploads always complete, never got a
> single access denied from ssh.

It seems to be temporary, but when it is temporary it affects everybody/lots of
people[* delete as appropriate] on #maemo simultaneously.
Comment 9 Andrew Flegg (reporter) maemo.org 2008-07-22 23:08:41 UTC
For the denial-of-service attack theory to hold water, the server logs show an
attack when etrunko's having a problem on 2008-07-16?
Comment 10 Benjamin Podszun 2008-07-22 23:14:50 UTC
Just some random thoughts:

- Do you keep any logs regarding the machine load? I've seen rejected ssh
connections when the machine had "other problems", cause perhaps by a build or
something
- Are any script-kiddie automated iptables rules inplace, that might be
overreacting? Adding people for X connects in Y timeframes or something?

My bet so far would be the first point?
Comment 11 Niels Breet maemo.org 2008-07-23 01:54:08 UTC
*** Bug 3487 has been marked as a duplicate of this bug. ***
Comment 12 Ferenc Szekely maemo.org 2008-07-25 15:16:20 UTC
Added some iptables rules following the tips from
http://www.debian-administration.org/articles/187 .
Also added 'MaxStartups 10' to sshd config.
Comment 13 Ferenc Szekely maemo.org 2008-07-25 15:50:59 UTC
(In reply to comment #9)
> For the denial-of-service attack theory to hold water, the server logs show an
> attack when etrunko's having a problem on 2008-07-16?
> 
Not exactly at the same time. The logs show attacks on that day, but not when
etrunko connected or tried to connect. His connections simply ended after about
15 seconds.
Comment 14 Graham Cobb maemo.org 2008-07-25 15:51:37 UTC
What parameters did you configure in the iptables rules?  I would like to make
sure my upload script doesn't trigger them accidentally. For example, it would
be quite easy to do 5 uploads in one minute when uploading the 20 pacakges for
GPE, but if I know what the rules are I can put appropriate delays in the
script.
Comment 15 Ferenc Szekely maemo.org 2008-07-25 15:54:35 UTC
(In reply to comment #5)
> I have added the following line to the .ssh/config file: 'LogLevel DEBUG3' and
> have attached the log obtained with scp (well it was with dput, but dput uses
> scp). I don't know if the log is useful, but here we go.
> 
Unfortunately I could not check your connection attempts. The logs from 'Jul  8
08:17:14'  until 'Jul 11 08:48:48' are vanished... I will ask the ISP for the
backups. This is very interesting indeed.
Comment 16 Ferenc Szekely maemo.org 2008-07-25 15:56:23 UTC
(In reply to comment #14)
> What parameters did you configure in the iptables rules?  I would like to make
> sure my upload script doesn't trigger them accidentally. For example, it would
> be quite easy to do 5 uploads in one minute when uploading the 20 pacakges for
> GPE, but if I know what the rules are I can put appropriate delays in the
> script.
> 
Max 3 connections are allowed from within a minute. We can change that to a
more reasonable number, but I would like to see first if this helps the
original problem first.
Comment 17 Ferenc Szekely maemo.org 2008-08-28 15:58:59 UTC
Marking it fixed. If you still experience the proble, please reopen.
Comment 18 Antonio Aloisio 2009-07-06 22:29:57 UTC
(In reply to comment #17)
> Marking it fixed. If you still experience the proble, please reopen.
> 

Hi,
Does it happen again? I've being trying to upload it for more than 30 mins.. 

gnuton@iron:/sb/FREMANTLE/Qt/PKG$ scp qt4-x11_4.5.2+git20090622-0maemo1.tar.gz
gnuton@garage.maemo.org:/var/www/extras/incoming-builder/fremantle
Permission denied (publickey,keyboard-interactive).
lost connection
Comment 19 Andrew Flegg (reporter) maemo.org 2009-07-15 00:17:46 UTC
As of 2009-07-14T21:16Z, this is happening again:

andrew@serenity:/scratchbox/users/andrew/home/andrew/src/mud-builder/trunk/upload$
for j in horizon; do for i in diablo fremantle; do scp "${j}_"*.tar.gz
"${j}_"*.diff.gz "${j}_"*.changes "${j}_"*.dsc
jaffa@garage.maemo.org:/var/www/extras-devel/incoming-builder/$i/; done; done
Permission denied (publickey,keyboard-interactive).
lost connection
Permission denied (publickey,keyboard-interactive).
lost connection
Comment 20 Andrew Flegg (reporter) maemo.org 2009-07-15 10:45:04 UTC
*** Bug 4241 has been marked as a duplicate of this bug. ***
Comment 21 Andrew Flegg (reporter) maemo.org 2009-07-29 23:48:08 UTC
(In reply to comment #19)
> As of 2009-07-14T21:16Z, this is happening again

And again at 2009-07-29T20:47T. Given it's much easier to set up scripts and
shortcuts to use the scp interface; it's pretty annoying to not be able to rely
on it.
Comment 22 Till Harbaum 2009-08-04 22:00:15 UTC
I also frequently have this problem. The last time a few minutes ago.
Comment 23 Mikko Vartiainen 2009-08-28 17:02:52 UTC
This is kind of reminder that this bug happens constantly. I would say that
failure rate from scp is higher than success rate.
Comment 24 Fred Lefévère-Laoide 2009-08-31 18:56:09 UTC
Still alive and kicking as of today ...
Comment 25 Fred Lefévère-Laoide 2009-09-02 17:45:14 UTC
it worked then not : frustrating !
Comment 26 melunko 2009-09-02 20:24:48 UTC
I am facing same problem today.
Comment 27 Andrew Flegg (reporter) maemo.org 2009-09-24 00:14:03 UTC
Can someone take ownership of this and sort it? The problem seems to be getting
worse - I haven't been able to upload via dput/SSH for ages now: whenever I try
I hit this.
Comment 28 Niels Breet maemo.org 2009-09-24 11:39:26 UTC
We are going to move to a different ISP on different hardware soon, until then
there is not much I can do. This seems load related and as this box runs at a
load of 10 by default, it is just an up hill battle.
Comment 29 Jeff Moe 2009-12-07 23:12:47 UTC
FWIW I saw this today, but it worked subsequently.
Comment 30 Till Harbaum 2010-02-11 12:48:59 UTC
And it's still not working for me ...
Comment 31 Niels Breet maemo.org 2010-02-11 12:58:51 UTC
@Till You noticed that we changed to drop.maemo.org, right? Garage doesn't
handle any ssh connections anymore.
Comment 32 Niels Breet maemo.org 2010-03-15 15:51:26 UTC
The move to drop.maemo.org fixed this issue.
Comment 33 xbenlau 2010-06-20 16:31:00 UTC
I am using drop.maemo.org , but I got the same problem. I never can upload a
package to it successfully. I must use the web interface. 

$ dput  -f fremantle-extras-builder  penpen_0.2-1_armel.changes
Uploading to fremantle-extras-builder (via scp to drop.maemo.org):
This is a restricted account.
Warning: The execution of '/usr/bin/scp' as
  'scp -p /scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1.dsc
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1.tar.gz
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1_armel.deb
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1_armel.changes
benlau@drop.maemo.org:/var/www/extras-devel/incoming-builder/fremantle'
  returned a nonzero exit code.
Error while uploading.