Comment 1 Ferenc Szekely 2008-07-02 22:42:29 UTC

This could have happened when I was struggling with some script kiddies who
kept on bombing garage thru ssh. I myself could not get a shell. I applied some
iptables rules and changed the ssh configuration slightly. After that I had to
restart ssh a couple of times. I guess this might have caused the connection
breaks. I just did not have time to mail the list that 'hey guys, i am
restarting ssh, so your connections might drop etc etc.' It was more urgent to
make garage accessible for us, admins. Sorry for this.

It looks like the kids have gone away, for a while, and garage seem to be
responsive again, at least ssh and scp works fine for my test user. I have to
mark this bug WORKSFORME, but it does not mean that I am ignoring the bug or
you, so don't get me wrong.

Please re-open it as soon as you see this happening again. I would also
appreciate -if possible- starting the scp with -vvv to get debug messages. If
you use dput for package upload then you might try to configure scp verbosity
elsewhere, like in your ~/.ssh/config file by adding  LogLevel DEBUG3 to the
'garage' section.

Comment 2 Niels Breet 2008-07-04 12:44:39 UTC

This is still happening. I have seen > 30 connection problems in the logs for
yesterday.

This happens to me also, i just keep retrying and sometime later, sometimes
only a few seconds, sometimes much more, it is working again - without any
change on my side.

Created an attachment (id=815) [details]
Scp log resulting in permission denied

I have added the following line to the .ssh/config file: "LogLevel DEBUG3" and
have attached the log obtained with scp (well it was with dput, but dput uses
scp). I don't know if the log is useful, but here we go.

Comment 6 Eduardo Lima (Etrunko) 2008-07-16 18:46:11 UTC

Created an attachment (id=826) [details]
Scp log with success

After several attempts, I've been able to upload the packages. The attached
file shows the scp log.

Comment 7 Ferenc Szekely 2008-07-22 21:03:14 UTC

I can not reproduce this problem. My uploads always complete, never got a
single access denied from ssh.

Comment 8 Andrew Flegg (reporter) 2008-07-22 21:17:14 UTC

> I can not reproduce this problem. My uploads always complete, never got a
> single access denied from ssh.

It seems to be temporary, but when it is temporary it affects everybody/lots of
people[* delete as appropriate] on #maemo simultaneously.

Comment 9 Andrew Flegg (reporter) 2008-07-22 23:08:41 UTC

For the denial-of-service attack theory to hold water, the server logs show an
attack when etrunko's having a problem on 2008-07-16?

Just some random thoughts:

- Do you keep any logs regarding the machine load? I've seen rejected ssh
connections when the machine had "other problems", cause perhaps by a build or
something
- Are any script-kiddie automated iptables rules inplace, that might be
overreacting? Adding people for X connects in Y timeframes or something?

My bet so far would be the first point?

Comment 11 Niels Breet 2008-07-23 01:54:08 UTC

*** Bug 3487 has been marked as a duplicate of this bug. ***

Comment 12 Ferenc Szekely 2008-07-25 15:16:20 UTC

Added some iptables rules following the tips from
http://www.debian-administration.org/articles/187 .
Also added 'MaxStartups 10' to sshd config.

Comment 13 Ferenc Szekely 2008-07-25 15:50:59 UTC

(In reply to comment #9)
> For the denial-of-service attack theory to hold water, the server logs show an
> attack when etrunko's having a problem on 2008-07-16?
> 
Not exactly at the same time. The logs show attacks on that day, but not when
etrunko connected or tried to connect. His connections simply ended after about
15 seconds.

Comment 14 Graham Cobb 2008-07-25 15:51:37 UTC

What parameters did you configure in the iptables rules?  I would like to make
sure my upload script doesn't trigger them accidentally. For example, it would
be quite easy to do 5 uploads in one minute when uploading the 20 pacakges for
GPE, but if I know what the rules are I can put appropriate delays in the
script.

Comment 15 Ferenc Szekely 2008-07-25 15:54:35 UTC

(In reply to comment #5)
> I have added the following line to the .ssh/config file: 'LogLevel DEBUG3' and
> have attached the log obtained with scp (well it was with dput, but dput uses
> scp). I don't know if the log is useful, but here we go.
> 
Unfortunately I could not check your connection attempts. The logs from 'Jul  8
08:17:14'  until 'Jul 11 08:48:48' are vanished... I will ask the ISP for the
backups. This is very interesting indeed.

Comment 16 Ferenc Szekely 2008-07-25 15:56:23 UTC

(In reply to comment #14)
> What parameters did you configure in the iptables rules?  I would like to make
> sure my upload script doesn't trigger them accidentally. For example, it would
> be quite easy to do 5 uploads in one minute when uploading the 20 pacakges for
> GPE, but if I know what the rules are I can put appropriate delays in the
> script.
> 
Max 3 connections are allowed from within a minute. We can change that to a
more reasonable number, but I would like to see first if this helps the
original problem first.

Comment 17 Ferenc Szekely 2008-08-28 15:58:59 UTC

Marking it fixed. If you still experience the proble, please reopen.

(In reply to comment #17)
> Marking it fixed. If you still experience the proble, please reopen.
> 

Hi,
Does it happen again? I've being trying to upload it for more than 30 mins.. 

gnuton@iron:/sb/FREMANTLE/Qt/PKG$ scp qt4-x11_4.5.2+git20090622-0maemo1.tar.gz
gnuton@garage.maemo.org:/var/www/extras/incoming-builder/fremantle
Permission denied (publickey,keyboard-interactive).
lost connection

Comment 19 Andrew Flegg (reporter) 2009-07-15 00:17:46 UTC

As of 2009-07-14T21:16Z, this is happening again:

andrew@serenity:/scratchbox/users/andrew/home/andrew/src/mud-builder/trunk/upload$
for j in horizon; do for i in diablo fremantle; do scp "${j}_"*.tar.gz
"${j}_"*.diff.gz "${j}_"*.changes "${j}_"*.dsc
jaffa@garage.maemo.org:/var/www/extras-devel/incoming-builder/$i/; done; done
Permission denied (publickey,keyboard-interactive).
lost connection
Permission denied (publickey,keyboard-interactive).
lost connection

Comment 20 Andrew Flegg (reporter) 2009-07-15 10:45:04 UTC

*** Bug 4241 has been marked as a duplicate of this bug. ***

Comment 21 Andrew Flegg (reporter) 2009-07-29 23:48:08 UTC

(In reply to comment #19)
> As of 2009-07-14T21:16Z, this is happening again

And again at 2009-07-29T20:47T. Given it's much easier to set up scripts and
shortcuts to use the scp interface; it's pretty annoying to not be able to rely
on it.

I also frequently have this problem. The last time a few minutes ago.

This is kind of reminder that this bug happens constantly. I would say that
failure rate from scp is higher than success rate.

Still alive and kicking as of today ...

it worked then not : frustrating !

I am facing same problem today.

Comment 27 Andrew Flegg (reporter) 2009-09-24 00:14:03 UTC

Can someone take ownership of this and sort it? The problem seems to be getting
worse - I haven't been able to upload via dput/SSH for ages now: whenever I try
I hit this.

Comment 28 Niels Breet 2009-09-24 11:39:26 UTC

We are going to move to a different ISP on different hardware soon, until then
there is not much I can do. This seems load related and as this box runs at a
load of 10 by default, it is just an up hill battle.

FWIW I saw this today, but it worked subsequently.

And it's still not working for me ...

Comment 31 Niels Breet 2010-02-11 12:58:51 UTC

@Till You noticed that we changed to drop.maemo.org, right? Garage doesn't
handle any ssh connections anymore.

Comment 32 Niels Breet 2010-03-15 15:51:26 UTC

The move to drop.maemo.org fixed this issue.

I am using drop.maemo.org , but I got the same problem. I never can upload a
package to it successfully. I must use the web interface. 

$ dput  -f fremantle-extras-builder  penpen_0.2-1_armel.changes
Uploading to fremantle-extras-builder (via scp to drop.maemo.org):
This is a restricted account.
Warning: The execution of '/usr/bin/scp' as
  'scp -p /scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1.dsc
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1.tar.gz
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1_armel.deb
/scratchbox/users/benlau/home/benlau/src/buildarea/penpen_0.2-1_armel.changes
benlau@drop.maemo.org:/var/www/extras-devel/incoming-builder/fremantle'
  returned a nonzero exit code.
Error while uploading.