| *** jamesmcarthur has quit IRC | 00:02 | |
| *** yamamoto has quit IRC | 00:28 | |
| *** longkb has joined #openstack-infra | 00:39 | |
| *** eumel8 has quit IRC | 00:52 | |
| *** longkb has quit IRC | 01:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add Fedora 29 testing https://review.openstack.org/618671 | 01:14 |
|---|---|---|
| *** jklare has quit IRC | 01:23 | |
| *** calbers has quit IRC | 01:23 | |
| *** calbers has joined #openstack-infra | 01:24 | |
| *** jklare has joined #openstack-infra | 01:24 | |
| *** auristor has quit IRC | 02:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 02:18 |
| *** auristor has joined #openstack-infra | 02:19 | |
| *** owalsh has quit IRC | 02:25 | |
| *** mrsoul has joined #openstack-infra | 02:27 | |
| *** d0ugal has quit IRC | 02:33 | |
| *** owalsh has joined #openstack-infra | 02:39 | |
| *** tonyb has quit IRC | 02:49 | |
| *** tonyb has joined #openstack-infra | 02:52 | |
| *** ianychoi has quit IRC | 02:57 | |
| *** ianychoi has joined #openstack-infra | 02:57 | |
| *** psachin has joined #openstack-infra | 03:12 | |
| *** udesale has joined #openstack-infra | 03:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:33 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:33 | |
| *** bhavikdbavishi has quit IRC | 03:37 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:37 | |
| *** ykarel|away has joined #openstack-infra | 03:40 | |
| *** ykarel|away is now known as ykarel | 03:44 | |
| *** jamesmcarthur has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has quit IRC | 04:03 | |
| ianw | yay! seems the f29 networkmanager was a systemd boot race ... look like i'm getting good tests in the gate now. could not replicate this outside the gate, which probably makes sense now | 04:13 |
| *** ramishra has joined #openstack-infra | 04:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 04:28 |
| *** janki has joined #openstack-infra | 04:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add Fedora 29 testing https://review.openstack.org/618671 | 04:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing https://review.openstack.org/618671 | 04:44 |
| *** fuentess has quit IRC | 04:58 | |
| *** ykarel has quit IRC | 05:07 | |
| *** psachin has quit IRC | 05:13 | |
| *** psachin has joined #openstack-infra | 05:14 | |
| *** ykarel has joined #openstack-infra | 05:24 | |
| *** witek has quit IRC | 05:30 | |
| *** witek has joined #openstack-infra | 05:33 | |
| *** tonyb[m] has joined #openstack-infra | 05:36 | |
| *** pcaruana has quit IRC | 05:44 | |
| *** witek has quit IRC | 05:50 | |
| *** witek has joined #openstack-infra | 05:51 | |
| *** janki has quit IRC | 05:55 | |
| *** janki has joined #openstack-infra | 05:56 | |
| *** witek has quit IRC | 05:57 | |
| *** witek has joined #openstack-infra | 05:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing https://review.openstack.org/618671 | 06:11 |
| *** chandankumar has joined #openstack-infra | 06:23 | |
| *** ralonsoh has joined #openstack-infra | 06:23 | |
| *** chandankumar is now known as chkumar|rover | 06:25 | |
| *** chkumar|rover is now known as chkumar|ruck | 06:25 | |
| *** quiquell|off is now known as quiquell | 06:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing https://review.openstack.org/618671 | 06:49 |
| *** jamesmcarthur has joined #openstack-infra | 06:58 | |
| *** kjackal has joined #openstack-infra | 07:00 | |
| *** jamesmcarthur has quit IRC | 07:02 | |
| *** david-lyle has joined #openstack-infra | 07:03 | |
| *** dklyle has quit IRC | 07:04 | |
| *** pcaruana has joined #openstack-infra | 07:08 | |
| *** dtantsur|afk is now known as dtantsur | 07:38 | |
| *** rcernin has quit IRC | 07:48 | |
| *** slaweq has joined #openstack-infra | 07:49 | |
| *** e0ne has joined #openstack-infra | 07:54 | |
| *** quiquell is now known as quiquell|brb | 07:56 | |
| *** rpittau has joined #openstack-infra | 08:02 | |
| *** jtomasek has joined #openstack-infra | 08:04 | |
| *** bobh has joined #openstack-infra | 08:08 | |
| *** ginopc has joined #openstack-infra | 08:11 | |
| *** quiquell|brb is now known as quiquell | 08:17 | |
| *** jpena|off is now known as jpena | 08:17 | |
| *** ykarel is now known as ykarel|lunch | 08:25 | |
| *** simon-AS5591 has joined #openstack-infra | 08:28 | |
| tobias-urdin | fungi: thank you :) | 08:32 |
| *** ramishra has quit IRC | 08:33 | |
| *** ramishra has joined #openstack-infra | 08:38 | |
| *** bobh has quit IRC | 08:39 | |
| *** gfidente has joined #openstack-infra | 08:46 | |
| *** bobh has joined #openstack-infra | 08:46 | |
| *** bobh has quit IRC | 08:50 | |
| *** d0ugal has joined #openstack-infra | 08:54 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: Add support for Gerrit v2.16's change URL schema https://review.openstack.org/619533 | 08:58 |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** mdbooth has joined #openstack-infra | 09:01 | |
| *** rossella_s has joined #openstack-infra | 09:02 | |
| *** mdbooth has quit IRC | 09:03 | |
| *** florianf is now known as florianf|pto | 09:04 | |
| *** e0ne_ has joined #openstack-infra | 09:07 | |
| *** e0ne has quit IRC | 09:09 | |
| openstackgerrit | Pierre Riteau proposed openstack/diskimage-builder master: Fix a typo in the help message of disk-image-create https://review.openstack.org/619679 | 09:09 |
| *** ykarel|lunch is now known as ykarel | 09:10 | |
| *** tosky has joined #openstack-infra | 09:15 | |
| *** bhavikdbavishi has quit IRC | 09:17 | |
| *** alexchadin has joined #openstack-infra | 09:19 | |
| *** emine__ has joined #openstack-infra | 09:26 | |
| *** alexchadin has quit IRC | 09:29 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** sshnaidm is now known as sshnaidm|off | 09:52 | |
| *** sambetts_ has joined #openstack-infra | 09:54 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: Add support for Gerrit v2.16's change URL schema https://review.openstack.org/619533 | 09:55 |
| openstackgerrit | chenge proposed openstack/diskimage-builder master: fix links https://review.openstack.org/619699 | 10:03 |
| *** longkb has joined #openstack-infra | 10:10 | |
| *** ramishra has quit IRC | 10:12 | |
| *** ramishra has joined #openstack-infra | 10:12 | |
| *** longkb has quit IRC | 10:20 | |
| *** chkumar|ruck has quit IRC | 10:22 | |
| *** simon-AS559 has joined #openstack-infra | 10:25 | |
| *** simon-AS5591 has quit IRC | 10:25 | |
| *** psachin has quit IRC | 10:26 | |
| *** psachin has joined #openstack-infra | 10:27 | |
| *** dpawlik_ has quit IRC | 10:32 | |
| *** chandan_kumar has joined #openstack-infra | 10:35 | |
| *** chandan_kumar is now known as chkumar|ruck | 10:35 | |
| *** dpawlik has joined #openstack-infra | 10:37 | |
| *** electrofelix has joined #openstack-infra | 10:38 | |
| *** lpetrut has joined #openstack-infra | 10:46 | |
| *** sambetts_ has quit IRC | 10:49 | |
| *** udesale has quit IRC | 11:07 | |
| *** jchhatbar has joined #openstack-infra | 11:13 | |
| *** janki has quit IRC | 11:15 | |
| *** ahosam has joined #openstack-infra | 11:20 | |
| *** psachin has quit IRC | 11:26 | |
| *** jchhatba_ has joined #openstack-infra | 11:29 | |
| *** rpittau_ has joined #openstack-infra | 11:31 | |
| *** aojea has joined #openstack-infra | 11:32 | |
| *** jchhatbar has quit IRC | 11:32 | |
| *** rpittau has quit IRC | 11:35 | |
| *** simon-AS5591 has joined #openstack-infra | 11:37 | |
| *** simon-AS559 has quit IRC | 11:38 | |
| *** rpittau_ is now known as rpittau | 11:43 | |
| *** alexchadin has joined #openstack-infra | 11:55 | |
| *** ahosam has quit IRC | 12:03 | |
| *** ramishra has quit IRC | 12:04 | |
| *** EmilienM is now known as EvilienM | 12:14 | |
| *** carl_cai has joined #openstack-infra | 12:20 | |
| *** jpena is now known as jpena|lunch | 12:20 | |
| *** shardy has joined #openstack-infra | 12:20 | |
| *** pcaruana has quit IRC | 12:26 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:41 | |
| *** fuentess has joined #openstack-infra | 13:09 | |
| *** jpena|lunch is now known as jpena | 13:15 | |
| *** bhavikdbavishi has quit IRC | 13:34 | |
| *** quiquell is now known as quiquell|lunch | 13:39 | |
| *** ykarel is now known as ykarel|away | 14:01 | |
| *** quiquell|lunch is now known as quiquell | 14:01 | |
| *** simon-AS5591 has quit IRC | 14:06 | |
| *** simon-AS559 has joined #openstack-infra | 14:08 | |
| *** simon-AS559 has quit IRC | 14:08 | |
| *** simon-AS5591 has joined #openstack-infra | 14:08 | |
| *** pbourke has quit IRC | 14:11 | |
| *** pbourke has joined #openstack-infra | 14:12 | |
| *** chkumar|ruck has quit IRC | 14:15 | |
| *** simon-AS5591 has quit IRC | 14:24 | |
| *** ykarel|away has quit IRC | 14:25 | |
| *** simon-AS559 has joined #openstack-infra | 14:25 | |
| *** carl_cai has quit IRC | 14:29 | |
| *** cdent has joined #openstack-infra | 14:33 | |
| cdent | gerrit cranky? | 14:33 |
| witek | I cannot push my change either | 14:35 |
| fungi | has it bogged down again? | 14:39 |
| cdent | fungi: it seems to have recovered, but for about 5 minutes about 5 minutes ago it was timing out | 14:39 |
| fungi | oh yeah, looks like memory usage in the jvm climbed steadily after i garbage-collected it yesterday and got back up around 45gb before it recovered again at 12:30z | 14:40 |
| fungi | now it's starting to climb again | 14:40 |
| cdent | that seems rather architecturally totally wrong | 14:42 |
| cdent | but I'm probably drawn wrong | 14:42 |
| fungi | yeah, i'm looking over a chart of memory utilization within the jvm over the past month and it did reach similar levels the week before the summit too | 14:43 |
| *** gtema has joined #openstack-infra | 14:43 | |
| AJaeger | infra-root, gtema mentioned some strange jobs failures on OVH-BHS1 - anybody around to check the cloud, please? | 14:45 |
| fungi | looks like nl04 is the one handling ovh | 14:47 |
| fungi | i see timeouts on node delete requests in gra1 | 14:49 |
| *** alexchadin has quit IRC | 14:49 | |
| fungi | which explains why we have a bunch of nodes in a delete state in that region | 14:50 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Migration to PHP 7.x https://review.openstack.org/616226 | 14:51 |
| fungi | ahh, here's a launch failure for bhs1... | 14:51 |
| fungi | yeah, the majority of the launch failures are bhs1 while the delete timeouts are gra1 | 14:52 |
| fungi | openstack.exceptions.SDKException: Error in creating the server. Compute service reports fault: No valid host was found. There are not enough hosts available. | 14:53 |
| fungi | so likely a capacity problem with the host aggregate they have us on (or at least i think that's what they've said was the issue when we've seen that error in the past?) | 14:54 |
| *** ykarel|away has joined #openstack-infra | 14:54 | |
| gtema | and the ones are there seems to be overloaded as well, since lots of jobs are getting sporadic timeouts | 14:54 |
| fungi | amorin: ^ perhaps you know (if you're around)? | 14:54 |
| *** trown|outtypewww has quit IRC | 14:54 | |
| *** rossella_s has quit IRC | 14:55 | |
| fungi | possible we're oversubscribed there and dropping our max-servers in bhs1 for now would improve performance, if we're being our own noisy neighbor | 14:55 |
| *** quiquell is now known as quiquell|off | 14:55 | |
| fungi | we've failed over 350 boot calls in bhs1 over the past ~2.5 hours due to "not enough hosts available" | 14:57 |
| amorin | hey | 14:57 |
| fungi | looking to see how far back this stretches | 14:57 |
| amorin | no valid host on BHS!/ | 14:57 |
| amorin | BHS1? | 14:57 |
| fungi | yep | 14:58 |
| amorin | checking | 14:58 |
| *** trown|brb has joined #openstack-infra | 14:58 | |
| fungi | looks like we've been seeing it fairly steadily when we're under load stretching back ~5 days | 14:58 |
| fungi | though actually it was only summit week where we weren't, so might just be due to making far fewer boot calls that week than usual | 14:59 |
| fungi | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&panelId=12&fullscreen&from=1531925989384&to=1542985189896&var-region=ovh-bhs1 | 15:00 |
| fungi | for a longer term view of the launch node error counts | 15:00 |
| fungi | so if the job timeouts in bhs1 have only cropped up relatively recently, i don't think the launch failures are likely related to the job timeouts there | 15:02 |
| fungi | probably two separate problems | 15:02 |
| *** dpawlik has quit IRC | 15:06 | |
| amorin | everyuthing is good but the aggregate is full | 15:07 |
| amorin | I will add a host | 15:07 |
| fungi | okay, thanks! | 15:07 |
| fungi | makes sense that we'd be getting that failure then if there weren't enough hosts in the aggregate to satisfy our quota | 15:08 |
| *** simon-AS5591 has joined #openstack-infra | 15:08 | |
| fungi | what sort of oversubscription ratio do we have there? | 15:09 |
| fungi | wondering if we should try setting our max-servers a bit under quota to see if that alleviates job timeouts, but hard to guess how much to try dropping it without knowing how oversubscribed the hosts are | 15:10 |
| *** simon-AS559 has quit IRC | 15:10 | |
| amorin | there is not oversubscription | 15:15 |
| amorin | no* | 15:15 |
| amorin | let me confirm | 15:16 |
| amorin | so, on CPU, we do x2 | 15:17 |
| amorin | no oversubscript on ram | 15:18 |
| fungi | given the reports seem to be about processes being generally slow in jobs i wouldn't be surprised if we're running into cpu contention. it might be a good experiment for us to try setting our bhs1 max-servers in nodepool to half and seeing if the job timeouts cease | 15:19 |
| *** lpetrut has quit IRC | 15:22 | |
| fungi | looks like we have bhs1 set to max-servers 159 and are running right up against that limit at the moment | 15:22 |
| *** cdent has quit IRC | 15:23 | |
| evrardjp | is it me or gerrit is in trouble? | 15:25 |
| *** bhavikdbavishi has joined #openstack-infra | 15:26 | |
| gtema | evrardjp: you are not alone | 15:26 |
| fungi | evrardjp: yeah, i think we're seeing a pileup of established connections on it eating memory, trying to see if there's a pattern but also troubleshooting ci issues concurrently | 15:26 |
| evrardjp | fungi: and you should be on holiday I guess :p | 15:27 |
| fungi | maybe some other infra-root folks will be around soon, but not sure since it's a holiday for many of them | 15:27 |
| evrardjp | it's start to be end of day for europe (it's a friday too, right?), so that timing is either really bad or perfect | 15:28 |
| fungi | heh, i'll take it! ;) | 15:28 |
| evrardjp | (depending on which side of the pond you're looking) | 15:28 |
| evrardjp | fungi: not sure what, but if it's about holidays, the timing is about right :p | 15:29 |
| evrardjp | hahaha | 15:29 |
| odyssey4me | yeah, gerrit seems to be hurting - sometimes a page comes back, sometimes it doesn't | 15:29 |
| evrardjp | I probably can't help you folks, but I am really behind you all | 15:29 |
| evrardjp | odyssey4me: yeah | 15:29 |
| evrardjp | you're not alone | 15:29 |
| evrardjp | I now regret not using gertty more :p | 15:29 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Halve ovh-bhs1 max-servers temporarily https://review.openstack.org/619749 | 15:31 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Halve ovh-bhs1 max-servers temporarily" https://review.openstack.org/619750 | 15:31 |
| fungi | established connections to the gerrit ssh api seem to account for a majority of the count reported by cacti | 15:35 |
| *** shardy has quit IRC | 15:36 | |
| fungi | top offenders are ip addresses at emc, netapp, red hat, vmware, cloudbase, fortinet, nec, lenovo, ibm... probably all third-party ci systems | 15:40 |
| *** rossella_s has joined #openstack-infra | 15:40 | |
| fungi | mmm, no i was looking at the wrong timeframe on the graph so i don't think the established connection counts are necessarily related | 15:42 |
| *** ykarel|away has quit IRC | 15:43 | |
| *** bhavikdbavishi has quit IRC | 15:43 | |
| *** aojea has quit IRC | 15:45 | |
| *** ykarel|away has joined #openstack-infra | 15:45 | |
| fungi | most of the recent errors reported by javamelody are related to queries the stackalytics-bot-2 account has been making, not sure if there's any correlation | 15:45 |
| aspiers | I guess folks already saw that Gerrit 2.16 is out? https://www.gerritcodereview.com/2.16.html | 15:49 |
| fungi | aspiers: awesome! i was not, no | 15:50 |
| aspiers | ah cool | 15:50 |
| * aspiers feels slightly less useless than normal ;-) | 15:50 | |
| aspiers | fungi: then you might also be interested in https://gitenterprise.me/2018/11/18/gerrit-user-summit-2018-2/ | 15:52 |
| aspiers | keep an eye on https://www.youtube.com/gerritforgetv for videos emerging | 15:53 |
| fungi | thanks!!! that's going to be a good read once things hopefully calm down today | 15:53 |
| *** e0ne_ has quit IRC | 15:55 | |
| fungi | #status log temporarily blocked stackalytics-bot-2 access to review.o.o to confirm whether the errors reported for it are related to current performance problems | 15:56 |
| openstackstatus | fungi: finished logging | 15:56 |
| fungi | i used `sudo ip6tables -I openstack-INPUT 3 -s 2600:3c01:0:0:f03c:91ff:fe69:4d6f -j DROP` to insert it just before the accept related,established rule but after the accept icmp rule | 15:57 |
| *** jamesmcarthur has joined #openstack-infra | 15:57 | |
| *** pcaruana has joined #openstack-infra | 16:00 | |
| *** cdent has joined #openstack-infra | 16:02 | |
| *** dpawlik has joined #openstack-infra | 16:04 | |
| *** armstrong has joined #openstack-infra | 16:05 | |
| fungi | memory usage plummeted shortly after i applied that block rule | 16:06 |
| fungi | inside the jvm anyway | 16:06 |
| fungi | going to keep an eye on it and see if that was only a temporary effect or if it restabilizes | 16:06 |
| *** dpawlik has quit IRC | 16:08 | |
| fungi | cdent: witek: evrardjp: (and anyone else) please let us know if gerrit response times are still a problem. hard to be sure it was related to the stackalytics queries | 16:10 |
| *** bhavikdbavishi has joined #openstack-infra | 16:11 | |
| cdent | fungi: so far good, will keep an eye on it | 16:11 |
| fungi | the webui is responding fine for me at the moment | 16:11 |
| fungi | but i know it's been in and out over the past few days | 16:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Halve ovh-bhs1 max-servers temporarily https://review.openstack.org/619749 | 16:11 |
| *** roman_g has joined #openstack-infra | 16:12 | |
| fungi | gtema: ^ hopefully we'll find out useful information if the job timeouts dry up after that goes into effect in the next half hour or so | 16:12 |
| gtema | thanks fungi | 16:13 |
| *** adriancz has quit IRC | 16:19 | |
| *** bobh has joined #openstack-infra | 16:19 | |
| evrardjp | fungi: good for me now | 16:28 |
| evrardjp | fungi: interesting to know that stackalytics is causing issues :p | 16:30 |
| evrardjp | now go eat some moar turkey! ;) | 16:30 |
| evrardjp | hahaha | 16:30 |
| evrardjp | I promise I am not jalous there! (oh god I am starving!) | 16:31 |
| fungi | well, i don't know for certain it's the fault of stackalytics querying, just testing a theory | 16:32 |
| evrardjp | aspiers: anything real good on the changelog? | 16:32 |
| aspiers | probably :) | 16:32 |
| evrardjp | If people don't come complaining it means it was a successful fix :) | 16:32 |
| evrardjp | aspiers: oh /files api endpoint that's kinda neat | 16:33 |
| evrardjp | learning gerrit at the same time :D | 16:33 |
| fungi | well, still far from a fix since the stackalytics operators are already used to rotating to new accounts because their socket handling doesn't clean up after itself and leaks eventually result in us refusing their connections anyway | 16:33 |
| fungi | if this seems to have helped we need to get up with them and find out what they've changed recently which could be contributing to the problem | 16:34 |
| evrardjp | shouldn't stackalytics be hosted by infra, when we think of it? | 16:34 |
| evrardjp | not sure what 'stackalytics operators' is :) | 16:35 |
| *** bobh has quit IRC | 16:44 | |
| *** kjackal has quit IRC | 16:44 | |
| *** rpittau has quit IRC | 16:45 | |
| fungi | still mirantis folks as far as i know | 16:48 |
| fungi | and yeah, memory utilization has stabilized after the plummet from when i added that block rule | 16:49 |
| fungi | evrardjp: we've tried to work with them to take over hosting and collaborate on development in the past and it never quite happened | 16:49 |
| fungi | there were people willing to contribute (in some cases already wrote specs/patches) to fixing some of the inherent design issues with it, but they didn't seem to want to give up control of the project nor did they seem to have time to review any such overhaul | 16:50 |
| fungi | we ran a stackalytics.openstack.org instance under infra team control for a while, which is what led to us discovering some of its problems like lack of persistent analysis state (leading it to re-query and reanalyze everything each time it was restarted) and lack of on-the-fly reconfiguration (leading to a need for frequent restarts for simple things like changing someone's employer affiliation) | 16:52 |
| fungi | i suppose we could have decided to fork it, but the people involved on this end lost interest after so much radio silence from the stackalytics maintainers and operators | 16:54 |
| *** jchhatba_ has quit IRC | 16:54 | |
| *** ginopc has quit IRC | 17:01 | |
| gtema | another job failed with timeout in devstack setup in ovh-bhs1 | 17:01 |
| *** jpich has quit IRC | 17:07 | |
| *** dtantsur is now known as dtantsur|afk | 17:07 | |
| *** gtema has quit IRC | 17:13 | |
| *** roman_g has quit IRC | 17:13 | |
| *** slaweq has quit IRC | 17:16 | |
| fungi | i wish gtema has linked to the log so i could check whether that ran before our utilization dropped in that region | 17:18 |
| fungi | er, had linked | 17:18 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed method getPresentationsBySummitAndRole Added Selection Plan to Presentation Serializer https://review.openstack.org/619775 | 17:19 |
| *** jamesmcarthur has quit IRC | 17:31 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed method getPresentationsBySummitAndRole Added Selection Plan to Presentation Serializer https://review.openstack.org/619775 | 17:31 |
| fungi | looks like we whittled the in use nodecount down to ~80 as of 17:00 utc, so if we see a lot of unanticipated timeouts from jobs starting in that region after that time then it's likely for some other reason than cpu contention | 17:32 |
| *** jpena is now known as jpena|off | 17:33 | |
| fungi | also gerrit jvm memory utilization seems to be holding steady still since the firewall rule went in at ~16:00z | 17:34 |
| *** emine has joined #openstack-infra | 17:35 | |
| *** Adri2000 has quit IRC | 17:38 | |
| *** emine__ has quit IRC | 17:38 | |
| *** jmccrory has quit IRC | 17:38 | |
| *** cmurphy has quit IRC | 17:38 | |
| *** Adri2000 has joined #openstack-infra | 17:39 | |
| *** jmccrory has joined #openstack-infra | 17:39 | |
| *** cmurphy has joined #openstack-infra | 17:44 | |
| fungi | though digging into the gerrit ssh log, i'm unsure the two are connected now since the last connection from stackalytics-bot-2 terminated at 14:37:43z | 17:44 |
| fungi | and memory utilization was still climbing steadily for ~1.5 hours after that | 17:45 |
| *** roman_g has joined #openstack-infra | 17:48 | |
| fungi | huh, looking at gerrit's javamelody http stats, someone cloned neutron directly from gerrit's java-based git service 53 times | 17:53 |
| *** robcresswell has joined #openstack-infra | 17:56 | |
| fungi | more than one somebody, looks like | 17:56 |
| fungi | an ip address assigned to pure storage did it 22 times, an ip address assigned to intel did it 16 times, 15 from one for citrix | 17:57 |
| fungi | 11 from ibm | 17:58 |
| fungi | average size of the request was 30mb | 17:58 |
| logan- | sounds like 3rd party ci | 17:59 |
| fungi | it does indeed, but i doubt this is out of the ordinary either since it seems to be the same order of magnitude for a variety of organizations | 17:59 |
| fungi | mean time to complete each of those requests was 45 seconds | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| fungi | anyway, also not a smoking gun but good to know we have rather a lot of third-party ci systems cloning neutron directly from gerrit's jgit/jetty | 18:01 |
| fungi | which is notoriously under-performant | 18:02 |
| fungi | i count 15 distinct addresses from what look like probably 10 different organizations | 18:02 |
| fungi | some more prolific than others mind you | 18:02 |
| fungi | over the past ~12 hours | 18:03 |
| *** roman_g has quit IRC | 18:04 | |
| logan- | yeah, I think the jenkins gerrit plugin does that | 18:05 |
| fungi | and that's just cloning neutron. i see similar patterns hitting nova, cinder, ironic | 18:05 |
| fungi | the usual suspects for 3ptyci | 18:06 |
| *** ralonsoh has quit IRC | 18:08 | |
| fungi | according to the apache logs there'massive amounts of data being queried about | 18:09 |
| fungi | gerrit reviewer accounts by some address at mcgill university | 18:09 |
| fungi | 34298 in the past 12 hours from bodiddley.ece.mcgill.ca | 18:10 |
| *** fuentess has quit IRC | 18:11 | |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| fungi | 147794 queries from that address in total over the last 12 hours | 18:12 |
| fungi | seems to have started at 2018-11-17 06:01:11 utc and performed as many as 330k requests a day to gerrit with various queries for broad ranges of change dates | 18:15 |
| fungi | get requests like changes/?q=after:%7B2011-07-01%2000:00:00.000%7D%20AND%20before:%7B2011-07-01%2023:59:59.999%7D&o=ALL_COMMITS&o=ALL_REVISIONS&o=ALL_FILES&S=0 | 18:15 |
| fungi | incrementing day at a time and then re-querying for details on specific changes (presumably returned by the earlier queries) | 18:17 |
| fungi | some script/tool using python-requests | 18:17 |
| fungi | not gertty, as that identifies itself with a specific user agent string | 18:18 |
| logan- | ouch | 18:18 |
| *** bobh has joined #openstack-infra | 18:18 | |
| fungi | the exciting things you find when you go spelunking in access logs | 18:19 |
| fungi | though that's only second place for the most requests today | 18:20 |
| fungi | top offender is some anonymous ip address in aws | 18:20 |
| *** bobh has quit IRC | 18:21 | |
| fungi | it too seems to be crawling change numbers | 18:22 |
| fungi | with a user agent string of "Default User Agent" | 18:22 |
| *** ykarel|away has quit IRC | 18:22 | |
| fungi | started 2018-11-22 23:25:21 utc by checking /robots.txt before continuing to crawl | 18:24 |
| *** jamesmcarthur has quit IRC | 18:25 | |
| *** cdent has quit IRC | 18:25 | |
| *** bobh has joined #openstack-infra | 18:28 | |
| *** e0ne has joined #openstack-infra | 18:31 | |
| *** d0ugal has quit IRC | 18:41 | |
| *** bobh has quit IRC | 18:42 | |
| *** d0ugal has joined #openstack-infra | 18:57 | |
| *** roman_g has joined #openstack-infra | 19:00 | |
| *** electrofelix has quit IRC | 19:06 | |
| *** e0ne has quit IRC | 19:08 | |
| *** bhavikdbavishi has quit IRC | 19:12 | |
| *** roman_g has quit IRC | 19:19 | |
| *** roman_g has joined #openstack-infra | 19:24 | |
| *** armstrong has quit IRC | 19:25 | |
| *** jtomasek has quit IRC | 19:40 | |
| *** rfolco has quit IRC | 19:40 | |
| *** e0ne has joined #openstack-infra | 19:42 | |
| *** roman_g has quit IRC | 19:43 | |
| *** david-lyle has quit IRC | 19:47 | |
| *** witek has quit IRC | 19:54 | |
| *** emine has quit IRC | 19:58 | |
| *** witek has joined #openstack-infra | 19:59 | |
| *** bobh has joined #openstack-infra | 20:05 | |
| *** bobh has quit IRC | 20:09 | |
| *** e0ne has quit IRC | 20:20 | |
| *** jamesmcarthur has joined #openstack-infra | 20:25 | |
| *** robcresswell has quit IRC | 20:25 | |
| *** gfidente has quit IRC | 20:25 | |
| *** jamesmcarthur has quit IRC | 20:30 | |
| *** slaweq has joined #openstack-infra | 20:40 | |
| *** kjackal has joined #openstack-infra | 20:53 | |
| *** dave-mccowan has joined #openstack-infra | 20:54 | |
| *** xek__ has quit IRC | 21:01 | |
| *** xek__ has joined #openstack-infra | 21:02 | |
| *** xek__ has quit IRC | 21:03 | |
| *** xek__ has joined #openstack-infra | 21:04 | |
| *** xek__ has quit IRC | 21:05 | |
| *** xek has joined #openstack-infra | 21:06 | |
| *** nsmeds has joined #openstack-infra | 21:08 | |
| *** xek has quit IRC | 21:09 | |
| *** xek has joined #openstack-infra | 21:09 | |
| *** xek has quit IRC | 21:11 | |
| *** xek_ has joined #openstack-infra | 21:11 | |
| *** dave-mccowan has quit IRC | 21:11 | |
| *** xek_ has quit IRC | 21:12 | |
| *** slaweq has quit IRC | 21:21 | |
| *** pcaruana has quit IRC | 21:46 | |
| *** jamesmcarthur has joined #openstack-infra | 22:17 | |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| *** lbragstad has joined #openstack-infra | 22:53 | |
| *** dhill_ has quit IRC | 22:56 | |
| *** kjackal has quit IRC | 22:57 | |
| *** jamesmcarthur has quit IRC | 23:04 | |
| *** eandersson has joined #openstack-infra | 23:10 | |
| *** slaweq has joined #openstack-infra | 23:18 | |
| *** xek has joined #openstack-infra | 23:25 | |
| *** rossella_s has quit IRC | 23:26 | |
| *** slaweq has quit IRC | 23:27 | |
| *** jamesmcarthur has joined #openstack-infra | 23:30 | |
| *** jamesmcarthur has quit IRC | 23:49 | |
| *** jamesmcarthur has joined #openstack-infra | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!