Monday, 2017-08-21

*** sbezverk has joined #openstack-infra00:01
*** yamamoto has joined #openstack-infra00:03
*** tosky has quit IRC00:04
*** dingyichen has joined #openstack-infra00:06
*** yamamoto has quit IRC00:08
*** thorst has joined #openstack-infra00:09
*** thorst has quit IRC00:13
*** sflanigan has joined #openstack-infra00:18
*** markvoelker has joined #openstack-infra00:19
pabelangerianw: thanks!00:27
*** mamitchl has joined #openstack-infra00:30
*** aeng has quit IRC00:32
*** iyamahat has quit IRC00:37
*** Guest39999 has quit IRC00:38
ianwpabelanger: i'll put the other override in and do a rebuild of centos00:40
*** Hal has joined #openstack-infra00:40
*** Hal is now known as Guest5029500:41
*** tuanluong has joined #openstack-infra00:42
*** thorst has joined #openstack-infra00:44
*** thorst has quit IRC00:45
pabelangerianw: yay, thanks00:47
openstackgerritIan Wienand proposed openstack-infra/system-config master: Remove internap clouds  https://review.openstack.org/49462900:47
ianwclarkb: also released i missed inap in https://etherpad.openstack.org/p/mirror-xenial, will look at now00:48
openstackgerritMerged openstack-infra/project-config master: Create glean@.service.d/override.conf  https://review.openstack.org/49366500:48
clarkbianw: inap was done by jebair as part of the name change I think00:48
ianwcool, if so i'll just mark it off00:49
*** aeng has joined #openstack-infra00:49
ianwclarkb: ok, it is xenial, but not "01" named00:50
openstackgerritMerged openstack-infra/system-config master: Move Lon1 and La1 mirrors to digitized names  https://review.openstack.org/49540800:50
ianwit's got attached volumes so not sure i want to fiddle00:52
clarkboh right that was before the digitizing00:56
clarkbmay be worth replacing for consistency but probably not urgent00:56
openstackgerritIan Wienand proposed openstack-infra/system-config master: Remove internap clouds  https://review.openstack.org/49462900:58
*** mat128 has joined #openstack-infra01:00
*** ianychoi has quit IRC01:02
*** liusheng has quit IRC01:04
*** mat128 has quit IRC01:08
*** cshastri has joined #openstack-infra01:11
*** hongbin has joined #openstack-infra01:11
*** liujiong has joined #openstack-infra01:12
*** aeng has quit IRC01:17
*** aeng has joined #openstack-infra01:17
ianwhmm, that's a new centos failure -> 2017-08-18 06:15:18.515 | ERROR neutron [01;35m[00mException: Could not retrieve schema from tcp:127.0.0.1:6640: Connection refused01:18
ianwthat's an ovs port01:18
*** gouthamr has quit IRC01:26
*** funzo has joined #openstack-infra01:27
*** bobh has joined #openstack-infra01:27
*** liusheng has joined #openstack-infra01:27
*** masber has joined #openstack-infra01:28
*** gcb has joined #openstack-infra01:30
*** funzo has quit IRC01:31
*** bobh has quit IRC01:32
*** zhurong has joined #openstack-infra01:36
*** gildub has joined #openstack-infra01:36
*** armax has joined #openstack-infra01:38
*** Guest50295 has quit IRC01:38
*** Hal has joined #openstack-infra01:39
*** Hal is now known as Guest5675301:39
*** sree has joined #openstack-infra01:46
*** thorst has joined #openstack-infra01:46
*** sree has quit IRC01:50
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add py36 devstack up/down job for Fedora 26  https://review.openstack.org/49567001:51
*** thorst has quit IRC01:51
*** yamamoto has joined #openstack-infra02:06
*** yamamoto has quit IRC02:11
*** armax has quit IRC02:19
*** funzo has joined #openstack-infra02:28
*** bobh has joined #openstack-infra02:28
*** funzo has quit IRC02:32
*** bobh has quit IRC02:32
*** zhurong has quit IRC02:34
*** dave-mccowan has quit IRC02:36
*** yamamoto has joined #openstack-infra02:45
*** thorst has joined #openstack-infra02:47
*** bobh has joined #openstack-infra02:50
*** thorst has quit IRC02:51
*** namnh has joined #openstack-infra03:17
*** armax has joined #openstack-infra03:17
*** hongbin has quit IRC03:33
*** Guest56753 has quit IRC03:35
*** Hal has joined #openstack-infra03:35
*** Hal is now known as Guest8613503:36
*** ramishra has joined #openstack-infra03:37
*** Dinesh_Bhor has joined #openstack-infra03:45
*** bobh has quit IRC03:46
*** felipemonteiro_ has joined #openstack-infra03:48
*** thorst has joined #openstack-infra03:48
*** felipemonteiro__ has joined #openstack-infra03:50
*** thorst has quit IRC03:52
*** felipemonteiro_ has quit IRC03:53
*** tonyb has quit IRC03:53
*** bobh has joined #openstack-infra03:54
*** links has joined #openstack-infra03:54
*** felipemonteiro__ has quit IRC03:55
*** ykarel__ has joined #openstack-infra03:57
*** tonyb has joined #openstack-infra04:00
*** bobh has quit IRC04:01
*** shardy has quit IRC04:06
*** liujiong_lj has joined #openstack-infra04:08
*** liujiong has quit IRC04:08
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add Python 3.6 jobs  https://review.openstack.org/49572804:09
*** shardy has joined #openstack-infra04:13
*** ykarel__ has quit IRC04:16
*** ykarel has joined #openstack-infra04:17
*** udesale has joined #openstack-infra04:21
ianwpabelanger: looking good04:23
ianw-bash-4.2# hostname04:23
ianwcentos-7-rax-ord-1053839904:23
ianw-bash-4.2# cat /etc/resolv.conf04:23
ianwnameserver 127.0.0.104:23
pabelangerianw: yay, awesome04:26
*** sree has joined #openstack-infra04:27
*** sree has quit IRC04:28
*** sree has joined #openstack-infra04:28
*** baoli has joined #openstack-infra04:28
*** funzo has joined #openstack-infra04:29
*** funzo has quit IRC04:33
*** ying_zuo has joined #openstack-infra04:36
*** gildub has quit IRC04:37
*** slaweq has joined #openstack-infra04:41
*** slaweq has quit IRC04:45
*** pbourke has quit IRC04:54
*** pbourke has joined #openstack-infra04:55
*** bhavik1 has joined #openstack-infra04:56
*** eranrom has joined #openstack-infra04:58
*** Guest86135 has quit IRC05:02
*** dhajare has joined #openstack-infra05:02
*** bhavik1 has quit IRC05:03
*** coolsvap has joined #openstack-infra05:05
*** baoli has quit IRC05:12
*** flwang has quit IRC05:15
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add Python 3.6 tox jobs  https://review.openstack.org/49572805:16
*** flwang has joined #openstack-infra05:18
*** kjackal has joined #openstack-infra05:23
*** e0ne has joined #openstack-infra05:31
*** e0ne has quit IRC05:35
*** eranrom has quit IRC05:39
*** baoli has joined #openstack-infra05:40
*** eranrom has joined #openstack-infra05:41
*** udesale__ has joined #openstack-infra05:42
*** baoli has quit IRC05:44
*** udesale has quit IRC05:45
*** ramishra has quit IRC05:48
*** thorst has joined #openstack-infra05:49
*** thorst has quit IRC05:54
*** pgadiya has joined #openstack-infra05:55
*** kjackal has quit IRC05:56
*** jascott1 has quit IRC06:03
*** rcernin has joined #openstack-infra06:04
*** ramishra has joined #openstack-infra06:06
*** LindaWang has joined #openstack-infra06:07
*** jascott1 has joined #openstack-infra06:08
*** namnh has quit IRC06:15
*** jaosorior has joined #openstack-infra06:16
*** liujiong_lj is now known as liujiong06:20
*** eranrom has quit IRC06:21
*** jtomasek has joined #openstack-infra06:21
*** AJaeger has quit IRC06:22
*** eranrom has joined #openstack-infra06:23
*** AJaeger has joined #openstack-infra06:26
*** funzo has joined #openstack-infra06:29
*** pcaruana has joined #openstack-infra06:30
*** funzo has quit IRC06:34
*** bogdando has joined #openstack-infra06:36
*** claudiub|3 has joined #openstack-infra06:44
*** thorst has joined #openstack-infra06:51
*** ykarel has quit IRC06:53
*** thorst has quit IRC06:55
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: neutron-fwaas/vpnaas-dashboard: publish to PyPI  https://review.openstack.org/49573806:57
bogdandoo/ PTAL e-r queries https://review.openstack.org/#/c/493535/ https://review.openstack.org/#/c/493525/  https://review.openstack.org/#/c/493520/ to reduce unknown/uncategorised numbers06:59
bogdandoI thought those queries for e-r are getting merged fast! :)06:59
bogdandomandre: hi. Do you mind to get https://review.openstack.org/#/c/494367/ accepted as is?07:04
bogdandoit's a drastic improvement for CI jobs time, kudos to stevebaker07:04
*** ykarel has joined #openstack-infra07:05
bogdandooh, sorry folks, wrong channel!07:06
*** kjackal has joined #openstack-infra07:06
*** florianf has joined #openstack-infra07:09
*** jbadiapa has quit IRC07:11
*** jbadiapa has joined #openstack-infra07:12
*** isaacb has joined #openstack-infra07:16
*** ramishra has quit IRC07:17
*** amoralej|off is now known as amoralej07:20
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Enable Octavia Variant  https://review.openstack.org/48792807:21
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Stop running broken trove tests  https://review.openstack.org/49574207:21
openstackgerritMerged openstack-infra/project-config master: enable qos in congress dsvm job  https://review.openstack.org/49435007:24
*** jbadiapa has quit IRC07:26
*** jpena|off is now known as jpena07:26
*** ramishra has joined #openstack-infra07:27
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Move experimental kuryr-k8s octavia job to check  https://review.openstack.org/49574407:28
openstackgerritMerged openstack-infra/project-config master: Fix layout for tripleo scenario007  https://review.openstack.org/49549307:29
*** gaurangt has joined #openstack-infra07:31
*** jpich has joined #openstack-infra07:32
*** cuongnv has joined #openstack-infra07:32
openstackgerritMerged openstack-infra/project-config master: Add splitted multinode job to watcher-tempest-plugin  https://review.openstack.org/49528907:33
*** Hal has joined #openstack-infra07:35
*** Hal is now known as Guest8372807:35
*** iyamahat has joined #openstack-infra07:38
*** gildub has joined #openstack-infra07:40
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add py36 devstack up/down job for Fedora 26  https://review.openstack.org/49567007:41
*** jbadiapa has joined #openstack-infra07:42
*** xinliang has joined #openstack-infra07:45
*** xinliang has quit IRC07:45
*** xinliang has joined #openstack-infra07:45
*** egonzalez has joined #openstack-infra07:45
*** aarefiev_afk is now known as aarefiev07:46
*** shardy has quit IRC07:49
*** shardy has joined #openstack-infra07:49
*** thorst has joined #openstack-infra07:51
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: neutron-fwaas/vpnaas-dashboard: publish to PyPI  https://review.openstack.org/49573807:53
openstackgerritMerged openstack-infra/project-config master: Split COE scenarios: 1 for each COE  https://review.openstack.org/49427207:54
*** Guest83728 has quit IRC07:54
*** ianychoi has joined #openstack-infra07:54
*** thorst has quit IRC07:56
openstackgerritMerged openstack-infra/project-config master: Add networking-sfc to Dragonflow's tempest job  https://review.openstack.org/49395007:56
*** iyamahat has quit IRC08:06
*** Hal has joined #openstack-infra08:09
*** Hal is now known as Guest7786408:10
*** udesale has joined #openstack-infra08:11
*** udesale__ has quit IRC08:12
*** isaacb has quit IRC08:13
*** larainema has quit IRC08:18
*** markus_z has joined #openstack-infra08:18
*** larainema has joined #openstack-infra08:21
*** isaacb has joined #openstack-infra08:22
*** derekh has joined #openstack-infra08:23
*** gaurangt_ has joined #openstack-infra08:23
*** lucas-afk is now known as lucasagomes08:25
*** gaurangt- has joined #openstack-infra08:26
*** gaurangt has quit IRC08:26
*** kukacz has quit IRC08:27
*** gaurangt_ has quit IRC08:28
*** funzo has joined #openstack-infra08:30
*** funzo has quit IRC08:35
*** Guest77864 has quit IRC08:35
*** jascott1 has quit IRC08:47
*** jascott1 has joined #openstack-infra08:47
*** tosky has joined #openstack-infra08:47
*** dingyichen has quit IRC08:48
openstackgerritAlexander Chadin proposed openstack-infra/project-config master: Remove old functional job for Watcher  https://review.openstack.org/49529608:48
*** bradm has joined #openstack-infra08:50
*** jascott1 has quit IRC08:52
*** thorst has joined #openstack-infra08:52
*** pcaruana has quit IRC08:53
*** baoli has joined #openstack-infra08:55
*** pcaruana has joined #openstack-infra08:55
*** thorst has quit IRC08:56
*** baoli has quit IRC08:59
*** ykarel has quit IRC09:11
*** Hal has joined #openstack-infra09:14
*** Hal is now known as Guest9239809:14
*** jbadiapa_ has joined #openstack-infra09:15
*** jbadiapa has quit IRC09:17
*** dtantsur has quit IRC09:18
*** pcaruana has quit IRC09:19
*** tosky_ has joined #openstack-infra09:19
*** tosky has quit IRC09:19
*** electrofelix has joined #openstack-infra09:19
*** dtantsur has joined #openstack-infra09:24
*** dtantsur has quit IRC09:24
*** dtantsur has joined #openstack-infra09:24
*** dtantsur has quit IRC09:24
*** dtantsur has joined #openstack-infra09:24
*** dtantsur has quit IRC09:24
*** dtantsur has joined #openstack-infra09:29
*** pcaruana has joined #openstack-infra09:32
openstackgerritFlavio Percoco proposed openstack-infra/tripleo-ci master: Collect mistral-ansible execution files from /tmp  https://review.openstack.org/48386709:33
*** kukacz has joined #openstack-infra09:34
*** nicolasbock_ has joined #openstack-infra09:42
*** tosky has joined #openstack-infra09:44
*** dtantsur has quit IRC09:44
*** pcaruana has quit IRC09:45
*** tosky_ has quit IRC09:45
*** nicolasbock_ has quit IRC09:47
*** markmcd has quit IRC09:49
*** dtantsur has joined #openstack-infra09:50
*** dims has quit IRC09:50
*** umbarkar has joined #openstack-infra09:51
*** markmcd has joined #openstack-infra09:51
*** tumbarka has quit IRC09:54
*** sambetts|afk is now known as sambetts09:57
*** LindaWang has quit IRC09:57
*** pcaruana has joined #openstack-infra09:58
*** sree has quit IRC09:58
*** kukacz has quit IRC09:58
*** sree has joined #openstack-infra09:58
*** egonzalez has quit IRC09:59
*** kukacz has joined #openstack-infra09:59
*** nicolasbock_ has joined #openstack-infra09:59
*** sree_ has joined #openstack-infra10:00
*** sree_ is now known as Guest8286010:00
*** cshastri has quit IRC10:01
*** strigazi has joined #openstack-infra10:01
*** cuongnv has quit IRC10:01
*** sree has quit IRC10:03
*** Guest82860 has quit IRC10:04
*** Qiming has quit IRC10:04
*** baoli has joined #openstack-infra10:05
*** efoley has joined #openstack-infra10:07
*** Qiming has joined #openstack-infra10:09
*** baoli has quit IRC10:09
*** ramishra has quit IRC10:16
*** ramishra has joined #openstack-infra10:18
*** egonzalez has joined #openstack-infra10:20
*** isaacb has quit IRC10:27
*** funzo has joined #openstack-infra10:28
*** wolverineav has joined #openstack-infra10:31
*** funzo has quit IRC10:32
*** udesale has quit IRC10:36
*** udesale has joined #openstack-infra10:38
openstackgerritMerged openstack-infra/project-config master: Remove old functional job for Watcher  https://review.openstack.org/49529610:39
*** jkilpatr has quit IRC10:41
*** shardy has quit IRC10:41
*** shardy has joined #openstack-infra10:41
openstackgerritArtur Basiak proposed openstack-infra/project-config master: Fix condition for logs pipeline  https://review.openstack.org/49581610:41
openstackgerritMerged openstack-infra/project-config master: Kuryr: Setup multinode gate  https://review.openstack.org/49494610:43
*** arturb has joined #openstack-infra10:44
*** e0ne has joined #openstack-infra10:44
*** gixxer1k has joined #openstack-infra10:45
openstackgerritMerged openstack-infra/project-config master: Add pike branch for OpenStack-Ansible gerritbot  https://review.openstack.org/49541010:47
*** wolverineav has quit IRC10:49
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Enable Octavia Variant  https://review.openstack.org/48792810:49
*** tuanluong has quit IRC10:50
*** shardy has quit IRC10:50
*** shardy has joined #openstack-infra10:51
*** thorst has joined #openstack-infra10:54
*** Julien-zte has quit IRC10:57
e0neHi team. can anybody please help with etherpad?10:57
e0neI can't open  can't open https://etherpad.openstack.org/p/horizon-pike-feedback  in any browser. any other links work good10:58
*** thorst has quit IRC10:58
*** kjackal has quit IRC11:01
*** jkilpatr has joined #openstack-infra11:01
*** mat128 has joined #openstack-infra11:07
*** eharney has joined #openstack-infra11:11
*** gcb has quit IRC11:11
*** wolverineav has joined #openstack-infra11:13
*** bnemec has joined #openstack-infra11:17
*** openstackgerrit has quit IRC11:17
*** bnemec has left #openstack-infra11:20
*** liujiong has quit IRC11:25
*** baoli has joined #openstack-infra11:26
*** egonzalez has quit IRC11:26
*** openstackgerrit has joined #openstack-infra11:29
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Enable Octavia Variant  https://review.openstack.org/48792811:29
*** baoli has quit IRC11:31
dmelladoe0ne: that's interesting11:31
dmelladohow about switching it for now to https://etherpad.openstack.org/p/horizon_pike_feedback11:31
dmellado?11:31
e0nedmellado: it opens new etherpad11:31
dmelladoe0ne: ugh11:31
*** hrubi has quit IRC11:33
*** hrubi has joined #openstack-infra11:34
*** ykarel|away has joined #openstack-infra11:36
*** ldnunes has joined #openstack-infra11:36
*** ykarel|away is now known as ykarel11:36
*** udesale has quit IRC11:36
*** udesale has joined #openstack-infra11:38
*** eharney has quit IRC11:38
*** ykarel has quit IRC11:43
*** lucasagomes is now known as lucas-hungry11:44
*** david-lyle has joined #openstack-infra11:50
*** dklyle has quit IRC11:51
*** kjackal has joined #openstack-infra11:52
*** dave-mccowan has joined #openstack-infra11:53
*** pgadiya has quit IRC11:53
*** Guest92398 has quit IRC11:54
*** thorst has joined #openstack-infra11:57
*** dprince has joined #openstack-infra11:57
*** dprince has quit IRC12:00
*** dprince has joined #openstack-infra12:00
*** jpich has quit IRC12:03
fungie0ne: we've seen pads get corrupted in the past. an infra-root sysadmin can probably retrieve the raw text content via the etherpad api so that it can be pasted into a new pad, but the formatting and author colors will be lost in the process12:04
*** egonzalez has joined #openstack-infra12:06
e0nefungi: I thinks it will be a better option than loss this data totally12:07
e0ne*think12:07
fungiif the corruption happened in a very recent edit it may be possible to roll the pad back, but i'm travelling all week and not in a position to investigate it myself12:07
e0nefungi: thanks. I'll ask somebody else later in US timezon12:09
*** e0ne has quit IRC12:09
*** mpaolino has joined #openstack-infra12:11
*** trown|outtypewww is now known as trown12:12
*** jpena is now known as jpena|lunch12:12
*** martinkopec has joined #openstack-infra12:12
*** e0ne has joined #openstack-infra12:15
*** isaacb has joined #openstack-infra12:16
*** LindaWang has joined #openstack-infra12:19
*** aviau has quit IRC12:19
*** aviau has joined #openstack-infra12:20
*** dhajare has quit IRC12:22
*** jpich has joined #openstack-infra12:26
*** Guest25108 has joined #openstack-infra12:26
*** amoralej is now known as amoralej|lunch12:27
*** lucas-hungry is now known as lucasagomes12:28
*** udesale has quit IRC12:29
openstackgerritThomas Morin proposed openstack-infra/project-config master: n8g-bagpipe: tune PROJECTS to match dependencies  https://review.openstack.org/49584112:29
*** ramishra has quit IRC12:30
*** udesale has joined #openstack-infra12:31
*** baoli has joined #openstack-infra12:35
*** rlandy has joined #openstack-infra12:36
*** ramishra has joined #openstack-infra12:37
*** baoli has quit IRC12:39
*** jbadiapa_ has quit IRC12:45
*** dprince has quit IRC12:51
*** dhajare has joined #openstack-infra12:53
openstackgerritThomas Morin proposed openstack-infra/project-config master: grafana updates for n8g-bgpvpn and n8g-bagpipe  https://review.openstack.org/49584912:54
*** coolsvap has quit IRC12:55
*** mat128 has quit IRC12:56
*** jcoufal has joined #openstack-infra12:58
*** mriedem has joined #openstack-infra13:00
*** jbadiapa_ has joined #openstack-infra13:01
*** gildub has quit IRC13:05
*** martinkopec has quit IRC13:06
*** aarefiev has quit IRC13:08
*** florianf has quit IRC13:09
*** udesale has quit IRC13:10
*** amoralej|lunch is now known as amoralej13:11
*** kgiusti has joined #openstack-infra13:11
*** aarefiev has joined #openstack-infra13:11
*** heroux has joined #openstack-infra13:11
*** florianf has joined #openstack-infra13:12
*** esberglu has joined #openstack-infra13:15
*** mattmceuen has joined #openstack-infra13:16
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Use direct calls to get_<resource>_by_id  https://review.openstack.org/49453513:16
*** dims has joined #openstack-infra13:21
*** srobert has joined #openstack-infra13:21
*** jpena|lunch is now known as jpena13:23
*** ldnunes has quit IRC13:24
*** ociuhandu has joined #openstack-infra13:27
*** Jeffrey4l has quit IRC13:27
*** Jeffrey4l has joined #openstack-infra13:28
*** jamesdenton has quit IRC13:28
*** bobh has joined #openstack-infra13:28
*** armax has quit IRC13:31
*** makowals has quit IRC13:31
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: DNM simulating project-config patch merged  https://review.openstack.org/48307713:31
*** yamamoto has quit IRC13:31
*** dprince has joined #openstack-infra13:32
*** dpawlik_ is now known as danpawlik13:33
*** makowals has joined #openstack-infra13:33
openstackgerritJohn Trowbridge proposed openstack-infra/project-config master: Revert "Revert "Change job type for 3nodes job to move to oooq runner""  https://review.openstack.org/49585313:33
*** alexchadin has joined #openstack-infra13:33
*** gcb has joined #openstack-infra13:36
*** atarakt has quit IRC13:39
*** atarakt has joined #openstack-infra13:39
*** ldnunes has joined #openstack-infra13:40
*** atarakt has quit IRC13:41
*** nhicher has joined #openstack-infra13:42
*** mat128 has joined #openstack-infra13:46
*** gcb has quit IRC13:47
*** gcb has joined #openstack-infra13:47
*** felipemonteiro_ has joined #openstack-infra13:48
*** baoli has joined #openstack-infra13:49
*** felipemonteiro__ has joined #openstack-infra13:49
*** Guest25108 has quit IRC13:50
*** markvoelker has quit IRC13:52
*** felipemonteiro_ has quit IRC13:52
*** baoli has quit IRC13:53
*** yamamoto has joined #openstack-infra13:57
*** yamamoto has quit IRC13:58
*** yamamoto has joined #openstack-infra13:59
*** yamamoto has quit IRC13:59
*** dmsimard|off is now known as dmsimard13:59
*** Goneri has joined #openstack-infra14:02
*** Hal has joined #openstack-infra14:05
*** Hal is now known as Guest4817914:06
*** dhajare has quit IRC14:07
*** armax has joined #openstack-infra14:08
*** Guest48179 has quit IRC14:10
*** alexchadin has quit IRC14:12
*** Jeffrey4l has quit IRC14:12
*** Jeffrey4l has joined #openstack-infra14:13
*** makowals has quit IRC14:15
*** makowals has joined #openstack-infra14:16
*** Jeffrey4l has quit IRC14:16
*** Jeffrey4l has joined #openstack-infra14:17
*** Jeffrey4l has quit IRC14:18
*** sree has joined #openstack-infra14:18
*** mpaolino has quit IRC14:19
*** Jeffrey4l has joined #openstack-infra14:21
*** arturb has quit IRC14:22
*** Jeffrey4l has quit IRC14:22
*** Jeffrey4l has joined #openstack-infra14:24
*** makowals has quit IRC14:24
openstackgerritMerged openstack-infra/tripleo-ci master: Copy Ceph logs from running containers  https://review.openstack.org/49434014:25
*** slaweq has joined #openstack-infra14:25
*** rbrndt has joined #openstack-infra14:26
*** florianf has quit IRC14:26
*** Hal has joined #openstack-infra14:27
*** Hal is now known as Guest3040614:27
*** _Cyclone_ has joined #openstack-infra14:29
*** makowals has joined #openstack-infra14:29
*** funzo has joined #openstack-infra14:29
*** ramishra has quit IRC14:31
*** Guest30406 has quit IRC14:31
*** danpawlik is now known as danpawlik_14:33
*** danpawlik_ is now known as dpawlik_14:33
*** funzo has quit IRC14:33
*** hongbin has joined #openstack-infra14:36
*** ihrachys has joined #openstack-infra14:40
*** florianf has joined #openstack-infra14:41
*** ltomasbo has joined #openstack-infra14:45
*** ltomasbo has quit IRC14:46
*** efoley has quit IRC14:48
openstackgerritGage Hugo proposed openstack-infra/project-config master: Skip integration/non-doc jobs in certain cases  https://review.openstack.org/49401814:48
*** jascott1 has joined #openstack-infra14:50
*** ltomasbo has joined #openstack-infra14:50
*** nicolasbock_ has quit IRC14:51
*** nicolasbock has joined #openstack-infra14:51
openstackgerritGage Hugo proposed openstack-infra/project-config master: Skip non-doc jobs in certain cases  https://review.openstack.org/49402714:51
*** gcb has quit IRC14:53
*** jascott1 has quit IRC14:54
*** xarses has joined #openstack-infra14:56
*** links has quit IRC14:56
*** ltomasbo has quit IRC14:56
*** isaacb has quit IRC14:57
*** yamamoto has joined #openstack-infra14:59
*** rlandy is now known as rlandy|mtg15:00
*** iyamahat has joined #openstack-infra15:00
*** yamahata has joined #openstack-infra15:00
*** signed8bit has joined #openstack-infra15:00
*** ltomasbo has joined #openstack-infra15:02
*** yamamoto has quit IRC15:05
*** ltomasbo has quit IRC15:05
*** eranrom_ has joined #openstack-infra15:07
*** ltomasbo has joined #openstack-infra15:07
*** eranrom has quit IRC15:07
*** Guest180 is now known as zeus15:11
*** zeus has joined #openstack-infra15:11
*** isaacb has joined #openstack-infra15:13
*** LindaWang has quit IRC15:15
*** rcernin has quit IRC15:15
*** ltomasbo has quit IRC15:15
openstackgerritPaul Belanger proposed openstack-infra/elastic-recheck master: Update queries/1708704.yaml to include non-voting  https://review.openstack.org/49590215:18
pabelangerclarkb: ^ updating our yum client failures to also include non-voting. With upgraded xenial mirror and dns fixes in place, things look really good.15:19
*** ltomasbo has joined #openstack-infra15:19
pabelangerclarkb: we still have the 502 Bad Gateway errors, hopefully I can look into that more in the coming days15:19
*** pcaruana has quit IRC15:22
*** nicolasbock has quit IRC15:23
*** sree has quit IRC15:25
*** slaweq has quit IRC15:28
*** dprince has quit IRC15:43
sshnaidmclarkb, fungi hi, an update about logs issues in tripleo15:45
sshnaidmclarkb, fungi we started using whitelist in configs collection: https://review.openstack.org/#/c/493973/15:46
sshnaidmclarkb, fungi so it's exactly you asked us.15:46
sshnaidmpabelanger, do you have a code that set up docker proxy in infra?15:46
*** baoli has joined #openstack-infra15:46
*** e0ne has quit IRC15:48
pabelangersshnaidm: yes, everything will be in openstack-infra/system-config part of our mirrors. http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n198 is the specific apache configuration15:50
sshnaidmpabelanger, so you cache docker images with apache?15:51
*** baoli has quit IRC15:51
pabelangersshnaidm: yes15:51
*** slaweq has joined #openstack-infra15:53
*** gixxer1k has quit IRC15:56
*** slaweq has quit IRC15:57
*** Apoorva has joined #openstack-infra15:57
*** nicolasbock has joined #openstack-infra16:02
*** markvoelker has joined #openstack-infra16:03
*** isaacb has quit IRC16:08
*** jascott1 has joined #openstack-infra16:09
*** dizquierdo has joined #openstack-infra16:09
*** egonzalez has quit IRC16:13
*** dprince has joined #openstack-infra16:14
*** dhajare has joined #openstack-infra16:18
*** david-lyle has quit IRC16:18
*** Hal has joined #openstack-infra16:18
*** david-lyle has joined #openstack-infra16:19
*** Hal is now known as Guest9848116:19
*** dizquierdo has quit IRC16:20
*** Guest98481 has quit IRC16:22
*** mattmceuen has quit IRC16:23
*** cody-somerville has joined #openstack-infra16:25
*** cody-somerville has quit IRC16:25
*** cody-somerville has joined #openstack-infra16:25
*** lucasagomes is now known as lucas-afk16:27
*** jascott1 has quit IRC16:28
openstackgerritFeilong Wang proposed openstack-infra/project-config master: Use python-jobs for Distil  https://review.openstack.org/48908216:29
*** funzo has joined #openstack-infra16:30
*** rlandy|mtg is now known as rlandy16:32
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Document execution_wrapper setting.  https://review.openstack.org/49544016:32
*** smarcet has joined #openstack-infra16:34
*** funzo has quit IRC16:35
smarcetfungi: afternoon16:35
smarcetseems that openstackid.org is not responding16:35
smarcettried to ssh but no dice16:35
*** signed8bit is now known as signed8bit_Zzz16:36
*** signed8bit_Zzz is now known as signed8bit16:42
*** filler has quit IRC16:43
*** baoli has joined #openstack-infra16:46
*** filler has joined #openstack-infra16:46
*** jpich has quit IRC16:48
*** baoli has quit IRC16:50
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Use artifact publication roles from zuul-jobs  https://review.openstack.org/49423116:52
*** gixxer1k has joined #openstack-infra16:52
*** links has joined #openstack-infra16:53
*** jamesmcarthur has joined #openstack-infra16:54
*** smarcet has quit IRC16:57
*** castulo has left #openstack-infra16:58
jamesmcarthurfungi: I think Sebastian already reached out to you, but openstackid.org is down.16:58
jamesmcarthurfungi: any chance someone there could have a look?16:59
*** derekh has quit IRC17:00
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Split envvar processing into its own script  https://review.openstack.org/49592717:03
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Make bootstrap copying a playbook  https://review.openstack.org/49592817:03
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Move logdir relocation to a playbook  https://review.openstack.org/49592917:03
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Move DEVSTACK_GATE_SETTINGS into process-env  https://review.openstack.org/49593017:03
*** yamamoto has joined #openstack-infra17:03
mordredjamesmcarthur: I agree - I cannot ssh to it - looking (althouhg I'm on an airplane so I may be slightly flakey)17:05
*** rcernin has joined #openstack-infra17:05
jamesmcarthurmordred: tahnk you! yeah, we're totally locked out.17:05
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Add TODO about improving key security  https://review.openstack.org/49593117:05
*** links has quit IRC17:06
*** shardy has quit IRC17:07
mordredjamesmcarthur, infra-root: there are console messages about task xenwatch being blocked17:08
*** yamamoto has quit IRC17:09
mordredI don't see anything else that looks out of sorts ...17:10
mordredinfra-root: anybody have additional thoughts other than "trigger a reboot"?17:10
jeblairmordred: is the console responsive? ie, can you hit enter and get a login prompt?17:12
*** baoli has joined #openstack-infra17:14
mordredjeblair: does not appear to be, no17:14
*** florianf has quit IRC17:14
jeblairmordred: that's all i got.  reboot sounds good17:15
mordredkk17:16
mordredrebooting openstackid.org17:17
jamesmcarthurmordred: I support reboot as well. Thank you. We can do a review of the logs after it's back up.17:17
*** Hal has joined #openstack-infra17:17
*** Hal is now known as Guest7021717:17
*** markvoelker has quit IRC17:18
*** baoli has quit IRC17:18
mordredjamesmcarthur: back up17:20
*** jamesmcarthur has quit IRC17:24
*** jamesmcarthur has joined #openstack-infra17:25
*** sambetts is now known as sambetts|afk17:26
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Increase the flavor size of the baremetal and undercloud nodes  https://review.openstack.org/49593517:27
*** yamahata has quit IRC17:28
*** jpena is now known as jpena|off17:28
*** iyamahat has quit IRC17:28
*** tosky has quit IRC17:28
*** harlowja has joined #openstack-infra17:29
*** jamesmcarthur has quit IRC17:29
*** trown is now known as trown|lunch17:31
*** ociuhandu has quit IRC17:35
*** Guest70217 has quit IRC17:36
*** Swami has joined #openstack-infra17:38
*** iyamahat has joined #openstack-infra17:40
*** signed8bit is now known as signed8bit_Zzz17:41
*** amoralej is now known as amoralej|off17:45
*** bogdando has quit IRC17:49
*** markvoelker has joined #openstack-infra17:50
*** yamahata has joined #openstack-infra17:58
*** dtantsur is now known as dtantsur|afk18:01
*** ociuhandu has joined #openstack-infra18:01
*** claudiub|3 has quit IRC18:03
*** rbrndt has quit IRC18:03
*** slaweq has joined #openstack-infra18:04
*** yamamoto has joined #openstack-infra18:05
*** slaweq has quit IRC18:08
*** yamamoto has quit IRC18:11
*** csomerville has joined #openstack-infra18:11
*** tosky has joined #openstack-infra18:12
*** electrofelix has quit IRC18:13
*** cody-somerville has quit IRC18:15
*** funzo has joined #openstack-infra18:16
*** rlandy is now known as rlandy|biab18:16
*** funzo has quit IRC18:20
mnaseralright18:23
mnaserwho broke launchpad18:23
*** jamesmcarthur has joined #openstack-infra18:24
jamesmcarthurmordred: thank you so much!18:25
*** Hal has joined #openstack-infra18:26
*** Hal is now known as Guest3966718:26
*** dave-mccowan has quit IRC18:30
*** trown|lunch is now known as trown18:30
*** dhajare_ has joined #openstack-infra18:30
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Add support for updating initramfs images in CI  https://review.openstack.org/47990018:32
*** xyang1 has joined #openstack-infra18:32
*** dhajare has quit IRC18:32
*** csomerville has quit IRC18:36
*** eranrom_ has quit IRC18:39
*** eranrom has joined #openstack-infra18:40
*** iyamahat has quit IRC18:41
*** iyamahat has joined #openstack-infra18:41
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add publish-openstack-python-branch-tarball job  https://review.openstack.org/49467218:42
openstackgerritMerged openstack-infra/zuul-jobs master: Add fetch-stestr-output role  https://review.openstack.org/49155918:45
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove publish-openstack-python-branch-tarball job  https://review.openstack.org/49467018:46
*** rbrndt has joined #openstack-infra18:47
*** dave-mccowan has joined #openstack-infra18:53
*** rlandy|biab is now known as rlandy18:54
*** jascott1 has joined #openstack-infra18:54
*** patriciadomin has quit IRC18:55
*** signed8bit_Zzz is now known as signed8bit18:56
*** signed8bit is now known as signed8bit_Zzz18:57
*** signed8bit_Zzz is now known as signed8bit18:57
*** slaweq has joined #openstack-infra18:59
*** claudiub|3 has joined #openstack-infra19:02
*** slaweq has quit IRC19:03
*** dhajare_ has quit IRC19:08
*** Guest39667 has quit IRC19:10
*** phenixrizen has quit IRC19:13
*** gixxer1k has quit IRC19:14
*** gixxer1k has joined #openstack-infra19:14
*** funzo has joined #openstack-infra19:16
*** gixxer1k has quit IRC19:17
*** gixxer1k has joined #openstack-infra19:18
*** marst has joined #openstack-infra19:19
*** gixxer1k has quit IRC19:21
*** funzo has quit IRC19:21
*** gixxer1k has joined #openstack-infra19:21
*** jtomasek has quit IRC19:21
*** gixxer1k has quit IRC19:24
*** gixxer1k has joined #openstack-infra19:25
*** gixxer1k has quit IRC19:28
*** gixxer1k has joined #openstack-infra19:28
*** gixxer1k has quit IRC19:31
*** jtomasek has joined #openstack-infra19:31
*** gixxer1k has joined #openstack-infra19:32
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add wrapper driver execution context  https://review.openstack.org/49543919:35
*** Hal has joined #openstack-infra19:36
*** Hal is now known as Guest6644919:37
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Write secrets to tmpfs  https://review.openstack.org/49544919:37
*** wolverineav has quit IRC19:37
*** wolverineav has joined #openstack-infra19:38
*** wolverineav has quit IRC19:39
*** wolverin_ has joined #openstack-infra19:39
*** jamesmcarthur has quit IRC19:39
*** jamesmcarthur has joined #openstack-infra19:39
*** slaweq has joined #openstack-infra19:40
*** wolverin_ has quit IRC19:43
*** mrhillsman has quit IRC19:43
*** wolverineav has joined #openstack-infra19:43
*** seongsoocho has quit IRC19:43
*** csomerville has joined #openstack-infra19:43
*** claudiub has joined #openstack-infra19:44
*** Guest66449 has quit IRC19:44
*** zhenguo has quit IRC19:44
*** claudiub|3 has quit IRC19:46
*** seongsoocho has joined #openstack-infra19:47
*** mrhillsman has joined #openstack-infra19:47
*** zhenguo has joined #openstack-infra19:47
*** wolverineav has quit IRC19:48
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add containers periodic test  https://review.openstack.org/47574719:48
openstackgerritJohn Trowbridge proposed openstack-infra/project-config master: Revert "Revert "Change job type for 3nodes job to move to oooq runner""  https://review.openstack.org/49585319:49
openstackgerritDavid Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section  https://review.openstack.org/49597119:54
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: WIP: Add upload-twine role  https://review.openstack.org/49597219:54
*** simondodsley has joined #openstack-infra19:56
openstackgerritMerged openstack-infra/puppet-zuul master: Remove zuul-executor pid file on stop  https://review.openstack.org/49466019:57
*** jtomasek has quit IRC19:58
*** jamesmcarthur has quit IRC20:02
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Use upload-twine role for testpypi playbook  https://review.openstack.org/49597320:03
*** jamesmcarthur has joined #openstack-infra20:03
*** gouthamr has joined #openstack-infra20:03
*** yamamoto has joined #openstack-infra20:04
*** vhosakot has joined #openstack-infra20:05
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Use upload-twine role for testpypi playbook  https://review.openstack.org/49597320:07
pabelangerjeblair: mordred: ^steps needed to publish wheels / tarballs to testpypi. Missing gpg keys still20:08
*** yamamoto has quit IRC20:11
*** slaweq has quit IRC20:15
*** slaweq has joined #openstack-infra20:15
clarkbpabelanger: are the remaining hits for the yum no more mirrors to try due to 502s on the proxy?20:15
pabelangerclarkb: a few today were still using old glean release20:16
clarkbah ok20:17
pabelangerthe few in infracloud-vanilla have been 502 bad gateway20:17
pabelangerclarkb: there are also kolla and ansible jobs still not using our mirrors20:17
clarkbgotcha20:18
openstackgerritDavid Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section  https://review.openstack.org/49597120:18
*** jcoufal has quit IRC20:19
openstackgerritDavid Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section  https://review.openstack.org/49597120:19
openstackgerritDavid Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section  https://review.openstack.org/49597120:20
openstackgerritDavid Shrewsbury proposed openstack-infra/infra-manual master: Add Zuul v3 quickstart section  https://review.openstack.org/49597120:20
Shrewsstupid fingers faster than my brain20:21
*** slaweq_ has joined #openstack-infra20:21
*** slaweq has quit IRC20:21
*** kgiusti has left #openstack-infra20:23
*** Hal has joined #openstack-infra20:23
*** Hal is now known as Guest6355420:24
*** armax has quit IRC20:25
slaweq_hello20:27
slaweq_I'm trying to setup shade repository on my local laptop to push some patches to review but I had issue with connection to gerrit:20:27
slaweq_Received disconnect from 104.130.246.91 port 29418:2: Too many authentication failures20:27
clarkbslaweq_: the two things to check on that are your username and ssh key20:28
slaweq_do You maybe know whith who I should talk to check if I'm blocked on review.openstack.org?20:28
*** slaweq has joined #openstack-infra20:28
clarkbI don't think you are blocked its just letting you know it failed because authentication failed (but we can check that your ccount isn't disabled too, can you give us the username or account id number?)20:28
slaweq_clarkb: I'm pretty sure that I'm using proper name and key because same key works for me from different host20:29
slaweq_my gerrit username is "slaweq"20:29
slaweq_I have configured remote: gerritssh://slaweq@review.openstack.org:29418/openstack-infra/shade.git20:29
slaweq_and it's working fine on server with Linux20:30
clarkbslaweq_: I've double check that account and it is active20:30
slaweq_clarkb: ok, thx a lot20:31
slaweq_so I will look on my side then20:31
clarkbslaweq_: can you try `ssh -p 29418 -vvv slaweq@review.openstack.org gerrit ls-projects` ?20:31
clarkbthat should give a bit more verbose information on what ssh is trying and where it is failing20:31
clarkbI would definitely double check the key thoug20:32
*** Guest63554 has quit IRC20:32
*** slaweq has quit IRC20:33
slaweq_clarkb: sure, result is on https://pastebin.com/x2AmpZw420:33
clarkbslaweq_: do you have many public keys that it is trying? looks like it tried about 6 or 7 of them and then the server said that is enough20:34
clarkbslaweq_: you might try pointing it at the specific key that is expected to work20:34
*** jkilpatr has quit IRC20:35
slaweq_strange thing is that I have it already configured in my .ssh/config file20:37
slaweq_clarkb: but thanks, I will try to figure it our on my side then :)20:37
*** smarcet has joined #openstack-infra20:38
*** claudiub has quit IRC20:42
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Retrieve filtered list of hosts for a task, not all hosts  https://review.openstack.org/49556120:45
*** rcernin has quit IRC20:47
*** vhosakot has quit IRC20:48
*** rockyg has joined #openstack-infra20:50
*** gagehugo has quit IRC20:50
slaweq_clarkb: found it :) it was some issue with my ssh-agent, I killed it and all starts working fine :)20:50
*** e0ne has joined #openstack-infra20:50
e0nehi team. can anybody please help with etherpad? I can't open https://etherpad.openstack.org/p/horizon-pike-feedback  in any browser. any other links work good20:51
clarkbe0ne: I can take a look, though I'll likely need to go find my ssh key20:52
e0neclarkb: great, thanks20:52
e0neclarkb: earlier today fungi sad: "we've seen pads get corrupted in the past. an infra-root sysadmin can probably retrieve the raw text content via the etherpad api so that it can be pasted into a new pad, but the formatting and author colors will be lost in the process"20:53
e0ne^^ just FYI20:53
*** vhosakot has joined #openstack-infra20:55
*** Goneri has quit IRC20:56
*** mat128 has quit IRC20:57
*** hichihara has joined #openstack-infra20:59
clarkbok, currently trying to see if there are any logged errors for that etherpad21:00
clarkband grep is saying no21:00
*** gouthamr has quit IRC21:01
*** trown is now known as trown|outtypewww21:02
e0neclarkb: can you somehow get therpad's data?21:03
pabelangerhttps://docs.openstack.org/infra/zuul/client.html21:03
pabelangerjeblair: clarkb: do you know why that is 404 now?21:03
*** thorst has quit IRC21:04
*** gixxer1k has quit IRC21:04
clarkbe0ne: yup via the admin api see http://paste.openstack.org/show/618953/21:04
*** jamesmcarthur has quit IRC21:04
clarkbe0ne: that is the content ^ I would try pasting that into a throwaway etherpad first to see if it somehow corrupts the etherpad21:05
*** srobert has quit IRC21:05
clarkbe0ne: and if not you can give it a more permanent home at eg horizon-pike-feedback-recovered or whatever21:05
e0neclarkb:great, thanks!21:05
clarkbpabelanger: looks like https://docs.openstack.org/infra/zuul/ is fairly empty21:06
clarkbpabelanger: note the name at the bottom left, its openstack-zuul-jobs, did we have that publishing over zuul docs somehow?21:06
pabelangerYa, looks like a namespace collision21:07
*** vhosakot has quit IRC21:07
*** vhosakot has joined #openstack-infra21:10
pabelangerk, think I see the issue21:11
*** yamamoto_ has joined #openstack-infra21:12
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix documentation nits  https://review.openstack.org/49431021:13
*** ldnunes has quit IRC21:15
*** e0ne has quit IRC21:15
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Add option to force delete cinder volume  https://review.openstack.org/49470521:15
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Add option to force delete cinder volume  https://review.openstack.org/49470521:16
*** smarcet has quit IRC21:16
*** funzo has joined #openstack-infra21:17
*** yamamoto_ has quit IRC21:18
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Fix infra-docs issue with (openstack-)zuul-jobs  https://review.openstack.org/49598421:20
pabelangerclarkb: jeblair: ^zuul docs fix21:20
*** Hal has joined #openstack-infra21:21
*** Hal is now known as Guest2084821:21
*** funzo has quit IRC21:22
clarkbpabelanger: commented21:23
*** hichihara has quit IRC21:23
pabelangerclarkb: replied21:24
clarkbpabelanger: I someow merged the project section above into that section21:25
clarkbmaybe I looked at hte sun too hard21:25
pabelanger:)21:25
pabelangerwe should retire zuul-base-jobs however21:25
*** Guest20848 has quit IRC21:25
*** vhosakot has quit IRC21:26
*** vhosakot has joined #openstack-infra21:26
jeblairnot yet21:28
*** slaweq_ has quit IRC21:31
*** jamesmcarthur has joined #openstack-infra21:34
pabelangerclarkb: https://bz.apache.org/bugzilla/show_bug.cgi?id=37770 is an interesting read, I think that is the same issue we are seeing21:36
openstackbz.apache.org bug 37770 in mod_proxy "proxy: error reading status line from remote server (null)" [Blocker,Reopened] - Assigned to bugs21:36
pabelangerclarkb: it is also only buildlogs.cdn that we have the issue, so maybe we need to tweak the default proxypass settings21:38
pabelangerProxyPass "/example" "http://backend.example.com" max=20 ttl=120 retry=300 for example21:38
flwanggreetings, could I get another +2 for this patch https://review.openstack.org/#/c/489082/ ? Thanks in advance21:38
clarkbpabelanger: or the remote is inappropriately closing keepalives?21:39
clarkb(or at leaset inappapropriate as far as the apache proxy is concerned may not be according to rfc)21:39
*** Apoorva_ has joined #openstack-infra21:39
openstackgerritMerged openstack-infra/project-config master: Fix infra-docs issue with (openstack-)zuul-jobs  https://review.openstack.org/49598421:39
pabelangerclarkb: well, if I am reading this correctly, the default for keepalive for proxypass is false21:39
*** dprince has quit IRC21:40
clarkbthe default is on iirc which is why we hit it with devstack21:40
clarkband there are two layers of keepalive, that in front of the proxy and the connections behind21:41
clarkbI think we are fialing on the connections behind based on how it is logged as an error in apache21:41
pabelangerThis parameter should be used when you have a firewall between your Apache httpd and the backend server, which tends to drop inactive connections. This flag will tell the Operating System to send KEEP_ALIVE messages on inactive connections and thus prevent the firewall from dropping the connection. To enable keepalive, set this property value to On.21:41
pabelangerclarkb: ya, and I think ^ would related to that?21:42
clarkbcould be21:42
*** heroux has quit IRC21:42
*** Apoorva has quit IRC21:42
clarkbcould explain why it happens in infracloud more often too since we have noticed networking blips there and possible firewall sillyness21:43
pabelangerya21:43
*** slaweq has joined #openstack-infra21:44
pabelangerso, thinking we could add infracloud-vanilla to emergency file and manually apply21:44
pabelangersee if we get any better results21:44
*** slaweq has quit IRC21:45
pabelangeror maybe we just enable keepalives across the board21:45
clarkbpabelanger: what bit are you wanting to change? that bug seems to have a whole handful of suggested workarounds21:45
clarkb(as an aside I just tcpdumped baremetal00 and don't see mixed traffic like before so wondering if hpe fixed it and just didn't tell us anything about it)21:46
clarkbs/fixed it/fixed hub like behavior on switches)21:46
pabelangerhttps://access.redhat.com/solutions/5457921:47
pabelangerclarkb: that is the issue we have right now ^21:47
pabelangererror reading status line21:48
clarkband you want to try the setenv? I'd be ok with that. Would be nice if we didn't have to set that across the board though. May want ot move this into its own vhost at some point21:48
pabelangerclarkb: well, I was going to first enable keepalived in proxypass just for buildlogs.cnd.centos.org21:49
pabelangersince that is disabled by default21:49
clarkbare you sure its disabled?21:49
clarkbpabelanger: do you mean the tcp keepalives are disabled?21:50
pabelangerhttps://httpd.apache.org/docs/2.4/mod/mod_proxy.html in worker|balancermember sections say it is default disabled21:50
*** Julien-zte has joined #openstack-infra21:51
clarkbah ok that looks like tcp and not persistent http (though they are related) ya I think we can enable that21:52
clarkbhowever we may have to tune the linux tcp settings to get those messages sent frequently enough to avoid the firewall killing the connection21:52
*** jascott1 has quit IRC21:52
*** jkilpatr has joined #openstack-infra21:53
clarkbproxy-nokeepalive is off by default (so keepalives at that level are on by default)21:53
*** xyang1 has quit IRC21:53
clarkbwe could also try to set the ttl so old unused connections are not used anymore21:54
pabelangerclarkb: also maybe we should also include a ttl, to attempt to close them from ourside21:54
clarkb:)21:54
pabelangerya21:54
pabelangerttl=120 ?21:54
clarkbya 2minutes is probably short enough21:55
clarkbin that case I'm not sure we need keepalive at all since the connection should always be active for at least 2 minutes21:55
clarkb(and kernel default for keepalives is like 2 hours)21:55
pabelangerclarkb: do you want to apply this to all proxypass entries?21:56
clarkbya I think ttl should be safe to apply21:57
clarkbthe proxy-nokeepalive one is the one I would worry about using across the board21:57
*** jascott1 has joined #openstack-infra21:58
pabelangersame21:58
pabelangerwhat about keepalive=On?21:58
clarkbI don't think that will be necessarywith a 2 minute ttl because the kernel defaults for it will be 2 hours21:59
clarkbwhich is far longer than our 2 minutes21:59
clarkbmaybe start with ttl first and if that doesn't improve it then go to tuning keepalive21:59
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Enable ttl=120 for ProxyPass  https://review.openstack.org/49602022:00
jeblairit's zuul meeting time in #openstack-meeting-alt22:01
*** dmellado has quit IRC22:04
openstackgerritMerged openstack-infra/project-config master: Use python-jobs for Distil  https://review.openstack.org/48908222:07
openstackgerritMerged openstack-infra/project-config master: Fix condition for logs pipeline  https://review.openstack.org/49581622:10
*** yamamoto_ has joined #openstack-infra22:14
*** signed8bit is now known as signed8bit_Zzz22:17
openstackgerritMerged openstack-infra/project-config master: grafana updates for n8g-bgpvpn and n8g-bagpipe  https://review.openstack.org/49584922:19
*** yamamoto_ has quit IRC22:19
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip master: (WIP) Fix the wheel url in functional tests' mirror config  https://review.openstack.org/49602722:21
*** bobh has quit IRC22:22
flwangthanks team, after changed the jenkins job, how long should I wait to recheck to make sure it works? Thanks22:27
clarkbflwang: it should be updated within an hour after merging22:29
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Test v3 volume RBAC tests only  https://review.openstack.org/49602922:29
*** kjackal has quit IRC22:30
flwangclarkb: cool, thanks22:34
*** thorst has joined #openstack-infra22:35
*** cody-somerville has joined #openstack-infra22:35
*** signed8bit_Zzz is now known as signed8bit22:36
*** xarses_ has joined #openstack-infra22:38
*** csomerville has quit IRC22:38
*** xarses has quit IRC22:38
*** sbezverk has quit IRC22:39
*** pcrews has quit IRC22:39
*** xarses_ has quit IRC22:42
*** felipemonteiro__ has quit IRC22:43
*** xarses has joined #openstack-infra22:43
*** jamesmcarthur has quit IRC22:47
*** rockyg has quit IRC22:52
*** thorst has quit IRC22:53
openstackgerritPaul Belanger proposed openstack-infra/elastic-recheck master: Remove query 1469730  https://review.openstack.org/49603322:54
*** armax has joined #openstack-infra22:54
*** signed8bit is now known as signed8bit_Zzz22:56
*** rbrndt has quit IRC22:57
*** marst has quit IRC22:59
*** dingyichen has joined #openstack-infra23:02
*** xarses has quit IRC23:04
openstackgerritMerged openstack-infra/elastic-recheck master: Update queries/1708704.yaml to include non-voting  https://review.openstack.org/49590223:06
openstackgerritMerged openstack-infra/elastic-recheck master: Remove query 1469730  https://review.openstack.org/49603323:06
*** jamesmcarthur has joined #openstack-infra23:10
*** Apoorva_ has quit IRC23:11
*** Apoorva has joined #openstack-infra23:11
*** jamesmcarthur has quit IRC23:14
*** tosky has quit IRC23:16
*** vhosakot has quit IRC23:16
*** gildub has joined #openstack-infra23:16
*** yamamoto_ has joined #openstack-infra23:16
*** sshnaidm is now known as sshnaidm|off23:18
pabelangerclarkb: so, I think if we had the yum client try another attempt on 502 bad gateway, it would be interesting to see the results. The only way I think that can be done, is setup a mirrorlist pointing like 10 times to the same mirror23:19
clarkbya I would expect a new connection to be made at that point then succeed23:20
pabelangerya23:20
pabelangerhttps://review.openstack.org/496020/ gets us ttl=120 on all ProxyPass settings23:20
*** yamamoto_ has quit IRC23:21
clarkb+223:21
*** signed8bit_Zzz is now known as signed8bit23:25
*** signed8bit is now known as signed8bit_Zzz23:25
*** hongbin has quit IRC23:26
*** stakeda has joined #openstack-infra23:27
*** bobh has joined #openstack-infra23:28
*** pcrews has joined #openstack-infra23:31
*** thorst has joined #openstack-infra23:33
*** thorst has quit IRC23:33
openstackgerritEric Kao proposed openstack-infra/project-config master: add qos extension to all congress dsvm jobs  https://review.openstack.org/49603923:40
*** rlandy is now known as rlandy|bbl23:44
*** Mutter_ has joined #openstack-infra23:45
*** jascott1 has quit IRC23:46
openstackgerritMerged openstack-infra/project-config master: Add Python 3.6 tox jobs  https://review.openstack.org/49572823:47
*** Mutter_ has quit IRC23:49
openstackgerritMerged openstack-infra/project-config master: Add py36 devstack up/down job for Fedora 26  https://review.openstack.org/49567023:50
*** jascott1 has joined #openstack-infra23:51
*** bobh has quit IRC23:56
*** bobh has joined #openstack-infra23:59
*** thorst has joined #openstack-infra23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!