Sunday, 2016-03-27

sdakepabelanger did you end up with a mirror of centos in the gates?00:02
sdakewith this bindep thing or is that something else00:03
pabelangersdake: not yet. once we finish bindep, I'm going to start on it again00:04
sdakecool00:04
pabelangersdake: I hope to have it ready by the summit00:04
sdakeany idea when it will be ready?00:04
sdakecool00:04
sdakereading my mind again ;)00:04
*** ZZelle has quit IRC00:04
pabelangerYa, I might setup a hacking session in austin00:04
sdakeif you could let me know or le tthe ml know it would rock00:04
pabelangerya, I'll send out an email to openstack-infra ML once finished00:05
sdakemirrorss o the distro speed up our gates 35%00:05
pabelangerwheel mirror was done last week for centos7, so you can use that now too00:05
sdakewhere is it00:05
pabelangerhttp://mirror.ord.rax.openstack.org/wheel/centos-7-x86_64/ for example00:06
sdakegot any pointers or an announcement email?00:06
pabelangerbut we set it by default with pip.conf for config_mirrors.sh in nodepool00:06
sdakeso pip will just know about it then?00:07
*** thorst has joined #openstack-infra00:07
sdakethe issue is we use pip in our containers00:07
sdakei'll ssee how its done with ubuntu atm00:07
clarkbcopy thr hosts pip.conf or bind mount it00:07
pabelangerthat00:07
clarkbI think kolla already copies it fwiw00:07
sdakefor ubuntu00:07
sdakebut not for centos woudl be my guess00:08
sdakei'll read the code00:08
clarkbits not distro specific00:08
sdakehttps://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L2800:09
sdakethis then?00:09
clarkbya00:10
sdakeya then thats generic00:10
sdakewe also have per-distro setup but this is the generic setup00:10
sdakeso looks like we already are using it00:10
sdakethanks!00:11
*** thorst has quit IRC00:15
*** tphummel has quit IRC00:16
*** tphummel has joined #openstack-infra00:17
*** tphummel has quit IRC00:18
*** briancline has joined #openstack-infra00:22
sdakeclarkb looks like zuul shows the gate mostly passing now00:22
sdakeso yay ;)00:22
sdakeits not ideal but our gate scripts are not designed to be run outside of oepnstack infrastructure00:23
clarkbits more likely that docker or testr updated I think since we havent touched the devstack-centos7 images00:24
clarkbbut I still dont have a link to the fail so dunno00:24
*** jerryz has quit IRC00:25
*** Daviey has quit IRC00:25
sdakeclarkb someone mentioned the images were updated which is why i thought thatwas teh cause00:25
sdakeclarkb http://logs.openstack.org/87/297987/2/check/gate-kolla-dsvm-build-centos-binary/c957005/console.html#_2016-03-26_20_44_04_18700:27
clarkbso thats jumping out of the venv and not finding testr00:28
*** amitgandhinz has joined #openstack-infra00:28
clarkbPATH isnt being preserved?00:29
sdake-E preseves the environment when passed to sudo00:29
sdaketestr isn't installed in the images00:29
sdakeits only in the tox environment00:29
sdakeand sudo doens't knwo about that00:29
clarkbyes00:29
sdakeit was previously isntalled in the images00:30
clarkbexcoet that tox isnt a shell and may not pass envthings to sudo properly00:30
clarkbI dont knkw that it was previously installed in images00:30
clarkbdevstack-centos7 has not changed00:30
sdakewe are using dvsm00:30
sdakeis that devstack-centos?00:30
clarkbyes00:31
sdakeok00:31
clarkbyou can see the node name at the beginning of your job00:31
clarkbI suppose its possible we pulled it in transitively00:31
clarkband stopped doing so recently00:32
*** achanda has quit IRC00:32
clarkbbut devstack nodes should not have it preinstalled intentionally00:32
sdakeclarkb mordred suggested this whic hI guess worked and I thought it didn't originally00:32
sdakehttps://review.openstack.org/#/c/297945/1/tox.ini00:32
clarkbya thats what I would do00:33
clarkbthat explicitly uses the venc00:33
*** briancline has quit IRC00:36
sdakeif you toggle ci and check the gate jobs for 1 though it fails00:37
sdakepatch set one00:37
sdakehttp://logs.openstack.org/45/297945/1/check/gate-kolla-dsvm-build-centos-binary/fc68cdb/console.html#_2016-03-26_14_15_05_00600:37
sdakeno subunit module00:37
sdakeso ya that attempt definately failed00:38
sdakeopen to using mordred's solution if can get it to work00:39
sdakebut it looks like something is being damaged with the running it that way00:39
sdakeso i removed sudo00:39
sdakeentirely00:39
sdakeand just changed permissions on docker.sock to 66600:39
sdakeso anyone can write to it00:39
sdake(that is on the local node)00:40
*** amitgandhinz has quit IRC00:42
fungiis it possible we recently changed /etc/sudoers in our images to start sanitizing the environment (more)?00:42
pabelangerAh, didn't think of that00:43
pabelangerpersonally, I'd drop sudo if possible. Which sounds like what sdake did00:44
clarkbfungi: oh yes!00:44
pabelangersdake: can you not add the jenkins user to the docker group instead of chmod00:45
clarkbthe safe path thing00:45
clarkbugh00:45
fungimy work is done here00:45
fungion to cartoons and sleep00:45
clarkbmystery solved00:45
pabelangersomebody always knows00:46
sdakepabelanger yes i did that but the shell doesn't absorb it00:46
fungiand knowing's half the battle00:46
clarkbwas it kolla that requested it too?00:46
clarkbsomeone did00:46
sdakerequested which00:46
pabelangeransible team I think00:46
clarkbthe sudo safepath thing00:46
sdakewe pretty much need sudo to be wide open00:47
clarkbsecurepath?00:47
sdakewasn't kolla ;)00:47
*** pfallenop has quit IRC00:47
clarkbso that shouldnt require any perms or group changes if thats it00:50
clarkbjust use explicit path?00:50
sdakehow do you do that00:51
*** Goneri has joined #openstack-infra00:51
*** torandu has joined #openstack-infra00:51
sdakei dont like the sudo -E thing anyway00:51
sdakei prefer to get rid of it00:51
clarkbits the change you wrote thst mordred suggested00:52
clarkbya you shouldnt need -E00:52
sdakeya that didnt' work tho - it couldn't find any of its dependencies00:52
clarkbsince the path is getting reset regardless00:52
clarkbyou may need to also set pythonpath00:52
clarkbnot sure00:52
sdakei dont ike running testr with sudo tbh ;)00:53
pabelanger++00:53
sdakebut its what we implemented originally00:53
*** esker has joined #openstack-infra00:55
sdakewel thakns folks for the hep ;)00:56
sdakeI'm off to play some diablo00:56
*** jerryz has joined #openstack-infra00:58
*** briancline has joined #openstack-infra00:59
*** claudiub has joined #openstack-infra00:59
*** esker has quit IRC01:00
*** claudiub|2 has quit IRC01:02
*** clsacramento has quit IRC01:03
*** clsacramento has joined #openstack-infra01:03
*** briancline has quit IRC01:06
*** rguillebert has quit IRC01:07
*** mescanef_ has joined #openstack-infra01:07
*** armax has joined #openstack-infra01:10
*** thorst has joined #openstack-infra01:12
*** armax has quit IRC01:15
*** baoli_ has joined #openstack-infra01:18
*** mescanef has quit IRC01:18
*** mescanef_ has quit IRC01:18
*** thorst has quit IRC01:19
*** claudiub has quit IRC01:19
*** baoli has quit IRC01:19
*** briancline has joined #openstack-infra01:21
*** mescanef has joined #openstack-infra01:29
*** achanda has joined #openstack-infra01:32
*** briancline has quit IRC01:34
*** achanda has quit IRC01:37
*** esker has joined #openstack-infra01:37
*** amitgandhinz has joined #openstack-infra01:38
*** Jeffrey4l has joined #openstack-infra01:38
*** briancline has joined #openstack-infra01:39
*** esker has quit IRC01:42
*** mescanef has quit IRC01:43
*** pfallenop has joined #openstack-infra01:45
*** amitgandhinz has quit IRC01:52
*** amrith is now known as _amrith_01:53
*** baoli_ has quit IRC01:58
*** jerryz has quit IRC01:58
*** ayoung has joined #openstack-infra02:00
sdakehate ot be a bother02:04
sdakebut rax.org seems super flakey02:04
sdakeis it just me02:05
sdakeor is that infrastructure having troubles02:05
sdakeord02:05
sdakehttp://logs.openstack.org/45/297945/22/check/gate-kolla-dsvm-build-ubuntu-source/540b89d/console.html#_2016-03-27_01_05_52_57702:05
*** achanda has joined #openstack-infra02:06
sdakehttp://logs.openstack.org/45/297945/22/check/gate-kolla-dsvm-deploy-ubuntu-source/6bb735d/console.html#_2016-03-27_01_11_28_69002:09
sdakealways seems to be ubuntu on rax.ord02:09
sdakecentos seems to work fine02:09
clarkbcentos doesnt use apt02:11
*** baoli has joined #openstack-infra02:13
*** thorst has joined #openstack-infra02:17
*** esker has joined #openstack-infra02:19
*** cody-somerville has quit IRC02:19
*** moravec has joined #openstack-infra02:20
*** asettle has joined #openstack-infra02:21
*** baoli has quit IRC02:22
*** esker has quit IRC02:23
*** moravec has quit IRC02:24
*** thorst has quit IRC02:25
*** moravec has joined #openstack-infra02:25
*** mylu has quit IRC02:26
*** baoli has joined #openstack-infra02:26
*** baoli has quit IRC02:29
*** fedexo has quit IRC02:31
*** baoli has joined #openstack-infra02:31
*** cody-somerville has joined #openstack-infra02:32
*** krtaylor has quit IRC02:34
*** denisra has quit IRC02:34
*** denisra has joined #openstack-infra02:35
*** |-paul-| has quit IRC02:36
*** asettle has quit IRC02:38
*** baoli has quit IRC02:39
*** baoli has joined #openstack-infra02:40
*** baoli has quit IRC02:41
*** baoli has joined #openstack-infra02:42
*** krtaylor has joined #openstack-infra02:45
*** baoli has quit IRC02:48
*** amitgandhinz has joined #openstack-infra02:48
*** salv-orlando has joined #openstack-infra02:56
*** esker has joined #openstack-infra03:00
*** salv-orlando has quit IRC03:01
*** amitgandhinz has quit IRC03:01
*** Daisy has joined #openstack-infra03:02
*** mescanef has joined #openstack-infra03:04
*** esker has quit IRC03:05
*** Daisy has quit IRC03:05
*** mescanef has quit IRC03:13
*** mescanef has joined #openstack-infra03:13
*** achanda has quit IRC03:13
*** Douhet has joined #openstack-infra03:18
*** knikolla has quit IRC03:20
*** esker has joined #openstack-infra03:22
*** mescanef has quit IRC03:22
*** mescanef_ has joined #openstack-infra03:22
*** thorst has joined #openstack-infra03:22
*** baoli has joined #openstack-infra03:25
*** esker has quit IRC03:26
*** mescanef has joined #openstack-infra03:26
*** thorst has quit IRC03:29
*** baoli has quit IRC03:30
*** dimtruck is now known as zz_dimtruck03:31
*** mescanef has quit IRC03:37
*** esker has joined #openstack-infra03:42
*** jiggalanator has quit IRC03:47
*** netsin has joined #openstack-infra03:47
*** esker has quit IRC03:47
*** mescanef has joined #openstack-infra03:47
*** moravec has quit IRC03:49
*** ramishra has quit IRC03:49
*** ramishra has joined #openstack-infra03:51
*** mescanef has quit IRC03:54
*** moravec has joined #openstack-infra03:56
*** amitgandhinz has joined #openstack-infra03:58
*** Nikolay_St has quit IRC03:59
*** Nikolay_St has joined #openstack-infra03:59
*** salv-orlando has joined #openstack-infra04:03
*** mescanef has joined #openstack-infra04:04
*** ayoung has quit IRC04:04
*** Daisy has joined #openstack-infra04:06
*** mescanef has quit IRC04:11
*** amitgandhinz has quit IRC04:11
*** achanda has joined #openstack-infra04:14
*** salv-orlando has quit IRC04:15
*** achanda has quit IRC04:19
*** mescanef_ has joined #openstack-infra04:20
*** moravec has quit IRC04:22
*** mescanef_ has quit IRC04:23
*** Daisy has quit IRC04:23
*** esker has joined #openstack-infra04:24
*** thorst has joined #openstack-infra04:26
*** esker has quit IRC04:29
*** thorst has quit IRC04:34
*** moravec has joined #openstack-infra04:35
*** achanda has joined #openstack-infra04:37
*** mescanef has joined #openstack-infra04:41
*** oanson has joined #openstack-infra04:44
*** esker has joined #openstack-infra04:45
*** Goneri has quit IRC04:45
*** mescanef has quit IRC04:45
*** esker has quit IRC04:50
*** mescanef has joined #openstack-infra04:50
*** mylu has joined #openstack-infra04:51
*** mylu has quit IRC04:53
*** mylu has joined #openstack-infra04:53
*** mescanef has quit IRC04:57
*** moravec has quit IRC05:00
*** moravec has joined #openstack-infra05:03
*** moravec has quit IRC05:04
*** amitgandhinz has joined #openstack-infra05:08
*** mescanef has joined #openstack-infra05:11
*** mescanef has quit IRC05:18
*** amitgandhinz has quit IRC05:21
*** salv-orlando has joined #openstack-infra05:27
*** thorst has joined #openstack-infra05:31
*** Sukhdev has joined #openstack-infra05:32
*** abregman has joined #openstack-infra05:32
*** mescanef has joined #openstack-infra05:34
*** salv-orlando has quit IRC05:36
*** thorst has quit IRC05:39
*** esker has joined #openstack-infra05:47
*** esker has quit IRC05:52
*** tphummel has joined #openstack-infra05:53
*** tqtran has joined #openstack-infra06:05
*** tqtran has quit IRC06:10
*** jerryz has joined #openstack-infra06:15
*** tphummel has quit IRC06:18
*** amitgandhinz has joined #openstack-infra06:18
*** mescanef has quit IRC06:24
*** mescanef has joined #openstack-infra06:24
*** amitgandhinz has quit IRC06:31
*** infra-red has joined #openstack-infra06:34
*** thorst has joined #openstack-infra06:37
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/28884906:42
*** hieulq has joined #openstack-infra06:42
*** Daisy has joined #openstack-infra06:43
*** thorst has quit IRC06:44
*** Daisy has quit IRC06:51
*** Daisy has joined #openstack-infra06:52
*** Daisy has quit IRC06:54
*** Daisy has joined #openstack-infra06:54
*** nmagnezi has joined #openstack-infra06:57
*** Daisy has quit IRC06:57
*** Daisy has joined #openstack-infra06:58
*** Sukhdev has quit IRC06:58
*** Sukhdev has joined #openstack-infra06:58
*** salv-orlando has joined #openstack-infra07:03
*** Daisy has quit IRC07:07
*** mylu has quit IRC07:08
*** Daisy has joined #openstack-infra07:08
*** salv-orlando has quit IRC07:10
*** Daisy has quit IRC07:12
*** salv-orlando has joined #openstack-infra07:18
*** salv-orlando has quit IRC07:22
*** jerryz has quit IRC07:25
*** amitgandhinz has joined #openstack-infra07:28
*** esker has joined #openstack-infra07:30
*** Sukhdev has quit IRC07:32
*** esker has quit IRC07:34
*** asettle has joined #openstack-infra07:37
*** Sukhdev has joined #openstack-infra07:38
*** hieulq has quit IRC07:39
*** amitgandhinz has quit IRC07:41
*** asettle has quit IRC07:42
*** thorst has joined #openstack-infra07:44
*** yamamoto has joined #openstack-infra07:46
*** thorst has quit IRC07:50
*** denisra has quit IRC07:50
*** Cormite has joined #openstack-infra07:59
*** sshnaidm has quit IRC08:01
*** sshnaidm has joined #openstack-infra08:01
*** yamamoto has quit IRC08:02
*** mescanef has quit IRC08:02
*** yamamoto has joined #openstack-infra08:05
*** esker has joined #openstack-infra08:11
*** claudiub has joined #openstack-infra08:12
*** yamamoto has quit IRC08:13
*** claudiub|2 has joined #openstack-infra08:15
*** esker has quit IRC08:16
*** claudiub has quit IRC08:17
*** salv-orlando has joined #openstack-infra08:22
*** achanda has quit IRC08:23
*** achanda has joined #openstack-infra08:27
*** sdake_ has joined #openstack-infra08:28
*** achanda has quit IRC08:30
*** sdake has quit IRC08:31
*** salv-orlando has quit IRC08:31
AJaegerI just noticed failures to connect to git.openstack.org, last two were to rax-ord, one example http://logs.openstack.org/08/298008/1/check/gate-openstack-manuals-tox-checkniceness/984dac8/console.html08:31
AJaegerand http://logs.openstack.org/08/298008/1/gate/gate-openstack-manuals-tox-doc-publish-checkbuild/28eab5e/console.html08:31
*** achanda has joined #openstack-infra08:32
AJaegerinfra-root, is rax-ord broken?08:32
*** hieulq has joined #openstack-infra08:36
*** amitgandhinz has joined #openstack-infra08:38
*** achanda has quit IRC08:41
*** hieulq has quit IRC08:44
*** thorst has joined #openstack-infra08:47
*** amitgandhinz has quit IRC08:51
*** thorst has quit IRC08:54
*** mescanef has joined #openstack-infra09:04
*** salv-orlando has joined #openstack-infra09:04
*** mescanef has quit IRC09:06
*** salv-orlando has quit IRC09:12
*** yamamoto has joined #openstack-infra09:13
*** esker has joined #openstack-infra09:14
*** esker has quit IRC09:19
*** yamamoto has quit IRC09:21
*** yamamoto has joined #openstack-infra09:35
*** matrohon has joined #openstack-infra09:41
*** achanda has joined #openstack-infra09:41
*** maishsk has joined #openstack-infra09:42
*** achanda has quit IRC09:46
*** amitgandhinz has joined #openstack-infra09:48
*** akaszuba has joined #openstack-infra09:49
*** thorst has joined #openstack-infra09:53
*** thorst has quit IRC10:00
*** amitgandhinz has quit IRC10:01
*** yamamoto has quit IRC10:02
*** oanson has quit IRC10:03
*** darvon has quit IRC10:04
*** darvon has joined #openstack-infra10:07
*** tqtran has joined #openstack-infra10:07
*** yamamoto has joined #openstack-infra10:10
*** tqtran has quit IRC10:12
*** tlbr has quit IRC10:13
*** e0ne has joined #openstack-infra10:21
*** asettle has joined #openstack-infra10:22
*** kzaitsev_mb has joined #openstack-infra10:23
*** e0ne has quit IRC10:24
*** asettle has quit IRC10:27
*** ociuhandu has joined #openstack-infra10:27
*** tlbr has joined #openstack-infra10:27
*** ociuhandu has quit IRC10:33
*** Sukhdev has quit IRC10:36
*** yamamoto has quit IRC10:43
*** salv-orlando has joined #openstack-infra10:44
*** ociuhandu has joined #openstack-infra10:44
*** _amrith_ is now known as amrith10:46
*** SergK has quit IRC10:46
*** dkaigarodsev has quit IRC10:46
*** salv-orlando has quit IRC10:48
*** e0ne has joined #openstack-infra10:49
*** e0ne has quit IRC10:50
*** SergK has joined #openstack-infra10:50
*** dkaigarodsev has joined #openstack-infra10:52
*** matrohon has quit IRC10:54
*** thorst has joined #openstack-infra10:57
*** amitgandhinz has joined #openstack-infra10:57
*** rguillebert has joined #openstack-infra11:01
*** thorst has quit IRC11:04
*** amitgandhinz has quit IRC11:11
abregmanhello11:11
*** e0ne has joined #openstack-infra11:13
*** achanda has joined #openstack-infra11:14
*** HeOS has joined #openstack-infra11:15
*** yamamoto has joined #openstack-infra11:17
*** yamamoto has quit IRC11:19
*** achanda has quit IRC11:19
*** baoli has joined #openstack-infra11:22
*** masayukig has quit IRC11:22
*** infra-red has quit IRC11:22
*** asettle has joined #openstack-infra11:25
*** baoli has quit IRC11:27
*** masayukig has joined #openstack-infra11:27
*** e0ne has quit IRC11:28
*** e0ne has joined #openstack-infra11:28
*** kzaitsev_mb has quit IRC11:29
*** thorst has joined #openstack-infra11:35
*** e0ne has quit IRC11:39
*** kzaitsev_mb has joined #openstack-infra11:41
*** baoli has joined #openstack-infra11:41
*** thorst has quit IRC11:42
*** thorst has joined #openstack-infra11:42
*** baoli has quit IRC11:43
*** baoli has joined #openstack-infra11:49
*** thorst has quit IRC11:51
*** Cormite has quit IRC11:53
*** |-paul-| has joined #openstack-infra12:03
*** baoli has quit IRC12:04
*** yamamoto has joined #openstack-infra12:04
*** amitgandhinz has joined #openstack-infra12:07
*** |-paul-| has quit IRC12:19
*** amitgandhinz has quit IRC12:21
*** infra-red has joined #openstack-infra12:23
*** infra-red has quit IRC12:28
*** Nikolay_St has quit IRC12:34
*** anthonyper has quit IRC12:39
*** anthonyper has joined #openstack-infra12:41
fungiAJaeger: i don't see anything reported to https://status.rackspace.com/ for ord networking, but i know rackspace has been doing disruptive network maintenance over the past week12:42
*** asettle has quit IRC12:44
*** thorst has joined #openstack-infra12:48
*** thorst has quit IRC12:55
*** sdake has joined #openstack-infra12:56
AJaegerfungi ,thanks. Let's see whether we get more reports...12:58
*** sdake_ has quit IRC12:58
AJaegerInfra-root, one more thing: There's a project-config job since 3h in the fate - waiting for nodes for fedora23 and precise (layout check)12:59
*** akaszuba has quit IRC13:00
*** asettle has joined #openstack-infra13:00
*** asettle has quit IRC13:05
*** markvoelker has joined #openstack-infra13:11
*** markvoelker has quit IRC13:17
*** achanda has joined #openstack-infra13:17
*** amitgandhinz has joined #openstack-infra13:17
*** baoli has joined #openstack-infra13:19
*** achanda has quit IRC13:23
*** infra-red has joined #openstack-infra13:24
*** baoli has quit IRC13:24
*** infra-red has quit IRC13:30
*** amitgandhinz has quit IRC13:31
pabelangerBlue box leaked all fips again, so just cleaned it up13:35
pabelangerchecking osic while I am here13:36
*** salv-orlando has joined #openstack-infra13:37
pabelangerAJaeger: not sure why zuul didn't pick them up, have we have nodes online. I just delete the oldest precise and fedora-23 nodes to kickstart it again13:41
*** yamamoto has quit IRC13:42
pabelangerYa, something not right. We had 3 bare-precise nodes ready but our min-ready is set to 113:43
*** Nikolay_St has joined #openstack-infra13:45
*** salv-orlando has quit IRC13:48
pabelangerAJaeger: fungi: Okay, part of the issue is I didn't properly delete ready nodes for jenkins02 when I shut it down on Friday. So we have a few ready images that were mucking our demand count13:53
*** thorst has joined #openstack-infra13:53
pabelangerI should work on https://review.openstack.org/#/c/83980/ again: Delete surplus ready nodes if greater then min_ready13:54
openstackgerritMerged openstack-infra/project-config: Add new project "performa"  https://review.openstack.org/29165713:54
pabelangerI had that problem back in 2014, but never finished the patch13:54
*** yamamoto has joined #openstack-infra13:56
*** achanda has joined #openstack-infra13:57
*** thorst has quit IRC14:00
*** maishsk_ has joined #openstack-infra14:02
*** maishsk has quit IRC14:03
*** maishsk_ is now known as maishsk14:03
*** mtanino has joined #openstack-infra14:06
*** achanda has quit IRC14:08
AJaegerthanks, pabelanger14:09
*** tqtran has joined #openstack-infra14:09
pabelangerAJaeger: np14:10
pabelangerand time to make some chili. bbiab14:11
*** tqtran has quit IRC14:14
*** nadya has joined #openstack-infra14:14
fungiit's possible those nodes were assigned to a jenkins master which was having problems, and so never picked up any jobs14:21
fungiusually when i'm finding a need to delete some number of seemingly stale ready nodes, i check to see if there's a correlation to the same jenkins master14:22
fungimore often than not, there is14:24
abregmanwhat is a "blue box" exactly?14:26
*** infra-red has joined #openstack-infra14:26
*** amitgandhinz has joined #openstack-infra14:27
pabelangerabregman: http://grafana.openstack.org/dashboard/db/nodepool-blue-box14:30
pabelangerfungi: Ah, good thought.  I can check the masters and see what is going on with them14:30
abregmanpabelanger: thanks :)14:30
*** salv-orlando has joined #openstack-infra14:30
*** infra-red has quit IRC14:30
*** Nikolay_St has quit IRC14:34
fungiabregman: ibm donates one of these to us https://www.blueboxcloud.com/14:34
fungiwe're getting about 40 workers worth of test capacity out of it at present14:35
pabelangerjenkins06 looks to be at 3,574 threads14:35
*** salv-orlando has quit IRC14:35
pabelangerall others appear normal14:35
fungiany idea if that's where the "problem" nodes were assigned?14:37
pabelangeractually, jenkins05 doesn't look happy either.14:37
pabelangerI can check backscrool14:37
pabelangerscroll*14:37
*** nadya has quit IRC14:38
pabelangerlooks like jenkins05 had a few attached to it14:38
pabelangerand nodepool is currently not using it for some reason14:39
pabelangersorry, that is not correct. 1 slave is attached14:40
*** kzaitsev_mb has quit IRC14:40
*** amitgandhinz has quit IRC14:41
*** kzaitsev_mb has joined #openstack-infra14:41
*** kashyap has quit IRC14:41
pabelangerOdd, jenkins02 appears to be in shutdown mode14:41
fungithat can happen if the automated restart over the weekend is underway (or if it failed miserably)14:42
fungigiven that it has no jobs running, that's probably the latter14:42
pabelangershould I continue with the restart of it?14:42
fungialso it could have accounted for ready nodes not picking up any jobs, if it started entering shutdown while nodepool was still in the process of building new nodes targeting it14:42
fungiyeah, delete remaining slaves in the webui (since there aren't many) and stop it completely, make sure there are no java processes still running on the server, then start jenkins again14:43
pabelangerfungi: okay, coming back online14:46
fungithanks!14:47
*** piet has joined #openstack-infra14:48
pabelangerGoing to do jenkins05 too, as I think we are getting 500 Server error back from it14:49
pabelangerOh14:54
pabelangerjenkins05 is out of HDD14:55
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata  https://review.openstack.org/29795014:56
*** maishsk has quit IRC14:58
*** baoli has joined #openstack-infra14:58
*** thorst has joined #openstack-infra14:59
*** baoli has quit IRC15:03
pabelangerfungi: so, logging is pretty insane on jenkins05 ATM15:03
pabelanger15gb jenkins.log file15:04
pabelangerwith a 40GB hdd15:04
*** thorst has quit IRC15:05
*** achanda has joined #openstack-infra15:09
pabelangerand jenkins03 is full too15:09
pabelangercleaning it up15:09
pabelangerguess we need to get logrotate working here15:09
*** piet has quit IRC15:13
*** sc68cal has joined #openstack-infra15:14
*** achanda has quit IRC15:14
amrithanybody having issue connecting to gerrit? I get a screen about errors in openid auth15:14
amrith"Provider is not supported, or was incorrectly entered."15:15
pabelangeramrith: I see that too15:16
pabelangerlet me check logs15:16
amrithpabelanger, thanks15:16
amrithno hurry on my part, I'll just go back to drinking beer or something higher priority like that :)15:17
pabelangeramrith: I think launchpad.net might be down15:17
*** infra-red has joined #openstack-infra15:17
pabelangerwell, the SSO from launchpad.net15:18
pabelangerhttps://launchpad.net/+login15:18
amrithsounds good, I'll try again later. Thanks for confirming that it isn't my end that's broken.15:18
amrithhave a good weekend15:18
mordredyah. it's down again15:18
*** kzaitsev_mb has quit IRC15:19
*** Goneri has joined #openstack-infra15:25
pabelangerokay, I think everything is back to normal with the jenkins masters15:27
*** dims has quit IRC15:27
*** dims has joined #openstack-infra15:34
pabelangeremail sent to ML15:36
pabelangerand now I afk for a it15:36
pabelangerbit*'15:36
*** salv-orlando has joined #openstack-infra15:36
*** amitgandhinz has joined #openstack-infra15:37
*** nmagnezi has quit IRC15:40
*** kzaitsev_mb has joined #openstack-infra15:41
*** salv-orlando has quit IRC15:42
*** asettle has joined #openstack-infra15:47
*** mylu has joined #openstack-infra15:50
*** amitgandhinz has quit IRC15:50
*** asettle has quit IRC15:52
*** infra-re_ has joined #openstack-infra15:55
*** infra-red has quit IRC15:59
*** abregman has quit IRC15:59
*** abregman has joined #openstack-infra15:59
*** akshai has joined #openstack-infra15:59
*** baoli has joined #openstack-infra16:00
openstackgerritMerged openstack-infra/project-config: Silence distro package install a bit  https://review.openstack.org/29796116:01
*** thorst has joined #openstack-infra16:03
*** IlyaG has quit IRC16:04
*** baoli has quit IRC16:04
*** Goneri has quit IRC16:07
*** salv-orlando has joined #openstack-infra16:09
*** thorst has quit IRC16:10
*** achanda has joined #openstack-infra16:11
*** salv-orlando has quit IRC16:12
*** achanda has quit IRC16:16
*** nadya has joined #openstack-infra16:19
*** thorst has joined #openstack-infra16:20
*** infra-re_ has quit IRC16:21
*** thorst has quit IRC16:24
*** thorst has joined #openstack-infra16:25
*** kzaitsev_mb has quit IRC16:26
*** thorst has quit IRC16:29
*** lucasagomes has quit IRC16:30
*** lucasagomes has joined #openstack-infra16:35
*** jerryz has joined #openstack-infra16:38
*** kzaitsev_mb has joined #openstack-infra16:42
*** mylu has quit IRC16:44
*** mylu has joined #openstack-infra16:45
*** amitgandhinz has joined #openstack-infra16:47
krotscheckfungi: Heya- some reading material for ya. I don't think we're vulnerable, since our publish pipeline bounces through our own tarballs and doesn't execute the scripts, but it's good to keep in mind for the future: http://news.softpedia.com/news/node-js-package-manager-vulnerable-to-malicious-worm-packages-502216.shtml16:51
mordredkrotscheck: I believe that issue exists in pip, gem, rpm and deb as well16:54
krotscheckmordred: Oh yes.16:54
mordred:)16:54
krotscheckI think npm might be a bit more vulnerable though, given how fast and loose dependency management ususally is.16:55
*** sputnik13 has joined #openstack-infra16:55
*** sputnik13 has quit IRC16:55
*** Jeffrey4l has quit IRC16:56
mordredpossibly ... not so sure mr. saccone actually disclosed a thing as much as described a known feature of distributed packaging systems as it relates to the trust model between users and packagers16:56
mordredhowever, I totally agree it's quality of them that we should all be very aware of16:57
mordredand is a good article to link to when peope want us to include random repos in the gate :)16:57
*** amitgandhinz has quit IRC17:00
mordredsdake: I have deleted your branches17:02
*** IlyaG has joined #openstack-infra17:04
*** piet has joined #openstack-infra17:07
clarkbkrotscheck: where did your py34 info come from in that recent blog post? I am pretty sure it isnt quite accurate. Many projects run a basic does it import py34 test job and thats about it17:08
clarkbthey dont actually function under py3k17:08
*** nadya has quit IRC17:11
clarkbthis packaginh issue is why we build package in untrusted env then copy artifact via trusted env17:11
mordredclarkb: btw ... zomg vilanova17:12
clarkbI was too busy barbequing to watch but kansas got owned?17:14
mordredclarkb: well, not owned so much as defeated17:15
mordredclarkb: but even that I think it impressive at this point17:15
mordredtoday's ACC semi-finals amuse me17:15
*** asettle has joined #openstack-infra17:16
openstackgerritPaul Belanger proposed openstack-infra/project-config: [WIP] Add ubuntu-xenial DIB to nodepool  https://review.openstack.org/29798117:18
krotscheckclarkb: I pulled it from the tox envs.17:19
krotscheckclarkb: I'm guessing I should generate that from whether or not they have a py34 job and/or whether it's voting?17:20
*** asettle has quit IRC17:20
clarkbeven voting may not tell the whole story as many of them run a small subset of the tests17:20
clarkbits really a does it compile check17:21
clarkbusing the trove data is probably best right now17:21
AJaegerkrotscheck: thanks for sharing the npm issue. Yes, that is a broader issue. Some of that can be avoided with how you build - like in chroot or VM without networking access - but this as to be analyzed and worked around on the build systems.17:21
AJaegerkrotscheck: just have a look at tox.ini of swift, it invokes a far smaller set of tests for py34 than for testenv17:22
krotscheckclarkb: Trove data?17:23
sdakemordredthanks :)17:23
sdakehappy easter everyone17:23
clarkbkrotscheck: the package info for python packages it often tells you which versions are supported17:23
*** maishsk has joined #openstack-infra17:23
krotscheckclarkb: Ooh, good idea.17:23
krotscheckclarkb, AJaeger: I'll go ahead and update the table tomorrow morning when I'm on the clock.17:23
* krotscheck is also going to compose a step-by-step how not to be silly guide when publishing npm modules.17:24
*** thorst has joined #openstack-infra17:26
*** mylu_ has joined #openstack-infra17:26
*** mylu has quit IRC17:26
mordredkrotscheck: ++17:26
*** SumitNaiksatam has quit IRC17:28
*** piet has quit IRC17:29
*** SumitNaiksatam has joined #openstack-infra17:29
openstackgerritMonty Taylor proposed openstack-infra/shade: Go ahead and admit that we return Munch objects  https://review.openstack.org/28290817:30
*** nadya has joined #openstack-infra17:31
*** thorst has quit IRC17:33
*** esker has joined #openstack-infra17:33
*** lucasagomes has quit IRC17:34
krotscheckmordred: Random thought, maybe I should remove my publishign rights to our various NPM packages...17:38
openstackgerritMerged openstack-infra/system-config: Start namespacing ansible group_vars  https://review.openstack.org/28431817:38
mordredkrotscheck: ah. good thought17:38
mordredkrotscheck: I should do the same for the pypi packages I can publish17:39
mordredthat openstack also can publish17:39
*** lucasagomes has joined #openstack-infra17:40
openstackgerritMonty Taylor proposed openstack-infra/system-config: Remove old un-namespaced variables  https://review.openstack.org/28431917:41
*** kzaitsev_mb has quit IRC17:43
*** kzaitsev_mb has joined #openstack-infra17:45
AJaegerproject-config cores, here's an older cleanup patch of mine that could use some review love, please: https://review.openstack.org/#/c/287414/17:47
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Consume namespaced default values  https://review.openstack.org/28432017:50
*** afazekas has quit IRC17:52
*** afazekas has joined #openstack-infra17:52
*** mylu_ has quit IRC17:57
*** amitgandhinz has joined #openstack-infra17:57
AJaegerthanks, mordred !17:59
mordredAJaeger: sure nuff!18:01
openstackgerritMerged openstack-infra/project-config: Use ZUUL_REFNAME (1/2) for branch checking  https://review.openstack.org/28741418:03
EmilienMI found some timeout on our git servers18:05
EmilienMhttp://logs.openstack.org/13/297413/1/gate/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-trusty/938ad18/console.html#_2016-03-27_15_53_38_76218:05
EmilienMif you scroll up, you should notice git servers took lot of time to checkout a repo18:05
*** yamamoto has quit IRC18:05
EmilienM2016-03-27 18:05:51.742 | git.openstack.org[1: 2001:4800:7819:103:be76:4eff:fe06:63c]: errno=Network is unreachable18:06
EmilienMit's happenning right now18:06
EmilienMhttps://jenkins07.openstack.org/job/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-trusty/34/console18:06
*** markvoelker has joined #openstack-infra18:12
*** SumitNaiksatam has quit IRC18:12
AJaegerEmilienM: which cloud and region? rax-ord?18:12
AJaegerEmilienM: that information helps a lot ;)18:14
AJaegerFirst one seems vexxhost-ca-ymq, second is rax-ord again ;(18:14
*** achanda has joined #openstack-infra18:14
*** SumitNaiksatam has joined #openstack-infra18:15
*** markvoelker_ has joined #openstack-infra18:15
*** dims has quit IRC18:15
EmilienMAJaeger: ok, will know for next time ;)18:15
*** amitgandhinz has quit IRC18:15
*** markvoelker has quit IRC18:17
*** mylu has joined #openstack-infra18:18
*** mylu has quit IRC18:19
*** achanda has quit IRC18:20
*** dims has joined #openstack-infra18:22
*** alexpilotti has quit IRC18:24
*** alexpilotti has joined #openstack-infra18:24
*** IlyaG has quit IRC18:26
*** nadya has quit IRC18:28
*** thorst has joined #openstack-infra18:31
*** mylu has joined #openstack-infra18:32
*** lezbar has joined #openstack-infra18:34
*** achanda has joined #openstack-infra18:35
*** mylu has quit IRC18:36
*** thorst has quit IRC18:38
*** mylu has joined #openstack-infra18:38
*** piet has joined #openstack-infra18:42
*** mylu has quit IRC18:47
*** mylu has joined #openstack-infra18:47
*** rcarrillocruz has left #openstack-infra18:50
*** amitgandhinz has joined #openstack-infra18:50
*** kzaitsev_mb has quit IRC18:50
*** kzaitsev_mb has joined #openstack-infra18:51
*** mylu has quit IRC18:51
*** lezbar has quit IRC18:53
*** tphummel has joined #openstack-infra18:53
*** mylu has joined #openstack-infra18:55
*** mylu has quit IRC18:57
*** markvoelker has joined #openstack-infra18:58
*** markvoel_ has joined #openstack-infra18:59
*** mylu has joined #openstack-infra19:00
*** markvoelker_ has quit IRC19:00
*** mylu has quit IRC19:02
*** amitgandhinz has quit IRC19:03
*** markvoelker has quit IRC19:04
*** markvoel_ has quit IRC19:06
*** amitgandhinz has joined #openstack-infra19:06
*** yamamoto has joined #openstack-infra19:06
*** yamamoto has quit IRC19:15
*** rcarrillocruz has joined #openstack-infra19:16
*** piet has quit IRC19:17
*** sdake has quit IRC19:20
*** amitgandhinz has quit IRC19:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Silence zuul_swift_upload  https://review.openstack.org/29805919:24
AJaegerproject-config reviewers, please check that we can do this in one step ^19:24
*** fedexo has joined #openstack-infra19:25
*** IlyaG has joined #openstack-infra19:27
*** IlyaG has quit IRC19:32
*** thorst has joined #openstack-infra19:36
*** thorst has quit IRC19:43
*** cj has quit IRC19:44
*** mylu has joined #openstack-infra19:53
*** Goneri has joined #openstack-infra19:55
*** mylu has quit IRC19:56
*** mylu has joined #openstack-infra19:58
*** mylu has quit IRC19:59
*** fedexo has quit IRC20:00
*** mylu has joined #openstack-infra20:04
*** achanda has quit IRC20:06
*** mylu has quit IRC20:06
*** |-paul-| has joined #openstack-infra20:10
*** mylu has joined #openstack-infra20:11
*** mylu has quit IRC20:12
*** maishsk has quit IRC20:17
*** elo has joined #openstack-infra20:17
*** maishsk has joined #openstack-infra20:17
*** elo has quit IRC20:18
*** amitgandhinz has joined #openstack-infra20:19
*** e0ne has joined #openstack-infra20:25
*** amitgandhinz has quit IRC20:29
*** mikelk has joined #openstack-infra20:33
*** e0ne has quit IRC20:34
*** mikelk has quit IRC20:39
*** thorst has joined #openstack-infra20:41
*** thorst has quit IRC20:49
*** Goneri has quit IRC20:54
*** thorst has joined #openstack-infra20:56
*** thorst has quit IRC20:58
*** maishsk has quit IRC21:00
*** mikelk has joined #openstack-infra21:00
*** mylu has joined #openstack-infra21:03
*** mikelk has quit IRC21:04
*** mikelk has joined #openstack-infra21:04
*** mikelk has quit IRC21:06
*** Nikolay_St has joined #openstack-infra21:09
*** Nikolay_St has quit IRC21:14
pabelangerfungi: clarkb: when you have a moment, if you want to send over the syntax to restart zuul, I can handle that.  The gate is pretty clear today21:15
*** kzaitsev_mb has quit IRC21:16
*** cj has joined #openstack-infra21:17
*** imcsk8 has quit IRC21:21
*** imcsk8 has joined #openstack-infra21:21
*** amitgandhinz has joined #openstack-infra21:26
*** Nikolay_St has joined #openstack-infra21:26
*** Nikolay_St has quit IRC21:31
*** tphummel has quit IRC21:34
*** mylu has quit IRC21:35
*** mylu has joined #openstack-infra21:36
*** sdake has joined #openstack-infra21:36
*** amitgandhinz has quit IRC21:39
*** abregman has quit IRC21:39
*** mriedem has joined #openstack-infra21:41
*** mriedem_away has quit IRC21:42
*** tqtran has joined #openstack-infra21:43
*** alexpilotti has quit IRC21:45
*** tqtran has quit IRC21:46
*** jamesmcarthur has joined #openstack-infra21:48
*** _denisra has quit IRC21:52
*** thorst has joined #openstack-infra21:59
*** elo has joined #openstack-infra22:00
*** thorst has quit IRC22:08
*** salv-orlando has joined #openstack-infra22:09
*** kzaitsev_mb has joined #openstack-infra22:13
*** jamesmcarthur has quit IRC22:13
*** sdake has quit IRC22:18
*** SumitNaiksatam has quit IRC22:26
*** SumitNaiksatam has joined #openstack-infra22:26
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add simple script to help debug shade image uploads  https://review.openstack.org/28729622:31
*** amitgandhinz has joined #openstack-infra22:36
*** achanda has joined #openstack-infra22:39
*** kzaitsev_mb has quit IRC22:39
*** IlyaG has joined #openstack-infra22:43
*** IlyaG has quit IRC22:48
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory  https://review.openstack.org/10759722:48
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Add the ability to use ansible for ready scripts  https://review.openstack.org/28289522:48
*** amitgandhinz has quit IRC22:49
*** ZZelle has joined #openstack-infra22:51
*** salv-orl_ has joined #openstack-infra22:57
*** salv-orlando has quit IRC22:57
*** tphummel has joined #openstack-infra22:58
*** baoli has joined #openstack-infra23:01
*** thorst has joined #openstack-infra23:05
*** tphummel has quit IRC23:06
*** mylu has quit IRC23:08
*** tphummel has joined #openstack-infra23:09
*** mylu has joined #openstack-infra23:11
*** baoli has quit IRC23:11
*** thorst has quit IRC23:13
*** asettle has joined #openstack-infra23:19
*** baoli has joined #openstack-infra23:23
*** SumitNaiksatam has quit IRC23:23
*** SumitNaiksatam has joined #openstack-infra23:28
*** tphummel has quit IRC23:30
openstackgerritMerged openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs  https://review.openstack.org/28525823:32
*** Goneri has joined #openstack-infra23:34
*** kzaitsev_mb has joined #openstack-infra23:35
*** baoli has quit IRC23:37
*** tphummel has joined #openstack-infra23:38
*** kzaitsev_mb has quit IRC23:40
*** moravec has joined #openstack-infra23:43
*** thorst has joined #openstack-infra23:45
*** amitgandhinz has joined #openstack-infra23:46
*** thorst has quit IRC23:46
*** thorst has joined #openstack-infra23:47
*** moravec has quit IRC23:49
*** cfarquhar has quit IRC23:49
*** thorst has quit IRC23:55
*** tphummel has quit IRC23:56
*** amitgandhinz has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!