Friday, 2016-06-17

« Thursday, 2016-06-16 Index Saturday, 2016-06-18 »
mordredjeblair: I've never waited myself :)00:01
mordredjeblair: I will now approve those patches00:01
jeblairokiedokie00:01
anteayazaro: ah okay thank you00:02
anteayazaro: are you willing to create a patchset to put the its-storyboard plugin on review-dev yet?00:02
*** sarob has quit IRC00:02
*** ddieterly has joined #openstack-infra00:05
*** SumitNaiksatam has quit IRC00:05
*** piet has quit IRC00:06
zaroanteaya: working on it00:06
JayFAre apache logs stored for the gate anywhere, and I'm just missing it/00:07
anteayazaro: oh lovely, thank you00:07
ianwJayF: yes, they should be copied out00:10
openstackgerritPaul Belanger proposed openstack-infra/zuul: Enabled host_key_checking  https://review.openstack.org/33082700:10
pabelangerShrews: jeblair: mordred: I was thinking something like ^ for host_key_checking00:10
pabelangeruntested00:10
JayFianw: http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/ can you show me where? I've checked damn near everything :/00:10
pabelangerwell, I tested it locally in a playbook, untested in zuul-launcher00:11
JayFianw: looks like apache *configs* are there but not logs00:11
ianwJayF: ok ... for standard devstack-gate cleanup they *should* be there, but something might be going wrong ... looking00:12
JayFianw: honestly these failures look like some other service ninja'd use of port 8088 out from under ironic, but I can't tell with the gate logs provided :/00:12
JayFianw: http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/logs/devstacklog.txt.gz#_2016-06-16_23_14_51_07300:12
jeblairJayF: oh, yep, we do that.00:12
*** sflanigan has joined #openstack-infra00:12
ianwJayF: port 8088 ... streaming logs via netcat with zuul ... ummm00:12
JayFoh shit00:12
JayFThat's what's been breaking the ironic gate00:12
anteayaplease00:12
JayFSorry anteaya :(00:12
JayFneed to watch my language00:12
jeblairJayF: we can move it fairly easily00:13
fungiwe could probably pick a more esoteric port00:13
jeblair(what we can't do is predict what port a job is going to use)00:13
JayFEither that or Ironic will have to change the port we configure in Ironic00:13
jeblairpick a number00:13
JayF6553400:13
JayF:D00:13
fungiheh00:13
fungiwfm (and unlikely to have an ephemeral binding that early in the job)00:14
pabelangerobligatory: 2300:14
JayFSo you all are going to change the port you're using? It's just as simple for us to change the port as well00:14
mordredjeblair: what about some of the really esoteric things below 8000:14
fungipabelanger: spot on00:14
jeblairi'd like to avoid below 102400:14
mordredjeblair: but echo is 7 :)00:14
JayFhttps://bugs.launchpad.net/ironic/+bug/1590139 is the Ironic bug related in question00:14
openstackLaunchpad bug 1590139 in Ironic "gate job fails when unable to start apache service during horizon setup" [Critical,Triaged]00:14
JayFI'm going to put this information there00:14
fungithough this isn't really a telnet server00:14
jeblairsomeone who is not me pick a number :)00:14
fungijust a streaming socket00:15
mordredzomgzom00:15
mordredjeblair: netrjs-1        71/tcp                          # Remote Job Service00:15
fungiavoiding below 102400:15
JayF1988500:15
fungi65535 sounded fine00:15
JayF1 + keypad numbers for 'zuul'00:15
fungier, 6553400:15
JayFfungi: it being in the source port range is probably nonideal, but unlikely to actually matter /famous_last_words00:16
fungi3133700:16
*** sarob has joined #openstack-infra00:16
fungi;)00:16
pabelangerepic00:16
jeblairbut less esoteric00:17
*** roxanaghe has joined #openstack-infra00:17
*** amotoki has quit IRC00:17
fungimight trip turn up false positives for rootkit scanners00:17
pabelangerWhat was the issue with 8088? Breaking some jobs?00:17
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Use port 19885 for console streaming  https://review.openstack.org/33082900:18
jeblairJayF, fungi, mordred, pabelanger: ^00:18
JayFcloses-bug 1590139 if you care00:18
openstackbug 1590139 in Ironic "gate job fails when unable to start apache service during horizon setup" [Critical,Triaged] https://launchpad.net/bugs/159013900:18
JayFI'll paste that review into the bug myself though00:18
clarkbtests shoukd use port 0 imo00:18
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Use port 19885 for console streaming  https://review.openstack.org/33082900:18
jeblairclarkb: it's pretty hard to use port 0 for something and then go and configure apache to forward to it.  possible, but it's a lot of work.00:19
ianwJayF: hmm, so i can see around http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/logs/devstack-gate-cleanup-host.txt 2016-06-16 23:15:31.974 the apache logs are being copied, but don't end up in the final log output as you say00:19
clarkbjeblair: ironic is fronting something with apache?00:19
jeblair(i mean, you have to start your service, then configure proxy servers, which is backwards from the way people usually do something)00:20
JayFWe run apache to host our static ipxe rom00:20
JayFfor ipxe jobs00:20
clarkbhuh ehy nit port 80 forbthat00:20
JayFApache static serving is a dependency for Ironic00:20
JayFI don't know, this well predates me00:20
JayFI'm just troubleshooting :)00:20
clarkbugh phone typing. why not port 8000:20
jeblairJayF: "Huh, why not port 80 for that"00:20
* jeblair translates for clarkb00:20
clarkbjeblair: can read phone typing00:20
mordredor 4011 which is the pxe port00:21
*** roxanaghe has quit IRC00:21
ianwJayF: and ansible seems to report rsyncing them ... http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/_zuul_ansible/logs/ansible.log00:21
jeblairJayF: i wonder if ironic and i picked 8088 for the same reason (easy to type and remember intel nostalgia)00:22
pabelangerreview 330784 could use some love, removes jenkins from grafana00:22
*** amitgandhinz has joined #openstack-infra00:24
*** ddieterly has quit IRC00:25
JayFjeblair: mordred: seriously no idea why; if 4011 is the standard we can use that. I think there was a reason we didn't want to host it on :80, but that was a long time ago before I worked upstream this much :)00:27
jeblairi think what i would like to do to speed things up is to force-merge 330829, stop zuul, hard-stop all the launchers then restart00:27
jeblairthat will cleanup all the nodepool leaks at the same time as we switch ports00:28
*** pvaneck has quit IRC00:28
JayFif you don't mind putting a comment in the bug when it's all fixed up, I can be the recheck-fairy in the morning :)00:28
pabelangerjeblair: want me to get up the system-config change for firewalls?00:28
*** zul has joined #openstack-infra00:28
*** ddieterly has joined #openstack-infra00:28
jeblairpabelanger: oh, i forgot about those00:28
jeblairpabelanger: that's an image update isn't it00:28
pabelangerright00:28
*** amitgandhinz has quit IRC00:29
jeblairJayF: erm, it may actually be faster for you to change your port00:29
JayFjeblair: are we measuring in hours? days? weeks?00:29
jeblairJayF: i think infra may be multiple days.00:29
pabelangerif things go well, 11+hours to update images00:29
jeblairthey never go well00:29
pabelangerright00:29
jeblairplan for a week00:29
JayFlol00:29
JayFworks, I'll push up the ironic patch too00:30
pabelangeractually, a week :)00:30
clarkboh right firewall updates00:30
JayF4011 is the official proper port for serving up pxe images?00:30
*** Sukhdev has quit IRC00:30
clarkblooks like it is for proxy dhcp00:31
jeblairpabelanger: let's go ahead and put both ports in the iptables config so we can still change the port.  just slowly.00:31
pabelangerjeblair: ack00:31
*** isaacb has quit IRC00:32
JayFnetboot-pxe     3928/tcp    # PXE NetBoot Manager00:32
JayFgoing for that.00:33
openstackgerritPaul Belanger proposed openstack-infra/system-config: Use port 19885 for console streaming for zuul workers  https://review.openstack.org/33083100:34
*** lvdongbing has joined #openstack-infra00:38
JayFthe Ironic side fix for that port overlap: Jay Faulkner proposed openstack/ironic: Change port used for Ironic static http to 3928  https://review.openstack.org/33083200:39
pabelangerAn interesting argument could be made that zuul-launcher could check to see if the configured port for console streaming is open, and either give a warning or actually open it.00:39
jeblairpabelanger: it opens it before the job starts00:40
pabelangerjeblair: sorry, meaning the firewall port too00:40
jeblairpabelanger: ah, that's pretty site-specific00:40
*** bpokorny has quit IRC00:41
pabelangerya, I figure we're going to walk the line of bootstrapping the worker with DIB or ansible moving forward.00:41
*** ddieterly has quit IRC00:42
*** lvdongbing has quit IRC00:43
ianwjeblair: ok, maybe i'm going bananas here.  here's the rsync copy for the logs cleaned up -> http://paste.openstack.org/show/516752/00:43
ianwjeblair: the resulting log dir -> http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/logs/00:44
*** lvdongbing has joined #openstack-infra00:44
ianwjeblair: what happened to "apache/*" ?00:44
jeblairianw: what job?00:44
jeblairianw: oh you just linked it :)00:44
JayFianw: I suspect it's also possible that apache never starts at all00:44
JayFianw: there probably should be error logs, but given the port conflicts wouldn't surprise me if it's just 100% empty00:45
ianwJayF: but i see the rsync copying files?00:45
JayFJust offering up other alternatives to being bananas :)00:45
jeblairianw: is that in both rsyncs?00:46
openstackgerritSampath Priyankara proposed openstack-infra/project-config: Request to add new project Masakari  https://review.openstack.org/33037000:46
*** tphummel has quit IRC00:46
ianwjeblair: checking http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/_zuul_ansible/logs/ansible.log00:47
jeblairdrwxr-x---  2 jenkins jenkins   4096 Jun 16 23:15 apache00:47
jeblairianw: ^00:47
ianwi wonder if we can get those "\n" to actually be a newline00:47
*** tphummel has joined #openstack-infra00:47
jeblairthat's on the log server00:47
fungiheh, we preserve perms?00:47
jeblairwe are faithfully copying the permissions00:47
funginiiiiiice00:47
ianwahhh, so i guess devstack-gate should fix this in the cleanup00:48
ianwi can do that00:48
JayFThanks for the help! Was not expecting to get both things fixed and explained in a matter of minutes00:48
fungiwell, i'd argue that our publishers shouldn't publish files we can't... publish00:48
JayFA++ would reccomend openstack infra to a friend00:48
jeblairianw: i think that's probably best... arguably we could have zuul-launcher not preserve perms, but requires more thought than the d-g change (and d-g *intends* to fix that stuff up anyway, so is a good one)00:49
*** thorst_ has joined #openstack-infra00:49
ianwjeblair / fungi : yeah, d-g is trying to get these files available, so it seems to make sense even not in a zuul-launcher context00:50
openstackgerritPaul Belanger proposed openstack-infra/zuul: Enabled host_key_checking  https://review.openstack.org/33082700:50
*** ddieterly has joined #openstack-infra00:51
fungiianw: agreed. i think we have it possibly wrong in two places, but fixing one is enough and that one is easier00:51
*** xarses_ has joined #openstack-infra00:52
fungii mean, maybe we _do_ want the flexibility of uploading files from a job and having them not be world-readable, but i'm not coming up with one00:52
ianwahh, we already do a chmod, just not enough -> https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n96900:53
pabelangerfungi: I can see that being usefully for somebody.00:54
openstackgerritDerek Higgins proposed openstack-infra/system-config: Add rh2 cloud credentials  https://review.openstack.org/33083300:54
*** thorst_ has quit IRC00:54
openstackgerritMerged openstack-infra/system-config: Re-enable in-progress job url links  https://review.openstack.org/33081500:54
pabelangerand think fixing devstack to setup the permissions properly is likely the right first00:54
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2"  https://review.openstack.org/33051400:54
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2  https://review.openstack.org/33051500:54
*** thorst_ has joined #openstack-infra00:55
* pabelanger quickly refreshes status.o.o/zuul waiting for 330815 to go live00:55
* jeblair kicks static.openstack.org00:55
pabelanger\o/00:56
jrollthat ironic port thing was an amazing bug, thanks friends for the help there :)00:57
jrollwas wondering why frequency just kept increasing00:57
jeblairjroll: sorry about that!  thanks for saying "8088" :)00:57
*** sdake has joined #openstack-infra00:58
fungiwe should all say "8088" more often00:58
jrollheh00:58
jeblairoh, i didn't update system-config before kicking.  oh well, i'll just wait for the cron.00:58
*** thorst_ has quit IRC00:59
*** zhurong has joined #openstack-infra01:00
JayFclarkb: https://bugs.launchpad.net/ironic/+bug/1593510 filed to make our stuff play nicely on port :80 with the other pieces.01:01
openstackLaunchpad bug 1593510 in Ironic "[RFE] Host ipxe static assets in devstack on :80" [Undecided,New]01:01
*** tkelsey has joined #openstack-infra01:01
*** Apoorva has quit IRC01:01
*** linuxmaniac has quit IRC01:04
jeblairsome non-afs servers have slipped into the afs ansible playbook01:05
jeblairprobably a grouping error01:05
*** samP has joined #openstack-infra01:05
*** tkelsey has quit IRC01:05
*** linuxmaniac has joined #openstack-infra01:06
nibalizerjeblair: which ones?01:07
jeblairnibalizer: some afs clients -- mirror-update and release.slave01:08
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Set read/search perms recursively  https://review.openstack.org/33083501:08
jeblair(the afs playbook was for afs servers so we can serialize them so we don't lose quorum)01:08
ianwjeblair / fungi : ^ should do it.  TIL about "X" (as opposed to "x")01:08
nibalizerjeblair: hrm http://paste.openstack.org/show/516755/01:08
JayFianw: that is amazing01:09
JayFianw: X, I mean (vs x, I just googled it)01:09
openstackgerritMerged openstack/diskimage-builder: Ironic agent kernel should be owned by user building image  https://review.openstack.org/33078301:09
jeblairnibalizer: huh.  that looks like the intent.01:09
ianwJayF: it's often amazing what you find when you actually read the man pages01:09
nibalizerjeblair: what are you seeing/looking at?01:09
jeblairnibalizer: puppet run all log01:10
JayFianw: thanks for that fix too, I appreciate you digging into it01:10
JayFleaving for the day, ty all for the help getting these things fixed up o/01:10
jeblairJayF: np, thank you!01:10
pabelangerjeblair: nibalizer: I see the issue, we have an [afsadmin] group, and hosts: "afs*" in our playbook01:11
jeblairpabelanger: and the group matches the regex?01:11
pabelangerjeblair: looks like it01:11
fungiianw: great find! i too was oblivious to X01:11
jeblairneat01:11
pabelangermirror-update and release.slave are in afsadmin01:12
jeblairi guess we should call it adminafs :)01:12
jeblairpabelanger: status.o.o updated01:12
pabelangerYES01:13
pabelangernow to setup telnet:// map in chrome01:13
dmsimardCONSOLE LINKS01:13
dmsimard\o/01:13
nibalizerjeblair: http://paste.openstack.org/show/516757/ looking at it its probably the * in 'afs*'01:13
nibalizeroh pabelanger way ahead of me01:14
jeblair#status log follow-up mail about zuul-related changes: http://lists.openstack.org/pipermail/openstack-dev/2016-June/097595.html01:14
*** sdake has quit IRC01:14
openstackstatusjeblair: finished logging01:14
jeblairinfra-root: ^01:14
jheskethjeblair: I saw that... awesome stuff ! :-)01:15
jheskethamazing work to all01:15
jeblairjhesketh: thank you too -- i hope to look at the testing stuff next week01:15
jeblairand, you know, zuulv301:15
nibalizerneat01:15
clarkbthe telnet links totally work on android with connectbot01:15
jeblairclarkb: !!!!!01:15
clarkbnot really readable though01:16
*** sdake has joined #openstack-infra01:16
nibalizerclarkb: nice01:16
pabelangerclarkb: nice!01:16
jheskethjeblair: that's cool, I haven't made it as far as I had hoped with the testing... nor did as much as I had hoped on v25 (or v3 yet for that matter) sorry :-s01:16
jeblairnp01:16
dmsimardpabelanger: you figure out how to get chrome to handle those properly yet ? still looking myself01:16
pabelangerdmsimard: hacking the googles chromes now01:17
clarkbthe issue is we dont use telnet line breaks01:17
clarkbso the lines dont format well01:17
*** Goneri has joined #openstack-infra01:17
*** roxanaghe has joined #openstack-infra01:17
ianwfungi: i think learning today that with rebase you can specify --exec to run tox in between each rebased change is still my "flag of the day" find :)01:18
clarkbchrome has an ssh/telnet app01:19
clarkbyou can likely use that01:19
nibalizerwith bbsfox it seems to work01:19
nibalizersorta01:19
*** jsavak has joined #openstack-infra01:20
clarkbchrome secure shell01:20
fungiianw: yep, rebase --exec is fun, similar to using bisect01:21
ianwdmsimard: ok, so i'm googling how to make readable and your name comes up ... -> http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/_zuul_ansible/logs/ansible.log01:22
*** roxanaghe has quit IRC01:22
*** sdake_ has joined #openstack-infra01:22
ianwdmsimard: your "human_log" gist is best-practice? https://gist.github.com/dmsimard/cd706de198c85a8255f601:22
dmsimardlol01:23
dmsimardianw: sooooooo01:24
dmsimardianw: it's what I found worked best01:24
*** Jeffrey4l has quit IRC01:24
*** sdake has quit IRC01:24
dmsimardI forked the original version and made it a bit better and ansible 2 ready01:24
*** amitgandhinz has joined #openstack-infra01:24
dmsimardbut it kind of sucks, because it doesn't suppress verbose output.. it prints the verbose output and then the "optimized" output01:25
dmsimardso it makes for a super crowded shell01:25
dmsimardthis is the story of why I'm working on this: http://logs.openstack.org/76/330576/6/check/gate-ara-integration-ubuntu-trusty/6edcb0a/logs/build/01:25
jamielennoxhey, so submitting a patch  to oslo.context is telling me:  ! [remote rejected] HEAD -> refs/publish/master/deprecate-vars (project is read only) and there is no project config at https://review.openstack.org/#/admin/projects/openstack/oslo.context,access01:26
lifelessianw: you can also poke in a per-stage test permanently with a little config01:26
jamielennoxknown issue? something happening?01:26
jgriffithclarkb: jeblair fungi et'al  Well done, love the telnet access FWIW01:26
* jgriffith just needs to figure out how to implement all of it on his side :)01:27
lifelessianw: https://github.com/testing-cabal/mock/blob/master/docs/index.txt#L14701:27
jamielennoxhttps://review.openstack.org/gitweb?p=openstack/oslo.context.git;a=blobdiff;f=project.config;h=9e7a0f50b2759180ce6cc82773bdbbe998130279;hp=23af7117ff1ef93a9d20809fc97b1a15c6c005eb;hb=4dea369bdde28658a276e53bd6cb19b215055b94;hpb=15f83c3fc2ad5f6e098b7e5bdee490fa0dbb790101:28
*** baoli has joined #openstack-infra01:28
eliqiaohi AJaeger: thank you for helping update patch and give helps on irc renaming stuff :) , really appreciate of it?01:29
eliqiaosorry s/?/.01:29
*** amitgandhinz has quit IRC01:30
*** hongbin has joined #openstack-infra01:30
ianwit also seems colorization doesn't know when to stop -> http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/7306594/console.html01:31
*** kushal has joined #openstack-infra01:32
fungijamielennox: dhellmann: i wonder if this is the problem? http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n299601:32
jamielennoxfungi: yep, just found that myself01:33
*** Daisy has joined #openstack-infra01:33
jamielennoxfungi: is still pointing to the old incubator acls so has been shutdown - will propose something soon01:33
*** Apoorva has joined #openstack-infra01:33
*** Goneri has quit IRC01:34
fungijamielennox: yeah, missed in https://review.openstack.org/32068001:35
*** sarob has quit IRC01:36
openstackgerritMerged openstack-infra/system-config: Use port 19885 for console streaming for zuul workers  https://review.openstack.org/33083101:36
openstackgerritMerged openstack-infra/project-config: Ensure console-log is our last publisher  https://review.openstack.org/32998401:37
*** Jeffrey4l has joined #openstack-infra01:38
dmsimardianw: let me know if you have any questions regarding human_log or ara, can show you how both of them look like in practice01:40
openstackgerritJamie Lennox proposed openstack-infra/project-config: Add ACLs back for oslo.context  https://review.openstack.org/33084501:40
jamielennoxfungi: ^01:41
dhellmannjamielennox, fungi : looks good. harlowja_ ^^01:41
*** ddieterly has quit IRC01:42
ianwdmsimard: well, i was just going to stick it into zuul, but i guess it's GPL so that's a no-go01:42
dmsimardianw: don't see how that's a problem, Zuul has GPL stuff in it already.01:42
dmsimardhttps://github.com/openstack-infra/zuul/blob/master/zuul/ansible/plugins/callback_plugins/timeout.py01:43
dmsimardThere's an ongoing unclear debate as to whether or not stuff importing ansible has to be GPL or not01:43
*** yanyanhu has joined #openstack-infra01:44
dmsimardI have an ongoing thread with RH Legal and Ansible01:44
dmsimardAnsible *is* GPL01:44
ianwdmsimard: ok, well if we're doing that ... i guess just putting it in there would work and give us clearer output?01:44
*** mtanino has quit IRC01:45
dmsimardclearer is subjective, hang on, let me show you.01:45
dmsimardianw: https://ci.centos.org/job/weirdo-master-promote-packstack-scenario001/263/consoleText01:45
*** hongbin has quit IRC01:45
ianwdmsimard: ahh, so you get the two copies01:46
dmsimardianw: yeah, that's what I was saying. This is one task: https://etherpad.openstack.org/p/TvALrzw4ww01:46
*** claudiub has quit IRC01:47
dmsimardNow that everything is switched to zuul launcher, I was going to try to squeeze my ara project inside a job to see what it looks like01:47
*** Daisy has quit IRC01:47
*** Goneri has joined #openstack-infra01:48
*** jsavak has quit IRC01:48
*** Daisy has joined #openstack-infra01:48
*** Daisy has quit IRC01:52
*** ddieterly has joined #openstack-infra01:52
dmsimardI was also wondering if zuul launcher would be affected by ansible configuration changes (such as exporting config variables and such)01:54
dmsimardonly one way to find out :)01:54
*** banix has joined #openstack-infra01:56
*** Sukhdev has joined #openstack-infra01:57
ianwhmm, zuul, python3 + pep8 == ImportError: No module named 'extras'01:59
ianwdown the rabbit hole ...01:59
pabelangerdmsimard: got it working02:00
dmsimardpabelanger: oh, do tell02:00
*** Goneri has quit IRC02:00
pabelangerdmsimard: let me get pastebin02:00
pabelangerdmsimard: https://etherpad.openstack.org/p/google-telnet might be better02:01
pabelangerdmsimard: that should be it02:03
dmsimardThat's way more involved than than I thought it'd be, I'll try it02:03
pabelangerdmsimard: there likely is a better way02:04
pabelangerI came up with that just hacking02:04
*** SumitNaiksatam has joined #openstack-infra02:04
dmsimardwfm if it works02:04
*** Apoorva has quit IRC02:05
pabelangercalling it for tonight should have left hours ago02:05
*** Daisy has joined #openstack-infra02:06
jeblairianw: re zuul py3 -- see https://review.openstack.org/325661 and related patches02:08
jeblairsorry have to run02:08
ianwlifeless: pbr -> testrepository -> subunit -> ImportError: No module named 'extras' (http://paste.openstack.org/show/516764/) ... suggestion welcome :)02:09
*** chem|off has quit IRC02:10
ianwok, so extras is a python2 thing02:11
ianwi am guessing this "basepython" set in -> http://git.openstack.org/cgit/openstack-infra/zuul/tree/tox.ini#n21 is not entirely safe02:12
*** SumitNaiksatam has quit IRC02:12
*** Goneri has joined #openstack-infra02:15
*** sdake has joined #openstack-infra02:16
*** ddieterly has quit IRC02:17
ianwhave we lost the "toggle CI" button?02:18
*** roxanaghe has joined #openstack-infra02:18
*** sdake_ has quit IRC02:19
*** piet has joined #openstack-infra02:22
*** mtanino has joined #openstack-infra02:22
*** roxanaghe has quit IRC02:23
*** Goneri has quit IRC02:26
*** amitgandhinz has joined #openstack-infra02:26
ianwthe answer appears to be yes, and that we are looking for -> https://review.openstack.org/static/hideci.js?e=df833e6b9eb694b6da03c5d403d38a8a which doesn't exist02:27
phschwartzmordred: fungi: nibalizer: so all of those reviews merged and I have verified in my downstream CI that all things are working onces again02:27
phschwartzIf I find any more that need to change I will open reviews02:27
tonybIs there a way to see all the jobs run on the post-release pipline in say the last 34hours?02:28
*** Goneri has joined #openstack-infra02:28
ianwahh, seems to be the same issue as https://review.openstack.org/#/c/126482/ ... i'm guessing this isn't working02:29
ianwjhesketh: ^ around maybe?02:29
*** amitgandhinz has quit IRC02:31
*** jogo has quit IRC02:32
*** ddieterly has joined #openstack-infra02:33
jheskethianw: right, sometimes that doesn't work and we don't know why02:33
jheskethianw: the solution is to just touch the header02:34
jheskethI'll do that now02:34
openstackgerritzzxwill proposed openstack-infra/zuul: ename jenkins to zull  https://review.openstack.org/33085302:34
*** piet has quit IRC02:35
jheskethianw: the new link is https://review.openstack.org/static/hideci.js?e=3820a6091f0996731e3f0dd72527aa1402:36
ianwjhesketh: yay, the button is back!02:36
ianwyak shaved02:36
dmsimardBasic zuul runner/launcher question, where does Ansible actually run ? On "localhost" (the nodepool VM?) or elsewhere ? I can't find those files: https://github.com/openstack-infra/zuul/blob/aaa852e455a1b2135c53f4a6251b659cea2d41d4/zuul/launcher/ansiblelaunchserver.py#L10202:37
dmsimardhttp://logs.openstack.org/47/330847/3/check/gate-ara-integration-centos-7/b777804/console.html#_2016-06-17_02_26_26_82405502:37
jheskethianw: yeah I wonder if there is a race where jenkins continues to the notify command while the asset is still being syncd or something02:37
*** thorst_ has joined #openstack-infra02:38
dmsimardIt looks like it would run from a "central" location (zuul launcher servers?) but at the same time I can see those servers getting quite busy with the amount of running jobs, even if it's just Ansible. No ?02:39
dmsimardI guess we can still scale the zuul launchers fairly well horizontally02:40
*** thorst_ has quit IRC02:41
*** thorst_ has joined #openstack-infra02:41
*** somerville32 has quit IRC02:42
jheskethdmsimard: yes, that's right.. the launchers hold the playbooks and then run them on the slave nodes02:44
jheskethand yeah, scalling is pretty horizontal02:45
openstackgerritzzxwill proposed openstack-infra/zuul: Rename jenkins to zull  https://review.openstack.org/33085302:45
*** thorst_ has quit IRC02:45
*** jogo has joined #openstack-infra02:46
*** Goneri has quit IRC02:46
dmsimardjhesketh: thought it ran locally, that would've allowed me to test if Zuul could've used another ansible callback successfully :(02:47
dmsimardI would've submitted a patch to zuul to test it, but there's not really any integration test that'd allow me to see if it worked or not02:47
dmsimardnot sure how I'd test this tbh, I don't have a pocket zuul02:48
*** ddieterly has quit IRC02:48
*** mdrabe has joined #openstack-infra02:49
dmsimardHmm, I've never thought of that, but I guess Zuul also zuul-clones itself for code reviews02:50
dmsimardAh, it's not effective for the jobs that are ran in the review, though02:51
jheskethzuulv3 will allow for some of that stuff02:51
jheskethbut yeah, it's a bit limited at the moment02:51
dmsimardFor example https://review.openstack.org/#/c/330827/ adds some ssh config02:51
dmsimardbut if you look at the ansible.cfg of the jobs, that ssh config isn't there02:51
openstackgerritIan Wienand proposed openstack-infra/zuul: Add human-log filter plugin  https://review.openstack.org/33085602:52
ianwdmsimard: ^ something to argue over02:52
*** zhurong_ has joined #openstack-infra02:53
dmsimardianw: eh I'm not going to fight for it02:53
dmsimardianw: I'd love to integrate ARA into Zuul but I don't know how to test it02:54
dmsimardThe way I'd see this would be: add ara to requirements.txt, add the ara callback path to callback_plugins and then in the post-job where the console logs are saved, we also generate a static version of the ansible run so we save something like this in the logs of the job: http://logs.openstack.org/34/330834/3/check/gate-ara-integration-ubuntu-trusty/d954777/logs/build/02:55
nibalizerphschwartz: http://codesearch.openstack.org/?q=provider%20%3D%3E%20%27pip%27&i=nope&files=&repos= and http://codesearch.openstack.org/?q=provider%20%3D%3E%20*pip&i=nope&files=&repos= seem to imply you hit them all02:56
nibalizerat least under -infra02:56
dmsimardSuper simple but I'm not comfortable just hooking that up, I guess I'll send a review02:56
*** zhurong has quit IRC02:56
*** Goneri has joined #openstack-infra02:57
*** jamielennox is now known as jamielennox|away03:03
openstackgerritzzxwill proposed openstack-infra/zuul: Rename jenkins to zuul  https://review.openstack.org/33085303:06
*** gyee has quit IRC03:07
*** amotoki has joined #openstack-infra03:07
*** Goneri has quit IRC03:08
*** jcoufal has joined #openstack-infra03:08
*** amotoki has quit IRC03:09
*** gouthamr_ has quit IRC03:12
*** tphummel has quit IRC03:13
jlvillalWould the openstackgerrit bot not announce patches to stable/mitaka for openstack/ironic on #openstack-ironic for some reason?03:17
*** r-mibu has quit IRC03:17
*** jamielennox|away is now known as jamielennox03:19
*** roxanaghe has joined #openstack-infra03:19
mtreinishjlvillal: it's not configured to: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerritbot/channels.yaml#n94803:20
jlvillalmtreinish, Thanks. I was wandering around from project to project to figure out where that is configured :)03:20
*** Goneri has joined #openstack-infra03:21
*** roxanaghe has quit IRC03:24
*** baoli has quit IRC03:25
*** jogo has quit IRC03:26
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic: Have gerritbot announce for stable branches too  https://review.openstack.org/33086703:26
*** amitgandhinz has joined #openstack-infra03:27
*** Goneri has quit IRC03:28
*** jogo has joined #openstack-infra03:30
*** amitgandhinz has quit IRC03:31
pabelangerdmsimard: the playbook currently live on the zuul-launchers (eg: zl01.openstack.org). We are dynamically building them when we see a new node.  After the job is finished, we'll upload them into the _zuul_ansible folder for logs.o.o. Obviously moving forward, how we build and setup playbooks with change03:36
pabelangerdmsimard: with the URL you linked above, here are the playbooks for that job. http://logs.openstack.org/47/330847/3/check/gate-ara-integration-centos-7/b777804/_zuul_ansible/03:37
dmsimardpabelanger: yeah I eventually sort of figured that out03:37
dmsimardpabelanger: I'm going to submit a patch just to see what it'd look like, not expecting it to go it03:37
dmsimardgo in*03:38
*** r-mibu has joined #openstack-infra03:38
dmsimardthat said, I don't see ansible in Zuul's requirement.txt03:38
dmsimardhacks ?03:38
pabelangerwe've left it our for now03:38
pabelangerand manage it from puppet03:38
dmsimardah, okay. I'll look in the manifests, just interested in knowing the version03:38
pabelangermostly because we don't want to support v25 for other users :)03:38
pabelanger2.1.0.003:38
dmsimardalright03:39
*** furlongm has quit IRC03:41
*** furlongm has joined #openstack-infra03:41
*** Goneri has joined #openstack-infra03:46
*** markvoelker has quit IRC03:47
*** mdrabe has quit IRC03:48
*** gema has quit IRC03:51
*** mriedem has quit IRC03:51
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul: WIP: Integrate ARA reporting inside Zuul Launcher  https://review.openstack.org/33087403:54
dmsimardianw: ^03:54
*** Daisy has quit IRC03:55
*** Daisy has joined #openstack-infra03:55
*** Goneri has quit IRC03:56
*** Daisy has quit IRC03:58
*** Daisy has joined #openstack-infra03:58
*** amotoki has joined #openstack-infra04:00
*** tkelsey has joined #openstack-infra04:03
ianwdmsimard: cool04:05
*** tkelsey has quit IRC04:07
*** Daisy has quit IRC04:11
*** Daisy has joined #openstack-infra04:11
*** Daisy has quit IRC04:16
*** amotoki has quit IRC04:18
krotscheckfungi: I'll try that, thanks04:19
*** links has joined #openstack-infra04:20
*** _nadya_ has joined #openstack-infra04:22
openstackgerritRico Lin proposed openstack-infra/project-config: Fix error jenkins scripts path.  https://review.openstack.org/33088004:23
*** shahid_ has joined #openstack-infra04:23
*** yamahata has joined #openstack-infra04:27
*** amitgandhinz has joined #openstack-infra04:28
*** ociuhandu has quit IRC04:29
*** amitgandhinz has quit IRC04:33
*** psachin has joined #openstack-infra04:35
*** tqtran has quit IRC04:36
openstackgerritIan Wienand proposed openstack-infra/zuul: Add human-log filter plugin  https://review.openstack.org/33085604:38
*** yuanying has quit IRC04:41
*** bhavik has quit IRC04:41
*** ociuhandu has joined #openstack-infra04:42
*** amotoki has joined #openstack-infra04:45
*** yamamoto has quit IRC04:46
*** markvoelker has joined #openstack-infra04:47
*** sdake_ has joined #openstack-infra04:50
*** markvoelker has quit IRC04:52
*** sdake has quit IRC04:53
*** Daisy has joined #openstack-infra04:53
*** sflanigan is now known as sflanigan_afk04:59
*** Daisy has quit IRC05:02
*** Daisy has joined #openstack-infra05:02
openstackgerritMerged openstack/diskimage-builder: Pre-install pip/virtualenv packages  https://review.openstack.org/32747205:05
*** Daisy has quit IRC05:07
*** Daisy has joined #openstack-infra05:07
*** maishsk has quit IRC05:07
*** maishsk has joined #openstack-infra05:08
*** salv-orl_ has joined #openstack-infra05:09
*** Daisy has quit IRC05:10
*** Daisy has joined #openstack-infra05:10
*** salv-orlando has quit IRC05:12
*** pgadiya has joined #openstack-infra05:12
*** gema has joined #openstack-infra05:13
*** pc_m has quit IRC05:16
openstackgerritMerged openstack/diskimage-builder: Remove discover from test-requirements  https://review.openstack.org/32817005:19
*** banix has quit IRC05:19
*** roxanagh_ has joined #openstack-infra05:20
openstackgerritMerged openstack-infra/project-config: Move gate-manila-buildimage-docker job to check pipeline  https://review.openstack.org/33060405:22
*** roxanagh_ has quit IRC05:25
*** sflanigan_afk is now known as sflanigan05:25
*** yamamoto has joined #openstack-infra05:26
*** links has quit IRC05:27
*** sdake has joined #openstack-infra05:27
*** links has joined #openstack-infra05:28
*** amitgandhinz has joined #openstack-infra05:29
*** sdake_ has quit IRC05:31
*** Daisy has quit IRC05:32
*** Daisy has joined #openstack-infra05:32
*** _nadya_ has quit IRC05:33
*** amitgandhinz has quit IRC05:33
*** cfarquhar has quit IRC05:35
*** _nadya_ has joined #openstack-infra05:36
*** Daisy has quit IRC05:37
*** sdake_ has joined #openstack-infra05:38
*** sdake has quit IRC05:41
*** Daisy has joined #openstack-infra05:42
*** _nadya_ has quit IRC05:43
openstackgerritIan Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point  https://review.openstack.org/33089305:48
*** sdake_ has quit IRC05:50
*** _nadya_ has joined #openstack-infra05:51
*** amotoki has quit IRC05:56
*** ccamacho has joined #openstack-infra06:00
*** hashar has joined #openstack-infra06:01
*** yaume has joined #openstack-infra06:02
*** amotoki has joined #openstack-infra06:02
*** yuanying has joined #openstack-infra06:02
*** Daisy has quit IRC06:04
*** Daisy has joined #openstack-infra06:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add console-log validator  https://review.openstack.org/32998806:08
*** Daisy has quit IRC06:09
*** YorikSar has quit IRC06:09
*** yaume has quit IRC06:13
*** Daisy has joined #openstack-infra06:15
*** pc_m has joined #openstack-infra06:15
*** Sukhdev has quit IRC06:17
*** YorikSar has joined #openstack-infra06:19
*** _nadya_ has quit IRC06:20
*** aeng has quit IRC06:21
openstackgerritMerged openstack-infra/project-config: Add ACLs back for oslo.context  https://review.openstack.org/33084506:21
openstackgerritKhai Do proposed openstack-infra/system-config: Refactor gerrit commentlinks parameter  https://review.openstack.org/33090206:22
*** rcernin has joined #openstack-infra06:23
*** YorikSar has quit IRC06:23
*** mtanino has quit IRC06:24
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33090306:24
*** david-lyle has quit IRC06:26
*** andreas_s has joined #openstack-infra06:28
*** zeih has joined #openstack-infra06:28
*** pcaruana has joined #openstack-infra06:29
*** amitgandhinz has joined #openstack-infra06:30
AJaegeryolanda, https://github.com/openstack/fuel-plugin-datera-cinder has not been setup yet but change merged for it. Could you check it, please?06:31
*** YorikSar has joined #openstack-infra06:32
*** amitgandhinz has quit IRC06:34
*** _nadya_ has joined #openstack-infra06:35
openstackgerritMerged openstack-infra/project-config: Add console-log validator  https://review.openstack.org/32998806:36
*** chem|off has joined #openstack-infra06:36
*** ramishra has joined #openstack-infra06:39
*** _nadya_ has quit IRC06:40
*** _nadya_ has joined #openstack-infra06:42
*** ramishra has quit IRC06:44
*** ramishra has joined #openstack-infra06:47
*** Daisy_ has joined #openstack-infra06:50
*** dingyichen has quit IRC06:51
*** Daisy has quit IRC06:52
ttxjeblair: nice!06:53
*** arxcruz has joined #openstack-infra06:55
*** chem|off` has joined #openstack-infra06:56
*** kushal has quit IRC06:56
*** Daisy_ has quit IRC06:57
*** _nadya_ has quit IRC06:57
*** Daisy has joined #openstack-infra06:58
*** mixos has quit IRC06:59
*** chem|off has quit IRC06:59
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: No-op Change to force a re-publish  https://review.openstack.org/33091406:59
*** yuanying has quit IRC07:02
*** ihrachys has joined #openstack-infra07:03
*** amotoki has quit IRC07:03
*** ramishra has quit IRC07:04
*** tkelsey has joined #openstack-infra07:05
openstackgerritSachi King proposed openstack-dev/pbr: Restore warnerrors behavior and support Sphinx 1.4  https://review.openstack.org/22995107:07
*** ihrachys has quit IRC07:08
*** javeriak has joined #openstack-infra07:09
*** salv-orl_ has quit IRC07:09
*** chem|off` has quit IRC07:09
*** salv-orlando has joined #openstack-infra07:09
*** tkelsey has quit IRC07:09
*** yaume has joined #openstack-infra07:10
*** Na3iL has joined #openstack-infra07:10
*** tesseract has joined #openstack-infra07:10
AJaegerhttps://github.com/openstack/fuel-plugin-datera-cinder is ready now...07:11
*** hogepodge has quit IRC07:12
*** saravana has joined #openstack-infra07:12
*** pahuang has quit IRC07:13
*** bauzas is now known as bauwser07:13
*** bauwser is now known as bauzas07:14
saravanamy zuul status page shows status.json: Proxy Error07:14
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Add configuration section for its plugins  https://review.openstack.org/33092207:14
*** matrohon has joined #openstack-infra07:15
*** dtantsur|afk is now known as dtantsur07:15
openstackgerritMerged openstack-infra/irc-meetings: No-op Change to force a re-publish  https://review.openstack.org/33091407:16
*** bauzas has left #openstack-infra07:18
*** jlanoux has joined #openstack-infra07:20
*** dtardivel has joined #openstack-infra07:20
*** otherwiseguy has quit IRC07:20
*** ramishra has joined #openstack-infra07:21
*** otherwiseguy has joined #openstack-infra07:21
*** esikachev has joined #openstack-infra07:22
*** roxanaghe has joined #openstack-infra07:22
*** vincentll has joined #openstack-infra07:23
*** rossella_s has joined #openstack-infra07:25
openstackgerritKhai Do proposed openstack-infra/system-config: Configure its-storyboard plugin  https://review.openstack.org/33092507:25
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33090307:26
openstackgerritBabu Shanmugam proposed openstack-infra/project-config: Adding a new JOB for networking-ovn project  https://review.openstack.org/33092607:27
*** roxanaghe has quit IRC07:27
*** bauwser has joined #openstack-infra07:28
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu and CentOS gate voting for Kolla  https://review.openstack.org/33092707:30
*** amitgandhinz has joined #openstack-infra07:30
*** javeriak has quit IRC07:31
*** javeriak has joined #openstack-infra07:31
*** dmk0202 has joined #openstack-infra07:32
*** dmk0202 has quit IRC07:33
*** sorantis has joined #openstack-infra07:35
*** salv-orlando has quit IRC07:35
*** amitgandhinz has quit IRC07:35
*** _ioni is now known as ioni07:36
*** ioni is now known as Guest2717907:36
*** claudiub has joined #openstack-infra07:37
openstackgerritzzxwill proposed openstack-infra/zuul: Rename jenkins to zuul  https://review.openstack.org/33085307:38
*** dingyichen has joined #openstack-infra07:39
*** amoralej|off is now known as amoralej07:39
*** javeriak has quit IRC07:42
*** netsin has quit IRC07:45
*** javeriak has joined #openstack-infra07:45
*** rcernin has quit IRC07:45
*** Daisy has quit IRC07:46
*** Daisy has joined #openstack-infra07:46
*** jpich has joined #openstack-infra07:50
*** Daisy has quit IRC07:51
*** Daisy has joined #openstack-infra07:51
openstackgerritMasayuki Igawa proposed openstack/os-testr: Make unexpected success as fail  https://review.openstack.org/33093707:51
*** camunoz has quit IRC07:52
*** tkelsey has joined #openstack-infra07:53
*** ramishra has quit IRC07:53
*** saravana has quit IRC07:53
*** yolanda has joined #openstack-infra07:55
*** ramishra has joined #openstack-infra07:55
yolandagood morning07:56
*** ramishra has quit IRC07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** yamamoto has quit IRC08:01
*** mikelk has joined #openstack-infra08:03
*** ramishra has joined #openstack-infra08:04
*** yuanying has joined #openstack-infra08:05
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting  https://review.openstack.org/31259608:06
*** eezhova has joined #openstack-infra08:06
dtantsurlucasagomes, updated this a bit while you were out ^^^08:07
*** yamahata has quit IRC08:07
*** rcernin has joined #openstack-infra08:10
*** ramishra has quit IRC08:16
*** dizquierdo has joined #openstack-infra08:18
*** ameade has quit IRC08:21
*** dteselkin has quit IRC08:21
*** ramishra has joined #openstack-infra08:21
*** BobBall_AWOL is now known as BobBall08:21
*** redrobot has quit IRC08:21
*** jlvillal has quit IRC08:22
*** jlvillal has joined #openstack-infra08:22
*** tobiash_ has quit IRC08:22
*** ameade has joined #openstack-infra08:22
*** igormarnat has quit IRC08:22
*** igormarnat has joined #openstack-infra08:22
*** hferenc has quit IRC08:22
*** Daviey_ has quit IRC08:23
*** rbergeron has quit IRC08:23
*** roxanaghe has joined #openstack-infra08:23
*** Daviey has joined #openstack-infra08:23
*** tobiash has joined #openstack-infra08:23
*** Guest27179 is now known as ioni08:23
*** ioni is now known as Guest6234208:24
*** redrobot has joined #openstack-infra08:24
*** redrobot is now known as Guest3155308:24
*** rbergeron has joined #openstack-infra08:24
*** javeriak has quit IRC08:24
*** Daisy has quit IRC08:26
*** Daisy has joined #openstack-infra08:26
*** dmk0202 has joined #openstack-infra08:27
*** dteselkin has joined #openstack-infra08:27
*** roxanaghe has quit IRC08:28
*** daemontool has joined #openstack-infra08:28
*** gema has quit IRC08:29
*** Daisy_ has joined #openstack-infra08:29
openstackgerritMasayuki Igawa proposed openstack/os-testr: Fix to enable '--no-failure-debug' option for success tests  https://review.openstack.org/33095208:30
*** amitgandhinz has joined #openstack-infra08:31
*** Daisy has quit IRC08:33
ttxLooks like we have an issue with Launchpad auth on Gerrit08:34
*** SlickNik has quit IRC08:35
*** SlickNik has joined #openstack-infra08:35
*** icey has quit IRC08:35
*** amitgandhinz has quit IRC08:35
ttxsigning in does not redirect to UbuntuOne auth but to some "Sign In to Gerrit Code Review at review.openstack.org" canned page08:36
*** trash has joined #openstack-infra08:36
*** jokke__ has joined #openstack-infra08:36
*** mestery_ has joined #openstack-infra08:36
*** andreas_s_ has joined #openstack-infra08:36
*** tristanC_ has joined #openstack-infra08:37
ttx"Provider is not supported, or was incorrectly entered."08:37
*** kaisers has quit IRC08:37
*** pgadiya_ has joined #openstack-infra08:37
*** Jeffrey4l has quit IRC08:37
*** flepied1 has joined #openstack-infra08:37
*** jlvillal has quit IRC08:37
*** mandre_ has joined #openstack-infra08:38
ttxAJaeger, yolanda ^08:38
*** kaisers has joined #openstack-infra08:38
*** bswartz has quit IRC08:39
*** yamamoto has joined #openstack-infra08:39
*** LinuxJedi_ has joined #openstack-infra08:39
*** coreyob has quit IRC08:39
*** rodrigods has quit IRC08:40
*** andymccr_ has joined #openstack-infra08:40
*** alazarev_ has joined #openstack-infra08:40
yolandattx, looking08:40
*** xwizard___ has joined #openstack-infra08:40
ttxFWIW logging into LP works ok08:40
*** rodrigods has joined #openstack-infra08:40
yolandagoing to check logs08:41
*** harlowja_ has quit IRC08:41
ttxyolanda: can you reproduce ?08:41
*** pkoniszewski has quit IRC08:41
*** electrofelix has joined #openstack-infra08:41
yolandathere may be some connectivity problem, i cannot ssh to gerrit08:41
ttxAh. Interesting08:42
*** jlvillal has joined #openstack-infra08:42
*** Daisy_ has quit IRC08:42
*** jmccrory has quit IRC08:42
*** daemontool_ has joined #openstack-infra08:42
*** anteaya has quit IRC08:42
*** waynr has quit IRC08:42
*** HeOS has joined #openstack-infra08:42
*** Daisy has joined #openstack-infra08:42
*** andreas_s has quit IRC08:42
*** xwizard has quit IRC08:42
*** tristanC has quit IRC08:42
*** ekarlso has quit IRC08:42
*** LinuxJedi has quit IRC08:42
*** LinuxJedi_ is now known as LinuxJedi08:42
*** alazarev has quit IRC08:42
*** alazarev_ is now known as alazarev08:42
*** m3m0 has quit IRC08:42
*** Na3iL has quit IRC08:42
*** shahid_ has quit IRC08:42
*** shahid__ has joined #openstack-infra08:42
*** furlongm has quit IRC08:42
*** pkoniszewski has joined #openstack-infra08:43
*** trash_ has quit IRC08:43
*** flepied has quit IRC08:43
*** mestery has quit IRC08:43
*** mattymo has quit IRC08:43
*** jokke_ has quit IRC08:43
*** pgadiya has quit IRC08:43
*** andymccr has quit IRC08:43
*** tlbr has quit IRC08:43
*** mestery_ is now known as mestery08:43
*** zeroDivisible has quit IRC08:43
*** mandre has quit IRC08:43
*** kaisers1 has quit IRC08:43
*** mandre_ is now known as mandre08:43
*** rcarrillocruz has quit IRC08:43
*** ruhe has quit IRC08:43
*** SergeyLukjanov2 has joined #openstack-infra08:43
*** asselin_ has joined #openstack-infra08:43
*** SergeyLukjanov has quit IRC08:43
*** dingyichen has quit IRC08:43
*** persia has quit IRC08:43
*** furlongm has joined #openstack-infra08:43
*** zeroDivisible has joined #openstack-infra08:43
*** jmccrory has joined #openstack-infra08:43
*** SergeyLukjanov2 is now known as SergeyLukjanov08:43
*** ramishra has quit IRC08:43
*** waynr has joined #openstack-infra08:43
*** persia has joined #openstack-infra08:43
*** schang has quit IRC08:44
*** rcarrillocruz has joined #openstack-infra08:44
*** ruhe has joined #openstack-infra08:44
yolandattx, works now08:44
*** asselin__ has quit IRC08:44
*** mattymo has joined #openstack-infra08:44
*** mtreinish has quit IRC08:44
ttxindeed08:44
*** daemontool has quit IRC08:44
*** schang has joined #openstack-infra08:44
*** tlbr has joined #openstack-infra08:44
*** e0ne has joined #openstack-infra08:44
ttxfalse alarm08:44
*** ihrachys has joined #openstack-infra08:44
ttxstill the error was weird, must be some kind of fallback08:44
*** tristanC_ is now known as tristanC08:44
*** dingyichen has joined #openstack-infra08:45
*** kaisers has left #openstack-infra08:45
*** m3m0 has joined #openstack-infra08:45
*** kaisers1 has joined #openstack-infra08:45
*** anteaya has joined #openstack-infra08:45
*** kaisers1 has left #openstack-infra08:45
*** kaisers1 has joined #openstack-infra08:45
*** mtreinish has joined #openstack-infra08:46
*** coreyob has joined #openstack-infra08:46
*** MarkAtwood has quit IRC08:46
openstackgerritYaroslav Lobankov proposed openstack-infra/project-config: [Rally] Enable Ceilometer for one of the 'rally verify' jobs  https://review.openstack.org/32715708:46
*** Ng has quit IRC08:47
*** tpeoples has quit IRC08:47
*** dham1 has quit IRC08:47
*** teran has quit IRC08:47
*** sflanigan has quit IRC08:47
*** Daisy_ has joined #openstack-infra08:47
*** Jeffrey4l has joined #openstack-infra08:48
*** teran has joined #openstack-infra08:49
*** dham1 has joined #openstack-infra08:49
*** tpeoples has joined #openstack-infra08:49
*** MarkAtwood has joined #openstack-infra08:49
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuseast name for chocolate  https://review.openstack.org/33058008:49
*** Ng has joined #openstack-infra08:49
*** kzaitsev_mb has joined #openstack-infra08:50
*** Daisy has quit IRC08:50
*** ekarlso has joined #openstack-infra08:51
*** dizquierdo has quit IRC08:54
*** ramishra has joined #openstack-infra08:54
openstackgerritosman proposed openstack-infra/jenkins-job-builder: Adds support rerun matrix part to the naginator  https://review.openstack.org/33095908:56
*** Na3iL has joined #openstack-infra08:56
*** Daisy has joined #openstack-infra08:58
*** Daisy_ has quit IRC08:58
*** ramishra has quit IRC08:59
*** ramishra has joined #openstack-infra09:00
*** markusry has joined #openstack-infra09:01
openstackgerritMerged openstack-infra/project-config: Ironic: Have gerritbot announce for stable branches too  https://review.openstack.org/33086709:01
*** YorikSar has quit IRC09:01
*** kushal has joined #openstack-infra09:01
*** javeriak has joined #openstack-infra09:02
openstackgerritMerged openstack-infra/project-config: Enable storyboard for js-openstack-lib  https://review.openstack.org/33077909:02
*** yanyanhu has quit IRC09:03
*** YorikSar has joined #openstack-infra09:03
*** yanyanhu has joined #openstack-infra09:04
*** Daisy_ has joined #openstack-infra09:04
*** yanyanhu has quit IRC09:04
*** Daisy_ has quit IRC09:06
*** Daisy has quit IRC09:06
*** Na3iL has quit IRC09:06
*** Na3iL has joined #openstack-infra09:07
*** dtantsur is now known as dtantsur|bbl09:07
*** _nadya_ has joined #openstack-infra09:11
*** sambetts|afk is now known as sambetts09:12
openstackgerritMerged openstack-infra/project-config: Remove Jenkins masters from grafana dashboard  https://review.openstack.org/33078409:12
*** jyuso1 has quit IRC09:12
openstackgerritMerged openstack-infra/project-config: New Project Quark Plugin  https://review.openstack.org/31936809:12
*** e0ne has quit IRC09:13
openstackgerritMerged openstack-infra/project-config: Adds API reference publishing jobs for Swift  https://review.openstack.org/31301509:14
bauwsercongrats folks for getting rid of Jenkins !09:14
*** openstackgerrit has quit IRC09:18
*** openstackgerrit has joined #openstack-infra09:18
*** eezhova has quit IRC09:19
*** samP has quit IRC09:20
openstackgerritMerged openstack-infra/project-config: Add a functional job for neutron-lbaas  https://review.openstack.org/32281209:20
*** chem|off` has joined #openstack-infra09:21
*** chem|off` is now known as chem09:21
*** chem has quit IRC09:22
*** acoles_ is now known as acoles09:22
*** chem has joined #openstack-infra09:22
*** acoles is now known as acoles_09:23
*** eezhova has joined #openstack-infra09:24
*** Guest62342 is now known as ioni09:24
*** ioni is now known as Guest5818409:25
*** binbincong has quit IRC09:25
*** binbincong has joined #openstack-infra09:25
*** alex_xu has quit IRC09:26
*** AJaeger has quit IRC09:26
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Fix the destination for eavesdrop site  https://review.openstack.org/33098609:27
*** alex_xu has joined #openstack-infra09:28
*** acoles_ is now known as acoles09:29
*** flepied1 is now known as flepied09:31
*** amitgandhinz has joined #openstack-infra09:32
*** mhickey has joined #openstack-infra09:33
*** _milan_ has joined #openstack-infra09:33
_milan_guys, I want to introduce inspector experimental grenade gate. I want to base it on the ironic job from  devstack-gate.yaml jenkins definitnion in infra. I'm wondering how to  make the job enable inspector's grenade plugin (under review now) without having to insert ref spec in the grenade plugin line.09:33
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Fix the destination for eavesdrop site  https://review.openstack.org/33098609:33
jheskethany other infra-root's around to help merge ^ to unblock eavesdrop publishers?09:33
openstackgerritElena Ezhova proposed openstack-infra/project-config: Run neutron-lbaas-dsvm-functional-nv job only on master  https://review.openstack.org/33099609:34
_milan_e.g https://github.com/openstack-infra/project-config/blob/d09a1a898d34931f7fc49e027bf665950e66a2b2/jenkins/jobs/devstack-gate.yaml#L209609:35
_milan_vs https://github.com/dparalen/devstack-gate-test/blob/with_inspector/ansible/roles/devstack-gate/files/inspector-grenade.sh#L9909:35
*** amitgandhinz has quit IRC09:36
eezhovayolanda, Hi! It just came up to me that the new lbaas functional job should be run only on master, but the patch has already merged09:37
eezhovayolanda, so I created a follow-up patch https://review.openstack.org/#/c/33099609:37
eezhovayolanda, could you please take a look at it?09:38
*** jokke__ is now known as jokke_09:38
*** Guest58184 has quit IRC09:40
*** Guest58184 has joined #openstack-infra09:40
*** Guest58184 is now known as ioni09:40
*** kzaitsev_mb has quit IRC09:41
*** kzaitsev_mb has joined #openstack-infra09:42
*** dizquierdo has joined #openstack-infra09:42
yolandaeezhova, sure09:44
openstackgerritMerged openstack-infra/project-config: New repo for murano plugin  https://review.openstack.org/32993109:44
*** megm_ has quit IRC09:44
chemhi, on the tripleo-ci side we have all jobs that timeout, and it looks like something changed as we have new ansible logs, is there somebody that could help me find out ?09:44
openstackgerritMerged openstack-infra/system-config: Add compute014.useast metadata for better tracking  https://review.openstack.org/33042109:45
eezhovayolanda, thanks!09:45
jheskethyolanda: Howdy.. If you have a moment to look at https://review.openstack.org/#/c/330986/ that'd be appreciated09:48
yolandahi jhesketh09:48
*** mjb has quit IRC09:48
*** megm has joined #openstack-infra09:49
openstackgerritMilan Kováčik proposed openstack-infra/project-config: Introduce experimental Inspector grenade job  https://review.openstack.org/33100109:50
*** saravana has joined #openstack-infra09:52
_milan_dtantsur|bbl, ^09:55
*** savihou has quit IRC09:55
*** mjb has joined #openstack-infra09:55
*** mjb has quit IRC09:56
*** gouthamr has joined #openstack-infra09:59
*** mjb has joined #openstack-infra09:59
*** ramishra has quit IRC09:59
*** gouthamr_ has joined #openstack-infra10:00
*** eezhova has quit IRC10:01
*** zeih has quit IRC10:03
*** openstackgerrit has quit IRC10:03
*** openstackgerrit has joined #openstack-infra10:03
jheskeththanks yolanda10:03
jheskethyolanda: any objections to me self-merging?10:04
*** gouthamr has quit IRC10:04
yolandajhesketh, no problem10:04
*** ramishra has joined #openstack-infra10:05
jheskethcool, will do10:05
*** rcernin has quit IRC10:05
*** eezhova has joined #openstack-infra10:06
*** zhurong_ has quit IRC10:06
*** kzaitsev_mb has quit IRC10:06
*** esikachev has quit IRC10:07
*** lvdongbing has quit IRC10:08
*** zeih has joined #openstack-infra10:09
odyssey4mejhesketh I'm seeing an odd result from time to time where our 'linters' job log shows that it succeeded, but the job result shows as failed. An example: https://review.openstack.org/#/c/330973/10:10
*** e0ne has joined #openstack-infra10:12
jheskethodyssey4me: it looks like we're not reporting timeouts correctly... that's likely a bug10:14
jheskethin zuul that is10:14
jheskethyolanda: so I self-merged but it occurred to me I'll need to restart the launchers to apply the change :-(10:14
jheskethas far as I can tell that's a pause, wait then stop on each of the launchers10:14
odyssey4mejhesketh should I note this somewhere, or just recheck and move on?10:15
yolandajhesketh, so you can do it gradually?10:15
jheskethodyssey4me: recheck because I'm curious to see if it'll pass, and I'll look into zuul10:15
jheskethyolanda: I /think/ so if I do the stepping through with pauses etc10:16
jheskethyolanda: might take a while10:16
odyssey4mejhesketh I think that I had one of these yesterday, but I saw the transitioning away from jenkins and thought it might be a transitional failure, so I rechecked and it worked10:16
odyssey4meI'll do the same here10:16
openstackgerritThomas Goirand proposed openstack-infra/project-config: Fix upload path to tarballs.o.o for packaging  https://review.openstack.org/33100810:16
yolandajhesketh, can I help somehow? i'll need your guidance for that, because I have not been very involved with that changes lately10:16
zigoodyssey4me: Can you review that one liner path fix patch?10:17
jheskethyolanda: maybe... let me do a little more digging first :-)10:17
odyssey4mezigo looking10:17
zigoCheers.10:17
openstackgerritMerged openstack-infra/system-config: Fix the destination for eavesdrop site  https://review.openstack.org/33098610:18
*** mrunge has quit IRC10:18
yolandajhesketh, thanks10:18
*** Adri2000 has quit IRC10:18
yolandai need to find some time to study about how zuul 2.5 works now, to be able to help10:18
odyssey4mezigo I'd recommend linking to the eavesdrop logs referencing the conversation for anyone reviewing to provide context10:18
*** esikachev has joined #openstack-infra10:21
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Revert "No-op Change to force a re-publish"  https://review.openstack.org/33101010:21
odyssey4mezigo FYI, as I understand it, the general idea with the macros is to provide a generalised facility for multiple projects to use10:22
jheskethtonyb: don't do that thing yet10:22
jheskethtonyb: we need to restart the zuul launchers for the new config to apply10:22
odyssey4mezigo in the case where only one project is doing something you may not have to have a macro at all, and can probably just implement the publisher in your job definition instead10:22
jheskethtonyb: after that your revert will be a good test to check it's actually working10:22
jheskethtonyb: but at the moment we're not ready10:23
tonybjhesketh: okay10:23
odyssey4mezigo let me take a peek at your job definition10:23
*** ociuhandu has quit IRC10:23
tonybjhesketh: feel free to +W that revert once the restart has happened if I'm not around10:23
jheskethtonyb: thanks10:23
*** e0ne has quit IRC10:24
*** roxanaghe has joined #openstack-infra10:24
jheskethyolanda: I'm going to gracefully restart zl01'10:27
*** tosky has joined #openstack-infra10:27
*** mrunge has joined #openstack-infra10:28
*** roxanaghe has quit IRC10:29
*** Daviey has quit IRC10:29
*** mohankumar has joined #openstack-infra10:29
*** Daviey has joined #openstack-infra10:29
*** e0ne has joined #openstack-infra10:29
odyssey4mezigo I've provided some feedback.10:31
odyssey4me(in the review)10:31
*** amitgandhinz has joined #openstack-infra10:32
*** ramishra has quit IRC10:36
*** amitgandhinz has quit IRC10:37
*** ramishra has joined #openstack-infra10:37
*** gouthamr_ is now known as gouthamr10:40
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow tags to be filtered by name  https://review.openstack.org/33102710:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for Ubuntu Cloud Archive  https://review.openstack.org/33015210:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make the Tags resource look for existing tags to filter by  https://review.openstack.org/33103010:43
*** ociuhandu has joined #openstack-infra10:43
*** ramishra has quit IRC10:45
*** permalac has joined #openstack-infra10:47
*** rossella_s has quit IRC10:47
*** e0ne has quit IRC10:47
*** sree has joined #openstack-infra10:48
*** ramishra has joined #openstack-infra10:49
*** AJaeger has joined #openstack-infra10:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-ansible: Make services optional  https://review.openstack.org/33103610:50
*** ramishra has quit IRC10:50
*** mescanef has joined #openstack-infra10:51
*** dizquierdo has quit IRC10:53
*** hogepodge has joined #openstack-infra10:54
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Do not run any grenade jobs on branches older than mitaka  https://review.openstack.org/33104610:56
*** ramishra has joined #openstack-infra10:56
*** binbincong has quit IRC10:57
*** binbincong has joined #openstack-infra10:57
AJaegerinfra-root, there'S since 10 hours a change in the post queue for openstack-manuals  - two out of the 4 jobs are still queued. Could anybody check these, please?10:58
*** Adri2000 has joined #openstack-infra10:59
yolandaAJaeger, let me see logs. Unfortunately, i cannot help so much with new zuul-launcher, until I get more knowledge about it11:00
*** sree has quit IRC11:01
AJaegeryolanda: might as well be a bug - so feel free to wait for jeblair or pabelanger to check.... Just wanted to point it out...11:02
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Do not run any grenade jobs on mitaka and older branches  https://review.openstack.org/33104611:03
*** eezhova has quit IRC11:03
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Make the Tags resource look for existing tags to filter by  https://review.openstack.org/33103011:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for MariaDB  https://review.openstack.org/30783111:04
*** andreas_s_ has quit IRC11:04
*** dtantsur|bbl is now known as dtantsur11:04
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Use tag icon when tags are used as filter criteria  https://review.openstack.org/33105411:06
yolandaAJaeger, i see nodepool working fine, picking jobs. No stale ready nodes, no errors in log11:07
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Do not run any grenade jobs on mitaka and older branches  https://review.openstack.org/33104611:08
*** dizquierdo has joined #openstack-infra11:08
openstackgerritMerged openstack-infra/project-config: Add a build for the gerrit its-storyboard plugin  https://review.openstack.org/33079611:10
*** sree has joined #openstack-infra11:13
*** linuxmaniac has quit IRC11:15
*** linuxmaniac has joined #openstack-infra11:15
*** esikachev has quit IRC11:21
*** esikachev has joined #openstack-infra11:22
*** thorst_ has joined #openstack-infra11:23
*** saravana has quit IRC11:23
*** kzaitsev_mb has joined #openstack-infra11:25
*** roxanaghe has joined #openstack-infra11:25
*** amoralej is now known as amoralej|lunch11:26
*** kzaitsev_mb has quit IRC11:29
*** roxanaghe has quit IRC11:29
*** hashar is now known as hasharAway11:30
zigoodyssey4me: Replied. It will be used by other repos.11:33
*** amitgandhinz has joined #openstack-infra11:33
*** kzaitsev_mb has joined #openstack-infra11:35
*** aysyd has joined #openstack-infra11:35
*** ldnunes has joined #openstack-infra11:37
*** amitgandhinz has quit IRC11:38
*** lucasagomes is now known as lucas-hungry11:39
*** sree__ has joined #openstack-infra11:41
*** sree has quit IRC11:42
*** esikachev has quit IRC11:43
*** gouthamr has quit IRC11:43
*** DevBox has joined #openstack-infra11:44
*** esikachev has joined #openstack-infra11:45
pabelangermorning11:46
*** zhurong has joined #openstack-infra11:47
*** sree has joined #openstack-infra11:47
*** rcernin has joined #openstack-infra11:47
*** Jeffrey4l has quit IRC11:48
*** zhurong has quit IRC11:48
*** Jeffrey4l has joined #openstack-infra11:48
*** esikachev has quit IRC11:50
*** gouthamr has joined #openstack-infra11:50
*** sree__ has quit IRC11:51
*** eezhova has joined #openstack-infra11:52
*** eharney has joined #openstack-infra11:55
*** DevBox has quit IRC11:56
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class  https://review.openstack.org/32822811:58
*** jsavak has joined #openstack-infra11:58
jheskethyolanda: so zl01 restarted gracefully but it took a while... I'm going to start zl02, but I don't think I'll get through them all tonight... the fix for eavesdrop isn't critical though so I can pick it up tomorrow11:59
pabelangerjhesketh: this should be of interest: https://review.openstack.org/#/c/330184/12:00
jheskethMorning pabelanger :-)12:01
*** psilvad has joined #openstack-infra12:01
pabelangerjhesketh: which fix are you reverting too?12:01
jheskethpabelanger: huh, that is handy... I was about to write my own. Thakns!12:01
jheskethpabelanger: https://review.openstack.org/#/c/330986/12:01
*** gouthamr_ has joined #openstack-infra12:02
*** ramishra has quit IRC12:02
*** gouthamr_ has quit IRC12:02
*** gouthamr_ has joined #openstack-infra12:03
openstackgerritMerged openstack-infra/release-tools: fix the pattern matching in the constraint update logic  https://review.openstack.org/33054212:03
pabelangerjhesketh: Ah, ya. Didn't realize we'd need to graceful restart zuul to pick up new conf changes12:03
ikhudoshynAJaeger: aroung?12:04
jheskethpabelanger: yeah, we could look at reloading zuul.conf on reconfigure12:04
pabelangerjhesketh: Ya, about to say the same. Seems like a good things to add12:04
*** gouthamr_ has quit IRC12:04
jheskethpabelanger: yeah I'll work on that12:04
*** markvoelker has joined #openstack-infra12:04
*** gouthamr_ has joined #openstack-infra12:04
*** dmk0202 has quit IRC12:05
*** gouthamr has quit IRC12:05
pabelangerjhesketh: nice12:05
*** gouthamr_ is now known as gouthamr12:05
jheskethpabelanger: small nit on your playbook if you want to fix it12:05
pabelangerjhesketh: sure, looking12:06
*** weshay has joined #openstack-infra12:06
*** pradk has joined #openstack-infra12:06
*** pradk has quit IRC12:07
*** psilvad has quit IRC12:07
*** e0ne has joined #openstack-infra12:07
*** ramishra has joined #openstack-infra12:07
*** mtanino has joined #openstack-infra12:07
AJaegerikhudoshyn: now yes, what's up?12:08
*** sree has quit IRC12:08
*** rfolco has joined #openstack-infra12:08
AJaegeryolanda: thanks for looking.12:08
ikhudoshynAJaeger: it's about the patch, https://review.openstack.org/#/c/328249/. It's in a merge conflict12:08
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add playbook to graceful restart zuul-launchers  https://review.openstack.org/33018412:08
AJaegerpabelanger: is the openstack-manuals job in the post queue something for jeblair to check?12:08
ikhudoshyni'm about to rebase it, yet I promised you not to touch it12:08
AJaegerikhudoshyn: go ahead and rebase ;)12:09
ikhudoshynAJaeger: tnx)12:09
pabelangerjhesketh: odd, it didn't have the latest version in the review. I've updated it to use async for the wait_for command12:09
pabelangerAJaeger: Sure, I can look12:09
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users  https://review.openstack.org/32824912:09
ikhudoshynAJaeger: submitted new one ^^12:10
jheskethpabelanger: are you sure we need the timeout on the wait_for? won't the async timeout fail and then the any-errors-fatal kicks in?12:11
*** javeriak has quit IRC12:11
pabelangerjhesketh: Ya, I think we need to setup a timeout for it too, otherwise it defaults to 30 seconds.  But I agree, I'd be interested to see if that is actually the case.  When I tested locally, the syntax was valid12:12
pabelangerjhesketh: I should ask in #ansible12:12
*** javeriak has joined #openstack-infra12:12
*** javeriak has quit IRC12:12
*** pradk has joined #openstack-infra12:13
jheskethpabelanger: ah okay..12:13
pabelangerAJaeger: http://paste.openstack.org/show/516834/12:13
pabelangerthats from zuul.o.o12:13
pabelangerI think we recently fixed that12:13
jheskethpabelanger: yeah, there's a default of 300 http://docs.ansible.com/ansible/wait_for_module.html12:13
*** maishsk has quit IRC12:13
openstackgerritMerged openstack-infra/storyboard-webclient: New Favicon  https://review.openstack.org/32763912:14
pabelangerAJaeger: jeblair: I believe we need to restart zuul.o.o to fix the openstack-manuals post pipeline12:14
jheskethpabelanger: I'm doing graceful on zl0212:15
pabelangerjhesketh: okay12:15
pabelangerAJaeger: https://review.openstack.org/#/c/327274/ was the fix12:15
jheskethpabelanger: yeah, that'll need a restart :-(12:15
AJaegerpabelanger: thanks for checking12:16
pabelangerAJaeger: np12:16
*** ddieterly has joined #openstack-infra12:16
*** kzaitsev_mb has quit IRC12:17
*** rhallisey has joined #openstack-infra12:17
jheskethpabelanger: so the reconfigure for launchers is currently about reconfiguring jjb... should we provide two different types of reconfigures? the current one for jjb and anotehr one for launcher config (ie zuul.conf)?12:18
jheskethotherwise do them both at once maybe?12:18
pabelangerjhesketh: I think we could get away with doing 2 zuul-launchers at a time for restarts, but should wait to see what other infra-roots say.  I've see each zuul-launcher top out at 150 jobs each, so with 6 online we should be able to do 900 concurrent jobs12:18
*** jsavak has quit IRC12:18
jheskethpabelanger: I agree and would be more keen to do so if it was a more urgent change, but this isn't too bad12:18
pabelangerjhesketh: I think reconfigure is fine, but maybe jeblair has some thoughts12:19
openstackgerritosman proposed openstack-infra/jenkins-job-builder: Adds matrix restriction to trigger-builds plugin  https://review.openstack.org/33111212:19
jheskethokay, let me play with that..12:19
*** dprince has joined #openstack-infra12:19
pabelangerjhesketh: ya, I think in the case of urgent changes, we have the ability to use stop. that will simply terminal all active jobs, and requeue them into zuul.  Which simply means, the current patch will just longer over actually failing the job12:20
pabelangerterminate*12:20
jheskethyep12:20
pabelangerI like the fact, it will be hard for us to actually fail a job because of something we do to zuul-launcher configuration12:21
*** ramishra has quit IRC12:25
*** roxanaghe has joined #openstack-infra12:26
*** ddieterly has quit IRC12:26
jhesketh:-)12:29
*** roxanaghe has quit IRC12:30
AJaeger;)12:30
*** ramishra has joined #openstack-infra12:31
*** kzaitsev_mb has joined #openstack-infra12:31
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for standard zuul signals to launcher  https://review.openstack.org/33112312:34
*** jsavak has joined #openstack-infra12:34
jheskethpabelanger: first pass at reconfiguring ^12:34
*** edmondsw has joined #openstack-infra12:34
*** amitgandhinz has joined #openstack-infra12:34
*** kushal has quit IRC12:34
*** baoli has joined #openstack-infra12:34
pabelangerjhesketh: looking12:35
*** kushal has joined #openstack-infra12:35
*** baoli_ has joined #openstack-infra12:36
*** kzaitsev_mb has quit IRC12:36
*** piet has joined #openstack-infra12:37
openstackgerritJoshua Hesketh proposed openstack-infra/puppet-zuul: Reconfigure via sending a HUP  https://review.openstack.org/33112712:37
openstackgerritMerged openstack-infra/release-tools: treat missing metadata as a warning not an error  https://review.openstack.org/33076412:38
*** jsavak has quit IRC12:38
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Adds ironic v3 only job  https://review.openstack.org/30430612:39
*** amitgandhinz has quit IRC12:39
*** jsavak has joined #openstack-infra12:39
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Adds ironic v3 only job  https://review.openstack.org/30430612:40
*** rlandy has joined #openstack-infra12:40
*** alaski is now known as lascii12:40
*** baoli has quit IRC12:40
pabelangerjhesketh: left a comment12:41
*** kzaitsev_mb has joined #openstack-infra12:41
*** baoli_ has quit IRC12:41
*** kdas__ has joined #openstack-infra12:42
phschwartzSo I am running nodepool on a network that is in a private scope(192.168). I order to not try and get floating ip's assigned (we were out of floatings) I changed the networks name to public.12:42
phschwartzHow bad will it make things die if I rename the network since I now have floating ip's12:42
phschwartzAnd of course change my nodepool config12:42
*** bswartz has joined #openstack-infra12:43
*** Srinu has quit IRC12:44
pabelangerphschwartz: I wouldn't expect running nodes to be affected.  We recently did that to OSIC in nodepool.yaml and didn't need to schedule an outage12:44
jheskethpabelanger: thanks, replied12:44
phschwartzpabelanger: perfect. I can do the update then :)12:44
*** kushal has quit IRC12:44
*** kdas__ is now known as kushal12:45
*** kushal has joined #openstack-infra12:46
mordredpabelanger: you figured out telnet:// handling in chrome? any chance you accidentally figured out firefox too?12:46
*** gordc has joined #openstack-infra12:46
pabelangermordred: ya, only (badly) hacked chrome last night12:47
mordredpabelanger: I spent some time trolling the internet and so far have learned no useful information12:48
*** ramishra has quit IRC12:48
*** shahid__ has quit IRC12:49
*** pradk has quit IRC12:50
openstackgerritosman proposed openstack-infra/jenkins-job-builder: Adds matrix restriction to trigger-builds plugin  https://review.openstack.org/33111212:50
*** lucas-hungry is now known as lucasagomes12:50
pabelangermordred: if you can get firefox to use xdg-open for telnet://, I think you could use the same logic I do with chrome.12:51
pabelangerwill give a test in a few minutes12:51
jheskethpabelanger: I've got to head off... zl02 is shutting down still, could you please bring it back up once its done?12:51
pabelangerjhesketh: sure12:51
jheskethonly 10 more nodes to go12:52
openstackgerritMerged openstack-infra/storyboard-webclient: Add a way to make time-moment not use am-time-ago  https://review.openstack.org/32947812:52
*** maishsk has joined #openstack-infra12:52
jheskethI can do the other launchers tomorrow12:52
jheskethpabelanger: thanks :-)12:52
*** ddieterly has joined #openstack-infra12:53
*** ccamacho is now known as ccamacho|lunck12:53
*** matt-borland has joined #openstack-infra12:53
*** ccamacho|lunck is now known as ccamacho|lunch12:54
jheskethNight all!12:55
AJaegergood night, jhesketh !12:55
phschwartzis there a command to remove 1 job from zuul? I have never had a need to do it before, but it looks like an abandon'ed review has borked my check queue as it is enqueue without a jenkins job to run12:59
*** psachin has quit IRC13:00
pabelangermordred: got it working, let me get directions13:00
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class  https://review.openstack.org/32822813:01
*** dizquierdo has quit IRC13:02
phschwartzhmm, puppetdb added to openstackci... Interesting.13:02
rcarrillocruzvery rough13:02
rcarrillocruzi'm wip'ing13:03
rcarrillocruzi need systtem-config module to be bumped to 4.1 first13:03
rcarrillocruzcos that version exposes ssl params13:03
pabelangermordred: https://etherpad.openstack.org/p/firefox-telnet _should_ work13:03
*** amoralej|lunch is now known as amoralej13:03
*** trown|outtypewww is now known as trown13:04
*** zhs_ss has quit IRC13:05
*** inc0 has joined #openstack-infra13:07
mordredpabelanger: woot. that worked - I was missing expose = False before13:07
AJaegerdims: could you review 329440, please? I wonder whether all projects need these oslo-jobs jobs now - or what's the proper set of projects to run them13:08
*** zhurong has joined #openstack-infra13:08
pabelangermordred: nice13:09
EmilienMit seems like TripleO CI is really down13:09
fungiphschwartz: https://review.openstack.org/95035 but i believe it still needs some polish13:09
*** mdrabe has joined #openstack-infra13:09
mordredpabelanger: we should maybe write this down somewhere for people13:09
mordredpabelanger: infra-manual perhaps?13:09
fungiEmilienM: they were relocating the hardware, right? has that started yet?13:09
EmilienMfungi: ok, I missed some info here thx13:10
*** xyang1 has joined #openstack-infra13:10
pabelangermordred: Ya, we likely should do that.13:10
*** thiagop has joined #openstack-infra13:10
phschwartzfungi: I will see if I can get that rolling again, till then, I take it there is no way to remove the borked job, right?13:10
fungiphschwartz: restart zuul?13:11
mordredpabelanger: I'll start a patch13:11
phschwartzfungi: that will make us loose all queued behind the borked one which I was trying to prevent13:11
EmilienMfungi: I'm not sure it's related in fact, it fails in the middle of overcloud deployment13:11
openstackgerritMerged openstack-infra/storyboard-webclient: Rework timeline events into a timeline  https://review.openstack.org/32723113:11
EmilienMfungi: but nevermind, probably something on our side13:11
*** Hal has joined #openstack-infra13:11
*** ddieterly has quit IRC13:11
*** kgiusti has joined #openstack-infra13:12
*** Hal is now known as Guest2811313:12
fungiEmilienM: oh, sorry, i interpreted "down" in a more literal sense13:12
EmilienMfungi: sorry13:12
pabelangerEmilienM: fungi: agreed, I still see tripleo-centos-7 nodes online13:13
AJaegerEmilienM, fungi: See https://review.openstack.org/#/c/330514/ - so there's a planned move13:13
EmilienMAJaeger: can we land it?13:14
EmilienMslagle, dprince ^ is it ok to switch to rh2 ? ^13:14
AJaegerhttps://review.openstack.org/#/c/330833/ is the system-config change that needs to land first. fungi, pabelanger, do you want to review?13:15
dprinceEmilienM: I think that is a question for derekh. I believe he has good results on that cloud but unclear to me that switching today (when derek is on PTO) is the plan13:17
dprincesorry, best plan13:17
EmilienMdprince: well, tripleo.org/cistatus.html13:17
EmilienMthere is a problem atm ^13:17
EmilienMdprince: is it related to the move or something?13:17
dprinceEmilienM: switching to an entirely new cloud because we have a CI failure on the old one may break more than it fixes :)13:18
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class  https://review.openstack.org/32822813:18
dprinceEmilienM: or not. let me look at the failures13:18
EmilienMdprince: http://logs.openstack.org/85/330785/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/18f9b94/logs/postci.txt.gz#_2016-06-17_04_01_58_00013:18
EmilienMAJaeger: can we hold this one please? https://review.openstack.org/#/c/330514/13:21
EmilienMAJaeger: until dprince and I find out why our CI is broken first - thanks a lot13:21
AJaegerEmilienM: sure...13:22
EmilienMthanks13:22
AJaegerEmilienM: I set a WIP13:22
EmilienMperfect13:22
*** akshai has joined #openstack-infra13:23
rcarrillocruzso folks13:23
rcarrillocruzany recommendation for a chromium extension for telnet13:24
rcarrillocruzmordred , pabelanger ^13:24
rcarrillocruzyou using it already via browser ?13:24
pabelangerrcarrillocruz: https://etherpad.openstack.org/p/google-telnet13:24
pabelangermind you, pretty hacky13:24
rcarrillocruzthx13:24
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests  https://review.openstack.org/32959013:26
fungichromium doesn't just have a protocol to application mapping you can configure?13:26
*** roxanaghe has joined #openstack-infra13:27
*** binbincong has quit IRC13:27
fungi(map telnet protocol to "xterm -c nc $host $port" or whatever?)13:27
*** abregman|afk has joined #openstack-infra13:27
Shrewsdoesn't look like the launchers are running with all the latest goodies13:27
pabelangerfungi: it might, but that's what I got it working last night.  It was pretty late13:27
fungiShrews: "latest goodies" meanining job config updates? zuul patches? something else?13:30
mordredfungi: well, something has to split the url into $host $port13:30
Shrewsfungi: zuul patches13:30
fungimordred: fair point. wrapper script to |sed 's/:/ /'13:30
mordredyah13:31
fungi(mailercode excuse)13:31
*** inc0 has quit IRC13:31
*** mriedem has joined #openstack-infra13:31
*** roxanaghe has quit IRC13:31
*** inc0 has joined #openstack-infra13:31
*** jamesmcarthur has joined #openstack-infra13:32
fungiShrews: all have "zuul==2.1.1.dev321  # git sha aaa852e" installed but maybe some need restarts?13:33
fungii saw jhesketh mention something about working through graceful restarts before i awoke13:33
pabelangerShrews: zl01 was restarted my jhesketh this morning13:35
dimsAJaeger : asked a question and added Josh to the review13:35
pabelangerzl02 is in graceful mode now13:35
*** mtanino has quit IRC13:35
AJaegerthanks, dims13:36
openstackgerritMerged openstack-infra/system-config: Add rh2 cloud credentials  https://review.openstack.org/33083313:37
*** ddieterly has joined #openstack-infra13:38
EmilienMpabelanger: we noticed that tripleo jobs start to fail when decommissionning all jenkins slaves13:38
*** nwkarsten has joined #openstack-infra13:39
EmilienMwe're wondering if some ENV used by zuul or something might break us13:39
pabelangerEmilienM: Hmm, I thought I see tripleo-ci passing on zuul-launcher before13:39
dprinceyes, it would seem so. Does the zuul runner perhaps have different timeouts, env variables or something?13:40
pabelangerseen*13:40
EmilienMthis is an example of console.log of failing job: http://logs.openstack.org/94/330694/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/8fff5e5/console.html13:40
*** maishsk_ has joined #openstack-infra13:40
*** rbrndt has joined #openstack-infra13:40
*** mixos has joined #openstack-infra13:41
*** dansmith is now known as superdan13:41
pabelangerEmilienM: dprince: I don't see the job timing out, but simply failing: http://logs.openstack.org/94/330694/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/8fff5e5/_zuul_ansible/logs/ansible.log13:41
pabelangeratleast from zuul-launcher POV13:41
EmilienMhttp://logs.openstack.org/94/330694/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/8fff5e5/console.html#_2016-06-16_21_22_37_28428113:42
EmilienMis it a big deal ^?13:42
dprincepabelanger: true, just noticed the failed jobs running faster13:42
pabelangerEmilienM: no, we are still using jenkins, but will be switching to zuul in the future13:42
dprincepabelanger: perhaps new ENV variables are changing our job behavior somehow?13:42
pabelangerdprince: oh, that is possible. We aren't using java anymore to run the process on the remote host.  Commands are directly over SSH13:43
*** amitgandhinz has joined #openstack-infra13:43
*** maishsk has quit IRC13:43
*** maishsk_ is now known as maishsk13:43
pabelangerdprince: possible, but I am sure the I've seen tripleo CI passing on zuul-launcher over the last week13:43
dprincepabelanger: link to the job?13:44
*** _ari_ has joined #openstack-infra13:44
dprincepabelanger: perhaps that was experimental and we there is a slightly different setting that made it work or something13:44
EmilienMdprince: did timeout changed ?13:45
EmilienMhttp://logs.openstack.org/94/330694/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/8fff5e5/console.html#_2016-06-16_21_32_45_02472113:45
EmilienMon a successful job: http://logs.openstack.org/14/329714/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/5f84c27/console.html#_2016-06-16_19_21_13_88213:45
*** HeOS has quit IRC13:45
EmilienMfrom 153 to 97 min?13:45
*** piet has quit IRC13:45
*** links has quit IRC13:46
pabelangerdprince: looking for a job13:47
*** hongbin has joined #openstack-infra13:47
dprincepabelanger: no worries if you can't find it. Just be good to have something to compare too13:47
*** amitgandhinz has quit IRC13:48
pabelangerEmilienM: there was an issue where build-timeout wasn't being setup properly under zuul. but I thought we fixed that13:48
*** piet has joined #openstack-infra13:48
*** sarob has joined #openstack-infra13:48
*** piet has quit IRC13:49
*** dingyichen has quit IRC13:49
*** PollyZ has joined #openstack-infra13:50
pabelangerzl02 restarted13:50
*** Goneri has joined #openstack-infra13:50
*** e0ne has quit IRC13:51
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts  https://review.openstack.org/33114813:52
*** e0ne has joined #openstack-infra13:52
EmilienMit sounds like a timeout thing really13:52
*** rodrigods has quit IRC13:52
*** rodrigods has joined #openstack-infra13:52
EmilienMwhen we used jenkins: Job timeout set to: 153 minutes13:52
EmilienMand now Job timeout set to: 97 minutes13:53
EmilienMlooking at http://tripleo.org/cistatus.html13:53
pabelangermaybe we're not exporting BUILD_TIMEOUT correctly now13:53
EmilienMall jobs fail at ~ 1h30 (90 min)13:53
*** sarob has quit IRC13:53
EmilienMdprince: the thing is, I'm really surprised to see jobs > 1h3013:53
EmilienMdprince: nonha takes usually less time13:53
EmilienM~1h or so13:53
EmilienMbut looking at http://tripleo.org/cistatus.html it seems it took more recently13:54
*** armax has joined #openstack-infra13:54
EmilienMif you scroll down, you'll see some jobs taking 1h20 or so13:54
EmilienMso we're a bit longer now, probably because we add more services etc13:54
EmilienMpabelanger: I have no  idea who we can increase the timeout in zuul13:54
*** zhurong has quit IRC13:55
*** Goneri has quit IRC13:55
*** pradk has joined #openstack-infra13:56
*** zhurong has joined #openstack-infra13:56
pabelangerwe should be passing BUILD_TIMEOUT into the env for our subprocess13:56
pabelangerEmilienM: we use the build-timeout wrapper13:57
*** xarses_ has quit IRC13:57
*** mriedem has quit IRC13:58
*** mriedem has joined #openstack-infra13:59
*** burgerk has joined #openstack-infra13:59
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: DO NOT MERGE  https://review.openstack.org/33115113:59
*** zhurong has quit IRC14:00
EmilienMpabelanger: we'll know in ~12 min14:02
EmilienMpabelanger: the time to run this script14:02
openstackgerritPaul Belanger proposed openstack-infra/devstack-gate: DO NOT MERGE  https://review.openstack.org/33115614:03
*** PollyZ has quit IRC14:03
*** inc0 has quit IRC14:04
*** inc0 has joined #openstack-infra14:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote existing OSA non-voting Xenial tests to voting  https://review.openstack.org/33115714:06
*** ddieterly is now known as ddieterly[away]14:06
*** sigmavirus24_ is now known as sigmavirus2414:06
*** zhurong has joined #openstack-infra14:07
*** PollyZ has joined #openstack-infra14:07
pabelangerEmilienM: Yup, BUILD_TIMEOUT is not being passed properly http://logs.openstack.org/56/331156/1/check/gate-tempest-dsvm-full-mitaka-nv/1f7b5e0/console.html#_2016-06-17_14_06_18_27060414:07
*** eezhova has quit IRC14:07
openstackgerritPaul Belanger proposed openstack-infra/devstack-gate: DO NOT MERGE  https://review.openstack.org/33115614:07
EmilienMdamn14:08
EmilienMpabelanger: nice catch14:08
pabelangerEmilienM: checking to see why14:08
AJaegerstrange, what happend with https://review.openstack.org/#/c/328298/ ?14:09
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs  https://review.openstack.org/24643314:10
*** andreykurilin has joined #openstack-infra14:10
andreykurilinhi everyone!14:10
EmilienMpabelanger: can it be in zuul/launcher/ansiblelaunchserver.py ?14:11
pabelangerEmilienM: yes, this will be in zuul-launcher14:11
*** rbergeron has quit IRC14:11
*** rbergeron has joined #openstack-infra14:11
andreykurilinCould someone look at novaclient's change ( https://review.openstack.org/#/c/291370/ )? It had W+1, but verify check was not launched. Rebase did not help.14:11
EmilienMAJaeger: yeah I was looking at this one14:11
*** PollyZ has quit IRC14:12
AJaegerandreykurilin: could you remove the +A and add it back again?14:12
*** sdake has joined #openstack-infra14:12
andreykurilinAJaeger: done14:12
pabelangerEmilienM: I think I see the issue14:12
AJaegerandreykurilin: wrong solution, found it. The job is stacked on top of https://review.openstack.org/#/c/240277/5 is14:13
*** ddieterly[away] is now known as ddieterly14:13
AJaegerAnd as long as 240277 is not merged, it will not merge either...14:14
*** eezhova has joined #openstack-infra14:14
andreykurilinoh14:14
AJaegerandreykurilin: see "Related Changes" in upper left corner for the stacked changes14:14
andreykurilinAJaeger: yeah... I missed it sometimes in new gerrit ui:(14:14
AJaegerYeah ;(14:14
*** Jeffrey4l has quit IRC14:15
andreykurilinAJaeger: thanks14:15
*** zul has quit IRC14:16
openstackgerritPaul Belanger proposed openstack-infra/zuul: Fix typo with timeout wrappers  https://review.openstack.org/33117014:16
*** zul has joined #openstack-infra14:16
EmilienMwhat?14:17
EmilienMthe wrapper name is build-timeout14:17
EmilienMpabelanger: ^14:17
openstackgerritMerged openstack-infra/devstack-gate: Set read/search perms recursively  https://review.openstack.org/33083514:17
*** jamesmcarthur has quit IRC14:17
*** amitgandhinz has joined #openstack-infra14:17
pabelangerEmilienM: I am testing14:17
*** amitgandhinz has quit IRC14:18
*** jamesmcarthur has joined #openstack-infra14:18
*** jamesmcarthur has quit IRC14:18
*** amitgandhinz has joined #openstack-infra14:18
*** xarses_ has joined #openstack-infra14:18
EmilienMpabelanger: ok I think you're right, we're already in 'wrappers' and we're looking for the next key, not the wrapper name14:20
fungiyep, it's in wrappers[build-timeout][timeout]14:20
*** hemanthm|afk is now known as hemanthm14:20
pabelangerI've abandoned for now14:20
pabelangerbecause we are setting up timeout some how14:20
pabelangerwant to understand that first14:21
*** zul_ has joined #openstack-infra14:21
fungipabelanger: no, i think your change looks correct14:21
*** mtanino has joined #openstack-infra14:21
fungiwe're doing "for wrapper in wrappers..."14:21
EmilienMyes, I think it's ok too14:21
fungiso wrapper inside the loop is wrappers[built-timeout] already, hence we're looking for wrapper[timeout]14:22
*** ccamacho|lunch is now known as ccamacho14:22
pabelangermy concern is, ANSIBLE_DEFAULT_TIMEOUT is 720014:22
*** zul has quit IRC14:23
EmilienMit's 120 min14:23
pabelangerbut I see -e timeout=10800 -v in our process14:23
pabelangerwhere is 10800 coming from14:23
fungioh, er, your change isn't quite correct, but i think the code there is wrong nonetheless14:23
pabelangerya14:23
EmilienMI found where it comes from14:23
pabelangersomething is a little weird, let me see if I can better understand14:24
EmilienMhttp://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/elements/testenv-worker/testenv-worker#n18014:24
*** maishsk has quit IRC14:24
pabelangerEmilienM: no, this in on zuul-launcher14:24
EmilienMalso http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/testenv-client#n14114:24
*** kushal has quit IRC14:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote existing OSA non-voting CentOS-7 tests to voting  https://review.openstack.org/33119114:27
pabelangerI think I see the issue14:27
*** roxanaghe has joined #openstack-infra14:28
fungibasically, line 1089 is looking for wrappers[built-timeout][build-timeout] which won't exist and ends up with build_timeout set to {}14:28
pabelangeris it possible that code is looking at the job_template wrappers section?14:28
pabelangerwhich does have a wrappers: build-timeout14:28
jeblairo/14:28
pabelangerand a timeout var14:28
*** Jeffrey4l has joined #openstack-infra14:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote existing OSA non-voting CentOS-7 tests to voting  https://review.openstack.org/33119114:28
pabelangerbut, does not have timeout-var14:28
fungipabelanger: it's looking at the job, which will be formed from the job-template, which yes14:28
pabelangerjeblair: oah14:28
EmilienMjeblair: hey, we're looking https://bugs.launchpad.net/tripleo/+bug/159374014:29
openstackLaunchpad bug 1593740 in tripleo "CI jobs timeout at 97 minutes instead of 153" [Critical,Confirmed] - Assigned to Paul Belanger (pabelanger)14:29
pabelangerfungi: ya, I was assuming it was some how using the wrapper in macro.yaml14:29
fungipabelanger: so anyway, i think it may make more sense to move the timeout processing out of that for loop entirely14:29
pabelangerwhich is where timeout-var is setup14:29
*** flepied has quit IRC14:29
fungiwrappers = jjb_job.get('wrappers', [])14:30
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class  https://review.openstack.org/32822814:31
fungior something14:31
jeblair /win 2114:31
pabelangerjjb_job['wrappers']['wrapper']['build-timeout']['wrappers] ?14:32
sc68calfound the other irssi user14:32
*** david-lyle has joined #openstack-infra14:32
fungipabelanger: no, wait, i'm brainparsing yaml incorrectly14:32
pabelanger_shugs_ my brains can't parse14:32
AJaegerjeblair, fungi, pabelanger : Could you restart zuul later today? We have some changes in post queue that have no jobs run - according to pabelanger this is fixed since last night and needs zuul restart14:32
*** roxanaghe has quit IRC14:32
funginow i see wrappers is a list, whose elements are nested dicts, so i think the original code is correct14:33
*** e0ne has quit IRC14:34
pabelangerfungi: I think this is what it is parsing: http://paste.openstack.org/show/516872/14:34
pabelangerwhich lacks timeout-var14:34
*** PollyZ has joined #openstack-infra14:34
jeblairfungi, pabelanger: http://paste.openstack.org/show/516873/   produces timeout=600 for me14:34
fungibasically one of the elements of the 'wrappers' list will be { 'build-timeout': { 'timeout': 1234 } }14:34
pabelangerfungi: I was expecting: http://paste.openstack.org/show/516875/14:34
dprincepabelanger: so there is no switch back to Jenkins just for #tripleo jobs... while we sort this out option?14:35
*** e0ne has joined #openstack-infra14:35
pabelangerdprince: no, we've delete jenkins14:35
fungiAJaeger: pabelanger: what was the fix merged for the jobs stuck queued in post?14:35
fungidprince: there are no jenkins servers14:35
dprinceboom14:35
fungidprince: the last of them were deleted yesterday14:35
dprincegot it14:35
pabelangerfungi: yes, we need to restart zuul14:35
fungipabelanger: that's not what i asked14:36
fungiAJaeger: pabelanger: what was the fix merged for the jobs stuck queued in post?14:36
*** zz_dimtruck is now known as dimtruck14:36
*** zhurong has quit IRC14:36
fungi(change number or hyperlink, or project and title, something i can query)14:36
fungi(git sha)14:37
AJaegerfungi https://review.openstack.org/33072514:37
fungiwas it "Ansible launcher: delay node assignment under load" ?14:37
AJaegerfungi, sorry, wrong link ;(14:38
openstackgerritSean McGinnis proposed openstack-infra/project-config: Add driver compliance checks to check pipeline  https://review.openstack.org/33119214:38
pabelangerfungi: sorry, confused now. Is we are talking about post pipeline issue with manual jobs, I think it is hitting https://review.openstack.org/#/c/327274/14:38
* AJaeger tries again https://review.openstack.org/#/c/327274/14:38
jeblairwhy would that have caused something to get stuck in post?14:39
jeblairoh i think i see how it could be possible14:39
jeblairwhy do we think the tripleo problem is related to zuul launcher timeouts?14:40
AJaegerhave to leave now, might be back later...14:40
pabelangerjeblair: I don't believe we are setting BUILD_TIMEOUT correctly in the env now14:41
jeblairdid anyone push up a test that echos $BUILD_TIMEOUT?14:41
pabelangeryes14:42
*** dprince has quit IRC14:42
pabelanger331156 is a quick test14:42
smcginnisHas there been any talk/exploration of using containers for some gate tests? Seems like things like pylint and pep8 jobs could be a lot faster if they just all ran on one instance in their own containers.14:42
pabelangeradded env and echo14:42
*** bhavik has joined #openstack-infra14:42
pabelangerI currently believe, http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n1091 will always be None14:42
pabelangerbecause I didn't properly understand our jjb_job.get('wrappers', []) code14:43
jeblairoh i know the problem14:43
rcarrillocruzsmcginnis: there was, but nothing concrete...14:43
*** dprince has joined #openstack-infra14:43
pabelangerjeblair: great14:43
smcginnisrcarrillocruz: OK, cool. Just something that popped in my head. :)14:43
fungismcginnis: those already run in a virtualenv. what more isolation is a container going to provide there?14:44
smcginnisfungi: Just thinking of the cost of spinning up VM instances. Or are they in their own venv on one instance?14:44
*** ihrachys has quit IRC14:45
*** baoli has joined #openstack-infra14:45
fungismcginnis: we used to do that a long time ago. it turned out to be a lot more costly in sysadmin time to maintain those servers than it was to discard them and let projects run whatever they want in a test instead14:45
fungismcginnis: so you could certainly design a job that ran both unit tests and static analysis if you wanted14:45
fungipabelanger: i understand the wrappers loop now. i was misparsing the yaml for it in my mind14:46
fungipabelanger: and the code looks right to me14:46
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set timeout env var in ms  https://review.openstack.org/33119914:46
jeblairpabelanger, fungi: ^14:46
*** flepied has joined #openstack-infra14:48
*** jamesmcarthur has joined #openstack-infra14:48
mordredjeblair: yay for units!14:49
mordredjeblair: at least we didn't mess up space flight due to metric/imperial14:49
rcarrillocruzlol14:49
pabelangerjeblair: I still have a concern that BUILD_TIMEOUT isn't actually in our env however14:49
pabelangeris it possible line 1091 is always None?14:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote existing OSA experimental Xenial jobs to non-voting  https://review.openstack.org/33120314:50
*** hasharAway is now known as hashar14:52
*** sorantis has quit IRC14:52
*** mescanef has quit IRC14:52
fungipabelanger: you have to look into jjb for that14:52
pabelangerya, that is what I am testing now14:53
*** ajmiller has quit IRC14:53
*** ajmiller has joined #openstack-infra14:53
*** krtaylor has joined #openstack-infra14:53
*** ramishra has joined #openstack-infra14:54
*** pcaruana has quit IRC14:54
fungipabelanger: so looking at the macro, if we provide a build-timeout wrapper, we set timeout to whatever we passed as timeout in the job(-template) and then we set timeout-var to BUILD_TIMEOUT http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n87514:54
fungipabelanger: oh, hrm, so thinking back through this, 331170 may be necessary after all14:55
fungijeblair: ^14:55
fungiour macro name is build-timeout, but the actual wrapper is called timeout14:56
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure timeout values are the correct type  https://review.openstack.org/33121014:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote existing OSA experimental CentOS-7 jobs to non-voting  https://review.openstack.org/33121114:56
jeblairfungi, pabelanger: without 331210 i was able to cause jobs to not run at all, but did not see them run successfully without a build_timeout var14:57
jeblairit may be that we are still not setting it correctly, and once we figure it out, we would need 33121014:57
*** inc0 has quit IRC14:57
pabelangerhttp://paste.openstack.org/show/516882/14:57
jeblairi will look into what fungi is suggesting by pointing my local copy at the full job set and introspecting14:57
*** flepied has quit IRC14:57
*** inc0 has joined #openstack-infra14:57
pabelangerthat is what wrappers contains14:57
pabelangerwhich is the wrappers from the job_template14:58
pabelangernot macros14:58
pabelangerand we set BUILD_TIMEOUT in timeout-var14:58
fungipabelanger: is that from parsing the yaml directly, or echoing the expanded data structure from jjb?14:58
*** roxanaghe has joined #openstack-infra14:59
pabelangerfungi: Ya, echo. I did print self.jobs in loadLoads()14:59
pabelangerno where in the jobs, is timeout-var or BUILD_TIMEOUT exposed15:00
pabelangerself.jobs15:00
*** gouthamr has quit IRC15:01
*** gouthamr has joined #openstack-infra15:01
fungithere was some work done to make jjb libified enough to be able to extract the parsed and expanded tree, but i wonder if the routine that substitutes builders and publishers with macros got moved while the routine which does the same for wrappers didn't15:01
fungiwill take a few minutes to reacquaint myself with that part of jjb15:02
*** vincentll has quit IRC15:02
openstackgerritMerged openstack-infra/zuul: Ansible launcher: set timeout env var in ms  https://review.openstack.org/33119915:02
*** inc0 has quit IRC15:03
*** roxanaghe has quit IRC15:03
jeblairfungi: oh, i think you've got it15:04
jeblairfungi: i think we're missing wrapper expansion in zuul15:04
jeblairfungi: (because jjb actually can't do the thing you just said)15:05
*** tesseract has quit IRC15:05
fungithat explains why i couldn't find the relocation i was expecting within jjb15:06
fungididn't remember that ended up reimplemented in zuul15:06
*** rcernin has quit IRC15:07
pabelangerokay cool15:07
*** dprince has quit IRC15:08
*** amotoki has joined #openstack-infra15:09
jeblairthat may be at issue, but i'm not convinced that's it yet15:09
*** dprince has joined #openstack-infra15:09
jeblairjust looking at gate-tempest-dsvm-full -- it defines its own wrappers.  it has a build-timeout wrapper with no var15:09
openstackgerritDong Ma proposed openstack-infra/jenkins-job-builder: Use 'false' and 'true' in default docstrings  https://review.openstack.org/33122015:10
jeblairiow, i believe that jobs from particular job template will correctly not have a timeout var set15:10
*** dprince has quit IRC15:11
jeblairfungi, pabelanger: oh, wait -- where are we on the 'build-timout' vs 'timeout' name? :)15:12
pabelangerjeblair: confused :D15:12
pabelangerAs I understand it, we should be passing BUILD_TIMEOUT for every job that is using the build-timeout wrapper from macros.yaml15:13
pabelangerI believe that was the functionality in jenkins15:13
jeblairi think the wrapper is called timout, and the macro is called build-timeout15:14
pabelangerI believe that is correct15:14
*** electrofelix has quit IRC15:15
pabelangerin the case of 'gate-dg-hook-dsvm' in jenkins/jobs/devstack-gate.yaml. I believe zuul-launcher is using that wrappers to extract our timeout and timeout-var.  And because timeout-var is missing, we are never setting it in zuul-launcher15:16
jeblairso i think we are not substituting the macros and we have the wrong wrapper name (weirdly, the combination of those things causes this to partially work -- just not setting the env var)15:16
pabelangerdefinitely weirdness15:17
openstackgerritJulien Danjou proposed openstack-infra/project-config: Disable Tempest plugins gate for Telemetry in stable/liberty  https://review.openstack.org/33122515:19
fungijeblair: yes, i agree. pabelanger's 331170, your 331199 and a fix for expanding wrappers macros are all going to be needed to solve this15:20
*** sdake has quit IRC15:20
*** flepied has joined #openstack-infra15:21
*** mdrabe has quit IRC15:24
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Fix timeout processing  https://review.openstack.org/33121015:25
*** afazekas|dentist is now known as afazekas15:25
*** zeih has quit IRC15:25
jeblairpabelanger, fungi: ^ please look closely at that15:25
pabelangerjeblair: looking15:26
fungijeblair: i think there's still a missing level15:27
fungijeblair: the parsed wrapper will actually be { 'timeout': { 'timeout': 12345 } }15:27
jeblairfungi: ?15:27
*** mixos has quit IRC15:27
*** vgridnev has joined #openstack-infra15:28
fungiat least looking at the macro we have15:28
jeblairfungi: line 1089 is the 1st and 1092 is the 2nd15:28
vgridnevAJaeger, could you please take another look on https://review.openstack.org/#/c/330125/ ?15:28
fungijeblair: d'oh, yep. misread15:28
pabelangerthis is what I am seeing now: http://paste.openstack.org/show/516890/ which looks better15:29
pabelangerchecking patch now15:29
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Fix timeout processing  https://review.openstack.org/33121015:29
jeblairfungi, pabelanger: that's just a style change ^  (dropping the None default values)15:30
fungiyeah, .get() will default to None on its own15:30
fungii was just confirming that15:30
jeblairwe had a mixture of both styles together which is potentially confusing as to intent15:31
pabelangerokay, I think that is correct15:31
pabelangerseems to line up with what I am expecting now15:31
jeblairi don't normally mind that sort of thing, but two lines next to each other doing the same thing different ways is weird15:31
jeblair(and OrderedDict behaves the same way with .get())15:31
pabelangerjeblair: +215:32
*** rajinir has joined #openstack-infra15:32
*** pgadiya_ has quit IRC15:32
jeblairi love that the input to the build timeout plugin is seconds and the output is ms15:33
*** dizquierdo has joined #openstack-infra15:34
*** mriedem has quit IRC15:35
*** andreww has joined #openstack-infra15:35
*** esikachev has joined #openstack-infra15:36
*** xarses_ has quit IRC15:36
*** hashar is now known as hasharAway15:36
*** sree has joined #openstack-infra15:36
fungii thought the input to the build timeout plugin was in minutes?15:37
jeblairfungi: er, yes it is :)15:37
fungi(we have timeout * 60 in there)15:37
jeblaireven better15:37
*** vhosakot has joined #openstack-infra15:38
jeblairfungi, pabelanger: when 210 lands, i'm thinking we should go ahead and do a full system restart15:38
jeblairbecause we have a scheduler fix that needs to go in15:38
*** mriedem has joined #openstack-infra15:38
pabelangersure, we should test out our playbooks15:38
jeblairso might as well down all the launchers at the same time to speed that up15:38
jeblairpabelanger: no, i don't mean graceful15:39
pabelangerOh15:39
*** yamahata has joined #openstack-infra15:39
pabelangeractually, that would be fun to watch15:39
jeblairpabelanger: because we need to restart zuul, which is not graceful15:39
jeblair(well, it can be, but it would take all day)15:39
pabelangerright15:39
jesusaurI seem to have hit a strange error with my log_processor workers, most of them crashed without writing any error to log; has anyone else seen this happen?15:39
pabelangerwould could update the playbook to by dynamic in that case15:40
pabelangereither way15:40
*** tphummel has joined #openstack-infra15:40
pabelangerchange for another day15:40
krotscheckWhat is the purpose of yasfb in the infra specs module? I'm trying to figure out if I need it.15:40
krotscheck(for javascript docs)15:40
*** ajmiller_ has joined #openstack-infra15:40
*** hosanai has joined #openstack-infra15:41
mordredkrotscheck: rss feed generation15:41
krotscheckmordred: Ahhhh, that makes sense15:41
krotscheckmordred: Do we have an idea of how many people use that?15:42
mordredkrotscheck: I do not15:42
fungikrotscheck: makes some sense for specs, less sense for mostly static docs15:43
krotscheckfungi: I agree.15:43
rcarrillocruzhmm15:43
rcarrillocruznibalizer:15:43
rcarrillocruzaround15:43
*** bhavik has quit IRC15:43
rcarrillocruz?15:43
*** ajmiller has quit IRC15:43
rcarrillocruzhttps://docs.puppet.com/puppetdb/3.2/release_notes.html , shows puppetDB only supports puppet 4.x15:44
rcarrillocruzwe still on puppet 3.x on nodes, no?15:44
*** flwang has quit IRC15:44
krotscheckfungi: That suggestion of yours yesterday workd.15:45
krotscheckOne question: pip install Sphinx -> put that in the makefile? I'm thinking yes.15:45
njohnstonHi!  Could I get a second +2 for https://review.openstack.org/329570 please?  Thanks much!15:45
*** inc0 has joined #openstack-infra15:46
*** ddieterly is now known as ddieterly[away]15:47
*** flwang has joined #openstack-infra15:48
smcginnisAnyone have a moment to tell me what I'm doing wrong here: https://review.openstack.org/#/c/331192/15:48
fungikrotscheck: again, i would probably use distro packages for tools you need in your docs build, document that they're needed, and just assume they're preinstalled15:48
fungikrotscheck: i would be surprised if i ran a makefile and it downloaded things from the internet15:49
smcginnisLooks to me like it should be fine, but job fails with a message saying the job I added isn't defined.15:49
*** bpokorny has joined #openstack-infra15:49
openstackgerritMerged openstack-infra/zuul: Ansible launcher: Fix timeout processing  https://review.openstack.org/33121015:49
*** esikachev has quit IRC15:49
*** Guest31553 is now known as redrobot15:50
*** mdrabe has joined #openstack-infra15:51
*** bhavik has joined #openstack-infra15:51
smcginnisHmm, and it is listed as a job earlier in the output. Confused.15:51
*** sdake has joined #openstack-infra15:51
anteayasmcginnis: there has to be a coresponding job definition for the jenkins job builder which draws from http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n76315:52
jeblairfungi, pabelanger: i'll restart zuul and all launchers after the next puppet run -- should be around 16:1015:52
smcginnisanteaya: Ah, thanks. Too many moving parts I guess.15:52
anteayasmcginnis: and I've never heard of this job name before, you might have to build the definition15:52
pabelangerfungi: jeblair: ack, I need to step away from my desk but back at 16:3015:52
anteayasmcginnis: there is a lot yes, is this a brand new job?15:53
*** matrohon has quit IRC15:54
*** sree has quit IRC15:55
anteayasmcginnis: if this is a brand new job specific to cinder you are going to have to build a job-template for it, I'd suggest putting a new job template for cinder in here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/cinder.yaml15:55
*** _milan_ has quit IRC15:57
hosanaiHello, I have a question about jenkins configuration. I would like to install non-python package for python27 checking. And I tried to use bindep but it seems to work for only packages which provide from debian or other OS distributors.15:57
hosanaiThe package is a tgz file "http://apache.claz.org/spark/spark-1.6.1/spark-1.6.1.tgz" so I would like to download it and then extract it for jenkins. Do you know how to do it?15:57
mordredhosanai: you'll just have to write a script to download and install it in your build scripts15:58
mordredhosanai: jobs in openstack infra have sudo access on the hosts they run on, so you can do kindof whatever you want to15:58
*** tonytan4ever has joined #openstack-infra15:59
mordredhosanai: what project is this for?15:59
*** sree has joined #openstack-infra15:59
hosanaimordred: thanks for quick response. Monasca (Monasca-Analytics)15:59
*** sree has quit IRC15:59
*** lanakars has joined #openstack-infra16:00
*** _nadya_ has quit IRC16:01
hosanaimordred: I already have the script. where should I register it?16:01
*** bpokorny has quit IRC16:01
*** winggundamth_ has joined #openstack-infra16:02
*** bpokorny has joined #openstack-infra16:02
*** permalac has quit IRC16:03
lanakarsHi folks, can anyone please help to fix this gate-fuel-docs-docs job: https://review.openstack.org/#/c/329587/16:03
*** Na3iL has quit IRC16:03
mordredhosanai: you really just need to run it in your job - I'm guessing as a devstack plugin, or as part of the devstack plugin monasca already uses16:03
mordredhosanai: be aware that downloading things from the internet during your jobs will increase your exposure to the internet being the internet16:04
openstackgerritSean M. Collins proposed openstack-infra/devstack-gate: [WIP] Switch to new Neutron devstack  https://review.openstack.org/27841516:04
nibalizerrcarrillocruz: hiya16:05
rcarrillocruzhey, so, it seems puppetdb 3.x is for puppet 4??16:06
rcarrillocruzat least 3.2, the link i pasted you16:06
*** roxanaghe has joined #openstack-infra16:06
*** jlanoux has quit IRC16:06
jeblairhosanai: as an official project, monasca should probably try to use dependencies that are already packaged or working with upstream distros to get them packaged16:06
mordred++16:07
nibalizerrcarrillocruz: so I think that means they didn't bother solving the packaging issue, not that there is anything wrong16:07
*** sdake has quit IRC16:07
nibalizerespecially when you consider thta we own the script that is going to post to puppetdb, 'not being compatible' is basically impossible16:07
jeblairzuul is updated everywhere.  i'm going to restart it now16:08
*** dtantsur is now known as dtantsur|afk16:08
rcarrillocruzK16:08
anteayalanakars: looking in this file: http://logs.openstack.org/87/329587/3/check/gate-fuel-docs-docs/6bef22e/_zuul_ansible/logs/ansible.log16:09
*** dmk0202 has joined #openstack-infra16:09
anteayalanakars: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/zuul_runner.py'16:09
hosanaimordred: thanks! sorry i still don't clear... i wrote devstack plugin for monasca-analytics but i don't know who kick this script from jenkins. maybe jenkins/jobs/projects.yaml?16:09
*** eezhova has quit IRC16:10
anteayalanakars: I'm guessing having ansbile find a zuul_runner.py file is kind of important for the job to complete16:10
anteayaI'm not sure what is supposed to put it where ansible can find it16:10
*** javeriak has joined #openstack-infra16:10
jeblairanteaya: recheck it; it's a known rare problem16:11
anteayajeblair: thank you16:11
anteayalanakars: ^^16:11
*** jogo has quit IRC16:12
jrollhey friends, do you want to know about: gate-ironic-specs-docs   POST_FAILURE in 47s16:12
hosanaijeblair: thanks for the advise. monasca-analytics is a sub component of monasca and it will introduce new dependencies (there is no pip/apt-get packages)16:12
jrollI assume that's a log post failure, but given all the changes, thought I'd let you know16:12
anteayajroll: link to patch?16:13
jeblairjroll: it is, and we can find a clue as to what caused it by looking at the ansible log published along with the job16:13
jrollanteaya: jeblair: https://review.openstack.org/#/c/188370/1816:13
jeblair(i can't look right now, i'm in the middle of a zuul restart, but can in a minute)16:13
*** tkelsey has quit IRC16:14
pleia2good morning16:14
*** ddieterly[away] is now known as ddieterly16:14
anteayajroll: hmm, File Not Found for the logs link is less helpful that I had hoped16:14
anteayamorning pleia216:14
jrollanteaya: :)16:14
anteayapleia2: welcome to a brand new zuul world16:14
lanakarsanteaya, jeblair: thank you! =)16:14
anteayalanakars: welcome16:14
jeblairanteaya, jroll: oh, then we would have to look on the launcher.  i'll do that when finished here16:14
jrolljeblair: anteaya: I'm not concerned about the actual results, I rechecked and assumed it will past (just specs linting), but wanted to let you know. thanks :)16:15
anteayajeblair jroll thanks16:15
anteayajroll: yep, thanks for letting us know16:15
anteayajroll: I have to relearn how to read logs again16:15
jeblairwhat's the link that 404s?16:15
jrolljeblair: http://logs.openstack.org/70/188370/18/check/gate-ironic-specs-docs/8629a84/16:15
*** roxanaghe has quit IRC16:15
pleia2anteaya: indeed!16:16
anteayapleia2: getting some log groking practice in16:16
jeblairoh, the launchers take a long time to stop because they still run their post playbooks16:16
jeblairpabelanger: what's the status of the don't run post on abort change?16:17
pleia2anteaya: great :)16:17
anteaya:)16:18
EmilienMzuul launchers were restarted already?16:20
jeblairEmilienM: in the process16:20
EmilienMack16:20
*** inc0 has quit IRC16:20
*** degorenko is now known as _degorenko|afk16:21
pabelangerjeblair: I believe that is 32963116:21
*** mrhillsman has joined #openstack-infra16:21
*** ramishra has quit IRC16:22
*** roxanaghe has joined #openstack-infra16:22
fungilgtm16:23
jeblairsorry i expected a hard stop to be a bit faster than this.  hopefully 631 will fix that (we should test that it does)16:23
jeblairshould be just a couple more mins16:23
fungigood point, 329631 doesn't add a test for that16:25
jeblairfungi: we have no tests for zuul launcher16:25
*** jsavak has quit IRC16:25
pabelangerI did check -9 is the exit code of an aborted job.  But yes, I've not actually tested it with zuul-launcher yet16:26
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add the ability to permalink to a comment  https://review.openstack.org/28907816:26
*** jsavak has joined #openstack-infra16:26
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Remove pagination for timeline events  https://review.openstack.org/32768216:26
jeblairmostly i mean we should issue a zuul-launcher stop and make sure it takes that code path16:26
pabelangeragreed16:26
fungiyeah, their exit codes aren't clearly documented that i can find16:26
*** e0ne has quit IRC16:27
jeblairi think -9 may come from us because of the way we kill the pgrop16:27
jeblairpgroup16:27
*** mixos has joined #openstack-infra16:27
jeblairokay, all launchers have stopped16:27
fungiaha, was about to say i don't see it at https://github.com/ansible/ansible/blob/devel/bin/ansible#L99 either16:28
*** dprince has joined #openstack-infra16:28
pabelangerRight, I didn't check if stop will actually get there. I was looking from abortRunningJob16:28
jeblairwow16:28
jeblairzuul-launcher leaked 0 nodes :)16:28
fungium16:28
pabelangerimpressive16:28
jeblairlike, there are 0 nodes in nodepool assigned to zl* that are not in delete state16:28
fungijenkins looks worse every day16:28
pabelangerthat is very nice16:29
mordred\o/16:29
jeblairso i think with everything in nodepool in either building or delete, i will go ahead and start zuul without touching nodepool16:29
jeblair(i was prepared to do a db query to force delete nodes but do not think that is necessary)16:29
mordredI agree16:29
fungiseems not anyway16:29
*** mriedem is now known as mriedem_lunch16:30
clarkbdid we remove the jenkins restart cron? it will fire today if not16:30
jeblairzuul is started16:30
nibalizerclarkb: haha good point16:30
fungialso did we remove jenkins masters from dns? if not, i can do that now16:30
fungilooks like we did not16:31
mordredfungi: they still resolve for me16:31
fungi(no idea why i asked when it was easier to query)16:31
fungiyep16:31
mordredfungi: :)16:31
jeblairstatus notice zuul was restarted for a software upgrade; events between 16:08 and 16:30 were missed, please recheck any changes uploaded during that time16:31
jeblairhow does that look ^ ?16:31
fungiany objections to me removing them from dns now?16:31
fungijeblair: lgtm16:31
jeblairfungi: no objection16:31
anteayalooks good16:31
jeblair#status notice zuul was restarted for a software upgrade; events between 16:08 and 16:30 were missed, please recheck any changes uploaded during that time16:31
openstackstatusjeblair: sending notice16:31
pabelangerfungi: none here16:32
anteayaI have no objection16:32
-openstackstatus- NOTICE: zuul was restarted for a software upgrade; events between 16:08 and 16:30 were missed, please recheck any changes uploaded during that time16:33
smcginnisanteaya: Oh, duh. I'm obviously not paying enough attention today.16:34
anteayasmcginnis: it is Friday, it is common16:34
smcginnisanteaya: It should have struck me that I at least need to say what to run for the job somewhere. ;)16:34
openstackstatusjeblair: finished sending notice16:34
pabelangerused nodes are almost 100% equal across zuul-launchers now16:34
anteayasmcginnis: well we did just change everything, so I totally understand the wishful thinking approach to coding16:34
*** jpich has quit IRC16:34
jeblairpabelanger: woo!16:34
anteayasmcginnis: who knows, maybe that will work someday16:35
pabelangerdrift of about 2 nodes I am seeing16:35
smcginnisanteaya: haha, it was definitely wishful thinking on my part that it would be that easy.16:35
jlvillalIs the data that "CI Watch" (http://ci-watch.tintri.com/project?project=ironic&time=7+days) uses available as a CSV file or something? If I want to generate some statistics. Or some other way for me to relatively easily generate statistics.16:35
anteayasmcginnis: ha ha ha16:35
anteayaasselin_: mmedvede ^^16:35
*** tkelsey has joined #openstack-infra16:35
Shrewspabelanger: yeah, i was watching that myself16:35
*** Apoorva has joined #openstack-infra16:36
mmedvedejlvillal: no, it is not16:36
*** Apoorva has quit IRC16:37
mmedvedejlvillal: ciwatch uses sql database, and there is no rest api implemented for backend16:37
pabelangerjeblair: I would be interested to see the steps you just did in etherpad to mass restart zuul.  With nodepool other of the picture, it might be a straightforward playbook to automate that process16:37
jlvillalmmedvede: Darn :(16:37
jlvillalmmedvede: And thanks.16:37
EmilienMjeblair: is there a plan to have a graphic realtime console? like we had in jenkins, that we can see in web browser, instead of using nc?16:38
mordredEmilienM: yes16:39
jeblairpabelanger: save gate/check queue; service zuul stop; on all launchers: zuul-launcher stop; wait; service zuul start; service zuul-launcher start16:39
mordredEmilienM: I've got it half written, hope to finish it soon16:39
pabelangerjeblair: great, thanks16:39
EmilienMmordred: it's in gerrit?16:39
*** Goneri has joined #openstack-infra16:39
jeblairjroll: ah, that job failed due to a known, but rare issue16:39
mordredEmilienM: https://review.openstack.org/#/c/320563/16:39
EmilienMmordred: awesome16:39
jeblairjroll: our ansible module briefly disappears from the file system when we reinstall zuul (which we do every time we land a patch)16:40
clarkbI am pretty sure if you install the chrome ssh plugin it will just work16:40
*** tkelsey has quit IRC16:40
mordredEmilienM: I'm waiting for the 2.5 stuff to be mostly done so I can avoid lots of rebases and stuff - but I should be able to start working on it again either today or early next week16:40
pabelangerclarkb: which one did you used on your phone again?16:40
clarkbpabelanger: connectbot on phone16:40
pabelangerclarkb: thanks16:40
jeblairmordred: i keep saying this is "rare" often enough that i'm thinking we may need to fix it soon.16:41
EmilienMmordred: awesome feature16:41
mordredjeblair: I agree16:41
EmilienMmordred: for impatient people like us :P16:41
*** mohankumar has quit IRC16:41
*** PollyZ has quit IRC16:41
pabelangerwhat I'm currently doing for chrome: https://etherpad.openstack.org/p/google-telnet16:42
fungimmedvede: jlvillal: we could perhaps consider doing what we did for the subunut2sql db used by health.openstack.org and just provide a query:query read-only user available through a socket proxy16:42
pabelangerwarning, pretty hacky16:42
fungiin theory there's no sensitive data in there?16:42
*** gyee has joined #openstack-infra16:42
EmilienMpabelanger: /me trying16:43
mmedvedefungi: should be no sensitive data there, it is just an aggregate from jenkins event stream16:43
mmedvede*gerrit event stream16:44
fungimmedvede: or if it's a very small dataset, just hack a csv export call into ciwatch available from the web interface16:44
*** mikelk has quit IRC16:44
fungispeaking of which, i should plug https://github.com/harelba/q again16:45
fungiit's awesomely useful16:45
*** trown is now known as trown|lunch16:45
*** PollyZ has joined #openstack-infra16:45
pabelangerfungi: jeblair: just confirmed BUILD_TIMEOUT is passed correctly now16:45
mmedvedefungi: that is awesome, I was just recently googling for something that does it16:46
pabelangerfungi: jeblair: thanks for helping out16:46
fungiexcellent16:46
AJaegergreat, pabelanger !16:46
fungidprince: ^16:46
fungithe tripleo crowd will be relieved16:46
mordred\o/16:46
AJaegersmcginnis: do you now know what to do?16:46
mmedvedefungi: so ciwatch instance is currently maintained outside infra team. Any chance there is now some compute available to get a VM up and deploy it within infra?16:47
EmilienMfungi: thx16:47
smcginnisAJaeger: I think so. Working on it now. Seems mostly straightforward, so I'll give it a shot.16:47
fungimmedvede: shouldn't be a problem. usually we ask for a (very small) spec describing the service and its needs16:47
smcginnisAJaeger: Thanks16:47
EmilienMpabelanger: it works lol16:47
zaromorning16:48
AJaegersmcginnis: just commented on your review and gave an example what to do...16:48
smcginnisAJaeger: Oh, awesome. Thanks!16:48
dprincefungi: ack, thanks for this16:48
AJaegersmcginnis: add that to jenkins/jobs/projects.yaml16:48
mmedvedefungi: http://specs.openstack.org/openstack-infra/infra-specs/specs/deploy-ci-dashboard.html, the spec landed a while back, so might need revision16:48
fungidprince: jeblair and pabelanger did most of the work, but you're welcome!16:48
dprincethanks jeblair and pabelanger :)16:49
jeblairdprince: sorry for the inconvenience16:49
fungimmedvede: oh, already most of the way there then16:49
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only  https://review.openstack.org/31358816:49
EmilienMpabelanger: just missed the chmod +x16:49
*** ddieterly is now known as ddieterly[away]16:50
pabelangerEmilienM: update etherpad :)16:50
*** _degorenko|afk has quit IRC16:50
EmilienMpabelanger: done16:50
pabelangerEmilienM: but ya, hacked on that last night at 10:30pm EST16:50
pabelangershould revisit it this weekend16:50
smcginnisAJaeger: Should that be the jenkins/jobs/projects.yaml file or jenkins/jobs/cinder.yaml since this is Cinder specific (at least for now).16:50
AJaegerclarkb: could you review https://review.openstack.org/330821 - Revert "Remove skip test logic for project-config" , please?16:51
AJaegersmcginnis: you instantiate templates in projects.yaml16:51
EmilienMoops I see "gear.NoConnectedServersError: No connected Gearman servers"16:51
EmilienMdprince: ^16:51
EmilienMdprince: in tripleo jobs16:51
AJaegersmcginnis: just search for tox in that file16:51
smcginnisAJaeger: OK, got it. Thanks again.16:51
pabelangerclarkb: would connectbot open, and connect to the host once you used telnet:// directly from the browser? Or do you copy and paste?16:51
EmilienMdprince: not sure that's something we expect16:51
*** lucasagomes is now known as lucas-afk16:51
*** daemontool_ has quit IRC16:52
zaropleia2: can you advice about users required for this change?16:53
zarohttps://review.openstack.org/#/c/330925/16:53
*** harlowja has joined #openstack-infra16:53
fungiokay, all jenkins*.openstack.org dns records have been deleted16:54
mordred\o/16:54
*** degorenko has joined #openstack-infra16:54
pleia2zaro: looking16:55
zaropleia2: we basically need a bot user in storyboard for gerrit. wondering if we might already have one a general bot user that gerrit can use?16:55
pleia2zaro: gotcha, let's see what we use for updating launchpad16:56
openstackgerritzzxwill proposed openstack-infra/zuul: Rename jenkins to zull  https://review.openstack.org/33085316:57
fungipleia2: in lp it's the hudson-openstack account16:57
fungii don't think we need to duplicate that username in sb though16:57
*** pvaneck has joined #openstack-infra16:58
pleia2fungi: right, but what user on review.o.o interacts with that account?16:58
*** _ari_ has quit IRC16:58
*** timello has quit IRC16:58
dprinceEmilienM: other jobs are running, which log did you see the connection error in?16:58
fungipleia2: none. it's a gerrit hook script calling an entrypoint in jeepyb16:58
dprinceEmilienM: are they consistent, or just a one off16:59
pleia2oh I see, this is looking for a user *in storyboard*16:59
openstackgerritSean McGinnis proposed openstack-infra/project-config: Add driver compliance checks to check pipeline  https://review.openstack.org/33119216:59
*** inc0 has joined #openstack-infra17:00
pleia2zaro: we do seem to have an OpenStack Gerrit user17:00
EmilienMdprince: all17:00
EmilienMdprince: telnet://66.187.229.81:808817:00
pleia2zaro: user 240 for review.o.o, user 616 for review-dev17:01
pabelangerha, need to update weechat to parse telnet://17:01
zaropleia2: cool, would you mind comment on my change what user ino i should retrieve from heirs?17:01
EmilienMdprince: http://paste.openstack.org/show/518471/17:01
pleia2zaro: will do17:01
*** inc0 has quit IRC17:01
EmilienMdprince: I tested different jobs on different patches17:01
*** inc0 has joined #openstack-infra17:02
zaropleia2: thanks.17:02
*** Apoorva has joined #openstack-infra17:02
*** jsavak has quit IRC17:02
pabelangerEmilienM: dprince: I suspect your local gearman server is offline17:02
*** jsavak has joined #openstack-infra17:02
zaropleia2: now i’m wondering if review-dev.o.o has access to storyboard-dev.o.o?  i mean can you post to sb-dev using secure curl?17:03
EmilienMdprince: do you have access to it?17:03
dprinceEmilienM: yes... just have to remember how I get in again17:03
pleia2zaro: I don't know, review-dev's account is on storyboard.o.o (not storyboard-dev), noting in change17:04
*** inc0 has quit IRC17:04
SotKwe intend to populate storyboard-dev with data from storyboard.o.o at some point ftr17:05
clarkbpabelanger: it opened automagically17:05
pabelangerclarkb: thanks, I'll dig deeper why that's not working for me17:05
anteayaSotK: sure put that likely won't happen by next week's sprint17:06
anteayas/put/but17:06
*** bpokorny has quit IRC17:06
hosanaimordred: sorry for bother you. could you point out a script (devstack plugin) which is called by jenkins?17:06
zaropleia2: i’m concerned because sb-dev doesn’t have real certs so making requests might be an issue.17:07
anteayaSotK: I'd like to get this interaction in place on the dev servers for next week's sprint17:07
*** lanakars has left #openstack-infra17:07
zaropleia2: would we be able to test a POST to sb-dev from review-dev?17:07
anteayaand I'm so grateful to zaro and pleia2 for helping here17:07
pleia2zaro: could be17:07
*** javeriak_ has joined #openstack-infra17:07
*** javeriak has quit IRC17:07
*** tonytan4ever has quit IRC17:07
*** xyang1 has quit IRC17:08
dprinceEmilienM: not all our machines have public IPs, so I'll go in through the mirror server17:08
*** xyang1 has joined #openstack-infra17:08
zaroanteaya: gerrit sb plugin is unlikely to make it in by next week.  i’m finding that there will need to quite a few changes required for install and configuration17:09
anteayazaro: okay fair enough, is there anything I can do to help, I've looked at as many changes as I can find17:09
*** yamamoto has quit IRC17:10
pleia2zaro or anteaya: have a suggestion as to what to post? I haven't used the API much yet17:10
anteayazaro: are you using a common topic for all the patches needed to get this interaction in place?17:10
anteayapleia2: yes, let me get you a command in a paste17:10
zaroanteaya: not atm, i’ll let you know when the changes are all ready for review.17:10
anteayapleia2: I didn't want to interrupt your conversation with zaro, you were making such good progress17:10
anteayazaro: okay can you link them with a gerrit topic in the meantime, so I can at least be aware of status?17:11
zaroanteaya: sure can.17:11
pleia2anteaya: thanks :)17:11
anteayazaro: thank you17:12
*** e0ne has joined #openstack-infra17:12
*** amotoki has quit IRC17:13
*** timello has joined #openstack-infra17:13
*** ddieterly[away] is now known as ddieterly17:13
*** arxcruz has quit IRC17:13
AJaegerthere's a change to remove node-devstack from nodepool. Syntax looks fine but I'm not sure whether we want that. fungi, jeblair, pabelanger: Could either of you comment on https://review.openstack.org/318994 , please?17:16
*** bpokorny has joined #openstack-infra17:16
anteayapleia2: hopefully http://paste.openstack.org/show/518474/ gets you at least part of the way there17:17
pabelangeranteaya: left comment17:19
fungiAJaeger: yep, that's some good cleanup17:19
pabelangeranteaya: apologies, tabbed incorrectly17:19
anteayapabelanger: sorry, left comment on what17:19
pleia2anteaya: thanks17:19
anteayaoh okay17:19
pabelangerAJaeger: I have left comments17:19
anteayapleia2: let me know if you need more17:19
zaropleia2: i think we want to test without the ‘-key'17:20
pabelangerfungi: I think we should also remove nodepool/elements/node-devstack ?17:20
fungipabelanger: yes, though i'm not particular about whether it has to be in that change17:21
AJaegerthanks, pabelanger17:21
AJaegerthanks, fungi17:21
pabelangerfungi: sure, followup works too17:21
fungiif you're -1 on it, i'm happy to drop my workflow +117:21
pleia2zaro: yeah, I don't have a local key for this17:21
pabelangerfungi: sounds like work.  I'll do a follow up17:22
anteayapleia2: you probably don't need one, that was just what I needed for this to work on my local gerrit17:22
anteayapleia2: I think zaro wants to test POSTing without that bit to see if certs on both sides will allow the POST to happend17:23
anteayahappen*17:23
*** thiagop has quit IRC17:23
pleia2anteaya: exactly17:23
anteayacool17:23
pleia2trying to figure out why this command isn't returning, I might have a stray ' somewhere17:24
*** e0ne has quit IRC17:24
anteayaor I could have given you a bad paste17:24
pleia2anteaya: in the Authorization section, should it be 'Authorization: My Name <3243232>' ?17:24
anteayas/or//17:24
anteayano17:24
dprinceEmilienM: our tripleo testenv geard is restarted. Watching it now17:24
anteayayou have to get a token and paste it in17:25
pleia2anteaya: well yes :) I was using a fake one in that example17:25
pleia2but pretend 3243232 is real, I have 'Authorization: Elizabeth K. Joseph <3243232>'17:25
anteayapleia2: https://storyboard-dev.openstack.org/#!/profile/tokens17:25
anteayano, just the token17:25
pleia2anteaya: I know, I have my token, I was asking about syntax17:26
anteayasorry17:26
pleia2anteaya: what is "Bearer" ? is that part of the command?17:26
anteayanot the username17:26
anteayayes Bearer needs to be there17:26
pleia2gotcha, I thought it was a placeholder for name, thanks17:26
AJaegermriedem_lunch: is https://review.openstack.org/#/c/313588 now fine?17:26
*** PollyZ has quit IRC17:26
anteaya-H 'Authorization: Bearer 1234567'17:26
pleia2ah, no brackets either17:27
anteayano brackets17:27
anteayasorry the brackets were my attempt to say fill in this space17:27
pleia2makes sense17:27
anteayafill in this space and then let's talk about it, is what I should have said17:28
pleia2so curl definitely doesn't like it without a key, using --insecure gets me a 500 error17:29
anteaya:(17:29
pleia2zaro: ^^17:29
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add support for node revocation  https://review.openstack.org/33127317:30
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Revoke nodes via gearman on deletion  https://review.openstack.org/33127417:30
jeblairpabelanger, mordred, fungi: ^ that is not urgent but may come in handy.17:31
*** piet has joined #openstack-infra17:31
*** nwkarsten has quit IRC17:32
*** dizquierdo has quit IRC17:32
*** nwkarsten has joined #openstack-infra17:32
mordredjeblair: agree - looks good ... although fwiw when I read the commit messages at first I thought you were describing the opposite direction17:32
mordredthat's likely just the way i processed it rather than a real issue17:33
AJaegerproject-config cores, the dependent job is currently in gate queue, would be nice to have install-guides jobs soon. Could you review https://review.openstack.org/330725 , please?17:33
*** nwkarsten has quit IRC17:33
jeblairmordred: yeah, they're "assigned" by gearman, so i think they should also be "revoked" by gearman.  the commit message prose could probably be better, but i think the jobs are named right17:34
*** milan has joined #openstack-infra17:34
AJaegerpabelanger: do you want to abandon 326006, or is this still needed?17:34
mordredjeblair: I agree17:34
jeblairer, i think 's/gearman/nodepool' above17:34
pleia2anteaya: so the 500 error seems to be "ValueError: No JSON object could be decoded" (and python traceback along with it)17:34
jeblairmordred: do you have any ideas how we might deal with the missing zuul_runner thing?17:35
jeblairmordred: it seems like the pip install makes the file disappear for a significant enough amount of time17:36
zarocurl -v -H "Authorization:Bearer a6ya1ndTkEaDHqfp5aJsqjVP3VUahp" -H "Content-Type: application/json" -X POST -d '{"story_id”:7,"content":"mama!"}' https://storyboard.openstack.org/api/v1/stories/7/comments17:36
zaropleia2: try this with your token ^17:36
pleia2zaro: thanks17:36
*** dtardivel has quit IRC17:37
mordredjeblair: I'm trying _really_ hard to not say AFS17:37
zaropleia2: opps that should be storyboard-dev.o.o17:37
jeblairmordred: i admire your restraint17:37
mordredjeblair: of course, it seems that atomic releasing of a directory worth of contents is a thing this would benefit from17:38
openstackgerritMerged openstack-infra/project-config: Remove node-devstack from nodepool  https://review.openstack.org/31899417:38
EmilienMdprince: it looks better17:38
pleia2zaro: breaks without ssl certificate, running with --insecure I get the same json error as with anteaya's command17:39
*** banix has joined #openstack-infra17:39
*** sarob has joined #openstack-infra17:39
pleia2zaro: json error is in the logs, 500 error on the user side from my curl command17:39
zaropleia2: well i would expect —insecure to work though17:40
mordredjeblair: do we have a good handle anywhere that could trap for the "this file isn't here" error and retry?17:40
*** sarob has quit IRC17:40
*** ajmiller has joined #openstack-infra17:40
*** sdake has joined #openstack-infra17:40
*** hosanai has quit IRC17:40
jeblairmordred: well, it's in the middle of the playbook17:40
*** sarob has joined #openstack-infra17:40
mordreddarn.17:40
jeblairmordred: idea:17:41
fungipleia2: make sure you use straight quotes? zaro's example had a smart quote mixed in according to my font17:41
jeblairmordred: we could, when setting up the jobdir, copy the needed modules into it.  that's something we can retry-loop.17:41
*** yaume has quit IRC17:41
pleia2fungi: thanks, I'll check17:41
mordredjeblair: that makes me a little sad, but not excessively sad17:42
jeblairmordred: it means every job would get their own copy of the modules which is silly, and it's a bit extra io, but i guess they're not that big.17:42
jeblair(we would want to make sure we don't put them into the directory that gets uploaded, because we really do not need a copy of zuul_runner for every job on our logserver)17:42
mordred++17:42
*** ajmiller_ has quit IRC17:43
zarofirst time i’ve ever heard those things called 'smart'17:43
jeblairmordred: only other thing that jumps to mind is having puppet copy them to some location on the host.17:43
mordredjeblair: we could also do the pip install and then rsync the contents of lib/python to another location which is the path we actually run from17:43
*** PollyZ has joined #openstack-infra17:43
mordredjeblair: so as that we can avoid the delete-then-create behavior17:43
mordredyah17:43
mordredwhat you said17:43
jeblairmordred: oh, that makes me think of a combination of the two:17:43
jeblairmordred: what if we had *zuul* do that, not puppet17:44
fungizaro: ” vs "17:44
mordredah - interesting. yeah. so we only have one extra copy - but it's a safe extra copy17:44
*** sambetts is now known as sambetts|afk17:44
jeblairmordred: so zuul makes a sort of global tmpdir for its ansible deps, and periodically checks that it's up to date from its install location17:44
*** bpokorny has quit IRC17:44
mordredyes17:44
asselin_rcarrillocruz, https://ansible-role-cloud-launcher.readthedocs.org link gets forwarded to https://ansible-role-cloud-launcher.readthedocs.io/ which doesn't exist yet. Is that expected?17:44
anteayafungi: pleia2 for context zaro was able to get the storyboard-plugin to work: https://storyboard-dev.openstack.org/#!/story/717:44
fungizaro: what was the outstanding stack of changes for the sb/its plugin work for review-dev? happy to do some reviewing on those17:45
*** jed56 has quit IRC17:45
anteayawe just have to get it configured correctly on review-dev17:45
*** rbrndt has quit IRC17:45
fungianteaya: zaro: ooh, excellent17:45
pleia2anteaya: got your command to work, but only by using --insecure17:45
anteayayeah, so the fault was mine, but I don't know what I did wrong yet17:45
anteayapleia2: okay thanks17:45
fungipleia2: the json decoding error was quoting-related then?17:45
zarothat came straight out of my text editor, must be irccloud mucking it up on paste17:46
jeblairmordred: so like: every time a job starts, make sure /var/lib/zuul/ansible is up to date.  and use that for the ansible plugins/modules location.17:46
krotscheckI've got a couple of old-ish maniphest patches still in my queue, do we still care or shall I abandon them? https://review.openstack.org/#/q/status:open+project:openstack-infra/puppet-phabricator+branch:master+topic:maniphest17:46
zaropleia2: cool, so we need it to work without -insecure. i thinkg17:46
jeblairmordred: or, you know what, it's not critical they be up to date17:47
anteayakrotscheck: we still care as the ux folks will need the functionality there17:47
jeblairmordred: let's just do it on startup.17:47
anteayakrotscheck: we just haven't had the review time for them to make the top priority currently17:47
jeblair(in fact, it's slightly weird that they change when the launcher doesn't)17:47
krotscheckanteaya: Alright, can you take over those patches? I'm refocusing on non-infra things.17:47
anteayaah sure, or you can just put them in any state you like17:48
anteayaif you abandon them a gerrit admin can take them over17:48
anteayakrotscheck: thanks for letting us know17:48
krotschecks/infra/task-tracking/17:48
anteayakrotscheck: yep fair enough17:48
*** pblaho has quit IRC17:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add auto interval to nodepool dashboard  https://review.openstack.org/33128117:48
anteayakrotscheck: thanks for working on those patches17:49
pabelangerjeblair: AJaeger: ^ I'd like to give that a try, it should give the users the ability to adjust the interval ranges based on https://www.youtube.com/watch?v=CLXKlgHvOuo17:49
zaropleia2: ohh maybe not.  i think we only need to allow java access to do the post.17:49
*** _nadya_ has joined #openstack-infra17:49
pabelangerAJaeger: checking on review you linked17:50
AJaegerthanks, pabelanger.17:50
*** markusry has quit IRC17:51
zaropleia2: this is how i gave java access from my local machine http://paste.openstack.org/show/518481/17:51
*** ccamacho has quit IRC17:52
zaropleia2: i’m not sure if that’s legit to do on review-dev or not.  fungi? ^17:52
pleia2zaro: ah, this creates the storyboard-dev.crt that anteaya had in her command17:52
mordredjeblair: ++ to doing it on startup17:52
*** tosky has quit IRC17:52
anteayapleia2: yeah, zaro had me run those commands on my local machine17:53
pleia2anteaya: nods17:53
anteayapleia2: here was what he had given to me: http://paste.openstack.org/show/516325/17:53
zaroyep, curl can accept that key as well17:53
*** yamahata has quit IRC17:53
anteayanote the last command is run with sudo17:53
pleia2anteaya: yeah, since it's editing the global java keystore17:54
pleia2I can run the first and just reference the key without touching global stuff17:54
anteayaI had missed noticing that myself on my first pass17:54
anteayaor making that connection I should say17:55
*** trown|lunch is now known as trown17:55
*** gyee has quit IRC17:55
fungizaro: we ought to be able to puppet that onto review-dev17:56
pleia2zaro: ok, got your command to work by using that ssl cert (I didn't put it in the global config)17:56
fungiassuming paste.o.o ever responds to me17:57
pleia2fungi: yeah17:57
*** tonytan4ever has joined #openstack-infra17:57
* fungi is assuming what is in it without being able to look at it yet17:57
zarofungi: ok. i just wanted to verify that it’s not going to conpromise any security thingys17:57
fungioh, not at all. also it's just review-dev17:57
fungifor production we wouldn't need to do this because review will be able to trust the signature on storyboard.o.o17:58
anteayayes, we must consider the security thingys17:58
pleia2yeah, anything with a real cert is fine17:58
*** mriedem_lunch is now known as mriedem17:59
*** cody-somerville has joined #openstack-infra17:59
zarook. i’ll puppet it then. thanks all.17:59
pleia2you're welcome17:59
anteayazaro: awesome, thanks17:59
fungizaro: we can just put the cert contents in public hiera, no need for that first command to be run _on_ review-dev17:59
*** nwkarsten has joined #openstack-infra18:00
*** nwkarsten has quit IRC18:00
zaro+1 dat18:00
*** nwkarsten has joined #openstack-infra18:00
fungithe second command, because it's updating the jvm trust store, does need to run on review-dev obviously18:00
mriedemAJaeger: yeah, i don't know if it's going to pass, but it's non-voting to start18:01
mriedemso seems ok to me18:01
AJaegermriedem: thanks - let's try it out ;)18:01
*** ddieterly is now known as ddieterly[away]18:03
*** harlowja has quit IRC18:04
fungizaro: pleia2: anteaya: so basically put the results of that first command in public hiera for review-dev.o.o, and then create a file resource with that content somewhere and then subscribe the exec for that second command to it18:04
fungishould be pretty simple18:04
anteayayay simple!18:04
pleia2java couldn't just have a file listing of certs18:05
*** kzaitsev_mb has quit IRC18:06
pleia2I guess it's faster this way?18:06
*** dprince has quit IRC18:06
*** dprince has joined #openstack-infra18:06
fungipleia2: i'm guessing keytool -storepass changeit also notifies the running jvm18:07
pleia2fungi: ah, perhaps18:07
fungithough you can likely directly edit ${JAVA_JDK_HOME}/jre/lib/security/cacerts if you want (no clue what format it's in, i haven't looked)18:07
pleia2it's a binary file18:08
fungiwhy of course it is!18:08
zaropleia2: i’m not sure actually.  it’s been a while but last i tried this was the only solution i found on the internet18:08
fungiprobably the java equivalent of python pickle18:08
*** ccamacho has joined #openstack-infra18:08
*** javeriak_ has quit IRC18:09
*** yamamoto has joined #openstack-infra18:10
*** javeriak has joined #openstack-infra18:10
*** thorst_ has quit IRC18:13
*** xiangxinyong has quit IRC18:13
*** thorst_ has joined #openstack-infra18:14
*** xiangxinyong has joined #openstack-infra18:14
nwkarsteni'm trying to port the pyton-jobs unit tests into my ci system, but i am having issues. i have the jenkins/jobs/macros.yaml file copied exactly, but the zuul-git-prep-upper-constraints builder is failing. Here's the console output http://paste.openstack.org/show/518485/ does anyone know why that i can't access that git repo?18:14
*** sarob has quit IRC18:15
clarkbnwkarsten: probably a firewall on your end blocking port 941818:16
*** cody-somerville has quit IRC18:17
*** jamesmcarthur has quit IRC18:18
*** thorst_ has quit IRC18:18
*** thorst_ has joined #openstack-infra18:19
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (Local Loop)  https://review.openstack.org/31959118:22
*** yamamoto has quit IRC18:24
funginwkarsten: you could try using https:// instead of git:// for the url passed to zuul-cloner18:24
fungi(or even just http://)18:24
*** yamahata has joined #openstack-infra18:25
nwkarsten@fungi,@clarkb: i will give https a shot18:26
*** jamesmcarthur has joined #openstack-infra18:27
anteayanwkarsten: it isn't necessary to use @ in front of an irc nick18:27
anteayathat is hipchat or twitter18:27
*** ajmiller has quit IRC18:29
*** ajmiller_ has joined #openstack-infra18:29
*** Sukhdev has joined #openstack-infra18:32
*** jamesmcarthur has quit IRC18:32
*** hasharAway is now known as hashar18:33
zxiiroanyone know if it's possible to edit the build description from within a jenkins job?18:35
*** jamesmcarthur has joined #openstack-infra18:36
pabelangerzxiiro: see jenkins/jobs/defaults.yaml in project-config18:37
*** jsavak has quit IRC18:37
*** jamesmca_ has joined #openstack-infra18:37
*** jsavak has joined #openstack-infra18:38
zxiiropabelanger: no i don't mean the job description. I mean the specific build description. Like if you run a job and it creates build #5, you can edit the description of that build run.18:39
zxiiroI have a variable from a job i want to attach to the build #'s description18:39
zxiirotrying to find if there's a plugin to handle that but no luck yet... maybe have to do a groovy script or something18:39
*** inc0 has joined #openstack-infra18:40
mordredzxiiro: maybe with groovy18:40
mordredyah18:40
*** simondodsley has joined #openstack-infra18:40
*** sarob has joined #openstack-infra18:40
mordredzxiiro: you already said that :)18:40
*** jamesmcarthur has quit IRC18:41
*** hosanai has joined #openstack-infra18:41
*** amitgandhinz has quit IRC18:41
*** amitgandhinz has joined #openstack-infra18:42
jrolljeblair: ansible disappearing sounds fun :)18:43
mordredjroll: I mean ... fun is one word for it18:44
*** hosanai has quit IRC18:45
*** jamesmca_ has quit IRC18:46
zarozxiiro: i think that requires a plugin18:46
pabelangermordred: jroll: where did we see that?18:46
zxiirozaro: as you said that I think i just found it https://wiki.jenkins-ci.org/display/JENKINS/Description+Setter+Plugin18:47
jrollpabelanger: "our ansible module briefly disappears from the file system when we reinstall zuul (which we do every time we land a patch)"18:47
jrollknown issues according to jim18:47
*** andreww has quit IRC18:47
*** jamesmcarthur has joined #openstack-infra18:47
pabelangerEep18:47
jeblairpabelanger: i have an idea to fix it, i'll write a patch shortly18:48
*** ccamacho has quit IRC18:50
fungipabelanger: yeah, non-atomic replacement of files ansible is relying on, so it just sort of crumbles at that point18:50
*** jamesmcarthur has quit IRC18:51
*** ccamacho has joined #openstack-infra18:52
*** jsavak has quit IRC18:52
*** cody-somerville has joined #openstack-infra18:52
*** jsavak has joined #openstack-infra18:53
anteayafungi: jeblair storyboard has grown utc timestamps again: https://storyboard-dev.openstack.org/#!/story/818:53
fungianteaya: i saw that patch streak by... thanks SotK!18:53
anteayayay thanks SotK18:53
*** _nadya_ has quit IRC18:54
anteayaand Zara for reviewing and mergeing18:54
pabelangerjeblair: when you have a moment, would be interested in our feedback on 33082718:54
fungiabsolutely18:54
fungii need to step away and get some yardwork done before the rains come. bbiab18:54
anteayaenjoy sunshine18:54
anteayaand fresh air18:54
fungimore like grass clippings up my nose, but thanks!18:55
anteayawooo, even better18:56
anteayachlorophyl is good for you18:56
*** aysyd has quit IRC18:58
jeblairpabelanger: we should ask notmorgan to look at 330827 since he helped us make it so that we *don't* store host keys for workers :)18:58
zarozxiiro: cool, there’s also one that sets a build number, https://wiki.jenkins-ci.org/display/JENKINS/Next+Build+Number+Plugin18:59
notmorganjeblair: looking now18:59
*** javeriak has quit IRC18:59
zxiirozaro: neat!18:59
lifelessianw: a distro installed testtools or something perhaps? Got a pip install output to look at?18:59
zarozxiiro: python-jenkins has support for that one.19:00
*** sarob_ has joined #openstack-infra19:00
jeblairpabelanger, notmorgan: we did that because if we share a single known_hosts file, it will grow forever, and it will have old host keys from previous jobs.  so i think this approach of using a per-job file in 330827 is good.19:00
notmorganjeblair: sometimes i'm paying attention :)19:00
jeblairnotmorgan: i mostly said that to remind myself :)19:01
anteayajeblair: fungi so timeline will now look like this: https://storyboard-dev.openstack.org/#!/story/7 instead of https://storyboard.openstack.org/#!/story/2000407 since this patch has merged: https://review.openstack.org/#/c/327231/ can we call happiness on timelines then?19:01
notmorganhehehe19:01
notmorganso we are storing known hosts here - and we want to store them but per job?19:02
jeblairnotmorgan: i just put a comment in gerrit about *why* we want to do this19:02
jeblairnotmorgan: not sure if you saw that "excitement" from yesterday19:02
*** pvaneck has quit IRC19:02
notmorganright19:02
*** pvaneck has joined #openstack-infra19:02
*** ddieterly[away] has quit IRC19:03
*** amoralej is now known as amoralej|off19:03
*** aysyd has joined #openstack-infra19:03
*** sarob has quit IRC19:03
notmorganjeblair: that makes sense. the question is do we need to even store known hosts at all for the worker?19:04
jeblairnotmorgan: good question -- we are speculating that we were seeing problems with ansible logging into different instances competing for the same ip19:04
jeblairnotmorgan: and hoping/assuming that known_hosts would cause that to blow up if it happens again19:05
jeblairnotmorgan: but i do wonder about controlmaster19:05
notmorganjeblair: ahh19:05
notmorganok then.19:05
jeblairnotmorgan: like -- even if ansible does a bunch of ssh calls -- shouldn't controlmaster make them use the same connection anyway?19:05
notmorganthis makes a lot of sense :)19:05
jeblairbut we're off in the weeds here, so i dunno19:05
notmorganjeblair: yes.19:05
*** jamesmcarthur has joined #openstack-infra19:05
notmorganit should, but this is a good immediate metric of "oh hey, yeah this isn't happening"19:06
*** andreww has joined #openstack-infra19:06
jeblair(but maybe something was slow and controlmaster terminated and so we got a new connection)19:06
*** jamesmca_ has joined #openstack-infra19:06
jeblairnotmorgan, pabelanger, fungi, mordred: response on ticket: http://paste.openstack.org/raw/518495/19:07
*** e0ne has joined #openstack-infra19:08
notmorganhmm19:08
jeblairthere's some definite misunderstanding of the question there.  i don't think i am completely confident they have eliminated our hypothesis.19:08
*** javeriak has joined #openstack-infra19:08
notmorganok - well i have zero issue (just checking the code one more time) with this fix.19:08
mordredjeblair: I don't either19:08
notmorganjeblair: and we can continue to poke at the tickets / vms / etc19:08
jeblairi will ask if they can say whether the host that the old instance was on had any problems, or was removed from/added to service19:08
*** jamesmcarthur has quit IRC19:10
notmorganjeblair: oh ... uhm make sure HashKnowHosts is false19:10
notmorganjeblair: just in case we want to check the IP and not guess19:11
notmorganjeblair: otherwise this looks good.19:11
* notmorgan will add a comment19:11
notmorgani assume we're going to upload the knownhost file as well at the success/fail of the job?19:11
jeblairnotmorgan: yeah, the way the patch is written, that will happen19:12
odyssey4meif anyone has a moment to help me figure out what's wrong with a tox config patch ( https://review.openstack.org/331251 ) I'd appreciate it - the zuul log says it can't open the upper-constraints file, even though it also appears from http://logs.openstack.org/51/331251/1/check/gate-openstack-ansible-memcached_server-linters/22f608c/console.html#_2016-06-17_18_06_30_738546 that it should be there19:12
jeblairnotmorgan: (uploading will happen)19:12
*** sarob_ has quit IRC19:12
odyssey4mehrm, I just thought of the idea that perhaps it's not there even if an env var is set for it19:12
notmorganjeblair: +1 with the knownhosts comment - shouldn't matter, but hey...19:13
notmorgansometimes suspenders and a belt is the right answer19:13
jeblairnotmorgan: ah, your old patch put hash false for the zuul user in its .ssh/config19:15
jeblairso i think this is sufficient, but it would be good to make zuul more self-contained here19:16
notmorganjeblair: ++19:16
*** jamesmca_ has quit IRC19:16
*** jamesmcarthur has joined #openstack-infra19:17
*** gyee has joined #openstack-infra19:17
jeblairmordred: this would be so much nicer with your nova key thing19:20
*** jamesmcarthur has quit IRC19:22
*** sarob has joined #openstack-infra19:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Enable linters job to use upper-constraints  https://review.openstack.org/33131719:23
odyssey4meheh, found the problem - a speedy review by project-config cores on https://review.openstack.org/331317 would be appreciated19:24
sc68caloh yahoo. why u do dis? http://paste.openstack.org/show/518501/19:24
openstackgerritMerged openstack-infra/zuul: Enabled host_key_checking  https://review.openstack.org/33082719:25
*** piet has quit IRC19:25
*** piet has joined #openstack-infra19:26
rcarrillocruzasselin_: no, should not  be broken, not sure19:26
rcarrillocruzi'll take a look next week19:27
*** sarob has quit IRC19:27
*** ddieterly has joined #openstack-infra19:27
openstackgerritKhai Do proposed openstack-infra/system-config: run git gc instead of repack for git repos  https://review.openstack.org/32956619:28
pabelangerjeblair: notmorgan: Right, I too am hoping SSH blows up.19:28
anteayasc68cal: does invoking the word yahoo autmatically force the invoker to use text speak19:30
anteayasc68cal: I had given you credit for actually being able to speak in real sentences19:30
sc68calanteaya: heh. it's an internet meme / joke19:30
AJaegeranteaya: I think the zuul-git-prep-upper-constraints is fine19:30
AJaegerit just sets up a few more files - projects still need to use it19:31
zaroany infra-core want to enable git gc so we can see how gerrit perfoms next week?  https://review.openstack.org/#/c/32956619:31
anteayasc68cal: ah19:31
anteayaAJaeger: does project-config and system-config use constraints?19:31
anteayaI didn't think we did19:31
AJaegeranteaya: they don't - but no problem19:31
*** harlowja has joined #openstack-infra19:32
AJaegerthe change *allows* use to use constraints there.19:32
AJaegerbut that is still a choice in tox.ini19:32
AJaegerthe change just makes the file available19:32
AJaegerAnd then tox.ini needs to use it - and infra will just not use it19:32
*** gouthamr has quit IRC19:32
anteayaoh okay19:33
*** inc0 has quit IRC19:33
*** Jeffrey4l has quit IRC19:34
*** ociuhandu has quit IRC19:34
*** dimtruck is now known as zz_dimtruck19:34
odyssey4meAJaeger I haven't confirmed whether that job still does the work of the other job too... ie is it a wholesale replacement, just with some extras19:34
*** sarob has joined #openstack-infra19:35
AJaegerodyssey4me: pep8 and linters should be the same, just different tox invocation and name19:37
AJaegerodyssey4me: so, your change looks fine to me.19:37
odyssey4meAJaeger yeah, that's what I thought19:37
odyssey4meok cool19:37
anteayaodyssey4me: yeah that is what I am looking at: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n1319:37
mordredjeblair: many things would be nicer with my nova key thing19:38
pabelangerinteresting spike in zl05 used nodes: http://grafana.openstack.org/dashboard/db/nodepool19:38
odyssey4meanteaya ok, it seems pretty clear to me that zuul-git-prep and gerrit-git-prep are equivalent... they do the same thing in different ways19:38
pabelangernot sure what is happening there19:38
*** sarob has quit IRC19:39
odyssey4meanteaya and zuul-git-prep-upper-constraints is much the same, with the extra bit for upper constraints19:39
*** infra-red has joined #openstack-infra19:39
*** infra-red has quit IRC19:39
*** tonytan4ever has quit IRC19:39
AJaegeryep19:39
odyssey4meI also see now why my zuul clone mapping wasn't working. :p19:39
AJaegeras long as this is not run in a *post* job, it's fine - and that should be the case19:40
waynrzaro: any chance we can get a python-jenkins release any time soon?19:40
AJaegerbtw. could a project-config core review https://review.openstack.org/330725 , please?19:40
mordredpabelanger: the nodepool graph page is amazing19:40
*** hosanai has joined #openstack-infra19:41
pabelangermordred: ya, we're pretty even for used nodes across all our zuul-launchers. Which is awesome19:42
waynralso, does anyone know how we can get released versions of python-jenkins documentation on readthedocs? all I see (http://readthedocs.org/projects/python-jenkins/) is "master" and "stable"19:42
mordredit's also just pretty with the different streams crossing and stuff19:42
pabelangerbut, don't know why zl05 jumped by 30nodes at 19:00UTC19:42
anteayaodyssey4me AJaeger yes, I just needed to take the time to look at this19:43
pabelangermordred: https://review.openstack.org/#/c/331281/ should give us better control on the interval for the graphs too.  Right now we're doing 1m summarize, that adds up to 1d summarized19:43
mordredwoot19:44
*** rajinir has quit IRC19:44
*** hosanai has quit IRC19:46
*** mriedem has quit IRC19:46
jeblairpabelanger: wait, why do we need that?19:46
pabelangerjeblair: There is a video in the backscroll explaining what is does.  Let me find it19:47
pabelangerhttps://www.youtube.com/watch?v=CLXKlgHvOuo19:47
jeblairpabelanger: i know what it does which is why i'm asking :)19:47
*** Jeffrey4l has joined #openstack-infra19:47
odyssey4methanks AJaeger & anteaya :)19:47
pabelangerbasically, as you zoom out, I'd like to summarize to something other then secods19:48
rcarrillocruzjesusaur: hipchat died for me19:48
rcarrillocruzso19:48
pabelangerseconds*19:48
rcarrillocruzhttp://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/update_puppet.yaml19:48
*** jsavak has quit IRC19:48
anteayaodyssey4me: thank you19:48
rcarrillocruzgood thing is that you can pass to ansible-galaxy a yaml, containing what roles you want it to install19:48
jeblairpabelanger: those graphs are not a rate, so it does not make sense to summarize them19:48
anteayazaro: have you selected a gerrit topic you would like to use for the gerrit storyboard plugin patches?19:48
jesusaurrcarrillocruz: oh interesting19:49
*** samueldmq has quit IRC19:49
rcarrillocruzand the roles.yaml19:49
rcarrillocruzhttp://git.openstack.org/cgit/openstack-infra/system-config/tree/roles.yaml19:49
rcarrillocruzso that installs under /etc/ansible/roles19:49
rcarrillocruzcloud-launcher19:49
rcarrillocruzand puppet19:49
*** Apoorva has quit IRC19:49
rcarrillocruzyou could probably have a cron or something on the gozer ansible control machine19:49
hasharmordred: jeblair: congratulations on having phased out Jenkins and solely relying on ansible :]19:49
rcarrillocruzto have an up to date roles all the time19:49
pabelangerjeblair: Is there another function that allows use to change the frequency at which we plot on the graph?  I find 10sec to busy when I'm viewing over a day period19:50
anteayazaro: like to you consider the patch to fix gerrit plugin installation to be critial to getting the storyboard plugin on review-dev? https://review.openstack.org/#/c/259653/19:50
jeblairpabelanger: in fact, one of the nice things about grafana is that as you zoom out, those graphs aggregate automatically as needed19:50
jeblairpabelanger: it's based on the size of the graph and the time period being graphed19:50
jeblairpabelanger: change the time period in the top to, say 24 hours and you'll get something like 5 minutes for each point19:50
pabelangerjeblair: right, I was looking to adjust that data point more, so say 1 hour19:51
jesusaurrcarrillocruz: does galaxy add /etc/ansible/roles to the roles_path, or do I need to manage that myself?19:51
jeblairpabelanger: why do you want that?19:52
*** mriedem has joined #openstack-infra19:53
pabelangerjeblair: personal preference I guess.19:53
*** e0ne has quit IRC19:53
jeblairpabelanger: you know the default aggrecation function for summarize is 'sum'.  which means you will add all the values together.19:55
jeblairpabelanger: so that graph might look nicer, but will be wrong.19:55
*** markvoelker has quit IRC19:55
*** sarob has joined #openstack-infra19:55
jeblairpabelanger: so i think if you want to do that, you'll need to set it to avg19:56
jeblairpabelanger: https://graphite.readthedocs.io/en/0.9.10/functions.html#graphite.render.functions.summarize19:56
*** ddieterly is now known as ddieterly[away]19:57
jeblair(i like my reference documentation in the form of documentation, not videos :)19:57
pabelangerjeblair: right, it is possible I have it incorrect.  Let me explain what I was hoping for, and see if there is a better function to use19:57
*** raildo is now known as raildo-afk19:57
jeblairpabelanger: (also, consider smartSummarize over summarize)19:57
rcarrillocruzjesusaur: i believe the default of roles path is already /etc/ansible/roles19:57
rcarrillocruzbut regardless19:57
rcarrillocruzthat can be set on ansible.cfg19:57
openstackgerritMerged openstack-infra/project-config: Enable linters job to use upper-constraints  https://review.openstack.org/33131719:58
*** sarob has quit IRC19:58
rcarrillocruzhttp://docs.ansible.com/ansible/intro_configuration.html#roles-path19:58
nwkarstenclarkb, fungi: i think my issue was that the $ZUUL_PROJECT was being set to "nova" not "openstack/nova". do you know where to change that?19:58
*** sarob has joined #openstack-infra19:59
rcarrillocruzjesusaur: yeah, that's the default http://docs.ansible.com/ansible/galaxy.html#roles-path19:59
rcarrillocruzso it will just work as you expect19:59
pabelangerUsing Last 24 hours quick range, I would like to see the data plotted in the In Use panel to have a frequency of 1hr between data points.  That value, is the number of used nodes, we totaled for the hour19:59
pabelangerif that makes sense19:59
rcarrillocruzshould you need further locations for roles, just change roles_path on ansible.cfg19:59
*** piet has quit IRC20:00
jeblairpabelanger: you won't be able to get that from this metric -- you'll need a metrice that's a counter (number of nodes used incrementing over time) or a rate (number of nodes used in 10 second intervals)20:00
jeblairpabelanger: this metric is a guage, which means it's only updated when the number changes -- so you can't tell when an individual node is 'consumed'20:01
*** ddieterly[away] is now known as ddieterly20:01
pabelangerjeblair: okay, let me test in graphite.o.o using a different metric20:01
jeblairpabelanger: (for instance, if inside of a 10 second statsd interval, a node went from ready to used and a different node went from used to delete, this number will stay the name, and there's no way to tell what happened)20:02
jeblairpabelanger: yeah, i bet we have what you're looking for in a different metric :)20:02
*** nwkarsten has quit IRC20:06
pabelangerjeblair: Right, it looks like I incorrectly used that metric, because that was the only one (gauge) we used the node state.  We'd have to revisit nodepool and see if it made sense to add another data type20:06
jeblairpabelanger: i'd probably ask at this point what question you're trying to answer.... it's starting to sound like '# jobs zuul ran' which we have -- do you want '# of jobs zuul ran by launcher'?20:08
*** zz_dimtruck is now known as dimtruck20:08
pabelangerjeblair: I mostly was curious how many used nodes we had it a hour, in a day20:09
pabelangerthat how this all started20:09
*** nwkarsten has joined #openstack-infra20:09
jeblairpabelanger: that is fairly close to # jobs zuul ran (which does use summarize)... do you want to update that graph with the interval selector?20:09
pabelangerjeblair: but you are right, I likely can use another metric to see that data20:10
*** markvoelker has joined #openstack-infra20:10
jeblair(i mean the difference between those numbers is only the difference between two different ways of measuring the same thing.  they should be equivalent)20:10
jeblair(minus jobs run on static nodes)20:10
pabelangerjeblair: let me use graphite.o.o and see what that looks like, you are likely right20:11
*** piet has joined #openstack-infra20:13
*** ewindisch has joined #openstack-infra20:13
ewindischjust here to @mordred.20:14
*** ddieterly has quit IRC20:14
apuimedodhellmann: mriedem: If you could review https://review.openstack.org/#/c/331333/20:14
*** nwkarsten has quit IRC20:14
*** gouthamr has joined #openstack-infra20:14
apuimedoit would be very helpful20:14
apuimedokuryr is kind of blocked in its fullstack tests until this is fixed20:15
*** winggundamth_ has quit IRC20:15
mriedemapuimedo: you must have me confused with a requirements core20:16
mriedemthose guys told me i couldn't be part of the club20:16
mriedemthat my ideas were too 'radical'20:16
apuimedooh, sorry20:16
pabelangerjeblair: right, so I believe Zuul Jobs Launcher (per hour): http://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana/zuul-status.yaml#n161 provides that information I was looking for.  Minus the granularity of which launcher it happen on20:17
apuimedomriedem: I thouoght you were Matt20:17
mriedemapuimedo: no problem :) i'm joking about 1/2 of the above.20:17
jeblair💩ewindisch: hi there!20:17
*** akshai has quit IRC20:18
*** Apoorva has joined #openstack-infra20:18
* jeblair starts a new nick prefix style20:19
apuimedodims: could you check https://review.openstack.org/#/c/331333/20:19
dimsapuimedo : ack. will do20:19
apuimedodims: thanks a lot20:19
apuimedo:-)20:19
pabelangermtreinish: it looks like stackviz failed again, breaking our nodepool DIBs for the day.  Do you mind reviewing nodepool.o.o to see if we need to address anything?20:20
pabelangerall images were affected20:21
*** amitgandhinz has quit IRC20:21
timothyb89pabelanger: the cause was a broken dependency, I have already pushed a workaround: https://review.openstack.org/#/c/331256/20:21
mtreinishpabelanger: https://review.openstack.org/#/c/331256/20:22
timothyb89pabelanger: I'm testing DIB right now to be sure, but that patch got gate tests working again20:22
pabelangertimothyb89: thanks for the quick reply20:22
mtreinishoh, timothyb89 was faster20:22
*** ihrachys has joined #openstack-infra20:22
*** banix has quit IRC20:22
*** thorst_ has quit IRC20:23
mordredewindisch: yay me :)20:24
smcginnisAny zuul issues right now?20:30
smcginnisHad a patch get +A earlier and it never showed up in gate queue.20:30
smcginnisFigured it was the earlier restart.20:30
smcginnishttps://review.openstack.org/#/c/327285/20:30
smcginnisHad them do a reverify on it and it kicked off in the check queue again.20:30
smcginnisPassed but never picked up in the gate queue again.20:30
smcginnisNot a dependent patch either.20:31
*** acoles is now known as acoles_20:31
anteayasmcginnis: you figured out early that we programmed zuul to prevent your patches20:31
smcginnisanteaya: Haha :)20:31
*** chem` has joined #openstack-infra20:32
smcginnisif cinder: sys.exit()20:32
pabelangerzuul was restarted between 16:08 and 16:30 UTC, so events from gerrit during that time would not have been picked up by zuul20:32
anteayathat's the line!20:32
anteayacheck results posted as success at 19:4120:33
*** chem has quit IRC20:33
anteayaqueue lengths are 0, I don't see it on the status page20:34
odyssey4meI've seen a bit of chatter about some jobs passing tests as far as the log is concerned, but failing from a job standpoint... An example is http://logs.openstack.org/00/331100/1/gate/gate-openstack-ansible-repo_server-linters/54bbdb5/console.html / https://review.openstack.org/33110020:34
odyssey4meIt seems that it's a known issue, and the workaround at this point is simply to recheck.20:35
odyssey4meAny further pointers? Can we help in any way?20:35
anteaya2 conflicts with other patches, but no parents and no Depends-On:20:35
anteayasmcginnis: I think we are going to have to impose on an infra-root to check zuul logs for https://review.openstack.org/#/c/327285/20:36
pabelangerodyssey4me: this looks to be a timeout issue. If you look at http://logs.openstack.org/00/331100/1/gate/gate-openstack-ansible-repo_server-linters/54bbdb5/_zuul_ansible/logs/ansible.txt you can see that our async task did complete20:36
odyssey4mepabelanger ah: fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time"}20:37
odyssey4meI saw a patch go in recently to change a setting related to timeouts20:37
jeblairit kind of looks like that job took about 2 minutes to run though?20:37
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend  https://review.openstack.org/33067820:38
anteayasmcginnis: nikesh's reverify comment didn't help, but I don't think it hurt anything20:38
openstackgerritMikhail S Medvedev proposed openstack-infra/ciwatch: Cleanup before Jenkins to Zuul rename  https://review.openstack.org/33134620:38
openstackgerritMikhail S Medvedev proposed openstack-infra/ciwatch: Support Jenkins to Zuul rename  https://review.openstack.org/33134720:38
odyssey4mes/go in/be submitted or perhaps merge/20:38
jeblair2016-06-17 20:24:02,589 DEBUG zuul.NodeWorker.ubuntu-trusty-rax-ord-1777421: Ansible command: ['ansible-playbook', '/tmp/tmp3gMsXA/ansible/playbook', '-e', 'timeout=1800', '-v']20:38
*** ihrachys has quit IRC20:38
jeblairodyssey4me, pabelanger: ^ so it had an 1800 second timeout (30 minutes)20:39
smcginnisanteaya: Yeah, strange that it kicked off check again, but no luck with gate.20:39
odyssey4meI mentioned the same thing this morning: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-17.log.html#t2016-06-17T10:10:0820:40
jeblairsmcginnis, anteaya: i can look at it after i track this one down20:40
pabelangerI would be interested to see what timeout we passed into async for that task20:40
asselin_rcarrillocruz, read the scroll back on ansible & system-config. Thanks for the links. Will continue looking into it.20:40
anteayajeblair: yup, thanks20:40
odyssey4meanother bit I think might be related: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-17.log.html#t2016-06-17T12:11:0120:40
jeblairodyssey4me: this is a better case because we had much incorrect about timeouts this morning.  but this is recent enough it should have all fixes20:40
anteayasmcginnis: in case that reverify did interrupt things, do you want to try a +! workflow on it?20:40
anteaya+120:40
smcginnisanteaya: Worth a shot.20:41
anteayasmcginnis: we are waiting anyway20:41
jeblairpabelanger: agree, though i don't know a good way to get that.  we could do a zuul_log of "Remaining timeout {{ timeout }}"20:42
smcginnisanteaya: And there it is, happily waiting in the gate queue.20:42
odyssey4meok, I see the various tons of chatter from 13:00 onwards20:42
anteayasmcginnis: there it goes20:42
smcginnisanteaya: Thanks!20:42
anteayasmcginnis: welcome, looks like the reverify comment did short circuit things20:42
*** hosanai has joined #openstack-infra20:42
jeblairanteaya, smcginnis: did it already have a +1 or +2 before the recheck by any chance?20:43
jeblair(from jenkins)20:43
pabelangerjeblair: I think zuul_log would be a good location. Our only other way would be to add a debug task because each zuul_runner with the msg={{ timeout }}20:43
pabelangerwhich grows our playbooks20:43
smcginnisjeblair: Yeah, +2/+A, so I thought the reverify should give it a kick.20:43
smcginnisjeblair: But only moved after I removed my vote, then did +2/+A again.20:44
jeblairsmcginnis: it had a +2 from jenkins?20:44
anteayajeblair: it had a +1 Verifying from jenkins yes20:44
anteayathen a +2 from smcginnis, then +2 +A from xing20:44
jeblairanteaya: ah, a +1 makes more sense.  in that case, gerrit does not emit an event that zuul can process (because "nothing changed").  clarkb has a patch to gerrit to change that.  i don't know the status.20:44
anteayathen 'reverify'20:44
*** rhochmuth has joined #openstack-infra20:45
anteayathank you20:45
jeblairanteaya, smcginnis: in this case, -W +W is the way to get things going20:45
anteayathanks20:45
smcginnisjeblair: OK, thanks.20:45
anteayasmcginnis: and please tell nikesh to ease back on the recheck/reverify20:45
pabelangerjeblair: I'd be in favor of adding http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n165 by zuul_log for each zuul_runner task we do20:45
jeblairpabelanger: debug task might be good --20:46
pabelangerjeblair: let me do a quick test to see if -v will output it20:46
jeblairpabelanger: it'll end up in the ansible log, so people can grab it20:46
jeblairpabelanger: but if we want it in the console log, that works too20:47
jeblairpabelanger: i'm kind of on the fence (mostly wondering whether it's useful or noise in the console log)20:47
*** hosanai has quit IRC20:47
clarkbjeblair zaro got it in iirc but would be 2.1220:47
jeblairanteaya: woot ^20:47
pabelangerjeblair: same, I mean, it is useful now because there is an issue. But in that case, ansible.log makes more sense20:48
clarkbI am not sure any of the code I fiddled with made it but the fix is there for the future so yay20:48
jeblairpabelanger: cool, let's start there20:48
*** banix has joined #openstack-infra20:48
jeblairpabelanger: (and actually, now that we upload the log, maybe we should drop task exit code line20:48
anteayajeblair: clarkb zaro yay!20:48
*** thorst_ has joined #openstack-infra20:49
odyssey4mehmm, perhaps to ask the stupid question - why is the task async at all?20:49
jeblairodyssey4me: without it, we hit ssh timeout errors20:49
jeblairodyssey4me: (on, eg, huge devstack jobs)20:49
anteayaclarkb: sometimes fiddling with the code is enough to get the attention of the folks who will actually fix it20:49
*** mhickey has quit IRC20:49
anteayaclarkb: so thanks for that20:49
jeblairodyssey4me: (also, bonus, async seems to be the best way to have ansible time out a task)20:50
jeblairodyssey4me: (current situation notwithstanding :)20:50
openstackgerritPeter Stachowski proposed openstack-infra/project-config: [trove] Add trove-dashboard to PROJECTS  https://review.openstack.org/33135220:51
jeblairodyssey4me, pabelanger: yeah, i don't have an answer for this other than it sure does look like ansible decided to abort the task after 24 seconds.  i would feel a lot better about proceeding with debugging this if we had a log line right above that that either did or did not say that we set the timeout to 24 seconds.20:52
notmorganjeblair: so, i havea  question for you re: the py3 (encode/decode/encode) things. if we're still replicating all the py3 conversions to the zuulV3 branch, I'll do that cleanup there - if not i can do the more intelligent encode/decode stuff here.20:52
odyssey4mejeblair hmm, yeah I guess if you're doing huge monolithic scripts executed by the command/shell module then that would be an issue unless you have something external to kill the task20:52
notmorganjeblair: i am thinking of adding a20:52
notmorgan"auto-encode/decode" interface directly in gear so I don't need to battle with gear looking at something different/receiving different output that zuul does.20:53
pabelangerjeblair: odyssey4me: this is what we'd see in the ansible.log for example: http://paste.openstack.org/show/518616/20:53
pabelangerare we okay with that?20:53
rcarrillocruzasselin_: if by scrollback you mean hipchat, i'll check it out on Monday, cos I can't connect...20:53
odyssey4mejeblair the ssh timeout jobs, though, might be a different problem - we've seen that from time to time depending on various settings... perhaps with a little more info on that one we can find an alternative solution20:53
notmorganwhere a subclass is going to have to look a lot like that anyway.20:53
notmorganmordred: ^ cc re jeblair's comment on https://review.openstack.org/#/c/325661/20:54
*** trown is now known as trown|outtypewww20:54
asselin_rcarrillocruz, no I mean here (and the start over there). hipchat is down.20:54
notmorganShrews: ^ cc as well.20:54
rcarrillocruzah ok20:54
* rcarrillocruz checks20:54
odyssey4mepabelanger jeblair in terms of noise versus value, a few lines to give more debugging capacity versus no info at all... I'd rather have more lines20:54
jeblairnotmorgan: okay, i like the idea of being able to make this easier for everyone in gear -- i worry about what that interface will look like (which i don't have to worry as much about in a zuul subclass)... maybe you can sketch it out before doing all the work?20:54
notmorganjeblair: basically i was going to add supplemental interface that was an @property for the name/unique/etc with setter/getter that encodes/decodes20:55
notmorganjeblair: haven't set on the "name"20:55
*** kgiusti has left #openstack-infra20:56
asselin_rcarrillocruz, too bad those os_network ansible roles don't use my clouds.yaml config for cert verify=false20:56
notmorganjeblair: so if you reference .name, it's default like it is today - or make .name the auto-encode/decode and an internal property gear references20:56
notmorganjeblair: i haven't decided yet. was mulling on that already actually :)20:56
notmorganspecifically because of this patch.20:57
jeblairnotmorgan: yeah, i like that gear is very 'pythonic' [aside: the fact that it isn't in python3 suggests something to me about how 'pythonic' python3 is]20:57
rcarrillocruzasselin_: yeah, that's a bug20:58
jeblairnotmorgan: so i feel like just being able to say 'job.name' is important.  so figuring out how to make that sensible in the most cases is important.20:58
rcarrillocruzi would have sworn i hit that and mordred fixed it on shade...20:58
rcarrillocruzi will search on git20:58
rcarrillocruzasselin_: feel free to open a bug on ansible20:58
rcarrillocruzi'll take it20:59
asselin_launchpad?20:59
rcarrillocruzgithub20:59
notmorganjeblair: right so i'll plan job.name to be public interface and have job._gear_thing_whatever_it's_called [though also public if someone wants to poke at it] be the way gear manages the byte-requirement of the policy. but this does imply we'll be breaking anyone (only us?) with that upgrade and did it the old way.20:59
odyssey4mepabelanger jeblair I have a few more sample of data if you'd like them?20:59
notmorgans/policy/properties20:59
asselin_https://github.com/ansible/ansible ?20:59
rcarrillocruzi think should be here21:00
rcarrillocruzhttps://github.com/ansible/ansible-modules-core/issues21:00
jeblairnotmorgan: maybe we should have an email thread21:00
notmorganjeblair: or, i could just make a new class. anyway, I'll look at the options. But the auto-encode/decode should absolutely be part of gear itself.21:00
notmorganjeblair: ++.21:00
asselin_rcarrillocruz, ok, still getting up to speed on ansible and seeing how the pieces fit together.21:00
jeblairnotmorgan: well, i think doing this in gear can be good -- but i'm not 100% convinced it's the right/only way.  gear also wants to be a low-level protocol interface.21:01
rcarrillocruzyep21:01
rcarrillocruzlet me know how the launcher works for you21:01
rcarrillocruzalso21:01
rcarrillocruzhttps://review.openstack.org/#/c/320466/11/playbooks/launch_node.yml21:01
rcarrillocruzto replace the launch_node with the cloud launcher21:01
rcarrillocruzin case you want to look at more examples21:01
notmorganjeblair: sure. but it feels like this is a usability thing. we're not really changing the protocol, just how someone interfaces with the bytes vs string requirements in py3. In py2 this "just works", and i like that.21:02
notmorganjeblair: i'll get an email thread going on this though.21:02
*** davidlenwell has quit IRC21:02
jeblairnotmorgan: yeah, we spent a lot of time trying to get this right.  i'm surprised it's so terrible in python3.21:03
mordredjeblair: I like a ton of the new stuff in python3 - the bytes/string handling is not one of the things21:03
notmorganjeblair: the difference between py2 and py3 bytesvstext seems to be py3 is consistently bad in the same way - py2 was inconsistently bad.21:03
asselin_rcarrillocruz, I like the separation from infra-ansible. I'm getting stuck on creating the sub-network. http://paste.openstack.org/show/518619/21:03
notmorganjeblair: so it was much harder to "Get right" in py221:03
openstackgerritPaul Belanger proposed openstack-infra/zuul: Add debug task before zuul_runner  https://review.openstack.org/33135521:03
odyssey4mepabelanger jeblair ok, I'm out for the weekend... have a great one everyone - catch you next week!21:04
rcarrillocruzpaste me your resources.yml21:04
pabelangerjeblair: odyssey4me: ^ for our debug command21:04
notmorganjeblair: also..  b'abdc'[0] => 9721:04
mordrednotmorgan: yah - I think in py2 it's almost impossible to get non-ascii right. in py3 it's ALSO hard to get ascii right, but it's consistently hard no matter whether you're doing ascii or not21:04
*** javeriak has quit IRC21:04
notmorganjeblair: in py321:04
notmorganmordred: exactly21:04
odyssey4mepabelanger FYI https://review.openstack.org/#/c/331076/ and https://review.openstack.org/#/c/331100/ both have the timeout issue in their logs if that matters21:05
pabelangerodyssey4me: thanks, I will look at them21:05
asselin_rcarrillocruz, nice patch....I essentially did the same thing in a less elegant way. I'll try it out21:05
jeblairnotmorgan: i think we have some assumptions in gear that "strings" in py2 are to be encoded as utf821:06
jeblairnotmorgan: we may be in a slightly better position than we think to do automatic decoding21:06
*** burgerk has quit IRC21:06
jeblairnotmorgan: see the "convert_to_bytes" function in gear21:06
rcarrillocruzi want to put the generate tmpkeypair on the role itself21:06
notmorganjeblair: yeah.21:06
notmorganjeblair: i noticed that.21:07
rcarrillocruzif a server does not have a keypair passed as param, create a temp one and return it after the task finishe21:07
rcarrillocruzs21:07
jeblairnotmorgan: anyway, this is complex, and an email will i think give the space to help page everything in :)21:07
notmorganjeblair: which is why i just did encode to utf-8 directly vs trying to do platform locale lookups.21:07
rcarrillocruzneed to get to it21:07
notmorganjeblair: ++21:07
mordredjeblair: ++21:07
notmorganjeblair: i'll rebase the current patch and get that email sent- so we can keep working on moving forward.21:07
notmorganregardless, the rebase has to happen :)21:08
jeblairok21:08
jeblairodyssey4me: yes, i'd love more sample data! :)21:08
jeblairodyssey4me: ah i see you noted them for pabelanger, got it, thanks21:09
fungianteaya: SotK: Zara: new timelines on storyboard-dev.o.o lgtm. awesome!21:09
funginibalizer: crinkle: regarding https://review.openstack.org/329566 if you rename a cron resource do you need to ensure=>absent the old name?21:09
pabelangerjeblair: I need to step away for a bit, please take over 331355 if you need too21:10
*** croelandt has quit IRC21:10
*** hashar has quit IRC21:10
jeblairpabelanger, odyssey4me: it seems strange this happens so often on the linters job21:11
nibalizerfungi: yea i think so21:11
anteayafungi: yay, thank you for looking21:11
jeblairmordred: can you +3 https://review.openstack.org/331355 ?21:11
anteayaZara: SotK just knocking them out of the park, you are21:12
SotKfungi: woo, just need to get rid of pagination now!21:12
*** daemontool has joined #openstack-infra21:12
jeblairmordred: i just tried 355 and it doesn't look right to me21:12
anteayaSotK: you have a patch up for that21:12
*** thorst_ has quit IRC21:12
anteayaSotK: and it looks good to me21:12
*** davidlenwell has joined #openstack-infra21:13
jeblair2016-06-17 14:12:19,778 DEBUG zuul.NodeWorker.test: Ansible output:     "msg": "Async"21:13
jeblairmordred, pabelanger: i got that ^21:13
jeblairi think we need to use data structures, not strings.  respinning.21:13
anteayahttps://review.openstack.org/#/c/327682/421:14
mordredjeblair: I agree21:14
jeblairi'm just testing now with actual timeout values to make sure it dtrt21:15
*** Guest28113 has quit IRC21:15
*** piet has quit IRC21:15
*** andreww has quit IRC21:15
*** andreww has joined #openstack-infra21:16
jeblairmordred: hrm.  it outputs the passed timeout value every time, not the one that's updated by the timeout callback plugin21:17
jeblair(which is a "nonpersistent fact")21:18
jeblairmordred: i *thought* i had ensured that async was picking up the updated timeout value, now i am not certain of that either.21:18
*** ldnunes has quit IRC21:18
*** chem` has quit IRC21:19
*** cody-somerville has quit IRC21:20
*** gordc has quit IRC21:21
*** markvoelker has quit IRC21:21
*** amitgandhinz has joined #openstack-infra21:22
mordredjeblair: that testing implies the same concern to me21:22
*** gouthamr has quit IRC21:22
crinklefungi: yes21:24
fungicrinkle: thanks!21:24
jeblairmordred: the async_wrapper command line has a number on it which looks like it is the timeout.  it doesn't appear to change from one task to the next, so it also seems to be using the original rather than updated value21:26
*** amitgandhinz has quit IRC21:27
jeblairthis is annoying, because it means our timeouts are inaccurate -- but only in the direction of being too long.  so it should not account for the error odyssey4me saw21:28
*** lascii is now known as alaski21:28
*** rfolco has quit IRC21:32
*** dmk0202 has quit IRC21:33
mordredjeblair: yes. that is doubly annoying21:33
anteaya<-- going for a walk21:34
*** dmk0202 has joined #openstack-infra21:34
jeblairmordred: do you have any ideas?21:36
*** thorst has joined #openstack-infra21:36
*** mdrabe has quit IRC21:36
*** erlon has quit IRC21:36
*** dprince has quit IRC21:36
asselin_rcarrillocruz, do you know if creating and attaching volumes works? I didn't see it in the sample, and having issues with it.   "msg": "Volume {'name': 'gfs02', 'volume_type': '3par-vol', 'size': 500} is not a valid volume in gpc-asselin:region1"}21:37
*** thorst has quit IRC21:39
*** thorst has joined #openstack-infra21:40
*** abregman|afk has quit IRC21:41
*** bhavik1 has joined #openstack-infra21:42
*** bhavik has quit IRC21:44
*** bhavik1 is now known as bhavik21:44
*** PollyZ has quit IRC21:45
*** xyang1 has quit IRC21:48
*** thorst has quit IRC21:49
*** dmk0202 has quit IRC21:50
zarowaynr: i’ll do python-jenkins release early next week.21:51
zarowaynr: i thought RTD automatically builds docs from all the tags but it doesn’t look like it.  took a quick look but don’t see how i can change it to do that.21:51
*** daemontool has quit IRC21:53
zaroanteaya: none of my patches for storyboard plugin are ready for review yet so i have not assigned a topic yet.21:54
zaroanteaya: what was your question regarding change https://review.openstack.org/#/c/259653/21:55
mordredjeblair: I do not. I also stepped away for just a moment- back in a few21:56
jeblairmordred: i'm also not certain that the pre_tasks thing we did in 330827 is working21:56
*** mriedem has quit IRC21:59
*** jcoufal has quit IRC21:59
openstackgerritScott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend  https://review.openstack.org/33067822:00
openstackgerritJames E. Blair proposed openstack-infra/zuul: Revert "Enabled host_key_checking"  https://review.openstack.org/33136222:01
jeblairpabelanger, mordred, notmorgan: ^ i think the ansible is not right for that -- i don't see the keyscan running22:01
jeblairi'm going to push that through until we can be more sure of it22:01
*** esberglu has quit IRC22:02
openstackgerritMorgan Fainberg proposed openstack-infra/zuul: Python 3 Fixes: Bundled Fixes  https://review.openstack.org/32566122:05
openstackgerritMorgan Fainberg proposed openstack-infra/zuul: Update gear requirement  https://review.openstack.org/32617322:05
openstackgerritMorgan Fainberg proposed openstack-infra/zuul: Python 3 fix: GC final state inspection  https://review.openstack.org/32743622:06
*** vhosakot has quit IRC22:07
notmorganjeblair: hmm.22:11
asselin_rcarrillocruz, per your request: https://github.com/ansible/ansible-modules-core/issues/398122:11
notmorganjeblair: i don't know22:11
*** baoli has quit IRC22:13
openstackgerritKhai Do proposed openstack-infra/system-config: run git gc instead of repack for git repos  https://review.openstack.org/32956622:13
*** dimtruck is now known as zz_dimtruck22:13
*** banix has quit IRC22:13
*** pradk has quit IRC22:13
*** bhavik has quit IRC22:16
*** matt-borland has quit IRC22:18
*** zz_dimtruck is now known as dimtruck22:18
*** dimtruck is now known as zz_dimtruck22:20
*** tphummel has quit IRC22:23
*** amitgandhinz has joined #openstack-infra22:23
*** tphummel has joined #openstack-infra22:25
*** amitgandhinz has quit IRC22:27
*** tpsilva has quit IRC22:27
*** jcoufal has joined #openstack-infra22:30
anteayazaro: do you require 259653 to be merged for the storyboard plugin to be on review-dev22:32
*** banix has joined #openstack-infra22:32
zarono, that’s to fix upgrading plugins.  for example if a sb-1.0 already exists and we upgrade to sb-2.022:33
anteayazaro: okay well I still would really like the storyboard plugin to be on review-dev by the sprint next week, and am very keen to know what patches are required so I can review and help you plus recruit other reviewers22:33
anteayaah okay thanks22:33
anteayaregardless of the shape of them if I can at least envision how many are required, that you know of at this point, that would help me22:33
*** mixos has quit IRC22:35
zaroanteaya: topic is "its-storyboard"22:36
anteayathank you22:37
anteayathanks so much: https://review.openstack.org/#/q/topic:its-storyboard22:38
*** sdake has quit IRC22:38
*** rhochmuth has quit IRC22:41
*** sdake has joined #openstack-infra22:41
*** rlandy has quit IRC22:43
*** hosanai has joined #openstack-infra22:44
*** thorst has joined #openstack-infra22:45
*** aysyd has quit IRC22:45
*** hosanai has quit IRC22:48
mordredjeblair: I have a wild idea on the timeout issue22:49
pabelangerjeblair: sure. I'm going to test it again locally now22:49
mordredjeblair: perhaps there is a timing/scope issue with the non-persistent facts - and maybe they're not in-scope at the right time22:49
mordredjeblair: by maybe if we register the timeout value at the end of each task, the next task will pick up the value22:50
*** thorst has quit IRC22:53
openstackgerritMonty Taylor proposed openstack-infra/zuul: Register the timeout variable to pass to the next task  https://review.openstack.org/33137122:54
mordredjeblair: something like that ^^22:54
*** edmondsw has quit IRC22:59
dmsimardmordred: you can persist the fact, no need to keep it in-memory23:02
mordreddmsimard: what do you mean?23:03
dmsimardif the issue is that you don't want to lose the variable from task to task23:03
dmsimardyou can persist it23:03
*** jcoufal has quit IRC23:03
mordredin this case the variable is something that's injected by a callback plugin23:03
mordredwe start with a total timeout value, and then with each task we run it through a callback plugin and inject the new timeout value based on the elapsed time in the previous task23:04
dmsimardhm, I've never done something like that before, not sure my solution makes sense in that context23:05
mordredit's a pretty edge case :)23:05
*** rfolco has joined #openstack-infra23:05
dmsimardbut basically you can have /etc/ansible/facts.d/whatever.fact and then have ini-style fact declaration in there23:06
mordredyah. in this case it's the rewriting between tasks that's killing us I think23:06
dmsimardmordred: btw: https://review.openstack.org/#/c/330874/23:07
dmsimardI'm out for the weekend now though, catch you later23:08
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul: WIP: Integrate ARA reporting inside Zuul Launcher  https://review.openstack.org/33087423:11
*** IlyaG has joined #openstack-infra23:13
*** ffrank has joined #openstack-infra23:16
*** markvoelker has joined #openstack-infra23:16
pabelangerjeblair: I'll wait for you to debug locally ssh-keyscan, the only thing I can think of is maybe a path issue.  I was able to extract the code for a locally test and the playbook looked to be correct23:16
*** bmjen has joined #openstack-infra23:19
*** sarob has quit IRC23:19
*** sarob has joined #openstack-infra23:20
openstackgerritSpencer Krum proposed openstack-infra/vinz: Test commit  https://review.openstack.org/33137723:23
*** amitgandhinz has joined #openstack-infra23:24
*** rfolco has quit IRC23:24
fungiheh23:27
*** hongbin has quit IRC23:27
*** amitgandhinz has quit IRC23:28
mordredwow23:28
nibalizeri just wanted to test23:31
nibalizerhaha the tests are failing on that repo with no content23:31
*** andreww has quit IRC23:35
*** claudiub has quit IRC23:38
waynrzaro: cool23:43
*** denisra has quit IRC23:43
*** hosanai has joined #openstack-infra23:44
*** yamahata has quit IRC23:45
*** eharney has quit IRC23:46
*** gyee has quit IRC23:46
*** hosanai has quit IRC23:49
*** asselin_ has quit IRC23:49
*** zz_dimtruck is now known as dimtruck23:50
*** thorst has joined #openstack-infra23:51
*** weshay has quit IRC23:52
*** Sukhdev has quit IRC23:55
*** thorst has quit IRC23:59
« Thursday, 2016-06-16 Index Saturday, 2016-06-18 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!