| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 00:01 |
|---|---|---|
| *** boden has joined #openstack-infra | 00:02 | |
| *** baoli has quit IRC | 00:03 | |
| *** andreas_s has joined #openstack-infra | 00:04 | |
| *** claudiub|2 has quit IRC | 00:04 | |
| *** dingyichen has joined #openstack-infra | 00:05 | |
| *** rkukura has quit IRC | 00:07 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Make the git web url a template https://review.openstack.org/515168 | 00:08 |
| *** andreas_s has quit IRC | 00:08 | |
| *** andreas_s has joined #openstack-infra | 00:09 | |
| *** boden has quit IRC | 00:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Make the git web url a template https://review.openstack.org/515168 | 00:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3 https://review.openstack.org/515231 | 00:14 |
| ianw | pabelanger: ^ | 00:14 |
| *** andreas_s has quit IRC | 00:15 | |
| *** andreas_s has joined #openstack-infra | 00:15 | |
| *** Swami has quit IRC | 00:16 | |
| pabelanger | ianw: left comment | 00:16 |
| *** gildub has joined #openstack-infra | 00:16 | |
| ianw | pabelanger: but only want to add them when they're up right? | 00:17 |
| ianw | or does that only do firewall config | 00:18 |
| pabelanger | ianw: yah, that just opens firewall | 00:18 |
| pabelanger | should be safe to do now | 00:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3 https://review.openstack.org/515231 | 00:20 |
| ianw | ok, maybe if names don't resolve it's temporary failiure. ^ | 00:20 |
| *** florianf has quit IRC | 00:20 | |
| *** andreas_s has quit IRC | 00:21 | |
| *** andreas_s has joined #openstack-infra | 00:21 | |
| *** links has joined #openstack-infra | 00:23 | |
| *** wolverineav has quit IRC | 00:26 | |
| *** andreas_s has quit IRC | 00:27 | |
| *** wolverineav has joined #openstack-infra | 00:27 | |
| *** cody-somerville has joined #openstack-infra | 00:27 | |
| *** cody-somerville has joined #openstack-infra | 00:27 | |
| ianw | infra-root: ^^ anyone able to sanity check ... don't want to bring up new hosts and have puppet roll out the zuulv2 treatment to them | 00:27 |
| *** andreas_s has joined #openstack-infra | 00:27 | |
| *** csomerville has quit IRC | 00:29 | |
| jeblair | ianw: one nit. otherwise lgtm. we'll need to restart the relevant firewalls after the hosts are re-added to dns. | 00:30 |
| jeblair | (and after dns has propagated) | 00:31 |
| *** mummer has quit IRC | 00:31 | |
| *** wolverineav has quit IRC | 00:31 | |
| *** andreas_s has quit IRC | 00:32 | |
| *** Apoorva_ has joined #openstack-infra | 00:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3 https://review.openstack.org/515231 | 00:32 |
| ianw | jeblair: ^ ... firewalls is in the plan (https://etherpad.openstack.org/p/zuulv2-removal) :) | 00:33 |
| *** andreas_s has joined #openstack-infra | 00:33 | |
| *** Apoorva has quit IRC | 00:35 | |
| *** Apoorva_ has quit IRC | 00:36 | |
| *** baoli has joined #openstack-infra | 00:37 | |
| *** andreas_s has quit IRC | 00:44 | |
| dhinesh | what is the ideal location for ssh keys when building an image using nodepool-builder, so that i can ssh to that instance? | 00:48 |
| ianw | dhinesh: well ... that depends | 00:49 |
| ianw | are you using devuser to add a user, or all the infra elements? | 00:49 |
| *** bobh has joined #openstack-infra | 00:51 | |
| dhinesh | images: | 00:52 |
| dhinesh | - name: dpc | 00:52 |
| dhinesh | min-ram: 8192 | 00:52 |
| dhinesh | diskimage: dpc | 00:52 |
| dhinesh | username: jenkins | 00:52 |
| dhinesh | private-key: '/home/nodepool/.ssh/id_rsa' | 00:52 |
| dhinesh | i currently have a nodepool.yaml, with entries like above | 00:52 |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Add docs job for VPNaaS and VPNaaS dashboard https://review.openstack.org/514906 | 00:52 |
| dhinesh | ianw: sorry for the weird paste | 00:52 |
| dhinesh | ianw: but this does not seem to work | 00:52 |
| *** andreas_s has joined #openstack-infra | 00:53 | |
| ianw | dhinesh: it's going to pick up the ~/.ssh/id_rsa of the building user IIRC | 00:54 |
| *** kiennt26 has joined #openstack-infra | 00:55 | |
| ianw | but, if you're using glean, that should allow you to put other keys into the config-drive that will be added to root | 00:55 |
| *** namnh has joined #openstack-infra | 00:55 | |
| ianw | that's how we get the infra root's keys on there | 00:55 |
| *** cuongnv has joined #openstack-infra | 00:58 | |
| mnaser | is there a way of knowing if zuulv3 is running with a specific change merged? | 00:58 |
| mnaser | more specifically, https://review.openstack.org/#/c/514744/ | 00:58 |
| *** LindaWang has joined #openstack-infra | 00:59 | |
| clarkb | ianw: dhinesh it will pick up the ~/.ssh/id_rsa.pub | 01:00 |
| clarkb | image build sshould never touch te private key only ssh'ing into instances does that | 01:00 |
| clarkb | mnaser: you can check the zuul version on the status page | 01:01 |
| clarkb | mnaser: thta change is running iirc | 01:01 |
| mnaser | oo | 01:01 |
| mnaser | id have to somehow match the version which seems like a lot of work :p but ill take your word for it and report back if it breaks | 01:01 |
| *** rwsu has joined #openstack-infra | 01:02 | |
| clarkb | mnaser: git describe is the magic to check typically | 01:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 01:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 01:03 |
| mnaser | darn | 01:04 |
| mnaser | jeblair maybe i'm missing things but it looks like the zuulv3 issue we reported about job variants was moved to the fixed etherpad but i still see it occur | 01:04 |
| * mnaser has been out of the loop for a day or two | 01:04 | |
| jeblair | mnaser: yeah, restarted with that about 10 hours ago. | 01:05 |
| mnaser | i just tried a recheck on https://review.openstack.org/#/c/512835 and it tried to run puppet-openstack-integration-4-scenario004-tempest-{centos-7,ubuntu-xenial} again | 01:05 |
| mnaser | it hasn't reported back but i can see it in the zuul status page | 01:05 |
| *** dave-mccowan has quit IRC | 01:05 | |
| mnaser | and just reported back | 01:06 |
| jeblair | mnaser: you may be running into the problem that i tracked down with EmilienM earlier. is it a job that inherits from another job that has file exclusion matchers and is on multiple branches? | 01:06 |
| *** andreas_s has quit IRC | 01:06 | |
| mnaser | the job inherits another job indeed (re-using the base job which handles common logic) but no file exclusion matchers, however, we have a branch matcher on scenario004 | 01:08 |
| *** andreas_s has joined #openstack-infra | 01:08 | |
| mnaser | jeblair sorry, i lied, the base job has file exclusion matchers too | 01:09 |
| mnaser | so yeah, looks like its the same issue | 01:09 |
| jeblair | mnaser: okay, that's almost certainly it. the good news is that i just added a test for that to my WIP change. the bad news is it probably won't be in production before friday. | 01:09 |
| *** tinwood has quit IRC | 01:10 | |
| mnaser | jeblair its ok, as long as we have forward progress :D | 01:10 |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job https://review.openstack.org/511784 | 01:10 |
| mnaser | jeblair good luck and thanks again :) | 01:10 |
| *** tinwood has joined #openstack-infra | 01:11 | |
| jeblair | mnaser: if it's urgent, we can probably make it work by adding or removing file matchers, but if you can wait, it will reduce the amount of temp work we have to undo | 01:11 |
| *** dave-mccowan has joined #openstack-infra | 01:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 01:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 01:12 |
| mnaser | jeblair i know the issue is currently blocking the version bump for queens milestone 1 for some modules which puts us behind, but mwhahaha takes care of release so it's his call - https://review.openstack.org/#/q/topic:prep-release-queens-1+status:open --- or perhaps infra can merge this for us (if its possible and if mwhahaha prefers so we can move with release) | 01:13 |
| mnaser | i'll leave that up to him :> | 01:13 |
| dhinesh | ianw: in that case, is the key stored as part of the image build, if so what is the location? | 01:13 |
| *** andreas_s has quit IRC | 01:13 | |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.db job https://review.openstack.org/511412 | 01:14 |
| *** andreas_s has joined #openstack-infra | 01:14 | |
| ianw | dhinesh: i'm not sure exactly what you mean. but, if you try, from the host & user you build images with "ssh -i ~/.ssh/id_rsa jenkins@remote.host.com" you might have luck | 01:17 |
| *** salv-orl_ has joined #openstack-infra | 01:18 | |
| dhinesh | ianw: ok let me give it a shot, thanks | 01:20 |
| *** esberglu has joined #openstack-infra | 01:20 | |
| *** esberglu has quit IRC | 01:20 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3 https://review.openstack.org/515231 | 01:20 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change Congress meeting time https://review.openstack.org/513467 | 01:21 |
| dhinesh | ianw: i meant the location within the VM, where the key will be stored. | 01:21 |
| *** salv-orlando has quit IRC | 01:21 | |
| *** kiennt26 has quit IRC | 01:23 | |
| *** andreas_s has quit IRC | 01:24 | |
| *** huanxie has joined #openstack-infra | 01:24 | |
| ianw | dhinesh: the key in ~/.ssh/id_rsa is added to authorized-keys by one of the infra build elements -- see https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user | 01:25 |
| *** andreas_s has joined #openstack-infra | 01:26 | |
| *** daidv has joined #openstack-infra | 01:27 | |
| *** liujiong has joined #openstack-infra | 01:29 | |
| *** esberglu has joined #openstack-infra | 01:30 | |
| dhinesh | ianw: ok, from the code it looks like, the node yaml entry of private-key: '/home/nodepool/.ssh/id_rsa' would be overwritten in any case | 01:31 |
| clarkb | no those are separate things | 01:32 |
| *** raissa has quit IRC | 01:32 | |
| clarkb | the yaml private key is telling nodepool what key to use to ssh into the VMs that have been booted | 01:32 |
| clarkb | the disk image builder element is telling dib what public key to write on disk | 01:32 |
| clarkb | ultimately those two have to line up but one is explicitly the private key and the other is explicitly the public key (they just have to be a pair) | 01:33 |
| *** yamahata has joined #openstack-infra | 01:33 | |
| *** kiennt26 has joined #openstack-infra | 01:34 | |
| *** andreas_s has quit IRC | 01:35 | |
| *** andreas_s has joined #openstack-infra | 01:36 | |
| *** anup has quit IRC | 01:37 | |
| *** luzC has quit IRC | 01:38 | |
| dhinesh | clarkb: got it, thanks | 01:38 |
| *** psachin has joined #openstack-infra | 01:41 | |
| *** Qiming has quit IRC | 01:43 | |
| *** luzC has joined #openstack-infra | 01:44 | |
| *** anupn has joined #openstack-infra | 01:44 | |
| *** andreas_s has quit IRC | 01:45 | |
| *** esberglu has quit IRC | 01:46 | |
| *** raissa has joined #openstack-infra | 01:46 | |
| *** andreas_s has joined #openstack-infra | 01:47 | |
| *** dhinesh has quit IRC | 01:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Switch to late-binding inheritance https://review.openstack.org/511352 | 01:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 01:48 |
| *** gus has quit IRC | 01:50 | |
| *** gus has joined #openstack-infra | 01:51 | |
| *** gyee has quit IRC | 01:52 | |
| *** yamahata has quit IRC | 01:55 | |
| *** andreas_s has quit IRC | 01:56 | |
| *** Guest12599 has quit IRC | 02:00 | |
| *** lindycoder__ has joined #openstack-infra | 02:00 | |
| *** lindycoder__ has quit IRC | 02:00 | |
| *** Qiming has joined #openstack-infra | 02:01 | |
| *** spectr has quit IRC | 02:03 | |
| *** fanzhang__ is now known as fanzhang | 02:04 | |
| *** spectr has joined #openstack-infra | 02:05 | |
| *** mat128 has joined #openstack-infra | 02:05 | |
| *** Qiming has quit IRC | 02:05 | |
| *** camunoz has quit IRC | 02:08 | |
| *** rwsu has quit IRC | 02:11 | |
| *** ramishra has joined #openstack-infra | 02:11 | |
| *** rwsu has joined #openstack-infra | 02:14 | |
| openstackgerrit | zhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove useless legacy-puppet-check-jobs https://review.openstack.org/515263 | 02:16 |
| *** bobh has quit IRC | 02:18 | |
| *** felipemonteiro_ has joined #openstack-infra | 02:20 | |
| *** Sukhdev has joined #openstack-infra | 02:20 | |
| *** vhosako__ has quit IRC | 02:22 | |
| *** vhosakot has joined #openstack-infra | 02:22 | |
| *** dhinesh has joined #openstack-infra | 02:24 | |
| *** zerick has quit IRC | 02:26 | |
| *** hongbin has joined #openstack-infra | 02:27 | |
| *** zerick has joined #openstack-infra | 02:27 | |
| *** masuberu has quit IRC | 02:34 | |
| *** gongysh has joined #openstack-infra | 02:35 | |
| *** gongysh has quit IRC | 02:36 | |
| *** mrodden has quit IRC | 02:41 | |
| *** larsks has quit IRC | 02:42 | |
| *** zerick has quit IRC | 02:43 | |
| *** zerick has joined #openstack-infra | 02:43 | |
| *** mrodden has joined #openstack-infra | 02:44 | |
| *** larsks has joined #openstack-infra | 02:44 | |
| *** vhosakot_ has joined #openstack-infra | 02:47 | |
| *** vhosakot has quit IRC | 02:48 | |
| *** rosmaita has quit IRC | 02:53 | |
| *** rwsu has quit IRC | 02:54 | |
| *** masber has joined #openstack-infra | 02:55 | |
| *** masber has quit IRC | 02:57 | |
| *** cuongnv has quit IRC | 03:00 | |
| *** cuongnv has joined #openstack-infra | 03:01 | |
| tonyb | What is the right way to to clone a repo that isn't a required project? In the past I'd use zuul-cloner and point it at the /opt/git cache in the nodepool image | 03:02 |
| *** Qiming has joined #openstack-infra | 03:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 03:07 |
| *** Qiming has quit IRC | 03:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 03:09 |
| *** dave-mccowan has quit IRC | 03:10 | |
| clarkb | tonyb: you probably want to clone from cache then update origin and update head? | 03:11 |
| clarkb | it may be worth having a script/tool for that | 03:11 |
| *** Qiming has joined #openstack-infra | 03:12 | |
| tonyb | clarkb: Yeah I think that's what I need. I was thinking I could get way without respecting Depends-On but I can't :( | 03:14 |
| *** annp has joined #openstack-infra | 03:14 | |
| tonyb | clarkb: where is the cache? still in /opt/git? | 03:15 |
| *** Qiming has quit IRC | 03:16 | |
| clarkb | tonyb: yes | 03:17 |
| clarkb | tonyb: if you need to respect depends on then maybe just use required projects? | 03:17 |
| tonyb | clarkb: I can't :( | 03:18 |
| tonyb | clarkb: It's for validating $project uses check-requirements so I just need to grovel around in the global and per project zuul config | 03:19 |
| tonyb | clarkb: it's similar to what we're doign for releases and publish jobs | 03:19 |
| tonyb | frickler: Modified the tool to look in project-config but the per-projects stuff is a todo item | 03:20 |
| tonyb | clarkb: I guess we could punt but that seems poor form | 03:20 |
| clarkb | I see | 03:20 |
| *** nicolasbock has quit IRC | 03:22 | |
| *** yamamoto has joined #openstack-infra | 03:22 | |
| *** gouthamr has quit IRC | 03:25 | |
| *** armaan has quit IRC | 03:31 | |
| *** thorst has joined #openstack-infra | 03:35 | |
| *** thorst has quit IRC | 03:41 | |
| *** Qiming has joined #openstack-infra | 03:50 | |
| *** iyamahat has joined #openstack-infra | 03:54 | |
| *** sree has joined #openstack-infra | 03:56 | |
| ianw | hey clarkb -- the error you mentioned in http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-08-16.log.html#t2017-08-16T23:58:25 | 03:59 |
| ianw | puppet-agent[9836]: Could not request certificate: Failed to open TCP connection to puppet:8140 | 03:59 |
| ianw | was that a red-herring? | 03:59 |
| ianw | i'm seeing creation seem to get stuck on "copy system puppet modules" | 04:00 |
| *** erlon has quit IRC | 04:00 | |
| ianw | but it's worked for two prior hosts | 04:00 |
| *** felipemonteiro_ has quit IRC | 04:01 | |
| clarkb | ianw: ya it went on to work | 04:03 |
| clarkb | ianw: I htink what is happening is the xenial distro package for puppet3 tried to start the service | 04:03 |
| *** baoli has quit IRC | 04:05 | |
| *** hongbin has quit IRC | 04:05 | |
| ianw | clarkb: hmm, maybe i'll add a manual stop to launcher after it does install_puppet.sh | 04:06 |
| *** dhinesh has quit IRC | 04:08 | |
| *** armaan has joined #openstack-infra | 04:11 | |
| *** huanxie has quit IRC | 04:20 | |
| *** armaan has quit IRC | 04:20 | |
| ianw | that helps, it's still stuck syncronising puppet repos for some reason though | 04:22 |
| *** udesale has joined #openstack-infra | 04:22 | |
| *** markvoelker has quit IRC | 04:23 | |
| *** ykarel has joined #openstack-infra | 04:24 | |
| *** dhajare has joined #openstack-infra | 04:28 | |
| *** udesale has quit IRC | 04:34 | |
| *** mat128 has quit IRC | 04:35 | |
| *** kiennt26 has quit IRC | 04:36 | |
| ianw | do i want to know why "virtualenv" creates a python3 virtualenv on puppetmaster by default? | 04:39 |
| tonyb | ianw: forward thinking? :D | 04:40 |
| *** Hunner has quit IRC | 04:41 | |
| *** Hunner has joined #openstack-infra | 04:41 | |
| *** Hunner has quit IRC | 04:41 | |
| *** Hunner has joined #openstack-infra | 04:41 | |
| clarkb | it uses the python it is installed under by default iirc | 04:41 |
| clarkb | so maybe it got pip3 installed? | 04:41 |
| jeblair | tonyb: the only way to get depends-on is via required-projects. | 04:48 |
| *** dhinesh has joined #openstack-infra | 04:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Switch to late-binding inheritance https://review.openstack.org/511352 | 04:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 04:49 |
| *** claudiub|2 has joined #openstack-infra | 04:49 | |
| jeblair | tonyb: which job are you working on? | 04:50 |
| tonyb | jeblair: Thanks. We can't use required-projects. In practise it wont be anyt different to waht we had pre-zuulv3 | 04:50 |
| tonyb | jeblair: validate-projects for requirements | 04:50 |
| tonyb | jeblair: what it did pre-v3 was lok in layout.yaml to see if the project had check-requirements listed in the template section | 04:51 |
| tonyb | jeblair: (for every project in projects.txt) | 04:51 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Stop puppet during install https://review.openstack.org/515279 | 04:52 |
| tonyb | jeblair: We can look in zuul/projects.txt which has been done but I'm pretty sure that'd break longer term if $project was doing in repo config | 04:52 |
| ianw | clarkb: ^^ re that puppet-agent warning error | 04:53 |
| tonyb | jeblair: The Depends-On thing is a nice to have but based on what you said it's a non-starter | 04:53 |
| jeblair | tonyb: we're planning on, very soon, having a rest api where we should be able to interrogate zuul's running config. i wonder if that might be the best approach. (it still won't have depends-on support, but may be easier than the alternative) | 04:54 |
| *** dhajare is now known as dhajare|mtg | 04:54 | |
| tonyb | jeblair: For me that'd be close to perfect | 04:54 |
| *** markvoelker has joined #openstack-infra | 04:55 | |
| tonyb | jeblair: I asked yesterday if such a thing existed ... I should have been more forward thinking ;P | 04:55 |
| jeblair | tonyb: maybe go with checking project-config for short-term, don't worry about the in-repo config and hopefully we'll have the api before it's a problem? :) | 04:55 |
| tonyb | jeblair: That sir sounds like a bonza[1] plan | 04:56 |
| tonyb | [1] https://www.urbandictionary.com/define.php?term=Bonza | 04:56 |
| *** armaan has joined #openstack-infra | 04:57 | |
| tonyb | jeblair: I'd offer to help but I suspect I'd just slow y'all down | 04:57 |
| andreaf | jeblair, mordred: I still get no real logging from zuul about what happens to my post-tempest play here http://logs.openstack.org/72/509272/13/check/devstack-tempest/d8d2021/job-output.txt.gz#_2017-10-25_17_10_46_098874 | 04:58 |
| andreaf | jeblair, mordred: I'm trying to consume on tempest side the stage-output roles via depends-on - I probably did some silly mistake in my patch but without logging is hard to tell | 04:59 |
| *** huanxie has joined #openstack-infra | 04:59 | |
| jeblair | andreaf: wow the post-tempest playbook doesn't even show up in ara or the json | 05:00 |
| jeblair | lemme see what i can dig up off the executor | 05:00 |
| tonyb | andreaf: aren't you just copying them over themselves? | 05:01 |
| tonyb | andreaf: logs_root and stage_dir are the same thing | 05:02 |
| andreaf | tonyb: they go from /opt/stack/tempest/... into /opt/stack/logs | 05:03 |
| jeblair | andreaf: "ERROR! the role 'stage-output' was not found in ..." | 05:03 |
| jeblair | andreaf: i added an entry to the etherpad for mordred so when he's back around he can look at why that didn't end up in the job log | 05:03 |
| andreaf | jeblair: ok thanks | 05:04 |
| andreaf | jeblair: oh, the depends-on points to the top patch of the series that is not rebased, that's why the name is not visible | 05:04 |
| *** vhosakot_ has quit IRC | 05:04 | |
| andreaf | jeblair: thanks | 05:04 |
| *** vhosakot has joined #openstack-infra | 05:05 | |
| andreaf | jeblair, tonyb: I assumed the staging area could still be /opt/stack/logs... but it doesn't have to, it can be anywhere | 05:05 |
| tonyb | andreaf: okay. I see how that works now | 05:06 |
| jeblair | okay, it's past my bedtime. 'night :) | 05:06 |
| andreaf | jeblair: 'night :) | 05:07 |
| *** vhosakot_ has joined #openstack-infra | 05:12 | |
| *** armaan has quit IRC | 05:12 | |
| *** yamahata has joined #openstack-infra | 05:13 | |
| *** armaan has joined #openstack-infra | 05:13 | |
| *** vhosakot has quit IRC | 05:14 | |
| *** gcb has joined #openstack-infra | 05:16 | |
| ianw | #info zm[0-4].o.o rebuilt to xenial and added to zuulv3 | 05:16 |
| *** armaan has quit IRC | 05:17 | |
| *** yamamoto has quit IRC | 05:17 | |
| ianw | #log zm[0-4].o.o rebuilt to xenial and added to zuulv3 | 05:17 |
| ianw | #status log zm[0-4].o.o rebuilt to xenial and added to zuulv3 | 05:18 |
| openstackstatus | ianw: finished logging | 05:18 |
| ianw | durr | 05:18 |
| *** markvoelker has quit IRC | 05:24 | |
| *** rwsu has joined #openstack-infra | 05:24 | |
| *** armax has quit IRC | 05:24 | |
| *** armax has joined #openstack-infra | 05:24 | |
| *** markvoelker has joined #openstack-infra | 05:24 | |
| *** armax has quit IRC | 05:25 | |
| *** armax has joined #openstack-infra | 05:25 | |
| *** armax has quit IRC | 05:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move to a common lock-file directory https://review.openstack.org/514079 | 05:28 |
| *** markvoelker has quit IRC | 05:28 | |
| *** jbadiapa_ has quit IRC | 05:32 | |
| *** thorst has joined #openstack-infra | 05:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 05:39 |
| *** thorst has quit IRC | 05:41 | |
| *** nikhil has quit IRC | 05:45 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove pep8/pyflakes https://review.openstack.org/515041 | 05:55 |
| openstackgerrit | zhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy puppet openstack check jobs https://review.openstack.org/515263 | 05:56 |
| *** aeng has quit IRC | 05:59 | |
| openstackgerrit | zhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy puppet openstack check jobs https://review.openstack.org/515263 | 05:59 |
| *** sree has quit IRC | 06:00 | |
| *** sree has joined #openstack-infra | 06:00 | |
| *** salv-orlando has joined #openstack-infra | 06:03 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream https://review.openstack.org/515283 | 06:03 |
| *** salv-orl_ has quit IRC | 06:04 | |
| *** sree_ has joined #openstack-infra | 06:05 | |
| *** sree has quit IRC | 06:05 | |
| *** sree_ is now known as Guest13914 | 06:05 | |
| *** cshastri has joined #openstack-infra | 06:05 | |
| *** e0ne has joined #openstack-infra | 06:10 | |
| *** dhinesh has quit IRC | 06:14 | |
| *** liusheng has joined #openstack-infra | 06:15 | |
| *** Sukhdev has quit IRC | 06:19 | |
| *** andreas_s has joined #openstack-infra | 06:20 | |
| *** vsaienk0 has joined #openstack-infra | 06:20 | |
| *** kjackal_ has joined #openstack-infra | 06:21 | |
| *** e0ne has quit IRC | 06:28 | |
| *** udesale has joined #openstack-infra | 06:30 | |
| *** janki has joined #openstack-infra | 06:31 | |
| *** kiennt26 has joined #openstack-infra | 06:40 | |
| *** LindaWang1 has joined #openstack-infra | 06:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer https://review.openstack.org/515043 | 06:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream https://review.openstack.org/515283 | 06:47 |
| *** LindaWang has quit IRC | 06:49 | |
| *** LindaWang1 is now known as LindaWang | 06:49 | |
| *** yamamoto has joined #openstack-infra | 06:49 | |
| *** gongysh has joined #openstack-infra | 06:52 | |
| *** ociuhandu has joined #openstack-infra | 06:53 | |
| *** florianf has joined #openstack-infra | 06:56 | |
| *** jbadiapa_ has joined #openstack-infra | 06:57 | |
| *** martinkopec has joined #openstack-infra | 06:57 | |
| *** jbadiapa_ is now known as jbadiapa | 06:57 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer https://review.openstack.org/515043 | 07:00 |
| *** iyamahat has quit IRC | 07:04 | |
| *** rcernin has quit IRC | 07:04 | |
| *** ociuhandu has quit IRC | 07:05 | |
| *** rossella_s has joined #openstack-infra | 07:08 | |
| *** tesseract has joined #openstack-infra | 07:09 | |
| *** apetrich has quit IRC | 07:09 | |
| *** salv-orlando has quit IRC | 07:12 | |
| *** Hal has joined #openstack-infra | 07:12 | |
| *** Hal is now known as Guest31954 | 07:12 | |
| *** salv-orlando has joined #openstack-infra | 07:12 | |
| *** e0ne has joined #openstack-infra | 07:15 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update https://review.openstack.org/511175 | 07:15 |
| *** salv-orlando has quit IRC | 07:16 | |
| *** Goneri has joined #openstack-infra | 07:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix neutron-failure-rate grafana dashboard https://review.openstack.org/509291 | 07:18 |
| *** aviau has quit IRC | 07:19 | |
| *** ralonsoh has joined #openstack-infra | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream https://review.openstack.org/515283 | 07:23 |
| *** markvoelker has joined #openstack-infra | 07:25 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-output role https://review.openstack.org/509233 | 07:25 |
| *** yolanda has joined #openstack-infra | 07:27 | |
| *** gongysh has quit IRC | 07:28 | |
| *** huanxie has quit IRC | 07:31 | |
| frickler | tonyb: do you think we are ready to make the reqs job voting again? https://review.openstack.org/515007 | 07:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer https://review.openstack.org/515043 | 07:35 |
| *** amoralej|off is now known as amoralej | 07:36 | |
| tonyb | frickler: No I think we shoudl leave it non-voting until we can do the per-project query which jeblair says we'll be able to do via the REST api tomorrow (well maybe not that soon) | 07:37 |
| tonyb | frickler: I'm okay with manually verifying any failures, but you shoudl probably get an answer from prometheanfire | 07:37 |
| AJaeger | tonyb: we recommend that projects have check-requirements in project-config | 07:38 |
| tonyb | AJaeger: okay, but not require right? | 07:38 |
| AJaeger | tonyb: that's the policy we have documented, so I would be fine with the check as is and require that projects leave it in there | 07:38 |
| *** yamamoto_ has joined #openstack-infra | 07:38 | |
| AJaeger | tonyb: read https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 07:39 |
| AJaeger | tonyb: so we require it for governance projects. And the requirements team could require it for everybody ;) | 07:39 |
| *** gongysh has joined #openstack-infra | 07:39 | |
| AJaeger | would make life easier - and I have seen seldom a need to customize this at all, so no need to move in-repo. | 07:39 |
| *** huanxie has joined #openstack-infra | 07:39 | |
| tonyb | AJaeger: okay. | 07:40 |
| AJaeger | tonyb: but discuss with prometheanfire and rest of team. | 07:40 |
| frickler | also, if we allow a project to configure and drop it on their own power, that check seems to not make sense anymore at all to me | 07:40 |
| AJaeger | tonyb: we could also for now make the job voting and you could wait and see... | 07:40 |
| tonyb | AJaeger: sure. I guess if it's documented when it fails we just say "you did it wrong" and point them at the docs | 07:42 |
| tonyb | AJaeger, frickler: I've +1'd the review | 07:42 |
| *** yamamoto has quit IRC | 07:42 | |
| *** salv-orlando has joined #openstack-infra | 07:42 | |
| AJaeger | tonyb: works for me ;) | 07:42 |
| *** jpena|off is now known as jpena | 07:43 | |
| AJaeger | I'll approved the revert now... | 07:43 |
| *** ccamacho has joined #openstack-infra | 07:43 | |
| tonyb | frickler, AJaeger: Thanks | 07:43 |
| AJaeger | frickler: care to +2 https://review.openstack.org/#/c/514911/ some time today, please? | 07:47 |
| * AJaeger likes to get rid of newton jobs quickly ;) | 07:47 | |
| *** stakeda has joined #openstack-infra | 07:47 | |
| *** kiennt26 has quit IRC | 07:48 | |
| *** gildub has quit IRC | 07:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Make legacy-requirements-tox-validate-projects non-voting until fixed" https://review.openstack.org/515007 | 07:50 |
| jd_ | howdy | 07:51 |
| *** kiennt26 has joined #openstack-infra | 07:51 | |
| jd_ | we still have our Zuul playbooks issue we don't really understand on https://review.openstack.org/512700 where zuul complains about missing playbooks whereas we remove the jobs :) if anybody has an idea | 07:52 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/shade master: Remove enabling IDENTITY_V2 API https://review.openstack.org/515304 | 07:55 |
| vsaienk0 | folks after zuulv3 ennoblement we found that ironic-python-agent post job didn't upload images to tarballs.openstack.org. Looks like path was changed with migration can you please check? http://logs.openstack.org/8b/8bf0629749ac4947762b5307b8048087f928ff13/post/ironic-python-agent-buildimage-tinyipa/eedc93a/job-output.txt.gz#_2017-10-25_23_18_59_574930 | 07:57 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Default change and patchset to NULL in SQLReporter https://review.openstack.org/514423 | 07:58 |
| *** markvoelker has quit IRC | 07:59 | |
| *** gcb has quit IRC | 08:01 | |
| *** namnh has quit IRC | 08:04 | |
| *** namnh has joined #openstack-infra | 08:04 | |
| *** gildub has joined #openstack-infra | 08:07 | |
| frickler | jd_: I'm only guessing but it may be due to the way zuul processes this patch, it is trying to read the playbooks before noticing that the job definition is removed. the solution then would be to split the patch, first remove the job definition only and do the playbook removal in a second patch | 08:07 |
| *** vsaienk0 has quit IRC | 08:08 | |
| jd_ | frickler: hm but won't it runs the target in the first patch anyway? that'd be a problem | 08:08 |
| *** armaan has joined #openstack-infra | 08:08 | |
| frickler | jd_: because they are failing? then maybe you can make them non-voting first | 08:10 |
| *** vsaienk0 has joined #openstack-infra | 08:11 | |
| *** ykarel is now known as ykarel|lunch | 08:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job https://review.openstack.org/511784 | 08:11 |
| jd_ | frickler: O_O if that's it, it really sounds like a nifty Zuul bug/limitation | 08:11 |
| jd_ | AJaeger: opinion? | 08:11 |
| *** apetrich has joined #openstack-infra | 08:12 | |
| *** slaweq has joined #openstack-infra | 08:14 | |
| frickler | jd_: I agree that this looks like a bug or missing feature at least. can you submit a change with only your patch to .zuul.yaml to verify? we can then add it to our list at https://etherpad.openstack.org/p/zuulv3-issues for further evaluation | 08:18 |
| *** yamamoto_ has quit IRC | 08:18 | |
| *** slaweq has quit IRC | 08:18 | |
| *** CrayZee has joined #openstack-infra | 08:21 | |
| *** david-lyle has quit IRC | 08:24 | |
| *** hamzy has quit IRC | 08:24 | |
| *** david-lyle has joined #openstack-infra | 08:25 | |
| *** hamzy has joined #openstack-infra | 08:26 | |
| *** dbecker has joined #openstack-infra | 08:27 | |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.db job https://review.openstack.org/511412 | 08:27 |
| *** yamamoto has joined #openstack-infra | 08:28 | |
| *** huanxie has quit IRC | 08:29 | |
| *** shardy has joined #openstack-infra | 08:29 | |
| *** gildub has quit IRC | 08:31 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo https://review.openstack.org/515316 | 08:31 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d https://review.openstack.org/515317 | 08:31 |
| *** d0ugal has quit IRC | 08:32 | |
| *** d0ugal_ has joined #openstack-infra | 08:32 | |
| *** d0ugal_ has quit IRC | 08:33 | |
| *** yamamoto has quit IRC | 08:33 | |
| *** d0ugal has joined #openstack-infra | 08:33 | |
| *** d0ugal has quit IRC | 08:33 | |
| *** d0ugal has joined #openstack-infra | 08:33 | |
| *** huanxie has joined #openstack-infra | 08:35 | |
| *** hashar has joined #openstack-infra | 08:41 | |
| *** lucas-afk is now known as lucasagomes | 08:41 | |
| *** dingyichen has quit IRC | 08:41 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo https://review.openstack.org/515316 | 08:42 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d https://review.openstack.org/515317 | 08:42 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Updated templates for heat-dashboard https://review.openstack.org/514895 | 08:42 |
| *** derekh has joined #openstack-infra | 08:45 | |
| *** ociuhandu has joined #openstack-infra | 08:47 | |
| *** hashar is now known as hasharAway | 08:49 | |
| *** iyamahat has joined #openstack-infra | 08:53 | |
| *** yamahata has quit IRC | 08:56 | |
| *** markvoelker has joined #openstack-infra | 08:57 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Updated templates for heat-dashboard https://review.openstack.org/514895 | 08:57 |
| *** ykarel|lunch is now known as ykarel | 08:58 | |
| *** bhavik1 has joined #openstack-infra | 09:00 | |
| *** yamamoto has joined #openstack-infra | 09:01 | |
| *** electrofelix has joined #openstack-infra | 09:03 | |
| *** bhavik1 has quit IRC | 09:05 | |
| *** huanxie has quit IRC | 09:05 | |
| *** tosky has joined #openstack-infra | 09:05 | |
| vsaienk0 | infra-team please help to investigate https://bugs.launchpad.net/ironic-python-agent/+bug/1727652 after migration to zuulv3 we found that ironic-python-agent images are not uploaded to tarballs | 09:06 |
| openstack | Launchpad bug 1727652 in ironic-python-agent "post job doesn't work after migration to zuulv3" [Undecided,New] | 09:06 |
| *** panda|ruck|off is now known as panda|ruck | 09:06 | |
| jd_ | frickler: I think your theory does not hold because the same problem happens on the next patch https://review.openstack.org/513098 | 09:07 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Make version_string() lazy load version info https://review.openstack.org/450918 | 09:08 |
| *** iyamahat has quit IRC | 09:10 | |
| andreaf | AJaeger, frickler: I just noticed that the ARA folder is not copied in legacy-tempest-dsvm-neutron-full - is that a known issue? | 09:16 |
| vsaienk0 | AJaeger: frickler could you please help to investigate issue described above https://bugs.launchpad.net/ironic-python-agent/+bug/1727652 ^ | 09:17 |
| openstack | Launchpad bug 1727652 in ironic-python-agent "post job doesn't work after migration to zuulv3" [High,Confirmed] | 09:17 |
| frickler | andreaf: do you have a link to a failure handy? | 09:18 |
| andreaf | frickler: http://logs.openstack.org/72/509272/13/check/legacy-tempest-dsvm-neutron-full/cc0bfed/ | 09:19 |
| andreaf | frickler: there's no failure in the sense of error in the logs, I just don't see the ARA folder there | 09:19 |
| *** gongysh has quit IRC | 09:23 | |
| *** namnh has quit IRC | 09:24 | |
| *** namnh has joined #openstack-infra | 09:24 | |
| *** pgadiya has joined #openstack-infra | 09:26 | |
| *** david-lyle has quit IRC | 09:26 | |
| *** dklyle has joined #openstack-infra | 09:26 | |
| *** hamzy has quit IRC | 09:27 | |
| *** huanxie has joined #openstack-infra | 09:28 | |
| *** hamzy has joined #openstack-infra | 09:28 | |
| *** markvoelker has quit IRC | 09:29 | |
| *** eumel8 has joined #openstack-infra | 09:30 | |
| *** wolverineav has joined #openstack-infra | 09:30 | |
| frickler | andreaf: looks like it may be generated only when the job is failing, the folder does exist e.g. for http://logs.openstack.org/80/348080/23/check/legacy-tempest-dsvm-neutron-full/42bcc78/ | 09:31 |
| *** dtantsur|afk is now known as dtantsur | 09:31 | |
| andreaf | frickler: I see... that's a bit strange though - it's always generated for new style jobs | 09:32 |
| andreaf | frickler: but ok... | 09:32 |
| frickler | andreaf: I think this may have been changed to save space/inodes on the zuul+log servers, introduced in https://review.openstack.org/511918 | 09:35 |
| andreaf | frickler: ok got it | 09:35 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Add nodejs4-publish jobs https://review.openstack.org/515332 | 09:35 |
| andreaf | frickler: thanks | 09:36 |
| *** wolverineav has quit IRC | 09:37 | |
| *** yamamoto has quit IRC | 09:38 | |
| *** thorst has joined #openstack-infra | 09:39 | |
| *** sshnaidm is now known as sshnaidm|afk | 09:39 | |
| *** thorst has quit IRC | 09:43 | |
| *** pgadiya has quit IRC | 09:43 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Match job-output as console logs for getthelogs https://review.openstack.org/514560 | 09:45 |
| *** gongysh has joined #openstack-infra | 09:49 | |
| openstackgerrit | Niraj Singh proposed openstack-infra/project-config master: Add masakari-dashboard project https://review.openstack.org/515337 | 09:49 |
| openstackgerrit | gongysh proposed openstack-infra/project-config master: Remove Tacker legacy job https://review.openstack.org/515341 | 09:52 |
| *** rossella_s has quit IRC | 09:56 | |
| *** erlon has joined #openstack-infra | 09:57 | |
| andreykurilin | hi folks | 09:59 |
| andreykurilin | we had a critical issue | 09:59 |
| *** pgadiya has joined #openstack-infra | 10:00 | |
| andreykurilin | I missed that one of our rally branch was mentioned here http://lists.openstack.org/pipermail/openstack-dev/2017-September/122381.html (there was not rally tag in the subject) | 10:00 |
| andreykurilin | And an active branch (a stable branch for previous rally release) was removed yesturday | 10:00 |
| *** armaan has quit IRC | 10:00 | |
| andreykurilin | this branch was created just a half year ago and I'm not sure why it was marked as old | 10:01 |
| andreykurilin | The removing stable/0.9 affected some of our customers | 10:02 |
| andreykurilin | tonyb^ | 10:03 |
| *** sdague has joined #openstack-infra | 10:03 | |
| andreykurilin | I have this branch at local repo. Does it possible to push it back? | 10:04 |
| andreykurilin | Also, the tag 0.9-oel should be removed. I do not know how to do it | 10:04 |
| andreykurilin | Any thoughts on the subject or it is better to raise the questions at openstack-releases channel? | 10:04 |
| openstackgerrit | gongysh proposed openstack-infra/openstack-zuul-jobs master: Remove tacker legacy job https://review.openstack.org/515346 | 10:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 10:07 |
| openstackgerrit | gongysh proposed openstack-infra/project-config master: Remove Tacker legacy job https://review.openstack.org/515341 | 10:07 |
| *** cuongnv has quit IRC | 10:09 | |
| *** sambetts|afk is now known as sambetts | 10:09 | |
| *** jklare_ has joined #openstack-infra | 10:09 | |
| *** liujiong has quit IRC | 10:10 | |
| *** jklare has quit IRC | 10:12 | |
| *** jklare_ is now known as jklare | 10:12 | |
| *** andreas_s has quit IRC | 10:12 | |
| *** andreas_s has joined #openstack-infra | 10:13 | |
| *** namnh has quit IRC | 10:16 | |
| *** kiennt26 has quit IRC | 10:16 | |
| *** andreas_s has quit IRC | 10:17 | |
| *** bhavik1 has joined #openstack-infra | 10:17 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:18 | |
| *** pgadiya has quit IRC | 10:21 | |
| *** andreas_s has joined #openstack-infra | 10:24 | |
| AJaeger | andreykurilin: the tag was [all]. Best to followup via email to the removal email. AFAIK it's possible to revert parts of this (the tag is distributed, even if we delete it, everybody will have it) but best to discuss later with rest of the team. Right now nobody is around that can help. | 10:25 |
| *** markvoelker has joined #openstack-infra | 10:26 | |
| *** ldnunes has joined #openstack-infra | 10:27 | |
| *** e0ne has quit IRC | 10:28 | |
| *** andreas_s has quit IRC | 10:29 | |
| *** gmann is now known as gmann_afk | 10:29 | |
| andreykurilin | AJaeger: I'm ok about that someone already obtain the tag. I just do not want to continue distributing it (at pypi at least). | 10:29 |
| andreykurilin | *already obtained | 10:30 |
| *** e0ne has joined #openstack-infra | 10:30 | |
| *** e0ne has quit IRC | 10:30 | |
| *** e0ne has joined #openstack-infra | 10:31 | |
| *** e0ne has quit IRC | 10:32 | |
| *** e0ne has joined #openstack-infra | 10:33 | |
| *** e0ne has quit IRC | 10:33 | |
| *** pgadiya has joined #openstack-infra | 10:34 | |
| *** Guest13914 has quit IRC | 10:34 | |
| *** vhosakot_ has quit IRC | 10:34 | |
| *** sree has joined #openstack-infra | 10:35 | |
| *** andreas_s has joined #openstack-infra | 10:36 | |
| *** boden has joined #openstack-infra | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:39 | |
| *** sree has quit IRC | 10:40 | |
| *** apetrich has quit IRC | 10:41 | |
| *** pbourke has quit IRC | 10:42 | |
| *** gcb has joined #openstack-infra | 10:43 | |
| AJaeger | #status log we lost the docs.o.o central home page, somehow our publishing is broken | 10:43 |
| openstackstatus | AJaeger: finished logging | 10:43 |
| *** pbourke has joined #openstack-infra | 10:44 | |
| *** yamamoto has quit IRC | 10:44 | |
| *** ramishra has quit IRC | 10:45 | |
| *** yamamoto has joined #openstack-infra | 10:47 | |
| *** ramishra has joined #openstack-infra | 10:47 | |
| *** andreas_s has quit IRC | 10:47 | |
| *** andreas_s has joined #openstack-infra | 10:49 | |
| *** stakeda has quit IRC | 10:50 | |
| *** gongysh has quit IRC | 10:52 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Add nodejs4-publish jobs https://review.openstack.org/515332 | 10:52 |
| *** andreas_s has quit IRC | 10:54 | |
| *** andreas_s has joined #openstack-infra | 10:54 | |
| *** nicolasbock has joined #openstack-infra | 10:54 | |
| *** jpena is now known as jpena|lunch | 10:59 | |
| *** markvoelker has quit IRC | 10:59 | |
| *** bhavik1 has quit IRC | 11:02 | |
| *** amoralej is now known as amoralej|lunch | 11:03 | |
| *** huanxie has quit IRC | 11:03 | |
| *** vsaienk0 has quit IRC | 11:06 | |
| *** nicolasbock has quit IRC | 11:06 | |
| *** vsaienk0 has joined #openstack-infra | 11:07 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci master: WIP, test CI with experimental undercloud upgrade https://review.openstack.org/515361 | 11:07 |
| *** gildub has joined #openstack-infra | 11:09 | |
| *** andreas_s has quit IRC | 11:10 | |
| smcginnis | Ruh roh. I thought the home page was a static, manually generated portion of the site. | 11:12 |
| *** udesale has quit IRC | 11:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Do not publish docs when building fails https://review.openstack.org/515365 | 11:15 |
| AJaeger | infra-root, config-core, let's make our publishing robust ^ | 11:15 |
| AJaeger | smcginnis: it is build from jinja-templates and we just removed it due to a failure in tox run ;(, see 515365 for more details | 11:16 |
| smcginnis | Ouch. | 11:19 |
| smcginnis | AJaeger: Yeah, good change. There's probably more places where we should be checking that the previous step was a success. | 11:20 |
| *** nicolasbock has joined #openstack-infra | 11:20 | |
| AJaeger | smcginnis: yes | 11:20 |
| *** gildub has quit IRC | 11:21 | |
| AJaeger | #status log docs.o.o index page was lost due to broken build and publishing of broken build, suggested fix in https://review.openstack.org/515365 | 11:22 |
| openstackstatus | AJaeger: finished logging | 11:22 |
| *** tpsilva has joined #openstack-infra | 11:24 | |
| * AJaeger disappears again | 11:25 | |
| *** gongysh has joined #openstack-infra | 11:26 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Do not publish Zaqar to pypi https://review.openstack.org/515366 | 11:26 |
| *** andreas_s has joined #openstack-infra | 11:27 | |
| *** janki has quit IRC | 11:31 | |
| andreaf | mordred, jeblair: so this works now https://review.openstack.org/#/c/509233 and it's kind of proven by https://review.openstack.org/#/c/509272/ | 11:31 |
| andreaf | mordred, jeblair: but I was wondering if we should do some kind of unit testing of these roles - to validate the various conditions in the roles | 11:32 |
| andreaf | on my machine I use a simple play to test things, it should be possible to have something like that in the gate | 11:33 |
| *** links has quit IRC | 11:36 | |
| *** kjackal_ has quit IRC | 11:36 | |
| *** andreas_s has quit IRC | 11:36 | |
| *** andreas_s has joined #openstack-infra | 11:38 | |
| *** _ari_ has quit IRC | 11:38 | |
| *** smatzek has joined #openstack-infra | 11:39 | |
| *** lucasagomes is now known as lucas-hungry | 11:40 | |
| *** _ari_ has joined #openstack-infra | 11:43 | |
| *** annp has quit IRC | 11:46 | |
| *** andreas_s has quit IRC | 11:48 | |
| *** andreas_s has joined #openstack-infra | 11:50 | |
| *** edmondsw has joined #openstack-infra | 11:50 | |
| *** dave-mccowan has joined #openstack-infra | 11:51 | |
| AJaeger | So, https://docs.openstack.org is up again - this time build worked without failure | 11:51 |
| AJaeger | infra-root, please delete http://files.openstack.org/docs/publish-docs/ - this was broken publishing and is all wrong... | 11:52 |
| *** bobh has joined #openstack-infra | 11:54 | |
| *** markvoelker has joined #openstack-infra | 11:57 | |
| *** rossella_s has joined #openstack-infra | 11:58 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Remove ansible-role-jobs template from openstack-ansible-ops https://review.openstack.org/515377 | 11:59 |
| *** bobh has quit IRC | 12:00 | |
| *** ociuhandu_ has joined #openstack-infra | 12:01 | |
| *** thorst has joined #openstack-infra | 12:01 | |
| *** ociuhandu has quit IRC | 12:01 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Build tarballs for networking-{bagpipe,bgvpn} https://review.openstack.org/515381 | 12:02 |
| *** ociuhandu_ has quit IRC | 12:02 | |
| *** salv-orl_ has joined #openstack-infra | 12:06 | |
| *** salv-orlando has quit IRC | 12:09 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 12:13 |
| *** kjackal_ has joined #openstack-infra | 12:13 | |
| *** jpena|lunch is now known as jpena | 12:13 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 12:18 |
| *** shardy has quit IRC | 12:20 | |
| *** jcoufal has joined #openstack-infra | 12:20 | |
| *** dsariel has joined #openstack-infra | 12:21 | |
| *** sree has joined #openstack-infra | 12:24 | |
| dsariel | hi! https://review.openstack.org/#/c/515377/ has a list of gates ( openstack-tox-linters ) triggered by change request. Where it is specified what is the list of gates what should be triggred unit +1 is given | 12:25 |
| *** cody-somerville has quit IRC | 12:25 | |
| *** cody-somerville has joined #openstack-infra | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:27 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 12:28 |
| *** sree has quit IRC | 12:29 | |
| *** markvoelker has quit IRC | 12:29 | |
| *** wolverineav has joined #openstack-infra | 12:32 | |
| *** lucas-hungry is now known as lucasagomes | 12:34 | |
| frickler | dsariel: I'm not sure I understand your question correctly. did you read https://docs.openstack.org/infra/manual/zuulv3.html ? that should answer most questions regarding how checks are being defined now | 12:36 |
| *** markvoelker has joined #openstack-infra | 12:40 | |
| *** LindaWang has quit IRC | 12:40 | |
| dmsimard | AJaeger: for https://review.openstack.org/#/c/515365/, the only thing I wonder is if we should handle that at the role level rather than the playbook level | 12:41 |
| *** gcb has quit IRC | 12:42 | |
| *** gcb has joined #openstack-infra | 12:43 | |
| *** amoralej|lunch is now known as amoralej | 12:43 | |
| *** sree has joined #openstack-infra | 12:43 | |
| *** dprince has joined #openstack-infra | 12:44 | |
| *** sree_ has joined #openstack-infra | 12:44 | |
| *** sree_ is now known as Guest18426 | 12:45 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use infra proxy server for trunk.r.o in delorean-deps https://review.openstack.org/508884 | 12:45 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 12:46 |
| *** vsaienk0 has quit IRC | 12:47 | |
| dsariel | frickler, thanks for the helpful reply! :q | 12:47 |
| *** sree has quit IRC | 12:48 | |
| *** gcb has quit IRC | 12:48 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 12:49 |
| *** bh526r has joined #openstack-infra | 12:49 | |
| dsariel | :q | 12:50 |
| frickler | dmsimard: I wondered about that, too, but it seems its mostly jobs referencing the playbooks directly and I didn't see how one could handle the condition there | 12:52 |
| dmsimard | frickler: in the roles' tasks/main.yaml, check for zuul_success | 12:53 |
| dsariel | frickler, one more question if you don't mind openstack-zuul-jobs/playbooks/legacy/ have definitions of what is performed by those gates. is there somewhere a definition which gates are obligatory to pass in order to have +1 from gerrit? | 12:53 |
| *** baoli has joined #openstack-infra | 12:54 | |
| *** LindaWang has joined #openstack-infra | 12:54 | |
| *** shardy has joined #openstack-infra | 12:57 | |
| *** bobh has joined #openstack-infra | 12:58 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin https://review.openstack.org/509823 | 12:59 |
| *** Goneri has quit IRC | 12:59 | |
| *** apetrich has joined #openstack-infra | 13:00 | |
| *** vsaienk0 has joined #openstack-infra | 13:00 | |
| *** gongysh has quit IRC | 13:00 | |
| *** rhallisey has joined #openstack-infra | 13:00 | |
| mkolesni | hey, someone know why the gate jobs for py27/py35 are sporadically failing? | 13:01 |
| mkolesni | its really interfering with getting changes merged | 13:01 |
| *** camunoz has joined #openstack-infra | 13:02 | |
| frickler | dmsimard: oh, you mean within the write-root-marker and upload-afs roles. well, I was assuming that they were too generic to be coupled with zuul_success unconditionally | 13:02 |
| frickler | mkolesni: do you have links to failures? I'm not aware of any global issues regarding these | 13:03 |
| *** e0ne has joined #openstack-infra | 13:03 | |
| dmsimard | frickler: yeah, I don't have a strong opinion | 13:03 |
| mkolesni | frickler, this one failed on POST_FAILURE even though py27 itself ran - http://logs.openstack.org/31/514931/2/gate/openstack-tox-py27/2f78d5b/ | 13:03 |
| *** yolanda has quit IRC | 13:04 | |
| mkolesni | frickler, on recheck, this one will fail on py35 but still hanging right now - http://zuulv3.openstack.org/static/stream.html?uuid=bcfe8e0605e8405a92ec00874a6eee7a&logfile=console.log | 13:04 |
| frickler | dsariel: it is the set of all checks that are applicable for a patch, except those marked with "voting: false" | 13:04 |
| AJaeger | dmsimard: if you teach me how to do it differently ... I'm an ansible novice and need your help. Feel free to take over the chagne or propose a superior one separately | 13:05 |
| * AJaeger is busy the next 4 hours | 13:06 | |
| frickler | mkolesni: this looks like there might be networking issues in one of our clouds, I guess some infra-root will have to look into this | 13:06 |
| mkolesni | frickler, ok do you know who should i contact? | 13:07 |
| * fungi will see if he can track down the intermittent network connectivity issues to a particular provider/region | 13:08 | |
| dsariel | frickler, maybe I will rephrase my question: how is it ensured that gerrit will not vote +1 until all non-voting gates are passed? | 13:08 |
| frickler | fungi: citycloud-kna1 was the one with ssh failure here http://logs.openstack.org/31/514931/2/gate/openstack-tox-py27/2f78d5b/job-output.txt.gz#_2017-10-26_08_40_47_777422 | 13:09 |
| *** camunoz has quit IRC | 13:09 | |
| *** camunoz has joined #openstack-infra | 13:10 | |
| frickler | mkolesni: fungi is one of them, so it is already being handled | 13:10 |
| *** mriedem has joined #openstack-infra | 13:11 | |
| frickler | dsariel: iiuc not gerrit is voting, but zuul. and the definition of how and when zuul does this is here https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml | 13:12 |
| *** Goneri has joined #openstack-infra | 13:12 | |
| mkolesni | frickler, ok thanks :) | 13:12 |
| *** hemna_ has joined #openstack-infra | 13:14 | |
| *** lindycoder has joined #openstack-infra | 13:15 | |
| *** lindycoder has quit IRC | 13:16 | |
| *** jaosorior has quit IRC | 13:16 | |
| fungi | frickler: yeah, i think we only recently reenabled some citycloud regions after having disabled them a while back for network instability. the old issues there may still be present | 13:16 |
| *** ykarel is now known as ykarel|away | 13:17 | |
| *** lindycoder has joined #openstack-infra | 13:18 | |
| fungi | though looks like that kna1 region has been enabled for a while (months at least) | 13:20 |
| *** eharney has joined #openstack-infra | 13:20 | |
| *** sbezverk has joined #openstack-infra | 13:20 | |
| *** vsaienk0 has quit IRC | 13:20 | |
| *** ykarel|away has quit IRC | 13:21 | |
| *** vsaienk0 has joined #openstack-infra | 13:23 | |
| dsariel | frickler, thanks! | 13:23 |
| fungi | citycloud hasn't reached out to notify us of any issues there, though that's fairly typical for public cloud providers we work with so not especially surprising | 13:23 |
| *** mat128 has joined #openstack-infra | 13:24 | |
| *** yamamoto has quit IRC | 13:26 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin https://review.openstack.org/515408 | 13:26 |
| *** dhajare|mtg has quit IRC | 13:27 | |
| *** smatzek has quit IRC | 13:34 | |
| EmilienM | infra-root: outstanding request to promote https://review.openstack.org/#/c/515388/ in the gate, that would help us to sort things out in our gate - thank very much | 13:34 |
| *** rosmaita has joined #openstack-infra | 13:35 | |
| *** spzala has joined #openstack-infra | 13:35 | |
| *** gouthamr has joined #openstack-infra | 13:36 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin https://review.openstack.org/515408 | 13:38 |
| *** martinkopec has quit IRC | 13:39 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:40 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs" https://review.openstack.org/515411 | 13:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs" https://review.openstack.org/515411 | 13:41 |
| frickler | fungi: openstack-health data seems to be stale, last updated 7am yesterday, makes it difficult to see whether we might have a larger issue currently | 13:41 |
| fungi | frickler: yep, i'm at a more effective computer now and starting to see if i can spot some patterns the old-fashioned way | 13:42 |
| EmilienM | project-config core: please review https://review.openstack.org/515411 when you can so we can re-enable voting on 2 important jobs. Thanks! | 13:43 |
| *** pgadiya has quit IRC | 13:45 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove usage of 'tripleo' gate queue https://review.openstack.org/515070 | 13:46 |
| *** dosaboy has quit IRC | 13:48 | |
| *** annp has joined #openstack-infra | 13:50 | |
| *** e0ne has quit IRC | 13:53 | |
| *** ykarel|away has joined #openstack-infra | 13:53 | |
| *** jaosorior has joined #openstack-infra | 13:53 | |
| *** spzala has quit IRC | 13:54 | |
| fungi | EmilienM: infra-root: i've enqueued tripleo-quickstart-extras patchset 515388,2 into the gate pipeline and promoted it to the head of the tripleo change queue | 13:54 |
| *** hongbin has joined #openstack-infra | 13:56 | |
| fungi | mkolesni: frickler: scanning the running jobs right now, i'm not seeing evidence of a spate of job failures, so maybe this was a temporary condition which has since resolved itself? or maybe the frequency of incidence is just really low? | 13:56 |
| *** ihrachys has joined #openstack-infra | 13:57 | |
| mkolesni | fungi, i dont know its happening since yesterday but seems to be hitting different jobs | 13:57 |
| EmilienM | fungi: thanks very much | 13:58 |
| mkolesni | fungi, the problem is the ci is broken because of some external changes which im trying to fix with this patch to unblock all others but these failures have stopped me from merging it for a day and a half now.. | 13:58 |
| *** smatzek has joined #openstack-infra | 13:59 | |
| *** ramishra has quit IRC | 14:00 | |
| *** dosaboy has joined #openstack-infra | 14:00 | |
| *** smatzek has quit IRC | 14:00 | |
| *** smatzek has joined #openstack-infra | 14:01 | |
| *** smatzek has quit IRC | 14:01 | |
| *** smatzek has joined #openstack-infra | 14:01 | |
| prometheanfire | AJaeger: ya, centralized project config should enforce some tests, like check-reqs | 14:02 |
| *** ramishra has joined #openstack-infra | 14:02 | |
| fungi | mkolesni: is it at all possible that something about the unit tests actually impacts networking on the node where the tests run? are these tests fully mocked or do they interact with the kernel/perform actual networking tasks? | 14:06 |
| mkolesni | fungi, no theyre fully mocked, its an sdn controller and its not installed during these tests | 14:06 |
| *** spzala has joined #openstack-infra | 14:06 | |
| fungi | mkolesni: because it really looks in the currently timing-out job like the executor lost connectivity to the test node entirely | 14:06 |
| mkolesni | fungi, i gtg but please talk to mpeterson hes working with me on this | 14:07 |
| fungi | okay, just making sure. in that case we may have nova instances dropping off the network at random, but normally zuul should retry such jobs | 14:07 |
| *** esberglu has joined #openstack-infra | 14:07 | |
| *** jbadiapa has quit IRC | 14:07 | |
| * mpeterson is catching up with what you were talking about | 14:08 | |
| *** rcernin has joined #openstack-infra | 14:08 | |
| fungi | mpeterson: currently failing job's console stream is at http://zuulv3.openstack.org/static/stream.html?uuid=bcfe8e0605e8405a92ec00874a6eee7a&logfile=console.log | 14:08 |
| *** pblaho has quit IRC | 14:08 | |
| mpeterson | fungi: yeah, it's not the first time we see a job stuck at that phase | 14:09 |
| fungi | peering closer, looks like it the executor lost connectivity to the test node during a pre phase task, and each subsequent task is trying for 30 minutes before timing out. trying to track down potential connectivity issues in one or more of our donor providers' regions | 14:09 |
| mpeterson | yes, I've seen that as well. Where the `run` can't run because `pre` phases timeout or have a network issue | 14:11 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove unreachable code https://review.openstack.org/506137 | 14:12 |
| mpeterson | fungi: is there a way to detect those issues though and retry the checks? otherwise it adds like long delays | 14:12 |
| fungi | mpeterson: well, in this case i fully expect zuul to reschedule this particular build automatically | 14:13 |
| *** marst has joined #openstack-infra | 14:13 | |
| fungi | if it does not, that will significantly alter my expectations | 14:13 |
| mpeterson | fungi: I think I've never seen zuul reschedule builds on this kind of issues before. I might not have observed too closely though, as you know the ins and outs :) | 14:14 |
| *** dklyle has quit IRC | 14:15 | |
| *** Guest31954 has quit IRC | 14:15 | |
| fungi | mpeterson: you would only notice if watching the status page, you'd see a job with some percentage of completion go back to queued temporarily and then start its completion bar at 0 again once it got scheduled to a replacement node | 14:15 |
| fungi | though it might be nice if it didn't try each and every post task for 30 minutes before giving up (though i don't know how we'd actually go about correctly detecting that we should do that in cases like this one where it's completely lost connectivity to the test node) | 14:16 |
| fungi | i expect it to tick over to the next task any second now, it's been on the current one for 30 minutes | 14:17 |
| fungi | and there it goes | 14:17 |
| *** catintheroof has joined #openstack-infra | 14:17 | |
| mpeterson | like a swiss clock | 14:18 |
| fungi | as long as the rest of these are localhost tasks it should hopefully wrap up | 14:18 |
| *** armax has joined #openstack-infra | 14:18 | |
| fungi | okay, logs for that are now at http://logs.openstack.org/31/514931/2/gate/openstack-tox-py35/bcfe8e0 but hopefully the build got rescheduled to a new node automagically and 514931,2 is still in the gate... checking | 14:20 |
| mpeterson | yes | 14:20 |
| mpeterson | as you expected | 14:20 |
| *** rbrndt_ has joined #openstack-infra | 14:21 | |
| *** Swami has joined #openstack-infra | 14:22 | |
| *** vsaienk0 has quit IRC | 14:23 | |
| mpeterson | things seem to be going well now | 14:24 |
| fungi | that's good at least | 14:24 |
| fungi | as for the build that was timing out, that one seems to have run in inap | 14:25 |
| *** yamamoto has joined #openstack-infra | 14:26 | |
| mpeterson | damn | 14:27 |
| mpeterson | failed | 14:27 |
| fungi | that was quick | 14:28 |
| mpeterson | the job didn't fail yet, but I can see that the tox command failed installing packages | 14:28 |
| *** eumel8 has quit IRC | 14:28 | |
| fungi | got that stream url handy? | 14:29 |
| mpeterson | fungi: http://zuulv3.openstack.org/static/stream.html?uuid=2601c9f64c224b00a9b64b5b61917916&logfile=console.log | 14:29 |
| fungi | there's a failure-looking message pip is emitting under tox when installing from local source | 14:29 |
| mpeterson | but it's running the tests now mmm weird | 14:29 |
| fungi | scrolling back in the stream is slow | 14:30 |
| fungi | so much output from the tests | 14:30 |
| mpeterson | mmm I'll get you the line that I meant once I can read the logs | 14:30 |
| fungi | yeah, i have a feeling it was the one where pip complains because it doesn't find an origin remote in the git repo. if so, that's scary-looking yet entirely benign | 14:31 |
| mpeterson | I actually wrote me a note early today to silence all this logging | 14:31 |
| *** yamamoto has quit IRC | 14:32 | |
| fungi | i sort of wonder if the higher incidence of post failures for those unit tests is due to the massive console log | 14:32 |
| fungi | though that doesn't explain the node in inap we just saw disappear out from under zuul in the pre phase | 14:32 |
| mpeterson | that could actually be an issue? | 14:32 |
| mpeterson | yeah, specially because that inap node never failed in a different phase | 14:33 |
| fungi | mpeterson: well, we have to collect logs over the network, and network interactions are the flakiest part of this entire process because, you know, clouds | 14:33 |
| mpeterson | yeah, wth, who are the crazy guys behind those... | 14:33 |
| mpeterson | fungi: failed again in the log collection | 14:34 |
| fungi | ;) | 14:34 |
| fungi | :( | 14:34 |
| mpeterson | perhaps it's bening? | 14:34 |
| mpeterson | because a merged message was issued | 14:34 |
| fungi | well, now that we have logs, i can check and tell you | 14:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 14:35 |
| mpeterson | okey, so let me find you those lines... | 14:35 |
| fungi | i'm already digging around in the console log | 14:35 |
| mpeterson | fungi: one is this one http://logs.openstack.org/31/514931/2/check/openstack-tox-py35/336c1b8/job-output.txt.gz#_2017-10-26_10_54_23_151273 | 14:35 |
| mpeterson | oh, okey | 14:35 |
| fungi | 845kb compressed? fun | 14:35 |
| mtreinish | fungi, clarkb: I haven't seen any launchpad firehose events go by in a while (I was watching to check for something) can you check if the lpmqtt daemon is running | 14:36 |
| mpeterson | oh that's nothing... on the java part we usually get like 123M compressed | 14:36 |
| fungi | mpeterson: the "falling back to uneditable format,Could not determine repository location" messages are indeed benign, a side effect of pip getting confused by the git repos we've pushed onto the test nodes since they don't have an origin remote | 14:37 |
| mpeterson | fungi: okey, and the skipping tests part is weird but is obviously related | 14:38 |
| fungi | it still installs them, as noted there, it just can't provide detailed git urls for them but we don't need it to anyway | 14:38 |
| *** gordc has joined #openstack-infra | 14:38 | |
| fungi | mpeterson: the "py35: skipped tests" part is because the task that ran under wasn't the actual test run task but rather one named "tox-siblings : Run tox without tests" | 14:39 |
| jeblair | 13:58 < tobiash> jkt: you probably will want to clone it from the merger and update (clone if the merger doesn't have it cached) it from upstream | 14:39 |
| jeblair | 14:12 < openstackgerrit> Merged openstack-infra/nodepool feature/zuulv3: Remove unreachable code https://review.openstack.org/506137 | 14:39 |
| jeblair | 14:35 < openstackgerrit> James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 14:39 |
| jeblair | derp sorry | 14:39 |
| fungi | mtreinish: i'll take a look at the server now and see if it's still going | 14:39 |
| fungi | mtreinish: lpmqtt 14396 0.1 1.3 283444 26972 ? Ssl 2016 605:19 /usr/bin/python /usr/local/bin/lpmqtt /etc/lpmqtt.conf | 14:40 |
| fungi | mtreinish: maybe we have some sort of e-mail delivery problem with it? when do you last remember seeing an lp event in the stream? | 14:40 |
| mtreinish | fungi: hmm, I wonder if the email stopped | 14:40 |
| mtreinish | fungi: oh, just saw a batch | 14:41 |
| *** nikhil has joined #openstack-infra | 14:41 | |
| fungi | oh, good | 14:41 |
| * fungi stands down | 14:41 | |
| mtreinish | I think we might have too long of a retry window | 14:41 |
| mtreinish | and I'm just impatient | 14:41 |
| mpeterson | fungi: mmm there was an error in the stream that just doesn't appear here :/ | 14:41 |
| mtreinish | fungi: I'll check the config in the puppet and maybe decrease the wait time a bit | 14:41 |
| mtreinish | fungi: oh, and more alarming is the warning on: http://status.openstack.org/openstack-health/#/ | 14:42 |
| *** martinkopec has joined #openstack-infra | 14:42 | |
| fungi | mtreinish: yeah, frickler pointed that out a little bit ago | 14:42 |
| fungi | i wonder if something is awry with the subunit2sql workers | 14:43 |
| *** csomerville has joined #openstack-infra | 14:43 | |
| mtreinish | fungi: yeah, I would assume it's something on the workers | 14:43 |
| fungi | mtreinish: i'm tailing /var/log/subunit2sql/subunit-worker-A-debug.log but... nothing showing up yet | 14:44 |
| *** scottda_ has quit IRC | 14:44 | |
| fungi | and i see why | 14:44 |
| fungi | looks like it's not running | 14:44 |
| *** cshastri has quit IRC | 14:45 | |
| *** ianychoi_ has joined #openstack-infra | 14:45 | |
| mtreinish | oh, did it crash? | 14:45 |
| *** martinkopec has quit IRC | 14:45 | |
| fungi | though i also may be looking at the wrong logfiles... these were last modified in may | 14:45 |
| mtreinish | hah | 14:45 |
| *** cody-somerville has quit IRC | 14:46 | |
| fungi | where is it supposed to log? | 14:46 |
| openstackgerrit | Merged openstack-infra/system-config master: Add notes on Release Managers abandon permissions https://review.openstack.org/514863 | 14:46 |
| *** vsaienk0 has joined #openstack-infra | 14:47 | |
| mtreinish | fungi: looking at the puppet that looks like the right place | 14:47 |
| mpeterson | what is MERGER_FAILURE? | 14:48 |
| mtreinish | fungi: /var/log/subunit2sql/subunit-worker<%= suffix %>-debug.log | 14:48 |
| *** ianychoi has quit IRC | 14:48 | |
| mtreinish | is what's in the puppet template for the init file | 14:48 |
| fungi | mtreinish: yeah, jenkins-subunit-worker-A initscript does say -d /var/log/subunit2sql/subunit-worker-A-debug.log | 14:48 |
| *** david-lyle has joined #openstack-infra | 14:48 | |
| mtreinish | fungi: are you on subunit-worker 01 or 02? | 14:48 |
| fungi | 01. didn't realize we have an 02 | 14:48 |
| fungi | or, rather, forgot | 14:48 |
| mtreinish | we stopped using 01 because it's precise iirc | 14:49 |
| jeblair | mpeterson: can you point me at the change where you saw that? | 14:49 |
| fungi | okay, now tailing the log on the active server, and it is indeed running | 14:49 |
| mtreinish | and there was an issue with ssl on it | 14:49 |
| fungi | "ValueError: Python 2.7.9 and 3.2 are the minimum supported versions for TLS." | 14:49 |
| mpeterson | jeblair: https://review.openstack.org/#/c/515057/2 | 14:49 |
| fungi | mtreinish: methinks we got a new paho-mqtt | 14:49 |
| mtreinish | fungi: oh, fun | 14:50 |
| jeblair | new things are always better! | 14:50 |
| AJaeger | jeblair: unless the break everything ;) | 14:50 |
| mpeterson | jeblair: there are two errors there actually the MERGER_FAILURE and the first job of that batch | 14:50 |
| fungi | mtreinish: http://paste.openstack.org/show/624721/ | 14:50 |
| *** armax has quit IRC | 14:50 | |
| jeblair | mpeterson: indeed. i bet neutron really does have a master branch. :) | 14:51 |
| *** burgerk has joined #openstack-infra | 14:51 | |
| mtreinish | fungi: hmm, the last paho release was 10/9: https://pypi.python.org/pypi/paho-mqtt | 14:51 |
| *** felipemonteiro__ has joined #openstack-infra | 14:52 | |
| fungi | mtreinish: and pip freeze says that's what we've got | 14:52 |
| fungi | subunit-gearman-worker.py last restarted Oct17 too according to the process list | 14:53 |
| fungi | so you would think we'd have no data since at least 9 days if this was the cause | 14:53 |
| *** felipemonteiro_ has quit IRC | 14:55 | |
| mtreinish | fungi: hmm, well it says the last data is from the 25th | 14:55 |
| fungi | right, which is why i think this must not be (completely at least) breaking it | 14:55 |
| mpeterson | jeblair: I guess I need to ignore both those errors and see what the recheck does then? | 14:55 |
| jeblair | mpeterson: i'm looking into them | 14:55 |
| mpeterson | good, thanks | 14:56 |
| mtreinish | fungi: are we running multiple worker processes on subunit-worker02? | 14:56 |
| mtreinish | I can't remember if we ever got that working | 14:56 |
| fungi | nope, just the one | 14:57 |
| mtreinish | fungi: could we have been running jobs in the future? (time sync issue) | 14:57 |
| fungi | a remote possibility i suppose | 14:57 |
| mtreinish | that timestamp in the warning comes from the date inserted into the db which it gets from the subunit iirc | 14:58 |
| fungi | also i don't think it's a gearman restart, since the scheduler was restarted 6+ hours after the data stops | 14:58 |
| jeblair | mtreinish: what warning? i don't see anything on o-h | 14:58 |
| fungi | jeblair: tan box near the top: " The last data in the subunit2sql database, from "2017-10-25T08:17:15.000Z", is >1 day old. There might be an issue with result collection." | 14:58 |
| jeblair | nope. don't see it. | 14:59 |
| *** ykarel|away has quit IRC | 14:59 | |
| fungi | weird | 14:59 |
| mtreinish | jeblair: https://i.imgur.com/WQewf57.png | 14:59 |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: Update neutron-failure-rate grafana dashboard https://review.openstack.org/515439 | 14:59 |
| *** xarses has joined #openstack-infra | 14:59 | |
| jeblair | mtreinish: https://screenshots.firefox.com/lm7FkiWlAJozvQUE/status.openstack.org | 15:00 |
| *** slaweq_ has quit IRC | 15:00 | |
| mtreinish | jeblair: timezones? I might have screwed up the JS and its comparing localtime to utc | 15:00 |
| fungi | ahh | 15:01 |
| fungi | i would buy that | 15:01 |
| jeblair | ah | 15:01 |
| fungi | well, except that 08:17z is longer ago than the offset between us and jeblair | 15:02 |
| *** spectr has quit IRC | 15:02 | |
| mtreinish | heh, I'm in prague right now so all my TZs are screwed up :) | 15:02 |
| fungi | ahh, you mean it's earluer than 08:17pdt | 15:02 |
| *** jaosorior has quit IRC | 15:02 | |
| fungi | which it technically is for 15 more minutes | 15:02 |
| *** baoli has quit IRC | 15:02 | |
| fungi | so, yeah, maybe that explains the missing message at least | 15:02 |
| *** gordc has left #openstack-infra | 15:03 | |
| jeblair | i'll check in 13m. | 15:04 |
| AJaeger | mtreinish: prague? That has a very nice timezone, keep it ;) | 15:04 |
| *** witek has quit IRC | 15:04 | |
| *** rcernin has quit IRC | 15:04 | |
| *** baoli has joined #openstack-infra | 15:05 | |
| *** witek has joined #openstack-infra | 15:06 | |
| AJaeger | jeblair, mordred, is https://review.openstack.org/#/c/515365/ the right approach - or should I do those changes directly on the roles involved? | 15:06 |
| mtreinish | jeblair, fungi: https://github.com/openstack/openstack-health/blob/master/app/js/directives/fresh-check.js#L19-L28 I don't see anything about tz in there | 15:06 |
| mtreinish | AJaeger: heh, yeah for open source summit europe. It's been a nice trip so far | 15:07 |
| mtreinish | fungi: so should I push a patch to pin paho, or should we spin up a new worker on a newer host os to get python >2.7.9? | 15:07 |
| jeblair | mpeterson: the executor that ran that job ran out of space. that shouldn't happen very often, though it may happen occasionally until the container/image projects change their workflow so we're not using the executor as an intermediary. a recheck should be good. | 15:08 |
| mpeterson | jeblair: thank you! | 15:10 |
| mpeterson | fungi: also you! | 15:10 |
| mpeterson | you guys are great | 15:10 |
| jeblair | though also, we may have extra directories laying around from some debugging work i did yesterday taking up space | 15:10 |
| jeblair | i'll work on cleaning those up to reduce the chances of this | 15:10 |
| fungi | mtreinish: i can try to replace the old 01 with xenial since it's unused anyway | 15:10 |
| fungi | we don't still need anything off subunit-worker01's filesystem i'm assuming? it's just been sitting there idle since may-ish | 15:11 |
| mtreinish | fungi: I don't think so | 15:11 |
| fungi | i'll try to get started on it as soon as tc office hour winds up | 15:12 |
| fungi | or winds down. whichever | 15:12 |
| jeblair | fungi: find /var/lib/zuul/builds -maxdepth 1 -cmin +480 -exec rm -fr {} \; | 15:12 |
| frickler | fungi: seems openstack-health is indeed still processing data, just being well behind. the timestamp for "latest data" seems to be continually increasing, just slowly | 15:12 |
| jeblair | fungi: how's that look for deleting all build dirs older than 8 hours? | 15:12 |
| *** annp has quit IRC | 15:13 | |
| *** baoli has quit IRC | 15:13 | |
| fungi | jeblair: any reason not to use -type d? | 15:14 |
| fungi | worried the timestamps on the directories may be older than contents several levels deeper? | 15:14 |
| fungi | jeblair: maybe use -type f with -delete and then do a separate pass looking for -type d -empty? | 15:15 |
| *** e0ne has joined #openstack-infra | 15:15 | |
| fungi | oh, right, with -maxdepth 1 you presumably only get directories anyway | 15:16 |
| fungi | jeblair: yeah, i'm done talking to myself now. lgtm ;) | 15:16 |
| jeblair | \o/ | 15:17 |
| *** Guest18426 has quit IRC | 15:17 | |
| *** sree has joined #openstack-infra | 15:18 | |
| *** baoli has joined #openstack-infra | 15:18 | |
| mtreinish | fungi, frickler: hmm yeah: http://paste.openstack.org/show/624727/ there are some new ones in there from what o-h was previously reporting | 15:18 |
| *** catintheroof has quit IRC | 15:20 | |
| mtreinish | fungi: is the worker really stuck there in a loop reporting the paho issue, or could it sporadically inserting data between the mqtt errors? | 15:20 |
| *** catintheroof has joined #openstack-infra | 15:20 | |
| fungi | mtreinish: there's lots of other logging going on about gearman work requests/notifications | 15:20 |
| fungi | so it does appear to at least be trying to do stuff. this error message _may_ simply be benign if we don't have it configured to actually do ssl/tls | 15:21 |
| fungi | oh, or, yeah... it's just failing to report to the firehose | 15:22 |
| fungi | but possible these mqtt reporting failures are also bogging it down, no idea really | 15:22 |
| *** sree has quit IRC | 15:22 | |
| mtreinish | fungi: if there are log messages like: "Converting Subunit V2 stream: %s to SQL" that means it's running the db insert stuff | 15:23 |
| frickler | can this also be related to e-r being backed up because it is still waiting for some console.log to show up? or is that a completely different beast? | 15:23 |
| mtreinish | the worker is configured to handle exceptions in the outerloop by logging them but not exit on it | 15:24 |
| *** sree has joined #openstack-infra | 15:24 | |
| *** catintheroof has quit IRC | 15:25 | |
| fungi | 2017-10-26 15:23:32,825 Converting Subunit V2 stream: <_io.BytesIO object at 0x7f7a42ea9770> to SQL | 15:25 |
| fungi | so, yeah | 15:25 |
| mtreinish | frickler: that's likely related to zuul v3 changing the filename of the output | 15:26 |
| fungi | logging into the rackspace dashboard is always a fun experience. never fails to have an interesting trouble ticket or several waiting for us | 15:27 |
| mtreinish | fungi: hah, nice | 15:27 |
| mtreinish | frickler: although the logstash workers will likely be hit by the same paho mqtt issue on restart | 15:27 |
| mtreinish | as will any of our python things running mqtt on trusty | 15:28 |
| *** yamamoto has joined #openstack-infra | 15:28 | |
| *** dhinesh has joined #openstack-infra | 15:28 | |
| *** sree has quit IRC | 15:29 | |
| *** ykarel has joined #openstack-infra | 15:29 | |
| *** ijw has joined #openstack-infra | 15:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make tripleo undercloud-containers voting https://review.openstack.org/513163 | 15:31 |
| *** ijw has quit IRC | 15:32 | |
| fungi | #status log Provider maintenance is scheduled for 2017-10-30 between 06:00-09:00 UTC which may result in up to a 5 minute connectivity outage for the production Gerrit server's Trove database instance | 15:32 |
| openstackstatus | fungi: finished logging | 15:32 |
| fungi | infra-root: ^ | 15:32 |
| dmsimard | fungi: should we pre-emptively shut down services before the maintenance window in order to avoid corruption or otherwise an unclean crash ? | 15:33 |
| *** yamamoto has quit IRC | 15:33 | |
| dmsimard | fungi: or is it just network ? | 15:34 |
| clarkb | fwiw gerrit survived the unclean shutdown of live migration gone wrong not too long ago | 15:34 |
| clarkb | it will likely just 500 error while it can't hit the database | 15:35 |
| clarkb | and be fine afterwards | 15:35 |
| *** ykarel has quit IRC | 15:35 | |
| clarkb | and the db will be backed up at 0000UTC | 15:35 |
| *** fultonj_ has joined #openstack-infra | 15:35 | |
| *** fultonj_ has quit IRC | 15:36 | |
| clarkb | frickler: ya the elasticsearch indexing is almost completely independent. And really it is indexing we are just looking for the wrong things in e-r | 15:36 |
| clarkb | dmsimard: re ^ did a chance get pushed to support a config switch? | 15:36 |
| dmsimard | clarkb: I sent a patchset with another approach | 15:37 |
| dmsimard | which seemed ok to me | 15:37 |
| dmsimard | sec | 15:37 |
| mtreinish | clarkb: iirc it's already configurable we should just need to update the puppet | 15:37 |
| clarkb | dmsimard: we should probably have mtreinish review it if mtreinish isn't too busy in prague :) | 15:37 |
| fungi | dmsimard: yeah, i would mostly just make sure we keep an eye on things... maybe ianw can double-check things are still in sane condition before he hits eod on the 30th | 15:37 |
| clarkb | mtreinish: the problem is that console.html and the "last" log line were hardcoded iirc | 15:37 |
| dmsimard | clarkb: https://review.openstack.org/#/c/509313/ oh, frickler added some comments -- I missed that | 15:37 |
| dmsimard | mtreinish: ^ see that review | 15:37 |
| mtreinish | clarkb: I think that was one of the things ramy did for his hpe downstream e-r usage | 15:38 |
| *** fultonj has quit IRC | 15:38 | |
| dmsimard | clarkb: the fact that gerrit survived the unclean shutdown doesn't mean it will every time :p | 15:38 |
| *** baoli has quit IRC | 15:38 | |
| *** fultonj has joined #openstack-infra | 15:38 | |
| *** baoli has joined #openstack-infra | 15:39 | |
| clarkb | dmsimard: this is true | 15:39 |
| fungi | dmsimard: in this case it's not the gerrit server going offline, it's the trove database instance it talks to | 15:39 |
| clarkb | mtreinish: I know the base url and stuff was configurable | 15:39 |
| clarkb | mtreinish: in any case https://review.openstack.org/#/c/509313/ is dmsimard's change to address some of this | 15:39 |
| dmsimard | I'll check out frickler's comments and address them later today, need to wrap up something first | 15:40 |
| clarkb | dmsimard: thanks | 15:40 |
| mtreinish | clarkb: ok, I'll try to review it tomorrow | 15:40 |
| mtreinish | or late tonight depending on my mood then :) | 15:41 |
| clarkb | mtreinish: just drink the lager and not the pils they say its lighter ;) | 15:41 |
| fungi | lighter colored? lighter bodied? lighter in abv? | 15:42 |
| jeblair | less dense? | 15:42 |
| fungi | these things matter to fungi | 15:42 |
| *** witek has quit IRC | 15:42 | |
| clarkb | fungi: the four options in every place were "light pils wheat dark" | 15:42 |
| clarkb | you didn't get a choice of brewery or whatever | 15:43 |
| clarkb | the light and the pils were pretty indistinguishable to me | 15:43 |
| fungi | jeblair: lower specific gravity i suppose | 15:43 |
| *** baoli has quit IRC | 15:43 | |
| clarkb | ya I'm guessing its basically the same beer as the pils just scaled down a bit | 15:43 |
| *** gyee has joined #openstack-infra | 15:43 | |
| * fungi will eventually find his way to prague | 15:44 | |
| clarkb | I kind of enjoy a simple beer menu like that, makes ordering easy. But the equivalent here is bud light, bud, blue moon, guiness and not sure I could live in that world | 15:45 |
| jeblair | to be fair, you're very likely to get 'bud' there too. but the real bud. | 15:46 |
| *** andreas_s has quit IRC | 15:46 | |
| *** andreas_s has joined #openstack-infra | 15:47 | |
| *** makowals has quit IRC | 15:47 | |
| *** witek has joined #openstack-infra | 15:47 | |
| mtreinish | clarkb: I had "Dalešické májové" the other day. I don't know whether it's lager or pils (or something else) but I liked it | 15:48 |
| *** vsaienk0 has quit IRC | 15:48 | |
| mtreinish | I also have no idea how to pronounce it | 15:49 |
| *** felipemonteiro_ has joined #openstack-infra | 15:49 | |
| *** baoli has joined #openstack-infra | 15:49 | |
| fungi | jeblair: itym "the real kind" | 15:50 |
| *** catintheroof has joined #openstack-infra | 15:51 | |
| *** andreas_s has quit IRC | 15:51 | |
| jeblair | indeed | 15:51 |
| *** felipemonteiro__ has quit IRC | 15:52 | |
| *** andreas_s has joined #openstack-infra | 15:53 | |
| *** baoli has quit IRC | 15:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add implied branch matchers on 'master' https://review.openstack.org/514459 | 15:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch to late-binding inheritance https://review.openstack.org/511352 | 15:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 15:54 |
| *** hemna_ has quit IRC | 15:54 | |
| *** catintheroof has quit IRC | 15:55 | |
| clarkb | dmsimard: I left a comment on that change too. I think that approach should work though | 15:56 |
| *** rkukura has joined #openstack-infra | 15:58 | |
| *** vsaienk0 has joined #openstack-infra | 15:58 | |
| *** Apoorva has joined #openstack-infra | 15:59 | |
| *** catintheroof has joined #openstack-infra | 16:01 | |
| *** andreas_s has quit IRC | 16:03 | |
| *** Hal has joined #openstack-infra | 16:03 | |
| *** andreas_s has joined #openstack-infra | 16:03 | |
| *** Hal is now known as Guest87185 | 16:04 | |
| fungi | infra-root: the fix to get e-mail reporting working again merged earlier today; when would be a good time for a scheduler restart to pick that up? | 16:07 |
| *** kiennt26 has joined #openstack-infra | 16:07 | |
| *** baoli has joined #openstack-infra | 16:08 | |
| clarkb | fungi: I'm good with whenever, probably depends more on release team than anything else? jeblair did just +3 a change of mine that will allow us to configure cgit urls | 16:08 |
| jeblair | fungi: any time i think | 16:08 |
| clarkb | let me push up a config change now so that we can get both things in at the same time | 16:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make the git web url a template https://review.openstack.org/515168 | 16:09 |
| *** andreas_s has quit IRC | 16:09 | |
| *** vsaienk0 has quit IRC | 16:09 | |
| *** LindaWang has quit IRC | 16:09 | |
| fungi | cool, i'll see when the release team has achieved quiescence | 16:09 |
| clarkb | fungi: also all three of your pre release signed tarballs fix changes merged I think we just need to confirm they work as expected then can mark them off the list | 16:09 |
| fungi | oh, yeah i believe they confirmed that yesterday while i was on the road. i'll double-check that too thanks | 16:10 |
| *** andreas_s has joined #openstack-infra | 16:10 | |
| *** armax has joined #openstack-infra | 16:10 | |
| *** dprince has quit IRC | 16:12 | |
| pabelanger | able to support a scheduler restart too | 16:12 |
| pabelanger | also, I think ianw finished the rebuild of zm01 to zm04 servers. Going to see if he started them or not | 16:13 |
| *** sree has joined #openstack-infra | 16:14 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Set cgit url in zuul connection config https://review.openstack.org/515460 | 16:15 |
| clarkb | there is the config change to use cgit urls | 16:15 |
| clarkb | if we can bundle that into the restart that would be excellent | 16:15 |
| pabelanger | Hmm, for some reason we didn't create swap again on zm01-zm04 | 16:16 |
| *** slaweq has joined #openstack-infra | 16:16 | |
| *** baoli has quit IRC | 16:16 | |
| pabelanger | I'm going to work on that now | 16:16 |
| *** derekh has quit IRC | 16:17 | |
| *** psachin has quit IRC | 16:17 | |
| *** lucasagomes is now known as lucas-afk | 16:17 | |
| jeblair | the cgit url change is safe to approve now -- it's okay to add unused things to the config file | 16:17 |
| jeblair | i have +2d | 16:18 |
| fungi | my attempt at building the replacement subunit-worker01 failed when trying to gather facts during the "remove cloud-init" task. weird... Could not detect which package manager to use." | 16:18 |
| *** andreas_s has quit IRC | 16:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs" https://review.openstack.org/515411 | 16:19 |
| *** sree has quit IRC | 16:20 | |
| clarkb | for ansible file resources that are state directory will it create multiple directories if parent dirs are not created yet? or do we have to sequentially list each directory? | 16:20 |
| fungi | release team is cool with us restarting zuul any time now (and excited to have their e-mail error reporting return) | 16:21 |
| *** baoli has joined #openstack-infra | 16:21 | |
| *** baoli has quit IRC | 16:22 | |
| *** mummer has joined #openstack-infra | 16:23 | |
| clarkb | "all immediate subdirectories will be created if they do not exist" <- I'm not sure I understand what that means | 16:24 |
| clarkb | does that mean just one level? | 16:24 |
| clarkb | or anything in the path? | 16:24 |
| * clarkb goes with it | 16:24 | |
| *** andreas_s has joined #openstack-infra | 16:24 | |
| fungi | huh, trying to launch the new subunit-worker01 again, same failure gathering facts on the same cloud-init removal task | 16:25 |
| dmsimard | clarkb: needs recurse: true | 16:25 |
| fungi | i'll try running with ansible from a virtualenv instead of the one we have globally-installed and see if that works any better | 16:25 |
| dmsimard | clarkb: file: path: /some/deep/path state: directory recurse: yes | 16:26 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Use local cirros image in tempest https://review.openstack.org/515462 | 16:26 |
| clarkb | dmsimard: thanks | 16:26 |
| fungi | ugh, i think the launch progressed further by using a virtualenv'd install of ansible i found in ~root, but it's failing to get the paho-mqtt library installed on the server now. rerunning with --keep to see if i can figure out from its syslog what transpires | 16:30 |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| pabelanger | okay, zm01-zm04 have swap again, it looks like ianw had an old version of make_swap.sh when he created new servers | 16:30 |
| *** ccamacho has left #openstack-infra | 16:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Create dir needed to publish gerrit plugins https://review.openstack.org/515463 | 16:31 |
| clarkb | dmsimard: like that ^? | 16:31 |
| *** baoli has joined #openstack-infra | 16:32 | |
| *** iyamahat has joined #openstack-infra | 16:32 | |
| *** bhavik1 has joined #openstack-infra | 16:33 | |
| dmsimard | clarkb: yes | 16:34 |
| dmsimard | clarkb: that allows you to create a nested tree, but also to do things like set permissions and stuff. | 16:34 |
| *** Goneri has quit IRC | 16:34 | |
| *** yamamoto has quit IRC | 16:34 | |
| fungi | argh. --keep doesn't help because the manifest hasn't gotten far enough to add our admin shell accounts yet :/ | 16:35 |
| *** andreas_s has quit IRC | 16:36 | |
| clarkb | fungi: I think you can use the ssh key launch uses in that case | 16:36 |
| clarkb | to ssh as root? | 16:36 |
| fungi | oh, as root from the puppetmaster? yeah maybe | 16:36 |
| fungi | nope, prompting for root's password now | 16:37 |
| fungi | does the launch script use a different ssh key than the normal root ssh key on the puppetmaster? | 16:37 |
| clarkb | ya it generates one iirc | 16:38 |
| fungi | i see some files in .tmp which end in id_rsa so maybe | 16:38 |
| clarkb | but I think if you --keep it doesn't delete it | 16:38 |
| fungi | er, in /tmp | 16:38 |
| fungi | ooh, those uuids are the nova instance ids. perfect | 16:38 |
| fungi | cool, that gets me in. thanks for the idea clarkb | 16:39 |
| fungi | oh, i think the log entry about the missing paho-mqtt lib is just a warning | 16:41 |
| *** rossella_s has quit IRC | 16:42 | |
| fungi | it's still failing on "TASK [set_hostname : Remove cloud-init]" | 16:42 |
| fungi | fatal: [subunit-worker01.openstack.org]: FAILED! => {"changed": false, "failed" true, "msg": "Could not detect which package manager to use. Try gathering facts or setting the \"use\" option."} | 16:42 |
| *** e0ne has quit IRC | 16:43 | |
| fungi | any clue what could cause that?> | 16:43 |
| *** Goneri has joined #openstack-infra | 16:43 | |
| *** e0ne has joined #openstack-infra | 16:44 | |
| clarkb | fungi: I think we likely used the generic package module in ansible rather than package manager specific modules to remove that package so ansible needs facts to switch automatically on the right one | 16:44 |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: Fix Grafana periodic counters for neutron-lib dashboard https://review.openstack.org/514801 | 16:44 |
| clarkb | fungi: likely just needs to gather facts on that playbook | 16:44 |
| *** e0ne has quit IRC | 16:44 | |
| *** baoli has quit IRC | 16:44 | |
| fungi | seems like that would be impacting all our server launching though | 16:44 |
| pabelanger | which playbook is that? | 16:44 |
| *** ijw has joined #openstack-infra | 16:44 | |
| clarkb | pabelanger: the launch one I think | 16:45 |
| clarkb | fungi: its a fairly new change (I didn't realize we had merged it even) | 16:45 |
| *** baoli has joined #openstack-infra | 16:45 | |
| pabelanger | ah, puppetmaster.o.o | 16:45 |
| *** rkukura has quit IRC | 16:45 | |
| pabelanger | maybe we should setup fact cache and use smart facts there | 16:45 |
| *** Swami has quit IRC | 16:45 | |
| *** e0ne has joined #openstack-infra | 16:45 | |
| clarkb | pabelanger: I'm not sure that would change anything since this is fire and forget launch node right? | 16:46 |
| fungi | okay, so this being an extremely recent regression makes sense i suppose | 16:46 |
| *** e0ne has quit IRC | 16:46 | |
| clarkb | fungi: ya | 16:46 |
| *** cody-somerville has joined #openstack-infra | 16:46 | |
| fungi | like in between when ianw finished launching the replacement zuul mergers and now | 16:46 |
| pabelanger | clarkb: Oh, this is launch-node.py, now i understand | 16:47 |
| *** andreas_s has joined #openstack-infra | 16:47 | |
| *** e0ne has joined #openstack-infra | 16:47 | |
| clarkb | fungi: oh not sure if it is that recent | 16:47 |
| pabelanger | which verson of ansible are use using fungi ? | 16:47 |
| *** e0ne has quit IRC | 16:47 | |
| clarkb | system-config/playbooks/roles/set_hostname/tasks/main.yml is the playbook I think | 16:48 |
| *** csomerville has quit IRC | 16:48 | |
| *** slaweq has quit IRC | 16:48 | |
| clarkb | I don't see it overriding fact gathering | 16:48 |
| *** slaweq has joined #openstack-infra | 16:48 | |
| rajinir | Does recheck still work for Zuul Gate upstream? | 16:48 |
| *** bhavik1 has quit IRC | 16:49 | |
| clarkb | rajinir: should | 16:49 |
| rajinir | "reacheck" same as before? | 16:49 |
| *** ijw has quit IRC | 16:49 | |
| fungi | pabelanger: i tried the one globally installed on puppetmaster, and then the one in the virtualenv in ~root, both with the same result | 16:49 |
| *** baoli has quit IRC | 16:49 | |
| clarkb | rajinir: yes "recheck" same as before | 16:49 |
| rajinir | clarkb: thanks | 16:51 |
| fungi | pabelanger: so both 2.2.1.0 and 2.1.1.0 | 16:51 |
| *** andreas_s has quit IRC | 16:51 | |
| pabelanger | odd | 16:51 |
| pabelanger | ianw launched 4 new servers yesterday, don't think he seen that issue | 16:51 |
| pabelanger | possible he has a different ansible.cfg in is home directory that always gather facts, but not sure | 16:52 |
| *** dprince has joined #openstack-infra | 16:52 | |
| clarkb | gathering facts is the default though right? | 16:53 |
| clarkb | nothing seems to turn that off in the playbook | 16:53 |
| *** slaweq has quit IRC | 16:53 | |
| clarkb | or in the launch.py ansible invocation? maybe its in the /etc/ansible config? | 16:53 |
| pabelanger | maybe fungi has it disabled in ansible.cfg? | 16:53 |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: Fix Grafana neutron-lib dashboard https://review.openstack.org/514801 | 16:53 |
| *** florianf has quit IRC | 16:53 | |
| pabelanger | local ansible.cfg | 16:53 |
| fungi | i don't have a local ansible.cfg afaik | 16:54 |
| fungi | here's what i'm doing (as a non-root user): http://paste.openstack.org/show/624732/ | 16:54 |
| pabelanger | yah, looks correct | 16:55 |
| pabelanger | it is also possible ianw wasn't using latest master, as his version of make_swap.sh was out of date | 16:55 |
| fungi | you're welcome to repeat the exercise if you like. unfortunately the fact that it spams teh entire openstack inventory on stdout makes it pretty impossible to find what transpired before the last few steps unless you have a really huge terminal buffer | 16:55 |
| *** e0ne has joined #openstack-infra | 16:56 | |
| clarkb | cloud init removal merged 10/9 | 16:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging https://review.openstack.org/515470 | 16:56 |
| clarkb | I think our use of ansible-playbook rather than ansible may be part of this? | 16:58 |
| clarkb | we don't pass a hosts directive in our playbook | 16:58 |
| *** sambetts is now known as sambetts|afk | 16:58 | |
| clarkb | oh because it is a role? | 16:58 |
| * clarkb reads more | 16:59 | |
| *** jpena is now known as jpena|off | 16:59 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging https://review.openstack.org/515470 | 16:59 |
| clarkb | aha! | 16:59 |
| clarkb | system-config/playbooks/set_hostname.yml | 16:59 |
| clarkb | fungi: ^ that says gather facts no. I think you can just delete that line or comment it out and rerun and it should work | 16:59 |
| clarkb | and if it does push a patch I guess | 17:00 |
| fungi | is the playbook shared by other stuff besides the launch script? wondering if that's otherwise useful, or whether we simply merged a completely untested change for our server launching | 17:00 |
| *** slaweq has joined #openstack-infra | 17:00 | |
| clarkb | grep says we don't use it anywhere else | 17:01 |
| fungi | strange | 17:01 |
| fungi | well, running on a separate clone of system-config in my homedir now with that line removed | 17:02 |
| fungi | fingers crossed | 17:02 |
| *** Swami has joined #openstack-infra | 17:02 | |
| *** claudiub|2 has quit IRC | 17:03 | |
| *** ralonsoh has quit IRC | 17:04 | |
| fungi | i'm going to enqueue 515460 into the gate so we don't have to wait for the check backlog to burn down before we can restart zuul with it in place | 17:04 |
| clarkb | sounds good | 17:05 |
| *** dsariel has quit IRC | 17:06 | |
| *** andreas_s has joined #openstack-infra | 17:06 | |
| *** ijw has joined #openstack-infra | 17:07 | |
| pabelanger | ++ | 17:08 |
| fungi | i think the launch has made it past where it was erroring before, so that's a good sign | 17:08 |
| *** baoli has joined #openstack-infra | 17:09 | |
| *** kjackal_ has quit IRC | 17:09 | |
| *** kjackal_ has joined #openstack-infra | 17:10 | |
| *** cody-somerville has quit IRC | 17:11 | |
| fungi | well, we made it to running puppet anyway | 17:11 |
| *** hemna_ has joined #openstack-infra | 17:11 | |
| *** e0ne has quit IRC | 17:12 | |
| *** baoli has quit IRC | 17:12 | |
| *** kiennt26 has quit IRC | 17:13 | |
| fungi | now to figure out what's regressed in this manifest | 17:13 |
| *** rwsu has quit IRC | 17:13 | |
| fungi | ahh, something's breaking with `pip install -q --upgrade subunit2sql` | 17:14 |
| fungi | looks like pip's having trouble building netifaces | 17:15 |
| fungi | likely missing a dev package or something | 17:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Gather facts in set_hostname playbook https://review.openstack.org/515478 | 17:16 |
| *** andreas_s has quit IRC | 17:17 | |
| *** baoli has joined #openstack-infra | 17:17 | |
| *** jamesdenton has quit IRC | 17:17 | |
| *** jamesden_ has joined #openstack-infra | 17:17 | |
| *** jamesden_ is now known as jamesdenton | 17:18 | |
| *** rbrndt_ has quit IRC | 17:18 | |
| *** yamahata has joined #openstack-infra | 17:18 | |
| johnsom | Is the current recommendation to remove the "queue: " configuration in our projects? I thought I saw some talk of that here the other day. | 17:19 |
| *** iyamahat has quit IRC | 17:19 | |
| *** Goneri has quit IRC | 17:20 | |
| fungi | johnsom: i think the tripleo team (temporarily?) undid their shared change queue in an attempt to get changes for more of their different projects merging in parallel (at the risk of wedging some of them which are normally tested together) | 17:21 |
| johnsom | Ok, if it's not a general thing, I will leave ours. I don't think it's been a big problem for us. | 17:21 |
| *** dtantsur is now known as dtantsur|afk | 17:22 | |
| *** trown is now known as trown|lunch | 17:22 | |
| *** slaweq has quit IRC | 17:22 | |
| *** Goneri has joined #openstack-infra | 17:22 | |
| pabelanger | fungi: no, they just made jobs non-voting but have reverted that | 17:23 |
| *** rkukura has joined #openstack-infra | 17:25 | |
| *** rwsu has joined #openstack-infra | 17:26 | |
| fungi | pabelanger: oh, i saw a change fly by earlier for removing the tripleo shared queue from zuul | 17:26 |
| * fungi finds | 17:26 | |
| fungi | pabelanger: https://review.openstack.org/515070 but i guess it hasn't merged yet | 17:27 |
| fungi | and (justifiable) has opposition voiced on it | 17:27 |
| fungi | er, justifiably | 17:27 |
| *** panda|ruck is now known as panda|ruck|off | 17:30 | |
| *** yamamoto has joined #openstack-infra | 17:31 | |
| *** kjackal_ has quit IRC | 17:32 | |
| pabelanger | fungi: thanks, commented too. And agree | 17:32 |
| *** kjackal_ has joined #openstack-infra | 17:33 | |
| *** ijw has quit IRC | 17:33 | |
| *** yamamoto has quit IRC | 17:36 | |
| *** ijw has joined #openstack-infra | 17:36 | |
| *** andreas_s has joined #openstack-infra | 17:38 | |
| AJaeger | mordred, pabelanger, is https://review.openstack.org/#/c/515365/ the right approach - or should I do those changes directly on the roles involved? | 17:38 |
| *** ijw has quit IRC | 17:40 | |
| *** andreas_s has quit IRC | 17:43 | |
| pabelanger | AJaeger: left comments about create / delete afs token, we can likely skip that too | 17:44 |
| openstackgerrit | Merged openstack-infra/system-config master: Set cgit url in zuul connection config https://review.openstack.org/515460 | 17:44 |
| AJaeger | pabelanger: yes, we could - is there a way to skip the whole section? Or do I need this for each one? | 17:45 |
| *** tesseract has quit IRC | 17:45 | |
| pabelanger | AJaeger: you'd need to add them for each. If we moved to include_role, we could use a block and single conditional | 17:46 |
| *** amoralej is now known as amoralej|off | 17:46 | |
| AJaeger | ok | 17:46 |
| fungi | okay, 515460 merged a couple minutes ago, so getting prepped for scheduler restart now | 17:47 |
| *** dhajare|mtg has joined #openstack-infra | 17:47 | |
| fungi | jeblair: do we have any custom patches that need to be cherry-picked in, or just a normal restart is expected to be fine at this point? | 17:48 |
| *** sree has joined #openstack-infra | 17:48 | |
| fungi | any patched libs i should check that haven't been upgraded to non-patched versions or anything like that? | 17:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 17:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add pragma directive https://review.openstack.org/515483 | 17:49 |
| pabelanger | AJaeger: I'm not sure if we'd add it to zuul or not, but one idea would be to have a post-run playbook on success flag, then we wouldn't need to sprinkle the conditions inside a playbook. | 17:49 |
| jeblair | fungi: the only thing is the relp patch i've been manually adding, but i think i can live without it for a bit, so feel free to just use system-installed zuul at this point. | 17:49 |
| *** yamahata has quit IRC | 17:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging https://review.openstack.org/515470 | 17:49 |
| pabelanger | right now, I don't think you can write a playbook to have hosts: foo, when: something and skip all tasks / roles | 17:50 |
| fungi | jeblair: great, thanks. grabbing check/gate contents and will then restart | 17:50 |
| pabelanger | you'd just not run ansible-playbook on that host | 17:50 |
| *** andreas_s has joined #openstack-infra | 17:50 | |
| jeblair | fungi: mind restarting iptables while it's down? | 17:50 |
| fungi | gladly! | 17:50 |
| *** Apoorva has quit IRC | 17:51 | |
| fungi | what's the service name for it? not iptables nor iptables-persistent i guess | 17:51 |
| AJaeger | pabelanger: ok, will continue with the way for now | 17:51 |
| *** dhinesh has quit IRC | 17:52 | |
| clarkb | fungi: I think nftables | 17:52 |
| clarkb | on xenial /me looks | 17:52 |
| *** sree has quit IRC | 17:52 | |
| clarkb | fungi: netfilter-persistent | 17:53 |
| fungi | perfect, thanks clarkb! | 17:53 |
| clarkb | I guess that is because iptables is a lie at this point and it is all netfilter under the covers? | 17:53 |
| fungi | netfilter-persistent restarted, zuul-scheduler coming up now | 17:54 |
| *** ccamacho has joined #openstack-infra | 17:55 | |
| fungi | anything in particular i should wait to see in the logs before i start reenqueuing jobs? | 17:55 |
| *** andreas_s has quit IRC | 17:55 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Remove migrated octavia zuul v3 jobs https://review.openstack.org/515486 | 17:55 |
| clarkb | maybe just double check iptables -L -n shows the expected rules? | 17:55 |
| fungi | i did, they're as expected | 17:56 |
| clarkb | cool | 17:56 |
| SamYaple | can i get some love on https://review.openstack.org/#/c/512941/ and https://review.openstack.org/#/c/513591/ ? they have been sitting for a week | 17:56 |
| jeblair | fungi: the 5k cat runs will take a few minutes. during that time, you can submit a zuul enqueue but it won't complete until they do | 17:56 |
| clarkb | as for the cgit thing we need post jobs to run to know that is working so nothing else to check on my end | 17:56 |
| fungi | looks like the scheduler is about done loading configuration maybe | 17:56 |
| jeblair | fungi: so it's okay to go ahead and starte re-enqueing any time, or you can wait until it's done config. | 17:56 |
| fungi | okay, cool | 17:56 |
| *** andreas_s has joined #openstack-infra | 17:57 | |
| jeblair | (just be aware if you start early, it's going to hang for a few mins anyway) | 17:57 |
| fungi | yeah, seems to be processing events already. i'll go ahead and reenqueue | 17:57 |
| jlvillal | Did something just happen to Zuul? The status page now shows 3 total jobs??? | 17:57 |
| jeblair | jlvillal: yeah we're restarting it. we're re-enqueing the old changes | 17:57 |
| * jlvillal wonders if chat about re-enqueing is realted | 17:57 | |
| jeblair | jlvillal: shouldn't need to do anything | 17:57 |
| jlvillal | related | 17:57 |
| fungi | jlvillal: you'll see them back in the next few minutes | 17:57 |
| jlvillal | jeblair: fungi: Awesome! | 17:58 |
| fungi | i saved the contents of check, gate and check-tripleo just before stopping and am putting them all back in now | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: remove implied run https://review.openstack.org/515487 | 17:58 |
| *** Apoorva has joined #openstack-infra | 17:58 | |
| *** slaweq has joined #openstack-infra | 18:00 | |
| *** tosky has quit IRC | 18:00 | |
| clarkb | jeblair: re SamYaple's changes do you know if gerrit publishes a list of files on ref-updated events? | 18:01 |
| *** slaweq has quit IRC | 18:01 | |
| *** andreas_s has quit IRC | 18:02 | |
| clarkb | looks liek they aren't, are file inclusion/exclusion relevant in that case with zuul (it could look up the diff itself I guesss) | 18:03 |
| *** dhajare|mtg has quit IRC | 18:03 | |
| SamYaple | clarkb: oh thats a good question. i hadnt considered that | 18:05 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Copy Gerrit delay to GitHub driver https://review.openstack.org/515224 | 18:05 |
| SamYaple | clarkb: only to save infra resources. i really only need to trigger on upper-constraints.txt changes, but changing on all doesnt matter to me | 18:06 |
| *** iyamahat has joined #openstack-infra | 18:06 | |
| clarkb | SamYaple: I think it woudl require zuul to inspect the diff directly. With v2 I'm pretty sure it was only supported if the gerrit event had the data though | 18:06 |
| SamYaple | its in the post queue, so low priority and jeblair has some post queue optimizations | 18:06 |
| jeblair | clarkb: i believe you're correct | 18:06 |
| jeblair | i really want change-merged to have the merge commit sha :( | 18:06 |
| sdake | tonyb "This will NOT be a discussion on how we implement an LTS but the topic will certainly come up ;P" :-) | 18:07 |
| fungi | #status log zuul scheduler restarted and check/check-tripleo/gate pipeline contents successfully reenqueued | 18:07 |
| openstackstatus | fungi: finished logging | 18:07 |
| jeblair | fungi, pabelanger: do the firewalls on the new zm servers look right? | 18:08 |
| *** rbrndt_ has joined #openstack-infra | 18:08 | |
| fungi | jeblair: the rules on zm01 appear a bit... minimalist | 18:08 |
| *** eroux has joined #openstack-infra | 18:09 | |
| * SpamapS is sad that OpenStack is moving away from industry best practice of CD, and toward old world LTS releasing. | 18:09 | |
| clarkb | SpamapS: so I think we'll have to drop the files restriction based on jeblair's comment. I'll leave a note on the change | 18:09 |
| clarkb | er SamYaple ^ | 18:09 |
| * SpamapS S'niped clarkb's reply | 18:10 | |
| SamYaple | clarkb: will update it, thanks :) | 18:10 |
| jeblair | fungi: hrm, i thought we had a reboot at the end of the launch which should take care of it. | 18:10 |
| fungi | is it safe to stop the zuul-merger daemon on one and reboot it? | 18:12 |
| *** ijw has joined #openstack-infra | 18:12 | |
| jeblair | fungi: i think/hope so? | 18:12 |
| fungi | just to see what transpires | 18:12 |
| fungi | i'll do that on zm01 now | 18:12 |
| jeblair | i'm pretty sure i put in a think that waits till the current merge is complete before shutting down, so should be transparent. | 18:12 |
| jeblair | a thing | 18:12 |
| fungi | yeah, it's not running after a stop, so it stopped at any rate | 18:13 |
| clarkb | SamYaple: commented on the other change too | 18:13 |
| fungi | rebooting now | 18:13 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish requirements loci images to DockerHub https://review.openstack.org/512941 | 18:13 |
| *** catintheroof has quit IRC | 18:13 | |
| smcginnis | We have a couple of release-related project-config changes I'd like to see get through. | 18:14 |
| *** electrofelix has quit IRC | 18:14 | |
| smcginnis | https://review.openstack.org/515381 and https://review.openstack.org/515366 | 18:14 |
| smcginnis | When someone has the time... | 18:14 |
| SamYaple | clarkb: dang. i even updated the wrong section. that patch was for ceph, not docker | 18:14 |
| fungi | jeblair: the firewall rules look the same to me after a reboot too | 18:15 |
| *** andreas_s has joined #openstack-infra | 18:15 | |
| fungi | do we normally allow anything to them besides ping, ssh and cacti snmp? | 18:15 |
| jeblair | fungi: are we not enabling the pseudo-service or something? | 18:15 |
| *** jkilpatr has joined #openstack-infra | 18:16 | |
| *** jkilpatr has quit IRC | 18:16 | |
| jeblair | fungi: oh, for mergers, no | 18:16 |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Add stretch mirror for ceph https://review.openstack.org/513591 | 18:16 |
| smcginnis | Thanks jeblair | 18:16 |
| fungi | jeblair: oh, well then the firewall rules before the reboot looked as expected | 18:16 |
| jeblair | fungi: that should be correct. when you said minimal, i thought you meant 'open'. :) | 18:16 |
| fungi | no, no. the rules were just our minimal set | 18:16 |
| jeblair | fungi: thanks for testing reboot too! | 18:16 |
| fungi | indeed | 18:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Newton-EOL: Remove obsolete jobs and branches https://review.openstack.org/515492 | 18:16 |
| jeblair | cool, so it sounds like there isn't a systemic problem | 18:16 |
| *** trown|lunch is now known as trown | 18:16 | |
| fungi | jeblair: merger daemon started back up on its own... i suppose that's to be expected? | 18:16 |
| jeblair | fungi: yep | 18:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Newton-EOL: Remove obsolete jobs https://review.openstack.org/515493 | 18:17 |
| *** andreas_s has quit IRC | 18:20 | |
| *** yamahata has joined #openstack-infra | 18:22 | |
| *** andreas_s has joined #openstack-infra | 18:22 | |
| fungi | cool, i'll let the release team know they're open for business again | 18:23 |
| *** Sukhdev has joined #openstack-infra | 18:25 | |
| *** aviau has quit IRC | 18:25 | |
| *** aviau has joined #openstack-infra | 18:25 | |
| clarkb | fungi: pabelanger ianw I think tomorrow is the day I'm going to set aside for working on the various infra presentations at the summit. Probably going to start with fungi's onboarding presentty rst since thats mostly done then work on the infra update | 18:26 |
| *** slaweq has joined #openstack-infra | 18:26 | |
| clarkb | I think getting to the point where we have something then iterating in gerrit on infra publications is the way to go? | 18:26 |
| fungi | wfm | 18:27 |
| fungi | i'll be around most of tomorrow, though need to go do early voting at some point since i'll be out of the country on election day | 18:27 |
| clarkb | oh right I need to drop a ballot in the mail | 18:27 |
| fungi | you and your fancy state from the future | 18:27 |
| pabelanger | clarkb: wfm | 18:28 |
| clarkb | we use ancient technology to vote. Postal system, paper, ink | 18:28 |
| fungi | over here you still need to gouge your votes into clay tablets | 18:29 |
| clarkb | in washington postage is prepaid too iirc. But not in oregon | 18:29 |
| fungi | or at least show up in person anyway | 18:29 |
| *** slaweq has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Build tarballs for networking-{bagpipe,bgvpn} https://review.openstack.org/515381 | 18:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Do not publish docs when building fails https://review.openstack.org/515365 | 18:30 |
| AJaeger | pabelanger: ^ | 18:30 |
| AJaeger | is this good now? | 18:30 |
| *** andreas_s has quit IRC | 18:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch service projects to not publish to pypi https://review.openstack.org/515366 | 18:32 |
| *** yamamoto has joined #openstack-infra | 18:32 | |
| *** Sukhdev has quit IRC | 18:33 | |
| *** sree has joined #openstack-infra | 18:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Simplify queue names https://review.openstack.org/515496 | 18:37 |
| *** andreas_s has joined #openstack-infra | 18:38 | |
| *** yamamoto has quit IRC | 18:38 | |
| *** sree has quit IRC | 18:40 | |
| *** andreas_s has quit IRC | 18:42 | |
| *** dhinesh has joined #openstack-infra | 18:42 | |
| openstackgerrit | Merged openstack-infra/system-config master: Gather facts in set_hostname playbook https://review.openstack.org/515478 | 18:42 |
| AJaeger | infra-root, could one of you remove http://files.openstack.org/docs/publish-docs/ , please? That content is a publishing error that is fixed now | 18:43 |
| clarkb | AJaeger: the entier publish-docs directory needs to be deleted? | 18:44 |
| AJaeger | clarkb: yes | 18:44 |
| AJaeger | clarkb: missing / in rsync, so we published everything to publish-docs instead to / | 18:45 |
| AJaeger | the same content is just available one level higher up as well | 18:45 |
| clarkb | gotcha | 18:45 |
| clarkb | I'm about to grab lunch but can work to remove it afterwards | 18:45 |
| AJaeger | thanks, clarkb | 18:45 |
| *** jcoufal has quit IRC | 18:48 | |
| *** tosky has joined #openstack-infra | 18:49 | |
| *** catintheroof has joined #openstack-infra | 18:51 | |
| *** catintheroof has quit IRC | 18:56 | |
| dhinesh | i get the error, "SSHException: not a valid EC private key file", when trying to launch machines with nodepool | 18:58 |
| dhinesh | i do have the key pairs set on the user that i am running from | 18:59 |
| dhinesh | and also part of nodepool.yml, there is an entry "private-key: /home/nodepool/.ssh/id_rsa" | 19:00 |
| dhinesh | is there somewhere else i have to populate the keys | 19:00 |
| *** Goneri has quit IRC | 19:00 | |
| fungi | does that file have the usual begin and end markers in it? is it owned by the nodepool user? | 19:03 |
| fungi | and is the .ssh directory owned and readable by the nodepool user? | 19:05 |
| *** catintheroof has joined #openstack-infra | 19:05 | |
| *** slaweq has joined #openstack-infra | 19:06 | |
| dhinesh | fungi: https://pastebin.com/deQPcGAV this is the nodepool.yml i am using | 19:06 |
| fungi | by "the file" i meant your /home/nodepool/.ssh/id_rsa file | 19:07 |
| fungi | and please don't let us know the contents of /home/nodepool/.ssh/id_rsa just double check that it's properly formatted with begin/end markers in it | 19:08 |
| *** catintheroof has quit IRC | 19:09 | |
| fungi | you should see the first and last lines are something like -----BEGIN OPENSSH PRIVATE KEY----- and -----END OPENSSH PRIVATE KEY----- | 19:11 |
| dhinesh | fungi: yes that is -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- | 19:11 |
| dhinesh | should it be openssh instead? | 19:11 |
| fungi | it's probably fine | 19:11 |
| fungi | also when you generated the key you used an empty passphrase so it wouldn't be encrypted, right | 19:12 |
| dhinesh | yeah i did not add a keyphrase to it | 19:12 |
| *** cody-somerville has joined #openstack-infra | 19:12 | |
| *** cody-somerville has joined #openstack-infra | 19:12 | |
| pabelanger | trying using the ssh private key with ssh -v -i /home/nodepool/.ssh/id_rsa user@ip manually, see what SSH logs | 19:14 |
| fungi | right, openssh client tends to have a little better error reporting than the paramiko library | 19:15 |
| *** jcoufal has joined #openstack-infra | 19:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fuel branch cleanup https://review.openstack.org/515501 | 19:15 |
| *** sree has joined #openstack-infra | 19:17 | |
| *** HeOS has joined #openstack-infra | 19:20 | |
| dhinesh | fungi: pabelanger: these are the last four lines | 19:21 |
| dhinesh | debug1: Offering RSA public key: /home/nodepool/.ssh/id_rsa | 19:21 |
| dhinesh | debug1: Authentications that can continue: publickey | 19:21 |
| dhinesh | debug1: No more authentication methods to try. | 19:21 |
| dhinesh | Permission denied (publickey). | 19:21 |
| tobiash | dhinesh: does the node have the according public key in its authorized_keys file? | 19:22 |
| fungi | dhinesh: okay, that suggests that either the username or key nodepool is trying to ssh in with isn't valid for the image you booted | 19:22 |
| *** eharney has quit IRC | 19:23 | |
| pabelanger | agree | 19:24 |
| dhinesh | tobiash: yes i created the pair usinf ssh-keygen | 19:24 |
| pabelanger | should be able to look in dib build logs for your image to see which SSH public key was setup | 19:24 |
| pabelanger | or are you using glean to inject the keys? | 19:25 |
| AJaeger | config-core, could you review my newton-eol cleanups, please? https://review.openstack.org/#/q/status:open+topic:newton-eol I like to get those in quickly to avoid merge conflicts | 19:25 |
| *** sree has quit IRC | 19:25 | |
| *** hemna_ has quit IRC | 19:26 | |
| *** trown is now known as trown|brb | 19:26 | |
| dhinesh | fungi: pabelanger: ok let me check the logs, no i dont have glean setup yet | 19:27 |
| *** eumel8 has joined #openstack-infra | 19:27 | |
| *** catintheroof has joined #openstack-infra | 19:28 | |
| tobiash | dhinesh: if you have the public key in the image maybe the permissions of the authorized_keys file are too open | 19:29 |
| dhinesh | tobiash: but is authorized_keys file required in /home/nodepool/.ssh/ ? | 19:31 |
| tobiash | it's required in the home directory of the user you want connect to | 19:31 |
| *** gordc has joined #openstack-infra | 19:32 | |
| gordc | hi, had a question re:zuul. https://review.openstack.org/#/c/515327 remove job from .zuul.yaml but for some reason it still gets run in check? | 19:34 |
| gordc | is there a secondary place the jobs are? | 19:34 |
| *** yamamoto has joined #openstack-infra | 19:34 | |
| *** trown|brb is now known as trown | 19:35 | |
| pabelanger | gordc: which job is that? | 19:37 |
| gordc | pabelanger: the tox-py27-* jobs | 19:37 |
| pabelanger | gordc: check project-config, they might be defined there | 19:38 |
| AJaeger | pabelanger: looks like zuul v3 does not remove jobs, when they are deleted. | 19:38 |
| AJaeger | pabelanger: jd_ reported something similar earlier | 19:38 |
| AJaeger | pabelanger: those jobs are not in project-config | 19:38 |
| gordc | AJaeger: oh. it's probably the same thing thing. | 19:38 |
| pabelanger | AJaeger: you mean removal from .zuul.yaml? | 19:38 |
| pabelanger | I don't see why it wouldn't | 19:39 |
| *** yamamoto has quit IRC | 19:39 | |
| pabelanger | do you have .zuul.yaml on other branches? | 19:39 |
| gordc | pabelanger: all of them. | 19:39 |
| gordc | we did manage to remove jobs in another patch: https://review.openstack.org/#/c/512286/12/.zuul.yaml | 19:39 |
| pabelanger | sounds like it could be related to the work jeblair is working on with branch matchers | 19:40 |
| gordc | all of them (that isn't eol that is) | 19:40 |
| pabelanger | http://logs.openstack.org/27/515327/1/check/ceilometer-tox-py27-mongodb/d761ab3/zuul-info/inventory.yaml | 19:40 |
| pabelanger | job seems to be coming from stable/ocata | 19:40 |
| gordc | hmm.. i guess i'll have to wait for jeblair work | 19:41 |
| dhinesh | fungi: pabelanger: https://pastebin.com/mtxkMmJ0 these are the only entries on the logs present when i searched for "id_rsa" | 19:41 |
| gordc | there isn't a patch i could track is there? | 19:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use native v3 publishing for project-team-guide https://review.openstack.org/514053 | 19:42 |
| pabelanger | gordc: there is a ML post about the currently branch issues on openstack-infra, might be worth reading that. We also have zuulv3-issues etherpad, you can look and maybe add to | 19:42 |
| gordc | pabelanger: cool cool. thanks for the info | 19:42 |
| dhinesh | fungi: pabelanger: looks like it is not copying the private key? | 19:42 |
| * gordc going to try and remove branch references in branches to see if that works. | 19:43 | |
| pabelanger | dhinesh: what users did you use to ssh with? | 19:43 |
| AJaeger | gordc: jeblair said yesterday to expect something by Friday - can you wait another day or two? | 19:43 |
| gordc | AJaeger: yep :) | 19:44 |
| dhinesh | pabelanger: i tried nodepool and jenkins | 19:44 |
| pabelanger | dhinesh: try zuul user | 19:45 |
| pabelanger | that code path is http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user | 19:45 |
| dhinesh | pabelanger: cool that worked | 19:45 |
| pabelanger | \o/ | 19:46 |
| *** harlowja has quit IRC | 19:46 | |
| dhinesh | pabelanger: so does nodepool try logging in with a different user | 19:46 |
| pabelanger | dhinesh: depends on what you have setup in nodepool.yaml | 19:46 |
| dhinesh | pabelanger: yes got it | 19:46 |
| pabelanger | username: zuul | 19:47 |
| pabelanger | should be your fix now | 19:47 |
| *** ijw has quit IRC | 19:48 | |
| clarkb | looks like afs doc publishing is via a secret now. Is there an easy way for me to get the keytab file so that I can delete the dir AJaeger wants deleted? | 19:49 |
| clarkb | oh I suppose I can auth as my own user on one of the afs machines | 19:50 |
| clarkb | (no local afs for me) | 19:50 |
| pabelanger | yah | 19:50 |
| dhinesh | pabelanger: ok let me try it out | 19:53 |
| *** tpsilva has quit IRC | 19:53 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch kuryr back to publish-to-pypi https://review.openstack.org/515511 | 19:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Copy Gerrit delay to GitHub driver https://review.openstack.org/515224 | 19:56 |
| *** hasharAway has quit IRC | 19:56 | |
| *** ijw has joined #openstack-infra | 19:56 | |
| *** jcoufal has quit IRC | 19:56 | |
| smcginnis | Any issues going on that would be causing POST_FAILURES right now? | 19:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Set .root-marker variables for openstack-manuals https://review.openstack.org/515512 | 19:56 |
| clarkb | does 'aklog: a pioctl failed while obtaining tokens for cell openstack.org` mean I am on the wrong host for doing afs things? afs01.dfw.openstack.org fwiw | 19:58 |
| clarkb | smcginnis: not that I am aware of | 19:58 |
| AJaeger | smcginnis: where do you get one? | 19:59 |
| smcginnis | clarkb: OK, just had a bunch of release failures due to the releasenotes jobs all geting POST_FAILURE. | 19:59 |
| smcginnis | And this is a new error to me: http://logs.openstack.org/9d/9da719b17a84b93f1725cf358ec4a2ef1752e0bb/tag/publish-openstack-releasenotes/1666a37/job-output.txt.gz#_2017-10-26_19_57_41_748988 | 20:00 |
| clarkb | huh ya /afs/openstack.org doesn't exist there I am guessing that is it | 20:01 |
| *** vsaienk0 has joined #openstack-infra | 20:01 | |
| AJaeger | smcginnis: mmh, zuul.branch not set in the pipeline? | 20:01 |
| clarkb | AJaeger: that is the tag pipeline I'm not sure if it will be present there | 20:01 |
| AJaeger | need some zuul expert help on https://review.openstack.org/#/c/515512/1/zuul.d/jobs.yaml - why does this fail? | 20:02 |
| jeblair | yeah, there's no branch in the tag pipeline | 20:02 |
| AJaeger | so, how to fix http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/write-root-marker/tasks/main.yaml#n4 ? | 20:03 |
| jeblair | that's just for humans, let's stick a default filter on it | 20:03 |
| clarkb | I'm going to use ze01 for afs dir deletion | 20:03 |
| clarkb | since that is where things get afs anyways during job execution | 20:03 |
| jeblair | AJaeger: "{{ zuul.branch | default('') }}" should do it | 20:04 |
| AJaeger | jeblair: shall I fix it or are you on it already? | 20:04 |
| jeblair | or put anything else in there | 20:04 |
| jeblair | AJaeger: if you could that'd be great | 20:04 |
| AJaeger | on it... | 20:04 |
| clarkb | AJaeger: to double check I am deleting /afs/.openstack.org/docs/publish-docs and we don't need a backup copy or anything? | 20:05 |
| AJaeger | clarkb: correct, this is all one level above - only content published fro msecurity-doc, openstack-manuals and api-site. | 20:05 |
| clarkb | AJaeger: ok going to run rm now | 20:05 |
| AJaeger | jeblair: and the other variables are all set? zuul.build and zuul.ref? | 20:06 |
| AJaeger | clarkb: thanks | 20:06 |
| clarkb | ok that didn't work | 20:06 |
| *** Apoorva has quit IRC | 20:06 | |
| clarkb | oh right we have separate admin principals right? | 20:06 |
| AJaeger | clarkb: are you on the read-write volume? | 20:06 |
| clarkb | I don't use kerberos users often enough | 20:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Hande write-root-marker in tag pipeline https://review.openstack.org/515517 | 20:08 |
| *** eharney has joined #openstack-infra | 20:08 | |
| AJaeger | smcginnis, pabelanger, jeblair ^ | 20:08 |
| clarkb | that appears to have been it. delete in progress now | 20:09 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch octavia back to publish-to-pypi https://review.openstack.org/515518 | 20:09 |
| *** slaweq has quit IRC | 20:09 | |
| smcginnis | AJaeger: Thanks! | 20:09 |
| *** slaweq has joined #openstack-infra | 20:10 | |
| clarkb | AJaeger: ls: cannot access 'publish-docs': No such file or directory <- I'm inclined to let the next publish job do the afs publish if only to avoid racing on that (not sure if we can do concurrent publishes | 20:10 |
| clarkb | AJaeger: so it may be a bit before that reflects on the RO volume but it is done on RW now | 20:10 |
| AJaeger | clarkb: http://files.openstack.org/docs/publish-docs/ is gone, so I'm happy ;) | 20:10 |
| jeblair | happens every 5m if the volume is updated | 20:11 |
| clarkb | perfect | 20:11 |
| jeblair | clarkb: so you probably sucessfully waited for the next automatic publish | 20:11 |
| AJaeger | jeblair: yep - so, perfect timing ;) | 20:11 |
| *** vsaienk0 has quit IRC | 20:11 | |
| clarkb | and btw it was I kinit'd the non admin account first time around | 20:11 |
| clarkb | once that was sorted everything worked great | 20:11 |
| jeblair | yay perms are correct | 20:11 |
| AJaeger | jeblair: could you tell me why zuul encounters a syntax error on https://review.openstack.org/#/c/515512/1/zuul.d/jobs.yaml , please? | 20:12 |
| clarkb | I should make my local fileserver an afs client since it runs ubuntu | 20:12 |
| jeblair | AJaeger: you need quotes around {{}} | 20:13 |
| AJaeger | Ah! thanks | 20:13 |
| jeblair | we should probably intercept that error and handle it specially | 20:14 |
| *** slaweq has quit IRC | 20:14 | |
| clarkb | oh I just approved the change | 20:14 |
| *** hamzy has quit IRC | 20:14 | |
| clarkb | or are we talking about a different change | 20:14 |
| clarkb | I approved 515517 | 20:14 |
| jeblair | clarkb: different: 515512 | 20:14 |
| clarkb | 2 and 7 too similar at first glance | 20:14 |
| jeblair | because the yaml parser is confused (it thinks it's constructing yet another mapping started by the {{ characters ) | 20:15 |
| jeblair | that's why the message ("unhashable key") doesn't make sense | 20:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Set .root-marker variables for openstack-manuals https://review.openstack.org/515512 | 20:15 |
| AJaeger | jeblair: I see, thanks. Trying again ^ | 20:15 |
| jeblair | clarkb: that's why new versions of pep8 don't allow the use of those numbers | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| AJaeger | jeblair: that's an upcoming pep8 ;) | 20:16 |
| jeblair | big problem on those clocks too, where the 7 is an upside down 2 | 20:16 |
| AJaeger | Our publishing deleted by accident the docs.o.o index page due to uploading docs after build that failed. Could I get a second +2 on https://review.openstack.org/#/c/515365 , please? This way we only upload after successfull builds | 20:17 |
| jeblair | AJaeger: done | 20:19 |
| AJaeger | thanks, jeblair | 20:19 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Update testenv-worker for newer novaclient https://review.openstack.org/515519 | 20:19 |
| clarkb | fungi: it looks like zuul.conf may have updated 2 minutes after zuul was restarted? I'm not seeing the cgit urls apply properly, current guess is that the restart happened just a little too early | 20:22 |
| ianw | pabelanger / fungi : yeah, doh ... i was using my branch of launch-node.py from https://review.openstack.org/#/c/440928/ (which i actually find quite useful but not sure anyone else agrees) | 20:22 |
| clarkb | fungi: do you know if you checked that puppet had run and update the conf before it was restarted? want to make sure I don't blame that too early | 20:23 |
| clarkb | eg do I need to keep debugging or just wait for next restart | 20:23 |
| *** sree has joined #openstack-infra | 20:24 | |
| *** smatzek has quit IRC | 20:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add provider defaults to launch-node.py https://review.openstack.org/440928 | 20:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Do not publish docs when building fails https://review.openstack.org/515365 | 20:26 |
| *** armax has quit IRC | 20:26 | |
| *** camunoz has quit IRC | 20:26 | |
| *** anupn has quit IRC | 20:26 | |
| *** anupn has joined #openstack-infra | 20:27 | |
| *** smatzek has joined #openstack-infra | 20:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint https://review.openstack.org/515520 | 20:28 |
| *** sree has quit IRC | 20:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint https://review.openstack.org/515520 | 20:29 |
| *** jamielennox has quit IRC | 20:30 | |
| *** rbrndt_ has quit IRC | 20:30 | |
| *** smatzek has quit IRC | 20:32 | |
| *** harlowja has joined #openstack-infra | 20:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint https://review.openstack.org/515520 | 20:33 |
| fungi | clarkb: i probably forgot to make sure it had successfully applied, sorry :/ | 20:33 |
| clarkb | fungi: no problem I am sure there will be another restart :) | 20:33 |
| clarkb | just wanted to make sure I didn't need to debug more | 20:34 |
| AJaeger | pabelanger: testing ansible-lint in 515520... | 20:34 |
| *** burgerk has quit IRC | 20:34 | |
| * AJaeger waves good night | 20:34 | |
| *** jamielennox has joined #openstack-infra | 20:34 | |
| *** Apoorva has joined #openstack-infra | 20:35 | |
| fungi | clarkb: yeah, let's just assume it's waiting for the next restart for now | 20:35 |
| *** yamamoto has joined #openstack-infra | 20:35 | |
| jeblair | which will hopefully be tomorrow with the changes i'm writing | 20:35 |
| dmsimard | ok, finished extinguishing various fires.. | 20:35 |
| dmsimard | looking at elastic recheck patch now | 20:36 |
| jeblair | hi, if folks wanted to start looking at the inheritance changes, i think they're ready for review. i'm finishing up the implied-run "bonus" change now. but the others (514459, 511352, 515249, 515483) are ready. | 20:36 |
| clarkb | jeblair: AJaeger does https://review.openstack.org/#/c/515512/2/zuul.d/jobs.yaml need the same branch deafult fix that 515517 got? | 20:37 |
| clarkb | basically default to empty string if no value is present | 20:37 |
| *** ijw has quit IRC | 20:37 | |
| jeblair | good question. if it gets used in tag, yes it will. | 20:37 |
| AJaeger | clarkb: it only runs in post, so shouldn't. | 20:37 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images https://review.openstack.org/514668 | 20:38 |
| clarkb | AJaeger: thanks | 20:38 |
| clarkb | I guess we don't tag our manuals | 20:38 |
| *** ijw has joined #openstack-infra | 20:38 | |
| AJaeger | we could make it robust in case it gets used there at one day - right now we don#t tag... | 20:38 |
| AJaeger | bye for today | 20:39 |
| *** yamamoto has quit IRC | 20:40 | |
| clarkb | good night | 20:41 |
| *** ijw has quit IRC | 20:44 | |
| fungi | ianw: tonyb: not sure whether you're around yet, but you're probably best equipped to address http://lists.openstack.org/pipermail/openstack-dev/2017-October/124030.html | 20:46 |
| *** andreas_s has joined #openstack-infra | 20:48 | |
| *** catintheroof has quit IRC | 20:49 | |
| ianw | tonyb: hmm, i imagine just putting back eol tags ... ping me and we can do that | 20:50 |
| *** thorst has quit IRC | 20:50 | |
| *** jcoufal has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set .root-marker variables for openstack-manuals https://review.openstack.org/515512 | 20:53 |
| *** jamielennox has quit IRC | 20:54 | |
| clarkb | ianw: ya the tag is there could just branch off of that | 20:55 |
| clarkb | and point back at hte mailing list I guess | 20:55 |
| *** jamielennox has joined #openstack-infra | 20:55 | |
| *** smatzek has joined #openstack-infra | 20:55 | |
| fungi | the only catch is that when it does finally come time to eol that branch later, you need a slightly different tag | 20:56 |
| fungi | 9.0-eol-no-really-we-mean-it-this-time | 20:56 |
| pabelanger | ha | 20:56 |
| clarkb | eolv2 | 20:57 |
| jeblair | i guess we need a stricter explicit sign-off process | 20:58 |
| *** andreas_s has quit IRC | 20:58 | |
| *** jcoufal has quit IRC | 20:59 | |
| *** andreas_s has joined #openstack-infra | 20:59 | |
| *** smatzek has quit IRC | 21:01 | |
| openstackgerrit | Merged openstack/os-client-config master: Consume the osc-functional-devstack-tips job https://review.openstack.org/514847 | 21:01 |
| *** HeOS has quit IRC | 21:02 | |
| tonyb | andreykurilin: So sorry. We can lean on infra to correct that | 21:03 |
| *** HeOS has joined #openstack-infra | 21:03 | |
| *** mat128 has quit IRC | 21:03 | |
| dmsimard | Just curious, I remember a discussion at the PTG around moving the hardware from infracloud elsewhere (HPE datacenter to an university or something like that?), did that ever happen ? | 21:03 |
| tonyb | andreykurilin: My next question is what could I have reasonably done to ensure your saw and responed to my email? | 21:03 |
| andreykurilin | tonyb: nothing is lost, so np :) | 21:04 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Hande write-root-marker in tag pipeline https://review.openstack.org/515517 | 21:04 |
| andreykurilin | tonyb: there were too many lines with listed branches, so I even didn't read them. Mention all affected project can fix "attention" | 21:05 |
| clarkb | fwiw there is nothing on pypi related to that eol | 21:05 |
| clarkb | we can recreate the stable branch based on the tag | 21:06 |
| *** trown is now known as trown|outtypewww | 21:06 | |
| tonyb | andreykurilin: ok. | 21:06 |
| tonyb | andreykurilin: you could have searched for rally ... that would have gottent passed the too many line issue | 21:06 |
| jeblair | "[rally]" did appear to be missing from the subject line, so it's understandable | 21:07 |
| clarkb | https://pypi.python.org/simple/rally/ for pypi listing | 21:07 |
| andreykurilin | clarkb: cool. Also, I pushed the branch to my own public repo if needed | 21:07 |
| pabelanger | dmsimard: http://lists.openstack.org/pipermail/openstack-infra/2017-September/005585.html | 21:07 |
| tonyb | jeblair: It's true I didn't call it out specifically. I'll just make the subject longer next time | 21:07 |
| jeblair | tonyb: heh, yeah, all or none would probably be best :) | 21:08 |
| pabelanger | dmsimard: next step is to find hardware | 21:08 |
| tonyb | clarkb: I shoudl be able to crate a branch as a Release manager, shouldn't I? | 21:08 |
| clarkb | tonyb: maybe? I'll look | 21:08 |
| clarkb | it looks like rally-core can create branches themslves | 21:09 |
| clarkb | rally-core has create reference on refs/heads | 21:09 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/rally,access | 21:09 |
| *** andreas_s has quit IRC | 21:10 | |
| clarkb | release managers has this perm too so either group | 21:10 |
| *** mummer has quit IRC | 21:10 | |
| andreykurilin | tonyb: people usually lazy and if nothing attracts attention, no action will be performed. The topic attracted me, but I just saw tags for several projects and it led me ignoring global tag [all] | 21:10 |
| tonyb | andreykurilin: again I'm sorry. Do you want me to (re-)create the branch for you or will you do it? | 21:11 |
| *** pblaho has joined #openstack-infra | 21:11 | |
| andreykurilin | tonyb: how can I do it? | 21:11 |
| tonyb | andreykurilin: okay, next time I'll include all affected project teams. | 21:11 |
| andreykurilin | Can I just push the branch via gerrit? Do I have enough rights for this? | 21:12 |
| clarkb | no you can't push them | 21:12 |
| clarkb | you have to create them via the gerrit ui | 21:12 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images https://review.openstack.org/514668 | 21:12 |
| jeblair | and use "0.9-eol" for the value | 21:13 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/rally,branches is where you do it | 21:14 |
| *** lindycoder has quit IRC | 21:14 | |
| clarkb | branch name field should be stable/0.9 I think and then initial revision is 0.9-eol | 21:14 |
| *** anupn has quit IRC | 21:15 | |
| andreykurilin | it worked | 21:15 |
| andreykurilin | thanks | 21:15 |
| tonyb | \o/ | 21:15 |
| tonyb | andreykurilin: What is the exptect EOL for that branch? | 21:16 |
| *** anupn has joined #openstack-infra | 21:16 | |
| tonyb | andreykurilin: FWIW I now have the branch here | 21:17 |
| andreykurilin | tonyb: no special date. we just released a new major version(0.10.0) and need to support both for awhile, since 0.10.0 contains a lot of refactoring... | 21:18 |
| andreykurilin | Actually, I hope that I'll not any more 0.9.* releases and currect state of the branch will be the last | 21:18 |
| tonyb | andreykurilin: okay are you thinking 6months 12months ? | 21:18 |
| tonyb | andreykurilin: or will it approximately match the queens-eol date? | 21:19 |
| andreykurilin | ideally, < 6 month | 21:19 |
| *** andreas_s has joined #openstack-infra | 21:19 | |
| tonyb | Ahh okay, well when you're ready let me know and I'll include it in the closest batch if that works for you | 21:19 |
| tonyb | this was a one off clean all the "cruft" out so from now on it shoudl be more targetted | 21:20 |
| andreykurilin | tonyb: our branches doesn't correlate with openstack releases. the latest rally has several jobs with mitaka openstack and it still work. even with latest openstack clients:) | 21:20 |
| andreykurilin | tonyb: I'll ping you as soon as we decide that 0.9 is not required any more. as I mentioned, hope it will be soon | 21:22 |
| *** armax has joined #openstack-infra | 21:22 | |
| andreykurilin | tonyb: and what about 0.9-oel tag. does it possible to remove it? | 21:22 |
| tonyb | andreykurilin: Cool. | 21:22 |
| *** rlandy is now known as rlandy|brb | 21:23 | |
| clarkb | typically we don't delete tags because downstreams wno't pick up a deleted tag | 21:25 |
| clarkb | I'd be inclined to leave it particularly since chances are good we don't actually change that branch between now and actual eol | 21:25 |
| clarkb | in which case we acn just delete the branch and be done | 21:26 |
| *** aeng has joined #openstack-infra | 21:27 | |
| *** andreas_s has quit IRC | 21:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Newton-EOL: Remove obsolete jobs and branches https://review.openstack.org/515492 | 21:28 |
| *** dprince has quit IRC | 21:30 | |
| *** andreas_s has joined #openstack-infra | 21:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Simplify queue names https://review.openstack.org/515496 | 21:31 |
| cloudnull | qq - https://docs.openstack.org/pike/deploy/ that shows links to the kolla and osa deployment guides however it just links on-to itself, is there something we should do to fix that ? | 21:32 |
| *** CrayZee has quit IRC | 21:32 | |
| smcginnis | New POST_FAILURES on releases: http://logs.openstack.org/2f/2f8ed39c4e19fc0afd22525fa36f9de3cc02ca38/tag/publish-openstack-releasenotes/3fe77fb/job-output.txt.gz#_2017-10-26_21_29_41_778840 | 21:33 |
| smcginnis | Doesn't like the conditional publishing that just merged? | 21:33 |
| jeblair | smcginnis: ya | 21:34 |
| *** andreas_s has quit IRC | 21:34 | |
| jeblair | oh, missing a : | 21:34 |
| jeblair | pabelanger, jlk: how's the ansible-lint project coming? :) | 21:34 |
| jeblair | i'll work on a fix | 21:35 |
| *** CrayZee has joined #openstack-infra | 21:36 | |
| pabelanger | jeblair: needs some reviews: https://review.openstack.org/512715/ | 21:36 |
| smcginnis | jeblair: Thanks | 21:36 |
| smcginnis | linting sounds useful there. ;) | 21:36 |
| *** yamamoto has joined #openstack-infra | 21:37 | |
| pabelanger | smcginnis: that is my mistake, I reviewed that change from AJaeger | 21:37 |
| jeblair | pabelanger: is there an effort to create a generic ansible linter any repo can use? | 21:37 |
| jeblair | ansible linter job, that is | 21:38 |
| ianw | clarkb: https://review.openstack.org/#/c/515347 :/ we all missed that (ENABLE_KSM not using $). i guess it wasn't working | 21:38 |
| smcginnis | pabelanger: I looked at it too, but that wasn't obvious. | 21:38 |
| clarkb | ianw: oh good they pushed a patch up | 21:38 |
| clarkb | I got a personal email asking about it ans I suggested the patch it :) | 21:39 |
| cloudnull | also looks like our docs to the draft deploy guide are being redirected https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/ :( | 21:39 |
| pabelanger | jeblair: yah, we could document how a project could update tox.ini today, but I think mordred see us creating ansible-linters playbook job that any project could run, without tox.ini changes | 21:39 |
| *** askb has joined #openstack-infra | 21:39 | |
| jeblair | pabelanger: yes, is there a reason we need tox? | 21:40 |
| pabelanger | jeblair: no, it was an easy way to manage pip dependencies mostly | 21:40 |
| jeblair | pabelanger: what are the pip depes we need? | 21:40 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Newton removal: Remove periodic-newton https://review.openstack.org/514911 | 21:41 |
| pabelanger | jeblair: just ansible-lint | 21:41 |
| clarkb | cloudnull: I want to say all those redirects are controlled by htpasswd files in the projects themselves | 21:41 |
| pabelanger | with zuulv3, would be a simple playbook | 21:41 |
| clarkb | cloudnull: so I would look in the project deploy guide and kolla | 21:41 |
| jeblair | pabelanger: ok. i +2d that change because if we land it, it will save us so much time. but yeah, i think we should replace it with a job that doesn't use tox, just installs ansible-lint, and adds all required-projects to the ansible-roles-path automatically. | 21:41 |
| jeblair | pabelanger: then we could use the same job for any single repo or sets of repos | 21:42 |
| jeblair | clarkb: can you +3 512715? | 21:42 |
| *** baoli has quit IRC | 21:42 | |
| * clarkb looks | 21:42 | |
| dhinesh | am getting an error "HTTPError: HTTP Error 405: Method Not Allowed" while running nodepool, is there a way to see logs of HTTP requests given by nodepool? | 21:42 |
| *** yamamoto has quit IRC | 21:42 | |
| dhinesh | i was not able to find it in the debug logs | 21:43 |
| *** baoli has joined #openstack-infra | 21:43 | |
| *** baoli has quit IRC | 21:43 | |
| cloudnull | clarkb: it doesn't look like there's any htaccess file in https://github.com/openstack/kolla-ansible/tree/master/deploy-guide/source or https://github.com/openstack/openstack-ansible/tree/master/deploy-guide/source somewhere else I might need to look? | 21:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Fix missing "role:" mapping keys https://review.openstack.org/515529 | 21:46 |
| jeblair | clarkb, smcginnis, pabelanger: ^ | 21:46 |
| clarkb | cloudnull: https://github.com/openstack/kolla-ansible/tree/stable/pike is where I would look because https://docs.openstack.org/project-deploy-guide/kolla-ansible/pike/ is what is being redirected | 21:47 |
| dmsimard | fungi: I hear you had estimated the costs for shipping the infracloud hardware to OSU OSL ? | 21:48 |
| clarkb | cloudnull: but I don't see one there, could it be project-deploy-guide? | 21:48 |
| clarkb | cloudnull: dhellmann and AJaeger would probably undersatnd that best though | 21:48 |
| *** andreas_s has joined #openstack-infra | 21:48 | |
| clarkb | dhinesh: where are you seeing that error? | 21:48 |
| smcginnis | Hey, I got to tour the OSU OSL once. | 21:48 |
| cloudnull | thanks clarkb! I'll keep poking around. | 21:49 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/system-config master: Add #openstack-women to meetbot channels https://review.openstack.org/515531 | 21:50 |
| *** rlandy|brb is now known as rlandy | 21:51 | |
| dmsimard | smcginnis: nice place? :) | 21:52 |
| smcginnis | dmsimard: Yeah, it was pretty cool. | 21:53 |
| clarkb | dhinesh: as for seeing the logs of http requests you can edit your logging config to log python requests as a debug level to get everything. Be warned this may include raw passwords/tokens/etc | 21:53 |
| dhinesh | clarkb: its part of the debug.log, as part of nodepool, its able to ssh into the machine, and then it throws this error. | 21:54 |
| clarkb | dhinesh: do you have a full traceback you can share? | 21:55 |
| *** jascott1 has joined #openstack-infra | 21:55 | |
| dhinesh | clarkb: yes i do https://pastebin.com/Q23ZV3pF | 21:57 |
| jeblair | clarkb, pabelanger: can you review https://review.openstack.org/515529 ? it's urgent for release | 21:58 |
| *** andreas_s has quit IRC | 21:58 | |
| *** gouthamr has quit IRC | 21:59 | |
| *** marst has quit IRC | 22:00 | |
| *** thorst has joined #openstack-infra | 22:00 | |
| clarkb | dhinesh: iirc jenkins updated their API and changed some things from get to post | 22:01 |
| *** rbrndt_ has joined #openstack-infra | 22:01 | |
| *** jascott1 has quit IRC | 22:01 | |
| clarkb | dhinesh: I thought that latest nodepool master switched most jenkins api calls to python-jenkins which added support for this? | 22:01 |
| clarkb | bkero: ^ I seem to recall you were involved in that? do you know? | 22:01 |
| dmsimard | jeblair: looking | 22:01 |
| dmsimard | jeblair: our gate jobs should have caught that :( | 22:02 |
| clarkb | approved | 22:02 |
| jeblair | dmsimard: we just approved the ansible lint job that would have | 22:02 |
| dmsimard | ack | 22:02 |
| clarkb | dhinesh: what version of nodepool and what version of python-jenkins do you have? | 22:03 |
| bkero | clarkb: that was a while ago. I think it uses it, but not for everything | 22:03 |
| bkero | IIRC there are some features of python-jenkins that are duplicated by nodepool, although there was a big commit to get rid of a lot of that | 22:03 |
| hongbin | hi infra team, could anyone take a look at: https://review.openstack.org/#/c/515511/ ? we needs it to release the library | 22:03 |
| clarkb | bkero: ya pretty sure that big commit merged | 22:03 |
| clarkb | https://review.openstack.org/#/c/259157/ that one | 22:04 |
| *** andreas_s has joined #openstack-infra | 22:04 | |
| clarkb | whcih based on dhinesh's traceback isn't the code being used | 22:04 |
| clarkb | dhinesh: so I think you need to update | 22:04 |
| dhinesh | my nodepool version is 0.3.1 | 22:05 |
| *** thorst has quit IRC | 22:06 | |
| dhinesh | clarkb: how would you get the version of python-jenkins? | 22:06 |
| clarkb | dhinesh: pip freeze | 22:06 |
| dhinesh | clarkb: python-jenkins==0.4.15 | 22:07 |
| *** boden has quit IRC | 22:07 | |
| *** esberglu has quit IRC | 22:07 | |
| clarkb | but I think you need to update nodepool | 22:08 |
| clarkb | since myjenkins is the cause of the problem | 22:09 |
| *** MasterPiece has joined #openstack-infra | 22:09 | |
| *** andreas_s has quit IRC | 22:09 | |
| ianw | clarkb: https://review.openstack.org/#/c/515463/1/playbooks/legacy/its-storyboard-gerrit-plugin-stable-2.13/post.yaml -- does the recurse actually do anything? | 22:10 |
| *** smatzek has joined #openstack-infra | 22:10 | |
| clarkb | ianw dmsimard said it was necessary | 22:10 |
| clarkb | I think to apply the same atrirbutes the whole way? | 22:11 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create openstack-zuul-jobs-linters jobs https://review.openstack.org/512715 | 22:11 |
| dmsimard | clarkb: iirc you wanted to use recurse to create a nested directory tree | 22:11 |
| ianw | dmsimard: that's not really my reading of http://docs.ansible.com/ansible/latest/file_module.html, but, on occasion, i have been known to be wrong about things :) | 22:12 |
| *** spzala has quit IRC | 22:14 | |
| *** spzala has joined #openstack-infra | 22:15 | |
| *** andreas_s has joined #openstack-infra | 22:15 | |
| dmsimard | ianw: http://paste.openstack.org/raw/624753/ | 22:15 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Switch kuryr back to publish-to-pypi https://review.openstack.org/515511 | 22:16 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Switch octavia back to publish-to-pypi https://review.openstack.org/515518 | 22:16 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Remove migrated octavia zuul v3 jobs https://review.openstack.org/515486 | 22:16 |
| dmsimard | ianw: you can send a PR to update the docs :) https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/file.py#L16 | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3 https://review.openstack.org/515249 | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add pragma directive https://review.openstack.org/515483 | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove implied run https://review.openstack.org/515487 | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support file extension in playbook path https://review.openstack.org/515535 | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update test fixtures to use explicit run https://review.openstack.org/515536 | 22:18 |
| ianw | dmsimard: yeah, but try it without "recurse" and it still works? | 22:18 |
| *** smatzek has quit IRC | 22:18 | |
| ianw | ~/tmp/ansible/bin/ansible localhost -m file -a "path=/tmp/some/long/path/with/dirs state=directory" | 22:18 |
| ianw | did for me, anyway | 22:18 |
| dmsimard | ianw: huh, you're right actually o_O | 22:18 |
| dmsimard | TIL | 22:18 |
| *** gildub has joined #openstack-infra | 22:19 | |
| dmsimard | Either I wrongly assumed recurse was required or maybe that changed at some point and I kept using it (which wouldn't surprise me) | 22:19 |
| *** spzala has quit IRC | 22:19 | |
| dmsimard | clarkb: I was wrong! | 22:20 |
| *** andreas_s has quit IRC | 22:20 | |
| *** andreas_s has joined #openstack-infra | 22:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix missing "role:" mapping keys https://review.openstack.org/515529 | 22:23 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create dir needed to publish gerrit plugins https://review.openstack.org/515463 | 22:23 |
| *** felipemonteiro_ has quit IRC | 22:24 | |
| dhinesh | clarkb: i upgraded nodepool using http://git.openstack.org/cgit/openstack-infra/nodepool/tree/README.rst | 22:27 |
| dhinesh | then tried to start nodepool service | 22:27 |
| dhinesh | got this ImportError: No module named nodepool.cmd.nodepoold | 22:28 |
| *** xarses has quit IRC | 22:28 | |
| *** bobh has quit IRC | 22:30 | |
| *** spzala has joined #openstack-infra | 22:31 | |
| *** bobh has joined #openstack-infra | 22:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Validate that a job has a run playbook on freeze https://review.openstack.org/515537 | 22:31 |
| ianw | jeblair / clarkb: https://review.openstack.org/#/c/513595/ -- moving swap/opt role into o-z-j ... any opinions here? this is what we agreed to do in the meeting a week or so ago. i'd like to call it from dib jobs to make sure /opt is nice and big | 22:31 |
| jeblair | ianw: did we discuss maybe putting it in plain-old zuul-jobs? | 22:32 |
| *** xarses has joined #openstack-infra | 22:32 | |
| jeblair | oh i see comments... | 22:33 |
| *** xarses has quit IRC | 22:33 | |
| *** rlandy has quit IRC | 22:33 | |
| jeblair | +2 | 22:33 |
| *** gordc has quit IRC | 22:34 | |
| *** xarses has joined #openstack-infra | 22:35 | |
| *** bobh has quit IRC | 22:36 | |
| *** xarses has quit IRC | 22:36 | |
| *** xarses has joined #openstack-infra | 22:38 | |
| *** yamamoto has joined #openstack-infra | 22:39 | |
| *** leyal has quit IRC | 22:39 | |
| *** lihi has quit IRC | 22:39 | |
| clarkb | dhinesh: doe that have a traceback too? (it is much easier to debug these things with more context) | 22:41 |
| *** leyal has joined #openstack-infra | 22:41 | |
| *** lihi has joined #openstack-infra | 22:41 | |
| ianw | jeblair: yeah, the only "guarantee" i'd give is that it does something useful on hosts inside clouds that infra uses ... to be z-j i'd think that would "does something useful on every cloud" | 22:42 |
| jeblair | ianw: maybe we can get there, but we don't have to do that today | 22:42 |
| clarkb | ianw: jeblair configure-swap is the same name in devstack and in ozj, will that cause lookup problems? | 22:43 |
| clarkb | roles can be shared | 22:43 |
| *** apetrich has quit IRC | 22:43 | |
| jeblair | clarkb: there's a lookup order. in this case it doesn't matter which wins, they're the same. we'll remove the devstack one when this lands. | 22:43 |
| ianw | the devstack one should overwite until we remove it | 22:44 |
| ianw | oh, what jeblair said :) | 22:44 |
| *** yamamoto has quit IRC | 22:44 | |
| clarkb | ok approved | 22:44 |
| dhinesh | sure, https://pastebin.com/mdF4W81m | 22:45 |
| jeblair | (roles are prepended as you go down the inheritance hierarchy, so if i child job specifies a role repo, it will take precedence over its parent. the roles form part of the playbook context though, so a child can't override the set of roles available to a parent playbook) | 22:46 |
| *** apetrich has joined #openstack-infra | 22:46 | |
| clarkb | dhinesh: that module is certainly there in the git repo | 22:46 |
| *** sshnaidm is now known as sshnaidm|off | 22:46 | |
| clarkb | dhinesh: does pip freeze show it is installed? is the version different? | 22:47 |
| ianw | jeblair / clarkb: anyway, if you'd like https://review.openstack.org/#/c/513597/ removes it from devstack too :) | 22:47 |
| *** andreas_s has quit IRC | 22:48 | |
| *** claudiub|2 has joined #openstack-infra | 22:48 | |
| *** rbrndt_ has quit IRC | 22:49 | |
| *** andreas_s has joined #openstack-infra | 22:49 | |
| *** jaypipes has quit IRC | 22:51 | |
| dhinesh | clarkb: when i do pip freeze this shows up "-e git://git.openstack.org/openstack-infra/nodepool@3bd21837001c77873b7cf4827de6d898fed567e1#egg=nodepool" | 22:52 |
| dhinesh | but nodepool --version shows 0.4.1 | 22:52 |
| dhinesh | looks like it did not install properly | 22:53 |
| clarkb | those two versions look right actually. 3bd2183 is tip of master and 0.4.0 was the last tag (so we are 0.4.1 for the actual version) | 22:53 |
| *** salv-orl_ has quit IRC | 22:53 | |
| clarkb | was this installed in a virtualenv or to the system? | 22:54 |
| *** andreas_s has quit IRC | 22:54 | |
| *** salv-orlando has joined #openstack-infra | 22:54 | |
| dhinesh | clarkb: it was installed to the system, i used puppet to initiate it the first time i installed it, maybe it has some dependencies on it | 22:58 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add configure-swap role https://review.openstack.org/513595 | 22:58 |
| *** salv-orlando has quit IRC | 22:59 | |
| clarkb | dhinesh: if you run python and then import nodepool.cmd.nodepoold does that work? | 22:59 |
| clarkb | maybe it installed to a different python? | 22:59 |
| *** andreas_s has joined #openstack-infra | 23:00 | |
| dhinesh | clarkb: yeah that works fine, python version is 2.7 | 23:01 |
| dmsimard | clarkb, frickler: for https://review.openstack.org/#/c/509313/3/elastic_recheck/config.py -- in v2 it's not necessary to pick up the failure messages because we can just use build_status:"FAILURE" | 23:01 |
| dmsimard | it's shorter and more reliable IMO | 23:01 |
| clarkb | dmsimard: I think it is using the console log output to make sure we've actually indexed it properly | 23:02 |
| *** xarses has quit IRC | 23:02 | |
| clarkb | but ya I think in practice using the build status there is probably fine | 23:03 |
| dmsimard | clarkb: that's ok, we're matching [Zuul] Job complete | 23:03 |
| dmsimard | clarkb: I'll reply to the comments and send another patchset | 23:03 |
| clarkb | ya that should work | 23:03 |
| clarkb | I still think you want to group teh v2 query bits together and OR them with the v3 query bits | 23:03 |
| dmsimard | yeah I got that, that's fine | 23:04 |
| dhinesh | clarkb: i tried upgrading 0.4.1 using puppet, it failed, but when i tried 0.4.0, it works now | 23:04 |
| *** xarses has joined #openstack-infra | 23:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update test fixtures to use explicit run https://review.openstack.org/515536 | 23:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove implied run https://review.openstack.org/515487 | 23:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Validate that a job has a run playbook on freeze https://review.openstack.org/515537 | 23:05 |
| dhinesh | clarkb: https://pastebin.com/dVXAH3ia logs on puppet | 23:06 |
| clarkb | dhinesh: yes 0.4.1 isn't tagged yet | 23:06 |
| *** mriedem is now known as mriedem_away | 23:07 | |
| clarkb | but since we haev 0.4.0 tagged any commits beyond it have to be at least 0.4.1 so that is what the version shows as when installing from the branch tip | 23:07 |
| clarkb | it doesn't look like 0.4.0 has the myjenkins fix though | 23:07 |
| dmsimard | clarkb: oh, er, frickler has a point in the other file comments.. there's also the *jenkins* version of the v2 to match | 23:07 |
| dmsimard | [Zuul] Job complete and the jenkins equivalent | 23:07 |
| * dmsimard pouts | 23:07 | |
| *** xarses has quit IRC | 23:07 | |
| clarkb | dmsimard: ya so its (filename:"console.html" AND (message:"foo" OR message:"bar")) | 23:08 |
| *** tosky has quit IRC | 23:11 | |
| *** andreas_s has quit IRC | 23:11 | |
| *** andreas_s has joined #openstack-infra | 23:13 | |
| *** wolverineav has quit IRC | 23:14 | |
| *** wolverineav has joined #openstack-infra | 23:15 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add support for Zuulv3-specific parameters in elastic-recheck https://review.openstack.org/509313 | 23:19 |
| *** pblaho has quit IRC | 23:19 | |
| *** wolverineav has quit IRC | 23:19 | |
| dmsimard | clarkb: ^ I middle fingered pep8 because the darn thing was not readable, I'm sure the legibility improvement is worth it :) | 23:20 |
| *** wolverineav has joined #openstack-infra | 23:20 | |
| clarkb | ya ship it | 23:21 |
| clarkb | I just +2'd | 23:21 |
| dmsimard | mtreinish: do you think you can take a look at https://review.openstack.org/#/c/509313/ ? | 23:21 |
| *** wolverineav has quit IRC | 23:24 | |
| *** andreas_s has quit IRC | 23:27 | |
| *** andreas_s has joined #openstack-infra | 23:28 | |
| *** claudiub|2 has quit IRC | 23:28 | |
| *** jascott1 has joined #openstack-infra | 23:28 | |
| *** jamesdenton has quit IRC | 23:28 | |
| *** thorst has joined #openstack-infra | 23:29 | |
| *** lindycoder has joined #openstack-infra | 23:30 | |
| *** jamesdenton has joined #openstack-infra | 23:31 | |
| *** sdague has quit IRC | 23:33 | |
| *** andreas_s has quit IRC | 23:33 | |
| *** andreas_s has joined #openstack-infra | 23:35 | |
| *** dingyichen has joined #openstack-infra | 23:36 | |
| *** yamamoto has joined #openstack-infra | 23:40 | |
| *** thorst has quit IRC | 23:43 | |
| *** andreas_s has quit IRC | 23:44 | |
| *** yamamoto has quit IRC | 23:46 | |
| *** links has joined #openstack-infra | 23:46 | |
| *** andreas_s has joined #openstack-infra | 23:47 | |
| jeblair | remote: https://review.openstack.org/515552 Zuul: add run attribute | 23:47 |
| jeblair | clarkb, ianw: ^ that might be good to get moving soon | 23:48 |
| jeblair | (i'm scanning all the repos in gerrit to find cases like that) | 23:49 |
| ianw | jeblair: ++ ... it can get hard to follow without that | 23:49 |
| *** salv-orlando has joined #openstack-infra | 23:54 | |
| *** MasterPiece has quit IRC | 23:57 | |
| *** andreas_s has quit IRC | 23:57 | |
| *** Swami has quit IRC | 23:59 | |
| *** salv-orlando has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
