| prometheanfire | ianw: we may have an update out | 00:01 |
|---|---|---|
| prometheanfire | iirc we redo our image a couple of times a week and one should be done tonight or tomorrow | 00:02 |
| prometheanfire | gentoo is still nv right? | 00:02 |
| ianw | it is | 00:02 |
| *** tqtran has quit IRC | 00:03 | |
| prometheanfire | this is why :P | 00:03 |
| prometheanfire | I knew this would happen every now and then when a major package get's updated | 00:04 |
| prometheanfire | part of what dib does is update the system | 00:04 |
| prometheanfire | so gcc, and glibc might take a while, too long for gate I think | 00:04 |
| *** baoli has joined #openstack-infra | 00:06 | |
| ianw | 2016-06-22 15:40:59.704925 | x86_64-pc-linux-gnu-ranlib: libbackend.a: No space left on device <- seems to be the root cause | 00:06 |
| *** baoli has quit IRC | 00:06 | |
| ianw | i wonder if it's doing this on a tmpfs | 00:07 |
| *** lmiccini has joined #openstack-infra | 00:07 | |
| ianw | yeah, i think it is ... we should probably set no tmpfs in the job | 00:07 |
| fungi | i'm inclined to agree. currently on nodepool.o.o... /dev/mapper/main-nodepoolbuild 1008G 528G 480G 53% /opt | 00:08 |
| jesusaur | or tell portage to build gcc/glibc in a different location | 00:08 |
| fungi | so i don't think it's running out of /opt | 00:08 |
| fungi | or put differently, i seriously doubt it takes nearly 0.5tb of scratch space to build recent glibc ;) | 00:09 |
| prometheanfire | ianw: I'd +1 that | 00:10 |
| ianw | jesusaur: in this case, it's chrooted into a tmpfs so there's nowhere to go | 00:10 |
| fungi | all boxed in, frightened and vulnerable | 00:10 |
| prometheanfire | we should let it go | 00:10 |
| ianw | fungi: this is juts for the functional test jobs, i'm 99% sure we do set the "notmpfs" variable for dib on nodepool builds | 00:10 |
| fungi | oh, the jobs, not the image builds. got it | 00:11 |
| fungi | yeah, also our workers may be severely limited on available disk depending on where you're trying to set your scratch space | 00:11 |
| prometheanfire | sure it's glibc it's rebuilding? | 00:11 |
| ianw | prometheanfire: it's gcc -> http://logs.openstack.org/66/332866/1/check/gate-dib-dsvm-functests-centos-7-extras-nv/9375d7c/console.html | 00:12 |
| *** stevemar has quit IRC | 00:13 | |
| *** xarses has quit IRC | 00:14 | |
| *** stevemar has joined #openstack-infra | 00:14 | |
| fungi | c | 00:14 |
| fungi | hah, you're not my mutt window! | 00:14 |
| *** baoli has joined #openstack-infra | 00:14 | |
| ianw | puts on obi-wan cape ... "That's a failure I haven't seen in a long time" ... not since building x86-to-x86 "cross"-compilers for various dubious reasons for ESXi userspace | 00:14 |
| fungi | any reason involving esxi counts as "dubious" | 00:15 |
| *** baoli has quit IRC | 00:15 | |
| openstackgerrit | Merged openstack-infra/system-config: Refactor gerrit commentlinks parameter https://review.openstack.org/330902 | 00:17 |
| ianw | it was actually pretty cool in that the entire build system was versioned together with the code, sort of like http://aegis.sourceforge.net/ | 00:17 |
| prometheanfire | start here http://logs.openstack.org/66/332866/1/check/gate-dib-dsvm-functests-centos-7-extras-nv/9375d7c/console.html#_2016-06-22_15_27_27_675537 | 00:18 |
| *** stevemar has quit IRC | 00:18 | |
| prometheanfire | I think it's the multislot support that did it | 00:19 |
| *** amitgandhinz has joined #openstack-infra | 00:20 | |
| *** sarob_ has quit IRC | 00:20 | |
| prometheanfire | because it was just a reinstall | 00:20 |
| prometheanfire | ianw: this commit did it our side | 00:21 |
| prometheanfire | https://www.mail-archive.com/gentoo-commits@lists.gentoo.org/msg209167.html | 00:21 |
| prometheanfire | ianw: it's the --with-bdeps=y that causes us to rebuild | 00:22 |
| prometheanfire | it's best practice to do it, but not technically needed | 00:22 |
| *** akshai has quit IRC | 00:24 | |
| ianw | gotta -funroll-loops :) | 00:25 |
| *** amitgandhinz has quit IRC | 00:25 | |
| prometheanfire | basically | 00:26 |
| prometheanfire | technically correct or functionally good enough, which do you prefer? | 00:27 |
| *** achanda_ has quit IRC | 00:27 | |
| ianw | pabelanger: so can we move zuul user creation back into puppet? i feel like devuser is the wrong thing to be doing it -- for starters it's just picking up the authorized_keys of the build user which isn't really right | 00:27 |
| ianw | i know we don't like puppet, but it has adding users figured out at least | 00:27 |
| *** sarob has joined #openstack-infra | 00:28 | |
| *** sdague has joined #openstack-infra | 00:28 | |
| prometheanfire | that's the thing I need right? | 00:28 |
| ianw | but also it overwrites if someone is actually trying to create a "devuser" (i.e., me :) | 00:28 |
| *** jamesmcarthur has quit IRC | 00:28 | |
| *** stevemar has joined #openstack-infra | 00:29 | |
| *** rhochmuth has joined #openstack-infra | 00:31 | |
| *** lvdongbing has joined #openstack-infra | 00:31 | |
| *** sarob has quit IRC | 00:32 | |
| *** baoli has joined #openstack-infra | 00:33 | |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| *** roxanaghe has quit IRC | 00:35 | |
| *** nwkarsten has joined #openstack-infra | 00:36 | |
| *** baoli has quit IRC | 00:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Don't use tmpfs for DIB functional tests https://review.openstack.org/333070 | 00:41 |
| *** xarses has joined #openstack-infra | 00:41 | |
| *** baoli has joined #openstack-infra | 00:41 | |
| ianw | prometheanfire: ^ should help ... in theory | 00:41 |
| *** jamesmcarthur has quit IRC | 00:42 | |
| *** baoli has quit IRC | 00:42 | |
| *** Jeffrey4l has joined #openstack-infra | 00:44 | |
| *** ddieterly has quit IRC | 00:44 | |
| greghaynes | ianw: huh, dib should protect against that - its supposed to fallback to non tmpfs if you dont have the necessary ram | 00:47 |
| *** ddieterly has joined #openstack-infra | 00:47 | |
| ianw | greghaynes: but we do have enough ram ... it just defaults to a 2gb space | 00:48 |
| *** ddieterly has quit IRC | 00:48 | |
| *** thorst has joined #openstack-infra | 00:48 | |
| *** gongysh has quit IRC | 00:48 | |
| greghaynes | ah | 00:48 |
| *** mkoderer___ has quit IRC | 00:48 | |
| *** stevemar has quit IRC | 00:49 | |
| ianw | i guess we could bump it up ... not sure either way is better or worse | 00:50 |
| *** mkoderer__ has joined #openstack-infra | 00:50 | |
| *** stevemar has joined #openstack-infra | 00:50 | |
| *** sree has joined #openstack-infra | 00:50 | |
| greghaynes | ianw: or have dib check the tmpfs size in addition to ram | 00:50 |
| Shrews | mordred: i'm ok with you cutting a new shade release | 00:52 |
| *** fbo has quit IRC | 00:52 | |
| *** baoli has joined #openstack-infra | 00:54 | |
| openstackgerrit | Jeff Grafton proposed openstack-infra/jenkins-job-builder: Add addTestResults to GitHub pull request builder trigger https://review.openstack.org/321300 | 00:54 |
| pabelanger | fungi: we haven't done the process for our long lived workers yet, so we actually need to add code for that. However, ianw is raising some concerns about how we create zuul in DIB. Let me think on the process tonight, and propose something to address both | 00:54 |
| *** anthonyper has quit IRC | 00:54 | |
| *** mhu has quit IRC | 00:54 | |
| prometheanfire | ianw: ya, it'll still take forever, but that may be ok | 00:55 |
| *** raunak has quit IRC | 00:55 | |
| pabelanger | ianw: we could, but I am surprised devuser is failing for you. I guess the element is mean to be included only once | 00:55 |
| openstackgerrit | Jeff Grafton proposed openstack-infra/jenkins-job-builder: Add addTestResults to GitHub pull request builder trigger https://review.openstack.org/321300 | 00:56 |
| *** thorst has quit IRC | 00:56 | |
| ianw | pabelanger: yeah, it wasn't really supposed to be a generic useradd mechanism | 00:56 |
| *** rfolco has quit IRC | 00:56 | |
| ianw | pabelanger: i mean there seems to be a lot of logic in things like jenkins::slave | 00:56 |
| pabelanger | ianw: okay, I'll revisit the process in the morning | 00:56 |
| pabelanger | maybe see about using ansible inside DIB :D | 00:57 |
| *** anthonyper has joined #openstack-infra | 00:59 | |
| *** stevemar has quit IRC | 01:00 | |
| *** stevemar has joined #openstack-infra | 01:00 | |
| *** nwkarsten has quit IRC | 01:02 | |
| *** baoli_ has joined #openstack-infra | 01:02 | |
| ianw | pabelanger: ahh ... node ansible_host=172.99.106.198 ansible_user=jenkins | 01:02 |
| ianw | i wondered how it was working | 01:03 |
| *** fbo has joined #openstack-infra | 01:03 | |
| *** baoli has quit IRC | 01:05 | |
| *** stevemar has quit IRC | 01:05 | |
| greghaynes | I have thought about making a thing to include ansible in dib elements, seems like it would be doable and useful | 01:08 |
| *** ddieterly has joined #openstack-infra | 01:09 | |
| ianw | greghaynes: i think you could actually rewrite dib in ansible | 01:09 |
| * ianw goes and hides | 01:10 | |
| greghaynes | heh | 01:10 |
| greghaynes | I mean, I wouldnt object to that. I worry it might not end up being easier to grok | 01:10 |
| greghaynes | ansible seems to get hard to parse when the logic gets at all complicated | 01:11 |
| *** zhurong has joined #openstack-infra | 01:11 | |
| *** mhu has joined #openstack-infra | 01:12 | |
| *** nwkarsten has joined #openstack-infra | 01:12 | |
| *** jamesmcarthur has joined #openstack-infra | 01:13 | |
| *** stevemar has joined #openstack-infra | 01:13 | |
| nibalizer | greghaynes: yep | 01:14 |
| *** shashank_hegde has quit IRC | 01:15 | |
| *** salv-orlando has joined #openstack-infra | 01:18 | |
| *** PollyZ has joined #openstack-infra | 01:19 | |
| *** amitgandhinz has joined #openstack-infra | 01:20 | |
| *** hongbin has joined #openstack-infra | 01:23 | |
| *** PollyZ_ has joined #openstack-infra | 01:24 | |
| *** amitgandhinz has quit IRC | 01:25 | |
| *** salv-orlando has quit IRC | 01:25 | |
| *** PollyZ has quit IRC | 01:26 | |
| *** rhochmuth has quit IRC | 01:27 | |
| *** rhochmuth has joined #openstack-infra | 01:27 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: TestSubCommand inherit from UpdateSubCommand. https://review.openstack.org/319619 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Don't expose YamlParser methods unnecessarily. https://review.openstack.org/319618 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move subcommand logic into subcommand classes. https://review.openstack.org/319617 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move 'update' yaml parsing to JenkinsJobs.execute. https://review.openstack.org/319616 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser. https://review.openstack.org/319623 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes https://review.openstack.org/319622 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove deprecated 'extended-choice' property. https://review.openstack.org/319621 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser references from JJB modules. https://review.openstack.org/319620 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules. https://review.openstack.org/319625 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings. https://review.openstack.org/319624 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method https://review.openstack.org/319752 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs. https://review.openstack.org/319753 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins https://review.openstack.org/319754 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module. https://review.openstack.org/319755 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Make internal builder.Jenkins attrs internal-only. https://review.openstack.org/319756 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move 'delete' yaml parsing to JenkinsJobs.execute. https://review.openstack.org/262954 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Make builder.Jenkins inherit from jenkins.Jenkins https://review.openstack.org/319751 | 01:27 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move load_files method to YamlParser from Builder. https://review.openstack.org/319613 | 01:27 |
| *** hongbin has quit IRC | 01:29 | |
| *** sree has quit IRC | 01:30 | |
| *** ddieterly has quit IRC | 01:31 | |
| *** sree has joined #openstack-infra | 01:31 | |
| *** raunak has joined #openstack-infra | 01:31 | |
| *** nwkarsten has quit IRC | 01:31 | |
| waynr | jeblair: hey we briefly talked about this a few weeks ago not sure if you remember but I finally got around to moving JJB macro expansion into the YamlParser so it is no longer dependent on XML expansion, here's the change: https://review.openstack.org/333076 | 01:32 |
| *** nwkarsten has joined #openstack-infra | 01:32 | |
| waynr | s/XML expansion/XML generation/ | 01:32 |
| *** ams__ has quit IRC | 01:33 | |
| *** sree has quit IRC | 01:35 | |
| *** banix has joined #openstack-infra | 01:35 | |
| *** esberglu has joined #openstack-infra | 01:36 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 01:36 |
| waynr | hopefully this makes JJB 2.0 desirable for Zuul work :) | 01:37 |
| *** jamesmcarthur has quit IRC | 01:37 | |
| *** stevemar has quit IRC | 01:40 | |
| *** stevemar has joined #openstack-infra | 01:40 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| *** sree has joined #openstack-infra | 01:42 | |
| *** sree_ has joined #openstack-infra | 01:43 | |
| *** baoli_ has quit IRC | 01:43 | |
| *** sree_ is now known as Guest9802 | 01:44 | |
| *** baoli has joined #openstack-infra | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:45 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** sree has quit IRC | 01:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet https://review.openstack.org/333080 | 01:48 |
| *** Guest9802 has quit IRC | 01:48 | |
| *** yamahata has quit IRC | 01:48 | |
| ianw | pabelanger: ^ my suggestion -- at least describes the issues i see with the current model anyway | 01:48 |
| *** nwkarsten has quit IRC | 01:50 | |
| *** nwkarsten has joined #openstack-infra | 01:51 | |
| *** sree has joined #openstack-infra | 01:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet https://review.openstack.org/333080 | 01:51 |
| *** jamesmcarthur has quit IRC | 01:52 | |
| *** thorst has joined #openstack-infra | 01:53 | |
| *** nwkarsten has quit IRC | 01:55 | |
| *** sree has quit IRC | 01:55 | |
| pabelanger | ianw: ack, will review in the morning | 01:56 |
| *** rhochmuth1 has joined #openstack-infra | 01:57 | |
| *** signed8bit_Zzz is now known as signed8bit | 01:57 | |
| *** sree has joined #openstack-infra | 01:57 | |
| *** rhochmuth has quit IRC | 01:58 | |
| pabelanger | jeblair: mordred: going to run ansible-playbook to restart zuul-launchers so we are good for the morning | 01:58 |
| pabelanger | jhesketh: loop you in too^ | 01:58 |
| pabelanger | it will be running on puppetmaster.o.o under screen | 01:58 |
| jhesketh | pabelanger: thanks for the heads up | 01:58 |
| pabelanger | doing zlstatic01.o.o first | 01:59 |
| *** thorst has quit IRC | 01:59 | |
| *** claudiub has quit IRC | 02:00 | |
| *** esberglu has quit IRC | 02:00 | |
| *** afred312 has quit IRC | 02:00 | |
| *** tqtran has joined #openstack-infra | 02:01 | |
| *** sree has quit IRC | 02:01 | |
| *** sree has joined #openstack-infra | 02:03 | |
| *** zul has quit IRC | 02:03 | |
| *** afred312 has joined #openstack-infra | 02:04 | |
| *** zul has joined #openstack-infra | 02:04 | |
| *** Apoorva has quit IRC | 02:04 | |
| *** tqtran has quit IRC | 02:05 | |
| *** thorst has joined #openstack-infra | 02:05 | |
| jeblair | pabelanger: thanks, just came back to do that myself | 02:05 |
| pabelanger | jeblair: jhesketh: running now on puppetmaster.o.o, zl03 is the first | 02:06 |
| jeblair | pabelanger, ianw: we don't have to use devuser, but we should create the user in dib, even if we make a new element for it | 02:06 |
| pabelanger | jeblair: ianw: Ya, we'll need to figure out the plan for long lived servers too, since we don't use DIB for them... yet | 02:07 |
| *** sree has quit IRC | 02:07 | |
| *** bhavik has quit IRC | 02:08 | |
| jeblair | i have to run now; more fun tomorrow | 02:09 |
| *** sree has joined #openstack-infra | 02:09 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Tidy up zuul-env-reqs https://review.openstack.org/199833 | 02:10 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Tidy up absent rule https://review.openstack.org/333082 | 02:10 |
| *** thorst has quit IRC | 02:11 | |
| *** stevemar has quit IRC | 02:14 | |
| *** sree has quit IRC | 02:14 | |
| *** tlian has quit IRC | 02:14 | |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/332281 | 02:17 |
| *** bhavik has joined #openstack-infra | 02:17 | |
| *** sree has joined #openstack-infra | 02:17 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move macro expansion into YamlParser. https://review.openstack.org/333076 | 02:18 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix copyright in docs https://review.openstack.org/333084 | 02:19 |
| *** hongbin has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:21 | |
| *** amitgandhinz has joined #openstack-infra | 02:21 | |
| *** sree has quit IRC | 02:22 | |
| *** zul has quit IRC | 02:23 | |
| openstackgerrit | Merged openstack-infra/shade: Treat DELETE_COMPLETE stacks as NotFound https://review.openstack.org/333026 | 02:23 |
| *** rajinir has quit IRC | 02:24 | |
| *** kushal has quit IRC | 02:26 | |
| *** amitgandhinz has quit IRC | 02:26 | |
| *** sree has joined #openstack-infra | 02:26 | |
| *** greghaynes has quit IRC | 02:27 | |
| *** nwkarsten has joined #openstack-infra | 02:28 | |
| *** raunak has quit IRC | 02:30 | |
| *** sdague has quit IRC | 02:30 | |
| *** sree has quit IRC | 02:31 | |
| *** sree has joined #openstack-infra | 02:33 | |
| *** stevemar has joined #openstack-infra | 02:34 | |
| *** aysyd has quit IRC | 02:35 | |
| *** zz_dimtruck is now known as dimtruck | 02:37 | |
| *** sree_ has joined #openstack-infra | 02:37 | |
| *** sree_ is now known as Guest91960 | 02:38 | |
| *** gongysh has joined #openstack-infra | 02:38 | |
| *** sree has quit IRC | 02:38 | |
| *** greghaynes has joined #openstack-infra | 02:38 | |
| *** stevemar has quit IRC | 02:39 | |
| *** inc0 has joined #openstack-infra | 02:39 | |
| *** sree has joined #openstack-infra | 02:41 | |
| *** Guest91960 has quit IRC | 02:41 | |
| *** rhochmuth1 has quit IRC | 02:43 | |
| *** sree has quit IRC | 02:46 | |
| *** inc0 has quit IRC | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:47 | |
| *** sree has joined #openstack-infra | 02:49 | |
| *** salv-orlando has joined #openstack-infra | 02:49 | |
| *** sree has quit IRC | 02:53 | |
| *** salv-orlando has quit IRC | 02:54 | |
| *** yanyanhu has joined #openstack-infra | 02:54 | |
| *** zhurong has quit IRC | 02:55 | |
| *** stevemar has joined #openstack-infra | 02:57 | |
| *** nwkarsten has quit IRC | 02:57 | |
| *** yamamoto_ has joined #openstack-infra | 03:02 | |
| *** achanda has joined #openstack-infra | 03:03 | |
| *** zhurong has joined #openstack-infra | 03:03 | |
| *** raunak has joined #openstack-infra | 03:04 | |
| ianw | jeblair: tbh i'm really not sure reinventing user adding in dib is a great idea, it seems like something that puppet does just fine. | 03:05 |
| *** nwkarsten has joined #openstack-infra | 03:07 | |
| *** sree has joined #openstack-infra | 03:07 | |
| *** achanda has quit IRC | 03:08 | |
| *** hongbin has quit IRC | 03:09 | |
| *** lvdongbing has quit IRC | 03:09 | |
| *** haaj has quit IRC | 03:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add zuul user for launching in puppet https://review.openstack.org/333080 | 03:10 |
| *** sree has quit IRC | 03:11 | |
| *** sree has joined #openstack-infra | 03:14 | |
| *** jamesmcarthur has joined #openstack-infra | 03:14 | |
| *** raunak has quit IRC | 03:14 | |
| *** bhavik has quit IRC | 03:15 | |
| *** bhavik has joined #openstack-infra | 03:15 | |
| *** thorst has joined #openstack-infra | 03:16 | |
| *** ramishra has joined #openstack-infra | 03:17 | |
| *** sree has quit IRC | 03:18 | |
| *** lvdongbing has joined #openstack-infra | 03:18 | |
| *** csomerville has quit IRC | 03:18 | |
| *** baoli has quit IRC | 03:19 | |
| *** jamesmcarthur has quit IRC | 03:19 | |
| *** raunak has joined #openstack-infra | 03:19 | |
| *** stevemar has quit IRC | 03:20 | |
| *** rhochmuth has joined #openstack-infra | 03:20 | |
| *** thorst has quit IRC | 03:21 | |
| *** amitgandhinz has joined #openstack-infra | 03:22 | |
| *** baoli has joined #openstack-infra | 03:23 | |
| *** nwkarsten has quit IRC | 03:25 | |
| *** nwkarsten has joined #openstack-infra | 03:25 | |
| *** amitgandhinz has quit IRC | 03:27 | |
| *** ramishra has quit IRC | 03:27 | |
| *** stevemar has joined #openstack-infra | 03:28 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Lower the log level in tests https://review.openstack.org/263364 | 03:31 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for sqlalchemy reporter https://review.openstack.org/223333 | 03:31 |
| *** stevemar has quit IRC | 03:32 | |
| *** ramishra has joined #openstack-infra | 03:32 | |
| *** sree has joined #openstack-infra | 03:33 | |
| *** garyk1 has joined #openstack-infra | 03:34 | |
| *** stevemar has joined #openstack-infra | 03:35 | |
| *** stevemar has quit IRC | 03:35 | |
| *** stevemar has joined #openstack-infra | 03:36 | |
| ianw | gate-infra-puppet-apply-ubuntu-xenial: queued <- do we not have a lot of xenial nodes? | 03:37 |
| *** sree has quit IRC | 03:37 | |
| *** rhochmuth has quit IRC | 03:39 | |
| *** gouthamr_ has quit IRC | 03:39 | |
| *** shashank_hegde has joined #openstack-infra | 03:41 | |
| jhesketh | ianw: not many... 9 total, 3 in hold, 5 used, 1 ready | 03:41 |
| *** raunak has quit IRC | 03:41 | |
| *** raunak has joined #openstack-infra | 03:43 | |
| *** signed8bit has quit IRC | 03:43 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Make ml2 job non-voting https://review.openstack.org/333097 | 03:44 |
| *** lvdongbing has quit IRC | 03:47 | |
| *** shashank_hegde has quit IRC | 03:49 | |
| *** nwkarsten has quit IRC | 03:49 | |
| *** nwkarsten has joined #openstack-infra | 03:50 | |
| *** links has joined #openstack-infra | 03:51 | |
| *** Nikolay_St has quit IRC | 03:52 | |
| *** ramishra has quit IRC | 03:53 | |
| *** ramishra has joined #openstack-infra | 03:54 | |
| *** winggundamth has quit IRC | 03:54 | |
| *** yamamoto_ has quit IRC | 03:54 | |
| *** nwkarsten has quit IRC | 03:54 | |
| *** sree has joined #openstack-infra | 03:57 | |
| *** nwkarsten has joined #openstack-infra | 03:59 | |
| *** burgerk has joined #openstack-infra | 04:00 | |
| *** salv-orlando has joined #openstack-infra | 04:00 | |
| *** sree has quit IRC | 04:02 | |
| *** baoli has quit IRC | 04:10 | |
| *** haaj has joined #openstack-infra | 04:11 | |
| *** cody-somerville has joined #openstack-infra | 04:13 | |
| *** salv-orlando has quit IRC | 04:14 | |
| *** sree has joined #openstack-infra | 04:17 | |
| *** achanda has joined #openstack-infra | 04:18 | |
| *** thorst has joined #openstack-infra | 04:19 | |
| *** shashank_hegde has joined #openstack-infra | 04:20 | |
| *** javeriak has joined #openstack-infra | 04:20 | |
| *** dimtruck is now known as zz_dimtruck | 04:21 | |
| *** amitgandhinz has joined #openstack-infra | 04:23 | |
| *** thorst has quit IRC | 04:25 | |
| *** amotoki has joined #openstack-infra | 04:25 | |
| *** cody-somerville has quit IRC | 04:27 | |
| *** rossella__ has joined #openstack-infra | 04:27 | |
| *** esikachev has joined #openstack-infra | 04:27 | |
| *** amitgandhinz has quit IRC | 04:27 | |
| *** Sukhdev has joined #openstack-infra | 04:28 | |
| *** PollyZ_ has quit IRC | 04:28 | |
| *** maishsk has quit IRC | 04:29 | |
| *** rossella_ has quit IRC | 04:30 | |
| *** raunak has quit IRC | 04:31 | |
| *** esikachev has quit IRC | 04:32 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd https://review.openstack.org/331123 | 04:33 |
| *** javeriak has quit IRC | 04:34 | |
| *** raunak has joined #openstack-infra | 04:35 | |
| *** psachin has joined #openstack-infra | 04:38 | |
| *** winggundamth has joined #openstack-infra | 04:38 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd https://review.openstack.org/331123 | 04:42 |
| waynr | hmm, it the "parallel" module in JJB seems to be a misnomer | 04:42 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Move the command processing into the launcher cmd https://review.openstack.org/331123 | 04:44 |
| *** yamamoto_ has joined #openstack-infra | 04:45 | |
| *** Liuqing has joined #openstack-infra | 04:46 | |
| *** javeriak has joined #openstack-infra | 04:46 | |
| *** burgerk has quit IRC | 04:47 | |
| *** javeriak has quit IRC | 04:50 | |
| *** salv-orlando has joined #openstack-infra | 04:50 | |
| *** banix has quit IRC | 04:52 | |
| *** shahid__ has joined #openstack-infra | 04:52 | |
| *** ramishra has quit IRC | 04:52 | |
| *** Sukhdev has quit IRC | 04:55 | |
| *** burgerk has joined #openstack-infra | 04:55 | |
| *** jaosorior has joined #openstack-infra | 04:55 | |
| *** ramishra has joined #openstack-infra | 04:58 | |
| *** salv-orl_ has joined #openstack-infra | 04:59 | |
| *** burgerk has quit IRC | 05:00 | |
| waynr | hmm, replacing threading.Thread with multiprocessing.Process causes unit tests to hang.... | 05:02 |
| *** salv-orlando has quit IRC | 05:02 | |
| waynr | i guess i'll just have to gut the parallel module to make it actually parallel...the fun never ends! | 05:02 |
| *** yamahata has joined #openstack-infra | 05:08 | |
| *** sree_ has joined #openstack-infra | 05:08 | |
| *** sree has quit IRC | 05:09 | |
| *** gongysh has quit IRC | 05:09 | |
| *** sree_ is now known as Guest77814 | 05:09 | |
| *** Nikolay_St has joined #openstack-infra | 05:12 | |
| *** ramishra has quit IRC | 05:23 | |
| *** thorst has joined #openstack-infra | 05:23 | |
| *** amitgandhinz has joined #openstack-infra | 05:24 | |
| *** abregman has joined #openstack-infra | 05:25 | |
| *** amitgandhinz has quit IRC | 05:28 | |
| *** thorst has quit IRC | 05:31 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Change commit message length validation to 72 chars https://review.openstack.org/316197 | 05:31 |
| *** coolsvap has joined #openstack-infra | 05:32 | |
| *** Liuqing has quit IRC | 05:33 | |
| *** shashank_hegde has quit IRC | 05:34 | |
| waynr | i wonder if i am just kidding myself about JJB 2.0 ever getting reviewed | 05:34 |
| *** shardy has joined #openstack-infra | 05:36 | |
| *** javeriak has joined #openstack-infra | 05:37 | |
| *** ramishra has joined #openstack-infra | 05:37 | |
| *** shashank_hegde has joined #openstack-infra | 05:38 | |
| *** kaisers1 has quit IRC | 05:38 | |
| *** shardy has quit IRC | 05:38 | |
| *** kaisers has joined #openstack-infra | 05:42 | |
| *** binbincong has quit IRC | 05:43 | |
| *** shahid__ has quit IRC | 05:43 | |
| *** Guest77814 has quit IRC | 05:43 | |
| *** sree has joined #openstack-infra | 05:44 | |
| *** roxanaghe has joined #openstack-infra | 05:44 | |
| *** ramishra has quit IRC | 05:44 | |
| *** nwkarsten has quit IRC | 05:45 | |
| *** sshnaidm_ has joined #openstack-infra | 05:45 | |
| *** nwkarsten has joined #openstack-infra | 05:45 | |
| *** roxanaghe has quit IRC | 05:45 | |
| openstackgerrit | Christian Fetzer proposed openstack-infra/jenkins-job-builder: Update Gitlab to >=1.2.0 https://review.openstack.org/333113 | 05:47 |
| *** ramishra has joined #openstack-infra | 05:49 | |
| *** nwkarsten has quit IRC | 05:50 | |
| *** lvdongbing has joined #openstack-infra | 05:50 | |
| *** camunoz has quit IRC | 05:51 | |
| *** rhochmuth has joined #openstack-infra | 05:51 | |
| *** _nadya_ has joined #openstack-infra | 05:54 | |
| *** raunak has quit IRC | 05:55 | |
| *** yolanda has joined #openstack-infra | 05:56 | |
| *** binbincong has joined #openstack-infra | 06:00 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin puppet-lint-absolute_classname-check to 0.1.3 https://review.openstack.org/333117 | 06:00 |
| *** tqtran has joined #openstack-infra | 06:02 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Handle locales install on Fedora 24 https://review.openstack.org/333118 | 06:04 |
| *** ramishra has quit IRC | 06:06 | |
| *** tqtran has quit IRC | 06:06 | |
| *** pgadiya has joined #openstack-infra | 06:08 | |
| *** rossella__ has quit IRC | 06:11 | |
| *** ramishra has joined #openstack-infra | 06:11 | |
| *** aeng has quit IRC | 06:12 | |
| *** dingyichen has quit IRC | 06:12 | |
| *** claudiub has joined #openstack-infra | 06:13 | |
| *** rcernin has joined #openstack-infra | 06:15 | |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Add networking-l2gw fullstack job https://review.openstack.org/326300 | 06:16 |
| *** linuxmaniac_ has joined #openstack-infra | 06:17 | |
| *** linuxmaniac has quit IRC | 06:17 | |
| openstackgerrit | Merged openstack-infra/project-config: python-saharaclient is required also in PROJECTS https://review.openstack.org/332797 | 06:17 |
| *** ramishra has quit IRC | 06:19 | |
| *** ramishra has joined #openstack-infra | 06:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't use tmpfs for DIB functional tests https://review.openstack.org/333070 | 06:21 |
| openstackgerrit | Merged openstack-infra/project-config: Request to add new project Masakari https://review.openstack.org/330370 | 06:21 |
| openstackgerrit | Merged openstack-infra/system-config: Use root to restart our zuul-launcher service https://review.openstack.org/332264 | 06:21 |
| openstackgerrit | Merged openstack-infra/project-config: Check git tag matches xstatic version https://review.openstack.org/330321 | 06:23 |
| *** hdaniel has joined #openstack-infra | 06:24 | |
| *** amitgandhinz has joined #openstack-infra | 06:24 | |
| *** maishsk has joined #openstack-infra | 06:27 | |
| *** Qiming has quit IRC | 06:28 | |
| *** andreas_s has joined #openstack-infra | 06:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove tests on magnum liberty. https://review.openstack.org/332780 | 06:28 |
| *** thorst has joined #openstack-infra | 06:28 | |
| *** amitgandhinz has quit IRC | 06:29 | |
| *** yolanda has quit IRC | 06:30 | |
| *** prabu has joined #openstack-infra | 06:30 | |
| *** maishsk has quit IRC | 06:31 | |
| *** isaacb has joined #openstack-infra | 06:33 | |
| *** sdake has joined #openstack-infra | 06:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate https://review.openstack.org/332766 | 06:34 |
| *** Qiming has joined #openstack-infra | 06:35 | |
| *** thorst has quit IRC | 06:36 | |
| *** _nadya_ has quit IRC | 06:36 | |
| *** javeriak has quit IRC | 06:43 | |
| *** sree has quit IRC | 06:44 | |
| *** changzhi has joined #openstack-infra | 06:44 | |
| *** sree has joined #openstack-infra | 06:44 | |
| *** Qiming has quit IRC | 06:47 | |
| *** zeih has joined #openstack-infra | 06:47 | |
| *** zeih has quit IRC | 06:48 | |
| *** zeih has joined #openstack-infra | 06:48 | |
| *** jamielennox is now known as jamielennox|away | 06:48 | |
| *** Hal has joined #openstack-infra | 06:51 | |
| *** Hal is now known as Guest25417 | 06:51 | |
| *** javeriak has joined #openstack-infra | 06:52 | |
| *** TaylorHuang has joined #openstack-infra | 06:52 | |
| *** gongysh has joined #openstack-infra | 06:52 | |
| *** Qiming has joined #openstack-infra | 06:54 | |
| *** TaylorH has quit IRC | 06:55 | |
| *** achanda has quit IRC | 06:56 | |
| *** esikachev has joined #openstack-infra | 06:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Retiring monitoring-for-openstack step 2 https://review.openstack.org/325122 | 07:00 |
| *** achanda has joined #openstack-infra | 07:01 | |
| *** shardy has joined #openstack-infra | 07:02 | |
| *** rcernin has quit IRC | 07:02 | |
| *** achanda has quit IRC | 07:06 | |
| *** samP has joined #openstack-infra | 07:07 | |
| *** isaacb has quit IRC | 07:08 | |
| *** ajmiller has quit IRC | 07:09 | |
| *** jpich has joined #openstack-infra | 07:11 | |
| *** tesseract- has joined #openstack-infra | 07:13 | |
| *** esikachev has quit IRC | 07:13 | |
| *** tesseract- has quit IRC | 07:14 | |
| *** tesseract- has joined #openstack-infra | 07:14 | |
| *** hdaniel has quit IRC | 07:17 | |
| *** rcernin has joined #openstack-infra | 07:17 | |
| *** salv-orl_ has quit IRC | 07:18 | |
| *** changzhi has quit IRC | 07:18 | |
| *** woodster_ has quit IRC | 07:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:20 | |
| *** Nikolay_St has quit IRC | 07:21 | |
| *** sdake_ has joined #openstack-infra | 07:21 | |
| *** rhochmuth has quit IRC | 07:22 | |
| *** zhurong has quit IRC | 07:22 | |
| *** furlongm_ has joined #openstack-infra | 07:22 | |
| *** furlongm has quit IRC | 07:22 | |
| *** zhurong has joined #openstack-infra | 07:23 | |
| *** sdake has quit IRC | 07:24 | |
| *** e0ne has joined #openstack-infra | 07:24 | |
| *** eezhova has joined #openstack-infra | 07:25 | |
| *** yaume has joined #openstack-infra | 07:25 | |
| *** amitgandhinz has joined #openstack-infra | 07:25 | |
| *** Na3iL has joined #openstack-infra | 07:26 | |
| *** jlanoux has joined #openstack-infra | 07:26 | |
| *** furlongm has joined #openstack-infra | 07:26 | |
| *** furlongm_ has quit IRC | 07:27 | |
| *** sree has quit IRC | 07:27 | |
| *** dizquierdo has joined #openstack-infra | 07:28 | |
| *** watanabe_isao has joined #openstack-infra | 07:29 | |
| *** Nikolay_St has joined #openstack-infra | 07:29 | |
| *** jlanoux has quit IRC | 07:29 | |
| *** amitgandhinz has quit IRC | 07:29 | |
| *** abregman has quit IRC | 07:30 | |
| *** abregman has joined #openstack-infra | 07:31 | |
| *** eezhova has quit IRC | 07:32 | |
| *** thorst has joined #openstack-infra | 07:34 | |
| *** jlanoux has joined #openstack-infra | 07:34 | |
| *** bhavik has quit IRC | 07:34 | |
| *** furlongm has quit IRC | 07:35 | |
| *** furlongm has joined #openstack-infra | 07:35 | |
| *** arxcruz has joined #openstack-infra | 07:36 | |
| *** thorst has quit IRC | 07:41 | |
| *** furlongm has quit IRC | 07:42 | |
| *** hashar has joined #openstack-infra | 07:42 | |
| *** watanabe_isao has quit IRC | 07:43 | |
| *** amoralej|off is now known as amoralej | 07:43 | |
| *** samP has quit IRC | 07:46 | |
| *** Na3iL has quit IRC | 07:46 | |
| *** matrohon has joined #openstack-infra | 07:48 | |
| *** kzaitsev_mb has quit IRC | 07:49 | |
| *** nmagnezi has joined #openstack-infra | 07:52 | |
| *** ifarkas has joined #openstack-infra | 07:53 | |
| *** bunting has quit IRC | 07:53 | |
| *** bunting has joined #openstack-infra | 07:54 | |
| *** sflanigan has quit IRC | 07:55 | |
| *** gus has quit IRC | 07:57 | |
| *** gus has joined #openstack-infra | 07:57 | |
| *** asselin has quit IRC | 07:58 | |
| *** isaacb has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** shahid__ has joined #openstack-infra | 08:02 | |
| *** jtomasek_ has joined #openstack-infra | 08:02 | |
| *** abregman has quit IRC | 08:03 | |
| *** zhurong has quit IRC | 08:03 | |
| *** yolanda has joined #openstack-infra | 08:04 | |
| *** salv-orlando has joined #openstack-infra | 08:05 | |
| *** markvoel_ has quit IRC | 08:05 | |
| *** dfflanders has quit IRC | 08:08 | |
| *** zhurong has joined #openstack-infra | 08:10 | |
| *** sree has joined #openstack-infra | 08:10 | |
| *** Na3iL has joined #openstack-infra | 08:10 | |
| *** isaacb has quit IRC | 08:11 | |
| *** e0ne has quit IRC | 08:11 | |
| *** _nadya_ has joined #openstack-infra | 08:15 | |
| *** ams__ has joined #openstack-infra | 08:16 | |
| *** _nadya_ has quit IRC | 08:16 | |
| *** _nadya_ has joined #openstack-infra | 08:16 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Add install-guide job https://review.openstack.org/333165 | 08:17 |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** abregman has joined #openstack-infra | 08:18 | |
| *** jtomasek_ has quit IRC | 08:20 | |
| *** dtardivel has joined #openstack-infra | 08:23 | |
| *** amitgandhinz has joined #openstack-infra | 08:26 | |
| *** acabot has quit IRC | 08:26 | |
| *** sree has quit IRC | 08:27 | |
| *** gongysh has quit IRC | 08:29 | |
| *** sree has joined #openstack-infra | 08:29 | |
| *** lucas-afk is now known as lucasagomes | 08:29 | |
| *** acabot has joined #openstack-infra | 08:29 | |
| *** shashank_hegde has quit IRC | 08:29 | |
| *** zhurong_ has joined #openstack-infra | 08:30 | |
| *** hosanai has joined #openstack-infra | 08:30 | |
| *** zhurong has quit IRC | 08:30 | |
| *** amitgandhinz has quit IRC | 08:31 | |
| *** Qiming has quit IRC | 08:31 | |
| *** daemontool has joined #openstack-infra | 08:31 | |
| *** sree has quit IRC | 08:34 | |
| *** jcooley has joined #openstack-infra | 08:35 | |
| *** sambetts|afk is now known as sambetts | 08:36 | |
| *** jcooley has left #openstack-infra | 08:36 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Add install-guide job https://review.openstack.org/333165 | 08:37 |
| *** thorst has joined #openstack-infra | 08:39 | |
| *** derekh has joined #openstack-infra | 08:44 | |
| *** HeOS has quit IRC | 08:44 | |
| *** Qiming has joined #openstack-infra | 08:44 | |
| *** thorst has quit IRC | 08:46 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Don't run tempest tests in zaqar gate https://review.openstack.org/333184 | 08:46 |
| *** Jeffrey4l has quit IRC | 08:47 | |
| *** eezhova has joined #openstack-infra | 08:48 | |
| *** salv-orlando has quit IRC | 08:49 | |
| *** rcernin has quit IRC | 08:50 | |
| *** salv-orlando has joined #openstack-infra | 08:50 | |
| *** amotoki has quit IRC | 08:53 | |
| therve | What's the procedure to exclude a job from master and forward? | 08:54 |
| openstackgerrit | Merged openstack-infra/storyboard: Add TAGS_ADDED and TAGS_DELETED to event_types.ALL https://review.openstack.org/331641 | 08:54 |
| therve | I see lots of example for not running on old stable branches, but not the other way around | 08:54 |
| *** salv-orlando has quit IRC | 08:55 | |
| *** dtantsur|afk is now known as dtantsur | 08:59 | |
| *** mhickey has joined #openstack-infra | 08:59 | |
| *** Jeffrey4l has joined #openstack-infra | 09:00 | |
| *** yolanda has quit IRC | 09:01 | |
| *** rcernin has joined #openstack-infra | 09:02 | |
| *** achanda has joined #openstack-infra | 09:03 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 09:03 |
| *** tqtran has joined #openstack-infra | 09:03 | |
| *** ramishra has quit IRC | 09:04 | |
| *** ramishra has joined #openstack-infra | 09:04 | |
| *** amotoki has joined #openstack-infra | 09:05 | |
| *** markvoelker has joined #openstack-infra | 09:05 | |
| *** infra-red has joined #openstack-infra | 09:06 | |
| *** achanda has quit IRC | 09:08 | |
| *** tqtran has quit IRC | 09:08 | |
| *** markusry has joined #openstack-infra | 09:08 | |
| *** dmk0202 has joined #openstack-infra | 09:09 | |
| *** markvoelker has quit IRC | 09:10 | |
| *** jaosorior has quit IRC | 09:11 | |
| *** Qiming has quit IRC | 09:11 | |
| *** jaosorior has joined #openstack-infra | 09:11 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master https://review.openstack.org/333200 | 09:13 |
| *** arif-ali has quit IRC | 09:15 | |
| *** binbincong has quit IRC | 09:16 | |
| *** binbincong has joined #openstack-infra | 09:16 | |
| *** electrofelix has joined #openstack-infra | 09:20 | |
| *** Qiming has joined #openstack-infra | 09:20 | |
| *** maishsk has joined #openstack-infra | 09:20 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 09:23 |
| *** zul has joined #openstack-infra | 09:23 | |
| *** arif-ali has joined #openstack-infra | 09:24 | |
| *** zhenguo_ has joined #openstack-infra | 09:25 | |
| *** amitgandhinz has joined #openstack-infra | 09:27 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Use Smaug configuration from Smaug's repository https://review.openstack.org/329305 | 09:27 |
| *** Jeffrey4l has quit IRC | 09:28 | |
| *** arif-ali has quit IRC | 09:29 | |
| *** amotoki has quit IRC | 09:30 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 09:30 |
| *** amotoki has joined #openstack-infra | 09:32 | |
| *** amitgandhinz has quit IRC | 09:32 | |
| *** ociuhandu has joined #openstack-infra | 09:32 | |
| *** yamahata has quit IRC | 09:33 | |
| *** amotoki has quit IRC | 09:33 | |
| *** infra-red has quit IRC | 09:33 | |
| *** abregman is now known as abregman|afk | 09:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:37 | |
| *** javeriak has quit IRC | 09:38 | |
| *** amotoki has joined #openstack-infra | 09:39 | |
| *** _milan_ has quit IRC | 09:39 | |
| *** acoles_ is now known as acoles | 09:41 | |
| *** Jeffrey4l has joined #openstack-infra | 09:42 | |
| *** javeriak has joined #openstack-infra | 09:42 | |
| *** salv-orlando has joined #openstack-infra | 09:42 | |
| *** isaacb has joined #openstack-infra | 09:42 | |
| *** sdake_ has quit IRC | 09:42 | |
| *** jamielennox|away is now known as jamielennox | 09:43 | |
| *** thorst has joined #openstack-infra | 09:44 | |
| *** esikachev has joined #openstack-infra | 09:44 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 09:47 |
| *** kzaitsev_mb has quit IRC | 09:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:49 | |
| *** amotoki has quit IRC | 09:50 | |
| andreykurilin | hi everyone! where I can find code of test-results publisher? https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L118 | 09:51 |
| *** thorst has quit IRC | 09:51 | |
| *** pcaruana has joined #openstack-infra | 09:51 | |
| *** ociuhandu has quit IRC | 09:55 | |
| *** tosky has joined #openstack-infra | 09:55 | |
| *** amotoki has joined #openstack-infra | 09:58 | |
| AJaeger | therve: just pin it to the current branches like "branch: ^(stable/(kilo|liberty)).*$" | 09:59 |
| therve | AJaeger, Yeah I think I found the solution, thanks | 10:00 |
| *** daemontool has quit IRC | 10:00 | |
| therve | https://review.openstack.org/333200 | 10:00 |
| *** shardy has quit IRC | 10:00 | |
| *** abregman|afk is now known as abregman | 10:01 | |
| *** sdake has joined #openstack-infra | 10:01 | |
| *** coolsvap has quit IRC | 10:02 | |
| *** lvdongbing has quit IRC | 10:02 | |
| AJaeger | therve: should it apply here as well? http://logs.openstack.org/00/333200/1/check/gate-project-config-layout/132489d/console.html#_2016-06-23_09_20_16_120451 | 10:03 |
| *** coolsvap has joined #openstack-infra | 10:03 | |
| AJaeger | therve: the rule for ^gate-heat-dsvm-functional-convg-mysql-.*$ overrides your v1 condition | 10:03 |
| therve | Ah, yeah... | 10:04 |
| AJaeger | therve: the codnition is correct otherwise. It's just the override... | 10:05 |
| therve | That's annoying :) | 10:05 |
| therve | I don't get why it can't do multiple matches, anyway | 10:05 |
| *** markvoelker has joined #openstack-infra | 10:07 | |
| therve | Urg that hurts my brain | 10:07 |
| *** eezhova has quit IRC | 10:09 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master https://review.openstack.org/333200 | 10:10 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/storyboard-webclient: Fix alignment of long-running simple events https://review.openstack.org/333225 | 10:10 |
| *** bhavik has joined #openstack-infra | 10:11 | |
| *** markvoelker has quit IRC | 10:11 | |
| *** yaume has quit IRC | 10:12 | |
| *** ociuhandu has joined #openstack-infra | 10:13 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: Add pytest_results.html to test-results publisher https://review.openstack.org/333226 | 10:14 |
| *** zhurong_ has quit IRC | 10:14 | |
| *** eezhova has joined #openstack-infra | 10:15 | |
| *** bhavik has quit IRC | 10:15 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: panko: enable functional testing https://review.openstack.org/333227 | 10:17 |
| *** dtantsur is now known as dtantsur|bbl | 10:17 | |
| AJaeger | therve: you have a pasto in there, otherwise should be fine. | 10:17 |
| therve | AJaeger, Thanks a ton! | 10:19 |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Remove Heat lbaasv1 from master https://review.openstack.org/333200 | 10:19 |
| *** _degorenko|afk is now known as degorenko | 10:20 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 10:22 |
| *** zul has quit IRC | 10:23 | |
| *** shardy has joined #openstack-infra | 10:24 | |
| *** HeOS has joined #openstack-infra | 10:25 | |
| *** amitgandhinz has joined #openstack-infra | 10:27 | |
| *** sdake_ has joined #openstack-infra | 10:28 | |
| *** dmk0202 has quit IRC | 10:29 | |
| *** sdake has quit IRC | 10:32 | |
| *** amitgandhinz has quit IRC | 10:32 | |
| *** zul has joined #openstack-infra | 10:35 | |
| *** eezhova has quit IRC | 10:38 | |
| *** zul has quit IRC | 10:38 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental functional job https://review.openstack.org/333236 | 10:40 |
| *** ramishra has quit IRC | 10:41 | |
| *** hrubi has quit IRC | 10:41 | |
| *** hrubi has joined #openstack-infra | 10:41 | |
| *** ramishra has joined #openstack-infra | 10:41 | |
| *** coolsvap has quit IRC | 10:42 | |
| *** coolsvap has joined #openstack-infra | 10:44 | |
| *** jpeeler has quit IRC | 10:44 | |
| *** thorst has joined #openstack-infra | 10:49 | |
| *** sorantis has joined #openstack-infra | 10:50 | |
| *** zul has joined #openstack-infra | 10:52 | |
| wznoinsk | hi infra, any issues with ci-watch or it's just me? | 10:52 |
| *** derekh has quit IRC | 10:53 | |
| wznoinsk | detailes: http://ci-watch.tintri.com/project?project=nova&time=24+hours gives no table with changes and results | 10:55 |
| *** rhochmuth has joined #openstack-infra | 10:55 | |
| *** thorst has quit IRC | 10:56 | |
| *** jistr is now known as jistr|mtg | 10:56 | |
| *** salv-orl_ has joined #openstack-infra | 10:58 | |
| *** coolsvap has quit IRC | 10:58 | |
| *** rhochmuth has quit IRC | 11:00 | |
| *** coolsvap has joined #openstack-infra | 11:00 | |
| *** salv-orlando has quit IRC | 11:01 | |
| *** YorikSar has quit IRC | 11:02 | |
| *** _nadya_ has quit IRC | 11:02 | |
| *** dtantsur|bbl is now known as dtantsur | 11:03 | |
| *** YorikSar has joined #openstack-infra | 11:04 | |
| *** achanda has joined #openstack-infra | 11:05 | |
| *** ddieterly has joined #openstack-infra | 11:05 | |
| *** sorantis has quit IRC | 11:05 | |
| *** markvoelker has joined #openstack-infra | 11:07 | |
| *** rhochmuth has joined #openstack-infra | 11:09 | |
| *** rhochmuth has quit IRC | 11:09 | |
| *** achanda has quit IRC | 11:09 | |
| *** yanyanhu has quit IRC | 11:10 | |
| *** sshnaidm_ is now known as sshnaidm | 11:11 | |
| *** markvoelker has quit IRC | 11:13 | |
| *** eezhova has joined #openstack-infra | 11:15 | |
| *** zul has quit IRC | 11:16 | |
| *** amrith has quit IRC | 11:16 | |
| *** zul has joined #openstack-infra | 11:17 | |
| *** amrith has joined #openstack-infra | 11:17 | |
| *** aviau has quit IRC | 11:17 | |
| *** aviau has joined #openstack-infra | 11:17 | |
| *** lucasagomes is now known as lucas-hungry | 11:19 | |
| *** amoralej is now known as amoralej|lunch | 11:19 | |
| *** jamesmcarthur has joined #openstack-infra | 11:19 | |
| *** sdake_ has quit IRC | 11:21 | |
| *** sdague has joined #openstack-infra | 11:22 | |
| *** lvdongbing has joined #openstack-infra | 11:23 | |
| *** sarob has joined #openstack-infra | 11:23 | |
| *** jamesmcarthur has quit IRC | 11:23 | |
| *** kzaitsev_mb has quit IRC | 11:25 | |
| *** ekarlso has quit IRC | 11:26 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: panko: enable functional testing https://review.openstack.org/333227 | 11:26 |
| *** daemontool has joined #openstack-infra | 11:28 | |
| *** amitgandhinz has joined #openstack-infra | 11:28 | |
| *** sarob has quit IRC | 11:28 | |
| *** snarwade is now known as snarwade|busy | 11:29 | |
| *** ddieterly has quit IRC | 11:31 | |
| *** prabu has quit IRC | 11:31 | |
| *** dizquierdo has quit IRC | 11:32 | |
| *** amitgandhinz has quit IRC | 11:33 | |
| *** dmk0202 has joined #openstack-infra | 11:33 | |
| *** ddieterly has joined #openstack-infra | 11:35 | |
| *** YorikSar has quit IRC | 11:36 | |
| *** YorikSar has joined #openstack-infra | 11:37 | |
| *** ddieterly has quit IRC | 11:38 | |
| *** ldnunes has joined #openstack-infra | 11:39 | |
| *** thorst has joined #openstack-infra | 11:41 | |
| *** _nadya_ has joined #openstack-infra | 11:42 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental functional job https://review.openstack.org/333236 | 11:43 |
| *** kzaitsev_mb has joined #openstack-infra | 11:43 | |
| *** gouthamr has joined #openstack-infra | 11:47 | |
| *** _nadya_ has quit IRC | 11:47 | |
| bkero | Good morning everyone | 11:48 |
| *** zhurong has joined #openstack-infra | 11:48 | |
| *** rhallisey has joined #openstack-infra | 11:49 | |
| *** ekarlso has joined #openstack-infra | 11:50 | |
| *** maishsk has quit IRC | 11:51 | |
| *** _nadya_ has joined #openstack-infra | 11:51 | |
| *** ddieterly has joined #openstack-infra | 11:51 | |
| *** armax has joined #openstack-infra | 11:53 | |
| *** rfolco has joined #openstack-infra | 11:53 | |
| *** coolsvap_ has joined #openstack-infra | 11:53 | |
| *** salv-orl_ has quit IRC | 11:53 | |
| *** coolsvap has quit IRC | 11:53 | |
| *** weshay has joined #openstack-infra | 11:54 | |
| *** sree has joined #openstack-infra | 11:54 | |
| *** daemontool_ has joined #openstack-infra | 11:54 | |
| *** daemontool has quit IRC | 11:55 | |
| *** eharney has joined #openstack-infra | 11:55 | |
| *** tpsilva has joined #openstack-infra | 11:56 | |
| *** banix has joined #openstack-infra | 11:56 | |
| *** coolsvap has joined #openstack-infra | 11:57 | |
| *** rhallisey has quit IRC | 11:57 | |
| *** rhallisey has joined #openstack-infra | 11:58 | |
| *** coolsvap_ has quit IRC | 11:58 | |
| *** ddieterly has quit IRC | 11:59 | |
| *** markvoelker has joined #openstack-infra | 11:59 | |
| *** dprince has joined #openstack-infra | 12:01 | |
| *** Srinu has joined #openstack-infra | 12:01 | |
| Srinu | fungi: hai | 12:01 |
| Srinu | jeblair:hi | 12:02 |
| *** lucas-hungry is now known as lucas-afk | 12:04 | |
| *** achanda has joined #openstack-infra | 12:06 | |
| Srinu | Hi. when i am running jenkins job in Jenkins CI while the test is in progress the slave went offline so my test report failure | 12:07 |
| *** snarwade|busy is now known as snarwade | 12:07 | |
| *** coolsvap has quit IRC | 12:08 | |
| *** eezhova has quit IRC | 12:09 | |
| *** achanda has quit IRC | 12:10 | |
| *** maishsk has joined #openstack-infra | 12:12 | |
| *** kgiusti has joined #openstack-infra | 12:12 | |
| *** aysyd has joined #openstack-infra | 12:13 | |
| *** zeih has quit IRC | 12:13 | |
| *** sshnaidm has quit IRC | 12:14 | |
| *** eezhova has joined #openstack-infra | 12:14 | |
| AJaeger | sdague: could you review some time this change for project-config? I fear it removes too many lines... https://review.openstack.org/329305 | 12:17 |
| *** zeih has joined #openstack-infra | 12:17 | |
| sdague | AJaeger: well, if they think it works for them | 12:18 |
| sdague | the real trick of it is what's in their devstackgaterc I think | 12:18 |
| *** sshnaidm has joined #openstack-infra | 12:18 | |
| *** ramishra has quit IRC | 12:19 | |
| *** lvdongbing has quit IRC | 12:21 | |
| *** jamesmcarthur has joined #openstack-infra | 12:21 | |
| *** jcoufal has joined #openstack-infra | 12:21 | |
| *** pradk has joined #openstack-infra | 12:21 | |
| *** sarob has joined #openstack-infra | 12:22 | |
| *** binbincong has quit IRC | 12:22 | |
| AJaeger | I would have expected the enable_plugin stay in project-config | 12:22 |
| *** amotoki has quit IRC | 12:23 | |
| AJaeger | sdague: seems they moved everything over: https://review.openstack.org/#/c/328941/2/devstack/devstackgaterc | 12:23 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: Add pytest_results.html to test-results publisher https://review.openstack.org/333226 | 12:24 |
| *** ddieterly has joined #openstack-infra | 12:24 | |
| *** amoralej|lunch is now known as amoralej | 12:25 | |
| EmilienM | jeblair, pabelanger: hey, could you log into nodes that fail (re: puppet ci, async issue) | 12:26 |
| *** sarob has quit IRC | 12:26 | |
| *** jcoufal_ has joined #openstack-infra | 12:26 | |
| *** amitgandhinz has joined #openstack-infra | 12:29 | |
| *** jcoufal has quit IRC | 12:29 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Added pre_hook for sahara cli tests https://review.openstack.org/333315 | 12:30 |
| *** erlon has joined #openstack-infra | 12:30 | |
| *** ramishra has joined #openstack-infra | 12:30 | |
| *** thiagop-afk has joined #openstack-infra | 12:31 | |
| *** thiagop-afk is now known as thiagop | 12:32 | |
| *** ddieterly has quit IRC | 12:32 | |
| *** lvdongbing has joined #openstack-infra | 12:33 | |
| Srinu | AJaeger: Hi. when i am running jenkins job in Jenkins CI while the test is in progress the slave went offline so my test report failure | 12:33 |
| *** armax has quit IRC | 12:33 | |
| *** amitgandhinz has quit IRC | 12:34 | |
| *** trown|outtypewww is now known as trown | 12:34 | |
| *** kushal has joined #openstack-infra | 12:38 | |
| *** matt-borland has joined #openstack-infra | 12:39 | |
| *** jamesmcarthur has quit IRC | 12:40 | |
| *** psilvad has joined #openstack-infra | 12:41 | |
| *** yamamoto_ has quit IRC | 12:41 | |
| *** ramishra has quit IRC | 12:41 | |
| *** ccamacho is now known as ccamacho|lunch | 12:42 | |
| *** banix has quit IRC | 12:44 | |
| *** kushal has quit IRC | 12:44 | |
| *** kushal has joined #openstack-infra | 12:46 | |
| *** pradk has quit IRC | 12:46 | |
| *** kushal has quit IRC | 12:47 | |
| *** kushal has joined #openstack-infra | 12:47 | |
| *** salv-orlando has joined #openstack-infra | 12:47 | |
| *** sdake has joined #openstack-infra | 12:48 | |
| *** tongli has joined #openstack-infra | 12:48 | |
| *** javeriak has quit IRC | 12:48 | |
| *** Ravikiran_K has joined #openstack-infra | 12:50 | |
| *** javeriak has joined #openstack-infra | 12:51 | |
| *** asselin has joined #openstack-infra | 12:51 | |
| *** rlandy has joined #openstack-infra | 12:52 | |
| *** nwkarsten has joined #openstack-infra | 12:53 | |
| *** zeih has quit IRC | 12:53 | |
| *** signed8bit has joined #openstack-infra | 12:54 | |
| *** garyk1 has quit IRC | 12:55 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 12:55 |
| *** tlian has joined #openstack-infra | 12:55 | |
| *** denisra has quit IRC | 12:55 | |
| *** hferenc has quit IRC | 12:56 | |
| *** xyang1 has joined #openstack-infra | 12:56 | |
| *** edmondsw has joined #openstack-infra | 12:57 | |
| *** zeih has joined #openstack-infra | 12:57 | |
| *** hosanai has quit IRC | 12:57 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts https://review.openstack.org/331148 | 12:59 |
| *** baoli has joined #openstack-infra | 13:01 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs https://review.openstack.org/246433 | 13:02 |
| *** shardy has quit IRC | 13:03 | |
| *** eharney has quit IRC | 13:03 | |
| *** Srinu has quit IRC | 13:08 | |
| *** lezbar__ has joined #openstack-infra | 13:08 | |
| *** mdrabe has joined #openstack-infra | 13:09 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:09 | |
| *** ociuhandu has quit IRC | 13:09 | |
| *** rhochmuth has joined #openstack-infra | 13:11 | |
| *** sree has quit IRC | 13:11 | |
| *** rhochmuth has quit IRC | 13:11 | |
| *** lezbar has quit IRC | 13:12 | |
| *** zz_dimtruck is now known as dimtruck | 13:15 | |
| *** yaume has joined #openstack-infra | 13:15 | |
| *** Srinu has joined #openstack-infra | 13:16 | |
| Srinu | Hi | 13:16 |
| mordred | Shrews, infra-root: releasing shade 1.10.0 in just a few unless someone screams | 13:16 |
| Shrews | mordred: k | 13:17 |
| Srinu | I am running jenkins jobs when the build is in progress the slave went offline this is the log of the slave http://paste.openstack.org/show/521666/ | 13:17 |
| Srinu | anyone face this issue before | 13:18 |
| Srinu | mordred:hi | 13:18 |
| mordred | hey Srinu ... I'm not really awake yet | 13:19 |
| *** signed8bit is now known as signed8bit_Zzz | 13:21 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:21 | |
| *** psilvad has quit IRC | 13:21 | |
| *** signed8bit is now known as signed8bit_Zzz | 13:21 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 13:24 |
| *** akshai has joined #openstack-infra | 13:25 | |
| *** pgadiya has quit IRC | 13:26 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:26 | |
| *** sree has joined #openstack-infra | 13:28 | |
| dims | AJaeger : looks like we have 2 +2's on https://review.openstack.org/#/c/332447/, can you please +W it? | 13:28 |
| *** coolsvap has joined #openstack-infra | 13:29 | |
| *** rossella__ has joined #openstack-infra | 13:30 | |
| *** _ari_ has joined #openstack-infra | 13:30 | |
| *** claudiub has quit IRC | 13:31 | |
| *** jistr|mtg is now known as jistr | 13:31 | |
| *** mriedem has joined #openstack-infra | 13:32 | |
| *** david-lyle_ has joined #openstack-infra | 13:32 | |
| *** inc0 has joined #openstack-infra | 13:32 | |
| dims | thanks! AJaeger | 13:33 |
| *** baoli_ has joined #openstack-infra | 13:33 | |
| *** ddieterly has joined #openstack-infra | 13:33 | |
| *** psilvad has joined #openstack-infra | 13:33 | |
| *** ccamacho|lunch is now known as ccamacho | 13:33 | |
| *** eharney has joined #openstack-infra | 13:33 | |
| *** esikachev has quit IRC | 13:34 | |
| SamYaple | morning | 13:34 |
| *** yolanda has joined #openstack-infra | 13:34 | |
| *** piet has joined #openstack-infra | 13:34 | |
| *** baoli has quit IRC | 13:35 | |
| *** achanda has joined #openstack-infra | 13:35 | |
| *** david-lyle has quit IRC | 13:36 | |
| *** hdaniel has joined #openstack-infra | 13:36 | |
| *** mriedem1 has joined #openstack-infra | 13:36 | |
| *** jpeeler has joined #openstack-infra | 13:37 | |
| *** esberglu has joined #openstack-infra | 13:37 | |
| openstackgerrit | Merged openstack-infra/project-config: import the release tools that need to run on secure nodes https://review.openstack.org/332447 | 13:37 |
| AJaeger | dims: you're welcome | 13:38 |
| AJaeger | mordred: you're releasing shade while not fully awake? ;) | 13:38 |
| * AJaeger sends some coffee to mordred | 13:38 | |
| *** mriedem1 has left #openstack-infra | 13:38 | |
| *** kushal has quit IRC | 13:39 | |
| *** mriedem has quit IRC | 13:39 | |
| mordred | AJaeger: mmm. coffee | 13:40 |
| *** salv-orlando has quit IRC | 13:40 | |
| *** javeriak has quit IRC | 13:41 | |
| *** salv-orlando has joined #openstack-infra | 13:41 | |
| *** mriedem has joined #openstack-infra | 13:41 | |
| *** dizquierdo has joined #openstack-infra | 13:41 | |
| tosky | AJaeger, yolanda: thanks for the help and sorry for all the changes to Sahara jobs, we are learning on the way and we almost done hopefully | 13:42 |
| pabelanger | morning | 13:42 |
| pabelanger | EmilienM: we need to enable more ansible debug logs, which we can enable today. jeblair landed a patch last night to help with the debugging | 13:43 |
| *** hashar has quit IRC | 13:43 | |
| pabelanger | rolling restarts are still on going | 13:43 |
| pabelanger | currently on zl05 | 13:43 |
| yolanda | hi tosky , no problem | 13:44 |
| EmilienM | pabelanger: ok | 13:44 |
| EmilienM | pabelanger: I noticed it's mostly on xenial jobs if not only | 13:44 |
| EmilienM | pabelanger: did you figure the same? | 13:44 |
| *** dimtruck is now known as zz_dimtruck | 13:44 | |
| pabelanger | EmilienM: yes, we picked that node yesterday too. | 13:44 |
| EmilienM | pabelanger: ok | 13:44 |
| EmilienM | pabelanger: did you see this thing on centos7 jobs? | 13:45 |
| bkero | Have we succeeded in keeping a failing node around to debug? | 13:45 |
| mordred | EmilienM: it's been a hard enough problem that we've now added a feature to zuul and a feature to nodepool to allow us to debug it | 13:45 |
| mordred | :) | 13:45 |
| mordred | bkero: we have! | 13:45 |
| bkero | \o/ | 13:45 |
| mordred | bkero: jeblair added a feature to tell nodepool not to delete X nodes after job Y fails | 13:45 |
| pabelanger | EmilienM: I only seen xenial yesterday, but I didn't look much into centos-7 | 13:45 |
| bkero | but I'm guessing no root cause analysis yet | 13:45 |
| *** rbrndt has joined #openstack-infra | 13:45 | |
| bkero | mordred: that sounds like a super useful feature for debugging failing jobs | 13:46 |
| *** achanda has quit IRC | 13:46 | |
| mordred | yup | 13:46 |
| mordred | we got a node yesterday, but after digging in needed the debug logs to be on | 13:46 |
| mordred | so jeblair then added a feature to zuul to allow us to toggle debug logs without having to restart zuul | 13:46 |
| pabelanger | bkero: clarkb creating this PR, https://github.com/ansible/ansible-modules-core/pull/4017 but I don't believe we have tested it yet | 13:46 |
| EmilienM | mordred: ack, thx | 13:47 |
| EmilienM | pabelanger, mordred: let me 5 min to dig all our CI jobs and see if centos7 has same issue | 13:47 |
| bkero | pabelanger: fun, I wonder when we'll get that in a released version | 13:48 |
| EmilienM | pabelanger, mordred: it's really on xenial | 13:48 |
| openstackgerrit | Merged openstack-infra/project-config: Added pre_hook for sahara cli tests https://review.openstack.org/333315 | 13:48 |
| bkero | and how many times that loop runs | 13:49 |
| *** coolsvap has quit IRC | 13:49 | |
| *** ramishra has joined #openstack-infra | 13:49 | |
| pabelanger | bkero: https://review.openstack.org/#/c/328649/ gives us a better way to track the loop runs, but still up for discussion if we want to do that | 13:50 |
| *** PollyZ has joined #openstack-infra | 13:50 | |
| *** rhochmuth has joined #openstack-infra | 13:50 | |
| *** coolsvap has joined #openstack-infra | 13:50 | |
| pabelanger | Would like to stabilize async_wrapper first | 13:51 |
| *** hashar has joined #openstack-infra | 13:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Use Smaug configuration from Smaug's repository https://review.openstack.org/329305 | 13:52 |
| openstackgerrit | Merged openstack-infra/project-config: panko: enable functional testing https://review.openstack.org/333227 | 13:52 |
| *** oferby has quit IRC | 13:53 | |
| Shrews | mordred: pabelanger: did anyone try executing the jobs locally now that we keep the ansible files around? | 13:53 |
| *** Ravikiran_K has quit IRC | 13:53 | |
| bkero | I still think the additional logging will be useful, even after async_wrapper is fixed | 13:53 |
| bkero | Whenever I see smaug I still think of https://github.com/bkero/smaug | 13:54 |
| pabelanger | Shrews: I didn't have much time yesterday to work on the held node. | 13:54 |
| *** hashar has quit IRC | 13:54 | |
| *** rossella__ has quit IRC | 13:55 | |
| *** itisha has joined #openstack-infra | 13:55 | |
| *** esberglu has quit IRC | 13:55 | |
| *** bhavik has joined #openstack-infra | 13:55 | |
| *** ddieterly is now known as ddieterly[away] | 13:56 | |
| *** yolanda has quit IRC | 13:56 | |
| *** esikachev has joined #openstack-infra | 13:56 | |
| *** pt_15 has joined #openstack-infra | 13:57 | |
| Shrews | i'd be very interested to see if zuul_runner exits the same when run locally | 13:57 |
| *** pradk has joined #openstack-infra | 13:57 | |
| *** zul_ has joined #openstack-infra | 13:59 | |
| *** ddieterly[away] is now known as ddieterly | 13:59 | |
| *** hdaniel has quit IRC | 13:59 | |
| *** zul has quit IRC | 14:00 | |
| *** rbradfor has joined #openstack-infra | 14:00 | |
| fungi | mordred: planning to stamp out a new glean too? i think cinerama and some others were hoping for a release with some recently merged fixes | 14:01 |
| *** sree has quit IRC | 14:01 | |
| *** zul_ has quit IRC | 14:01 | |
| *** esberglu has joined #openstack-infra | 14:02 | |
| *** sree has joined #openstack-infra | 14:02 | |
| *** zul has joined #openstack-infra | 14:02 | |
| *** rbradfor has quit IRC | 14:02 | |
| *** esberglu has quit IRC | 14:02 | |
| openstackgerrit | Merged openstack-infra/project-config: [magnum] Add install-guide job https://review.openstack.org/333165 | 14:03 |
| *** sree has quit IRC | 14:03 | |
| *** sree has joined #openstack-infra | 14:04 | |
| *** achanda has joined #openstack-infra | 14:05 | |
| *** rbradfor has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove Heat lbaasv1 from master https://review.openstack.org/333200 | 14:05 |
| openstackgerrit | Merged openstack-infra/project-config: Run Heat functional tests in Identity v3-only https://review.openstack.org/314643 | 14:05 |
| openstackgerrit | Merged openstack-infra/project-config: Don't run tempest tests in zaqar gate https://review.openstack.org/333184 | 14:06 |
| *** sree has quit IRC | 14:06 | |
| *** achanda has quit IRC | 14:07 | |
| *** sree has joined #openstack-infra | 14:07 | |
| *** shardy has joined #openstack-infra | 14:09 | |
| *** sree has quit IRC | 14:11 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 14:12 |
| *** woodster_ has joined #openstack-infra | 14:12 | |
| *** zqfan has quit IRC | 14:13 | |
| *** sdake has quit IRC | 14:13 | |
| *** zhurong has quit IRC | 14:15 | |
| *** kdas__ has joined #openstack-infra | 14:16 | |
| *** sdake has joined #openstack-infra | 14:16 | |
| pabelanger | jeblair: mordred: after 12 hours, all our zuul-launchers are now graceful restarted | 14:17 |
| pabelanger | happy that our playbook worked for 12 hours | 14:17 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in heat https://review.openstack.org/315214 | 14:18 |
| pabelanger | little sad it took 12 hours | 14:18 |
| *** zhurong has joined #openstack-infra | 14:18 | |
| *** baoli_ has quit IRC | 14:19 | |
| *** rhochmuth has quit IRC | 14:19 | |
| fungi | probably gets complicated to tell ansible to do them in parallel but not to have more than n launchers in graceful shutdown at a time? | 14:20 |
| *** baoli has joined #openstack-infra | 14:21 | |
| mordred | fungi: ansible actually knows about batching rolling things | 14:21 |
| mordred | fungi: so that shoudl be very easy | 14:21 |
| fungi | so when things are quiet, we could tell it to do up to 4 at a time? that would be handyu | 14:22 |
| *** eezhova has quit IRC | 14:22 | |
| mordred | fungi: serial: 1 | 14:22 |
| *** piet has quit IRC | 14:22 | |
| *** bhavik has quit IRC | 14:23 | |
| *** zz_dimtruck is now known as dimtruck | 14:23 | |
| mordred | fungi: is the line | 14:23 |
| *** amotoki has joined #openstack-infra | 14:23 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names https://review.openstack.org/282660 | 14:23 |
| *** amitgandhinz has joined #openstack-infra | 14:23 | |
| fungi | so that's an actual integer knob, not just a boolean flag? | 14:23 |
| *** kdas__ is now known as kushal | 14:23 | |
| mordred | yup | 14:23 |
| *** psilvad_ has joined #openstack-infra | 14:23 | |
| *** kushal has quit IRC | 14:23 | |
| *** kushal has joined #openstack-infra | 14:23 | |
| fungi | neat-o | 14:23 |
| *** psilvad has quit IRC | 14:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Make batch size configurable for z-l restart https://review.openstack.org/333377 | 14:25 |
| mordred | fungi: ^^ untested | 14:25 |
| EmilienM | puppet xenial jobs are too unstable for us, we need to land some patches and it's blocked I think I'm going to -nv them | 14:29 |
| pabelanger | mordred: added comment about using strategy | 14:30 |
| fungi | EmilienM: i've not been following closely unfortunately. have any idea what's unstable there? | 14:30 |
| EmilienM | fungi: async thing with zuul | 14:31 |
| fungi | EmilienM: oh, that. and it's only on xenial? | 14:31 |
| fungi | doesn't affect trusty? | 14:31 |
| fungi | weird | 14:31 |
| EmilienM | fungi: we don't run much trusty nodes lately | 14:32 |
| EmilienM | fungi: trusty nodes are run for mitaka/liberty branches | 14:32 |
| *** amotoki has quit IRC | 14:32 | |
| EmilienM | fungi: but since newton we deploy on xenial | 14:32 |
| *** salv-orlando has quit IRC | 14:35 | |
| *** salv-orlando has joined #openstack-infra | 14:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: xenial jobs are now -nv https://review.openstack.org/333381 | 14:36 |
| EmilienM | mwhahaha, degorenko, fungi ^ | 14:37 |
| mordred | pabelanger: oh yeah. totally agree | 14:37 |
| *** ayoung has quit IRC | 14:37 | |
| *** jistr is now known as jistr|mtg | 14:38 | |
| *** pblaho has quit IRC | 14:38 | |
| fungi | EmilienM: and you're not seeing the same instability on centos 7 jobs either? | 14:38 |
| EmilienM | fungi: nope | 14:38 |
| EmilienM | so, since zuul migration, we're having async issue with our integration jobs (they don't run ansible but puppet btw), on xenial nodes | 14:39 |
| *** burgerk has joined #openstack-infra | 14:39 | |
| *** zhurong has quit IRC | 14:40 | |
| fungi | but you run similar integration jobs on centos 7 and they've remained stable through the zuul migration? | 14:40 |
| *** amotoki has joined #openstack-infra | 14:40 | |
| *** inc0 has quit IRC | 14:41 | |
| *** links has quit IRC | 14:41 | |
| *** pcaruana has quit IRC | 14:41 | |
| *** claudiub has joined #openstack-infra | 14:43 | |
| *** baoli has quit IRC | 14:43 | |
| AJaeger | EmilienM: So, ansible+puppet on Xenial = broken is the current theory? | 14:44 |
| AJaeger | s/theory/observation/ | 14:44 |
| EmilienM | yes but this is only an observation, jeblair / pabelanger / mordred are investigating lately and today we should make progress | 14:44 |
| EmilienM | the thing is we could not login a failed node to see what's wrong | 14:44 |
| *** kushal has quit IRC | 14:45 | |
| EmilienM | and now jeblair did that feature in nodepool, we'll be able to ssh a failed node and see what happenned | 14:45 |
| *** jamesmcarthur has joined #openstack-infra | 14:45 | |
| EmilienM | but in the meantime, we can't continue to block our CI | 14:45 |
| EmilienM | newton cycle is going on and we have a backlog of patches to land | 14:45 |
| EmilienM | Fuel & TripleO are depending on Puppet modules, we can't block our CI anymore | 14:45 |
| *** esikachev has quit IRC | 14:45 | |
| AJaeger | making it non-voting will not block our continous testing - and you still have voting jobs, so +2 from my side. | 14:47 |
| mordred | ++ | 14:47 |
| EmilienM | yeah, we still want them to continue the debug | 14:47 |
| EmilienM | just make them non-voting will allow say "ok the patch is passing on centos7, and would also pass on ubuntu, so let's land it" | 14:47 |
| AJaeger | EmilienM: agreed | 14:48 |
| *** baoli has joined #openstack-infra | 14:48 | |
| EmilienM | thx AJaeger, mordred | 14:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs https://review.openstack.org/333065 | 14:48 |
| *** jlanoux has quit IRC | 14:50 | |
| *** inc0 has joined #openstack-infra | 14:51 | |
| *** ramishra has quit IRC | 14:51 | |
| *** coolsvap has quit IRC | 14:52 | |
| *** xarses has quit IRC | 14:52 | |
| *** baoli has quit IRC | 14:52 | |
| *** jaosorior has quit IRC | 14:53 | |
| *** deadnull_ has joined #openstack-infra | 14:53 | |
| *** Nikolay_St has quit IRC | 14:54 | |
| *** baoli has joined #openstack-infra | 14:54 | |
| *** coolsvap has joined #openstack-infra | 14:55 | |
| *** pilgrimstack has joined #openstack-infra | 14:55 | |
| *** armax has joined #openstack-infra | 14:57 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update config file provider docs to have default options https://review.openstack.org/333393 | 14:58 |
| *** deadnull_ has quit IRC | 14:58 | |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| *** david-lyle_ is now known as david-lyle | 14:59 | |
| *** shardy has quit IRC | 14:59 | |
| *** tbarron|errand is now known as tbarron | 15:00 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: xenial jobs are now -nv https://review.openstack.org/333381 | 15:01 |
| *** esikachev has joined #openstack-infra | 15:01 | |
| *** xarses has joined #openstack-infra | 15:01 | |
| *** mhickey has quit IRC | 15:02 | |
| *** markvoelker_ has joined #openstack-infra | 15:03 | |
| *** amotoki has quit IRC | 15:03 | |
| *** markvoelker has quit IRC | 15:03 | |
| *** amit213 has quit IRC | 15:04 | |
| *** Na3iL has quit IRC | 15:05 | |
| *** eezhova has joined #openstack-infra | 15:05 | |
| *** pblaho has joined #openstack-infra | 15:05 | |
| *** tqtran has joined #openstack-infra | 15:06 | |
| *** bhavik has joined #openstack-infra | 15:07 | |
| *** zeih has quit IRC | 15:07 | |
| *** ramishra has joined #openstack-infra | 15:08 | |
| *** shardy has joined #openstack-infra | 15:08 | |
| *** esberglu has joined #openstack-infra | 15:08 | |
| *** amotoki has joined #openstack-infra | 15:08 | |
| *** xarses has quit IRC | 15:10 | |
| *** tqtran has quit IRC | 15:10 | |
| cloudnull | o/ all | 15:10 |
| fungi | cloudnull: all's quiet on the osic front? i hope you're here for a social visit and no a bearer of bad tidings! | 15:12 |
| fungi | er, not a | 15:12 |
| *** eezhova has quit IRC | 15:12 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Store history of a comment when editing https://review.openstack.org/332208 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add a way to retrieve the history of edited comments https://review.openstack.org/333409 | 15:13 |
| cloudnull | fungi: not at all. | 15:13 |
| cloudnull | just here to say hi. | 15:14 |
| cloudnull | been on holiday for a while. so its good to hear the osic front is all quite :) | 15:14 |
| *** sdake_ has joined #openstack-infra | 15:14 | |
| fungi | cloudnull: absolutely. good job! | 15:15 |
| *** pilgrimstack has quit IRC | 15:15 | |
| fungi | you should go on holiday more often ;) | 15:15 |
| * cloudnull puts it on the books | 15:15 | |
| *** baoli has quit IRC | 15:15 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh1 https://review.openstack.org/333411 | 15:15 |
| *** sdake has quit IRC | 15:16 | |
| pabelanger | EmilienM: are we leaking FIPs in tripleo? | 15:16 |
| EmilienM | pabelanger: I discussed with derek yesterday, and he mentionned that he had to manually release floating ips from time to time | 15:17 |
| EmilienM | otherwise our CI would get stuck, since no job could run | 15:17 |
| EmilienM | I think it's related to what I'm trying to do | 15:18 |
| EmilienM | pabelanger: he wanted to setup a crontab that clean this up | 15:18 |
| EmilienM | pabelanger: but I'm sure we can solve it in another way | 15:18 |
| *** amotoki has quit IRC | 15:18 | |
| clarkb | EmilienM: nodepool has a built in cleanup fips thing we can enable on tht cloud | 15:19 |
| *** vhosakot has joined #openstack-infra | 15:19 | |
| clarkb | (assuming it got fixed in shade) | 15:19 |
| pabelanger | EmilienM: not sure I follow "related to what I'm trying to do" comment | 15:19 |
| *** sdake_ has quit IRC | 15:20 | |
| pabelanger | but honestly, I never looked at tripleo-test-cloud-rh1 for leaked IPs | 15:20 |
| *** jlanoux has joined #openstack-infra | 15:20 | |
| EmilienM | clarkb: https://review.openstack.org/333411 | 15:21 |
| EmilienM | clarkb: is it what you mentionned? | 15:21 |
| *** maishsk has quit IRC | 15:21 | |
| *** ddieterly is now known as ddieterly[away] | 15:21 | |
| EmilienM | pabelanger: what I mean is, I looked a nodepool and I found this feature, I thought it could help us to remove unsassigned floating ips | 15:21 |
| *** dizquierdo has quit IRC | 15:21 | |
| clarkb | yes but depends on shade being fixed and I dunno if it goy fixed | 15:21 |
| EmilienM | clarkb: do we have a launchpad or something? | 15:22 |
| greghaynes | ianw: looks like the dib cache change broke our CI | 15:22 |
| pabelanger | clarkb: Yup, shade was released with the fix | 15:22 |
| greghaynes | ianw: http://logs.openstack.org/11/331911/1/check/gate-dib-dsvm-functests-ubuntu-trusty/2d9c95b/console.html | 15:22 |
| clarkb | pabelanger: and we installed it and restarted nodepool? | 15:22 |
| *** dizquierdo has joined #openstack-infra | 15:22 | |
| pabelanger | clarkb: ya, it landed a while back. maybe 3+ weeks | 15:22 |
| clarkb | (nodepool was restarted yeaterday so if release is older we are good) | 15:22 |
| *** _nadya_ has quit IRC | 15:22 | |
| *** shahid__ has quit IRC | 15:23 | |
| EmilienM | https://github.com/openstack-infra/shade/commit/b1343ac6b65ac9b0423ad78c432f59cb1273f79f | 15:23 |
| EmilienM | this one? | 15:23 |
| *** baoli has joined #openstack-infra | 15:23 | |
| mordred | clarkb: can't use it with tripleo-ci | 15:23 |
| EmilienM | ah no https://github.com/openstack-infra/shade/commit/58faf12cc2092653685d08082f888cfa295678ba | 15:23 |
| pabelanger | clarkb: oh, maybe I am confusing with another fix | 15:23 |
| EmilienM | mordred: why? | 15:23 |
| *** baoli has quit IRC | 15:23 | |
| mordred | because we use nova api to do floating ips on the tripleo cloud | 15:23 |
| mordred | and it's not safe to run the cleanup fips with the nova api | 15:23 |
| clarkb | oh right | 15:23 |
| pabelanger | Ah, right. For got about that | 15:24 |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Make dib_cache dir with sudo in dib jobs https://review.openstack.org/333415 | 15:24 |
| greghaynes | ianw: ^ | 15:24 |
| EmilienM | damn, so I can abandon my patch? | 15:24 |
| mordred | we use the nova api because the neutron api for create/attach does not work on the tripleo cloud | 15:24 |
| *** ddieterly[away] is now known as ddieterly | 15:24 | |
| mordred | yah. sorry | 15:24 |
| EmilienM | mordred: can you comment on https://review.openstack.org/#/c/333411/ please? | 15:24 |
| mordred | EmilienM: I _think_ the neutron api doesn't work because tripleo-test-cloud-rh1 control plane is old and there is a bug? | 15:24 |
| *** baoli has joined #openstack-infra | 15:25 | |
| EmilienM | mordred: that's possible, I have no access on it, so who know? | 15:25 |
| mordred | EmilienM: I don't know if you're going to deploy a new control plane with the rh2 cloud or if it's just a move | 15:25 |
| EmilienM | dprince: can you confirm? | 15:25 |
| *** shardy has quit IRC | 15:25 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health: Add build_newrev to the list of ignored keys https://review.openstack.org/328467 | 15:25 |
| pabelanger | rh2 is neutron, according to clouds.yaml for nodepool | 15:25 |
| EmilienM | cool, so I -1 https://review.openstack.org/#/c/330514/ | 15:26 |
| EmilienM | we can add the boolean ^ | 15:26 |
| pabelanger | tested launch-node.py on it yesterday too, for AFS mirror | 15:26 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts https://review.openstack.org/331148 | 15:26 |
| pabelanger | worked the first time | 15:26 |
| EmilienM | pabelanger: see the patch for rh2, not goot then | 15:26 |
| mtreinish | nibalizer: ^^^ \o/ it finally landed | 15:26 |
| pabelanger | EmilienM: maybe a follow up patch? | 15:26 |
| mordred | pabelanger: cool - so clouds.yaml for rh2 doesn't have the nova override and auto_ip in launch_node worked properly? | 15:27 |
| mordred | that's great new | 15:27 |
| mordred | news | 15:27 |
| pabelanger | EmilienM: otherwise, we can start landing rh2 today | 15:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Allow comments to be edited by their owner https://review.openstack.org/333418 | 15:27 |
| EmilienM | pabelanger: ok, you can +A, and I'm proposing a following patch | 15:27 |
| pabelanger | mordred: Ya, this was their patch: https://review.openstack.org/#/c/330833/ | 15:27 |
| pabelanger | which I used with launch-node.py | 15:28 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs https://review.openstack.org/246433 | 15:28 |
| clarkb | pabelanger: and it floating IP'd and all that? | 15:28 |
| mordred | awesome | 15:28 |
| pabelanger | clarkb: Ya | 15:28 |
| mordred | so we shoudl be able to enable clean-fips for that cloud in nodepool.yaml then | 15:29 |
| pabelanger | http://mirror.regionone.tripleo-test-cloud-rh2.openstack.org/ too | 15:29 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2 https://review.openstack.org/333419 | 15:29 |
| EmilienM | pabelanger: ^ done | 15:29 |
| pabelanger | worked first try, which made me happy | 15:29 |
| pabelanger | okay, if no objections, I'll help get rh2 online today | 15:30 |
| pabelanger | I think we need to restart nodepool-builder too | 15:30 |
| EmilienM | pabelanger: excellent | 15:30 |
| mordred | pabelanger: it's almost like our automation works or something | 15:30 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2 https://review.openstack.org/333419 | 15:31 |
| *** sdake has joined #openstack-infra | 15:31 | |
| pabelanger | mordred: I can abandon my research on cloning | 15:32 |
| mordred | :) | 15:33 |
| *** salv-orlando has quit IRC | 15:33 | |
| *** ldnunes has quit IRC | 15:33 | |
| *** sdake_ has joined #openstack-infra | 15:34 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: stop merging tags from stable branches into master https://review.openstack.org/333421 | 15:34 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts https://review.openstack.org/331148 | 15:34 |
| *** coolsvap has quit IRC | 15:35 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts https://review.openstack.org/331148 | 15:36 |
| *** PollyZ has quit IRC | 15:36 | |
| *** PollyZ has joined #openstack-infra | 15:36 | |
| *** sdake has quit IRC | 15:36 | |
| *** ayoung has joined #openstack-infra | 15:37 | |
| *** Rockyg has joined #openstack-infra | 15:37 | |
| *** jamesmcarthur has quit IRC | 15:37 | |
| *** cody-somerville has joined #openstack-infra | 15:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a temporary tripleo cloud "rh2" https://review.openstack.org/330514 | 15:45 |
| *** ramishra has quit IRC | 15:45 | |
| *** ramishra has joined #openstack-infra | 15:46 | |
| *** dmk0202 has quit IRC | 15:46 | |
| *** jamesmcarthur has joined #openstack-infra | 15:46 | |
| *** coolsvap has joined #openstack-infra | 15:47 | |
| *** psachin has quit IRC | 15:47 | |
| _alastor_ | Hey, is this the right place to ask about the ci-watch tintri pages? | 15:48 |
| *** armax has quit IRC | 15:48 | |
| *** nwkarsten has quit IRC | 15:49 | |
| *** nwkarsten has joined #openstack-infra | 15:49 | |
| BobBall | I think so | 15:50 |
| _alastor_ | http://ci-watch.tintri.com/project?project=cinder&time=24+hours | 15:51 |
| _alastor_ | The cinder one hasn't posted any results here in at least 24-hours | 15:51 |
| mmedvede | _alastor_: the person who is supporting ci-watch deployment at tintri is apoorvad | 15:51 |
| mmedvede | but he is not on irc right now | 15:51 |
| _alastor_ | mmedvede: ok, I'll check with him when he comes online | 15:52 |
| _alastor_ | mmedvede: in the meantime, do we have an alternative place to look at the results? | 15:53 |
| jeblair | pabelanger, mordred, Shrews: good morning! | 15:53 |
| jeblair | Shrews: in the errors we are seeing, zuul_runner exits correctly at the end of the job and before the timeout. it's just that ansible erroneously gives up before that happens. | 15:53 |
| asselin | rcarrillocruz, is infra still using git+git://git.openstack.org/openstack-infra/ansible-role-puppet.git now that there's no puppetmaster? | 15:53 |
| *** ramishra has quit IRC | 15:53 | |
| pabelanger | jeblair: morning | 15:54 |
| mordred | asselin: yes | 15:54 |
| mordred | asselin: we use it to run puppet apply | 15:54 |
| mmedvede | _alastor_: I am not aware of any other place. I was working on getting ci-watch deployed on an infra VM, but very busy with other stuff right now | 15:54 |
| asselin | mordred, ok, docs state it requires a puppetmaster, maybe just out of date. http://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/README.md#n25 | 15:55 |
| fungi | _alastor_: i'm guessing not. it isn't currently managed by the infra team so there's not a lot of help we can provide. http://specs.openstack.org/openstack-infra/infra-specs/specs/deploy-ci-dashboard.html is the plan for solving that but i'm not sure how far along it's gotten | 15:55 |
| mordred | asselin: ah - one sec | 15:55 |
| jeblair | ianw, pabelanger: i don't care what creates the user on the image, as long as it's not system-config puppet. it should be completely self-contained and usable by both us and others. we need a good story for "if you want to use zuul with nodepool, you should create dib images like this..." personally, i think it would be better for the system as a whole to not depend on both ansible *and* puppet, so i'd rather it be created some ... | 15:56 |
| jeblair | ... other way. personally, as a dib user, i'd be pretty surprised if there wasn't a nice way to get a user on a disk image -- i mean, that's very much in the wheelhouse of something that creates disk images i think. | 15:56 |
| *** andreas_s has quit IRC | 15:56 | |
| mmedvede | fungi: there is puppet code that deploys ci-watch here https://review.openstack.org/#/c/238606/, need to get it merged | 15:56 |
| _alastor_ | fungi: Thanks. I'll keep an eye on that spec | 15:57 |
| asselin | mordred, rcarrillocruz fyi, i'm interested in using this: https://review.openstack.org/#/c/320466/13 so looking for docs/starting points to make it happen in my env. | 15:57 |
| *** xarses has joined #openstack-infra | 15:57 | |
| pabelanger | jeblair: completely agree | 15:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet: Update README with info about puppet apply https://review.openstack.org/333459 | 15:58 |
| mordred | asselin, rcarrillocruz ^^ | 15:58 |
| openstackgerrit | Merged openstack-infra/project-config: Clean floating IPs on tripleo-test-cloud-rh2 https://review.openstack.org/333419 | 15:58 |
| *** ldnunes has joined #openstack-infra | 15:58 | |
| dprince | EmilienM: I'd have to check. Our neutron is very old at this point | 15:58 |
| mordred | jeblair: yah- and the devuser element exists and whatnot for getting a user and a public key on | 15:59 |
| mordred | dprince: is the rh2 a new install? | 15:59 |
| asselin | mordred, adding an example playbook for puppet apply at the end would be helpful | 15:59 |
| mordred | asselin: agree | 15:59 |
| dprince | mordred: yes, we should just wait to use that | 15:59 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add new db api method to get recent successes by metadata https://review.openstack.org/333460 | 15:59 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs https://review.openstack.org/333065 | 15:59 |
| mordred | dprince: cool. yah - the neutron fip issue we see with rh1 is not present in rh2 - so I was assuming it was newer :) | 15:59 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Fix preseed script for the brave new world https://review.openstack.org/333461 | 15:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add a "ptl" command to show info about how to contact a PTL https://review.openstack.org/333462 | 16:01 |
| *** bhavik has quit IRC | 16:02 | |
| *** Nikolay_St has joined #openstack-infra | 16:03 | |
| *** markvoelker has joined #openstack-infra | 16:03 | |
| pabelanger | my only concern about devuser (and DIB elements in general) is the potential creep of configuration management that could be added into elements. I do believe tooling like puppet and ansible are better for that. And honestly, I feel if we had an run-ansible-playbook element we'd likely start adding playbooks for our zuul worker elements | 16:03 |
| *** markvoelker_ has quit IRC | 16:04 | |
| *** pilgrimstack has joined #openstack-infra | 16:05 | |
| *** e0ne has quit IRC | 16:05 | |
| *** jcooley has joined #openstack-infra | 16:05 | |
| greghaynes | I wouldnt make that an element, but if you want to run ansible in dib it should be trivial (just... run ansible). If theres a more consistent API wanted for that though it'd probably need to go in dib directly | 16:06 |
| mordred | greghaynes: I | 16:06 |
| mordred | greghaynes: I've sketched out a design for it a few times in my head ... the nice thing with dib-ansible is ansible's chroot connection plugin | 16:07 |
| mordred | which should make it pretty cool to deal with | 16:07 |
| mordred | greghaynes: I should maybe write down the things that are in my brain | 16:07 |
| pabelanger | Ya, I haven't tried chroot with ansible, but it does look interesting | 16:07 |
| greghaynes | Yea, it makes a lot of sense. Where itll get wierd is how you make it interleave with the existing script ordering | 16:07 |
| *** tphummel has joined #openstack-infra | 16:08 | |
| greghaynes | e.g. I want this playbook run before install.d/50-bar | 16:08 |
| mordred | greghaynes: yah | 16:09 |
| jeblair | if you look at this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n738 | 16:10 |
| jeblair | there's a list of elements there, that pretty clearly says "to make a fedora-23 image for nodepool, build it up from these parts" | 16:11 |
| greghaynes | I do disagree that elements and config management are somehow exclusive though, thats how the existing config mgmt stuff works. Really its a question of whether youre using elements for composition or youre using an element to call out to something else doing your composition. I think the former is a ton less complex so not sure why that isnt the goal | 16:11 |
| jeblair | and by and large, it's pretty legible | 16:11 |
| zaro | morning | 16:11 |
| jeblair | you can tell what each one is doing, and why you would want it on your nodepool image | 16:11 |
| jeblair | except 'puppet' | 16:11 |
| jeblair | that's sort of 'magic goes here' | 16:12 |
| greghaynes | Yep, exactly what I'm thinking :) | 16:12 |
| jeblair | so i'd love it if that went away, and we had 'install-zuul-user'... and, honestly, hopefully not much else :) | 16:12 |
| *** bhavik has joined #openstack-infra | 16:13 | |
| *** dtantsur is now known as dtantsur|afk | 16:14 | |
| pabelanger | greghaynes: for me, I've always seen elements used to create the base image. And call out to do configuration using another tool. | 16:16 |
| *** yamahata has joined #openstack-infra | 16:16 | |
| dhellmann | are folks aware of "disk quota exceeded" failures in the wheel jobs? | 16:16 |
| dhellmann | http://logs.openstack.org/83/83570952a107c37bbc3b27f51c18663c598d3973/release/wheel-build-centos-7-amd64-python2/eda5308/console.html#_2016-06-23_14_53_51_001597 | 16:16 |
| dhellmann | we're seeing a lot of failures on those now that the release team is watching the output by email | 16:17 |
| fungi | dhellmann: no, but i'll have a look on the server and see if it's piled up some cruft | 16:17 |
| *** tesseract- has quit IRC | 16:17 | |
| greghaynes | pabelanger: Thats not really realted to the issue is what I am getting at - The API for an element is intentionally 'something that is executable' so you can call out to whatever tool works best, the real issue is that composition needs to happen from elements not one element that composes all the things | 16:17 |
| dhellmann | fungi: I haven't done an exhaustive review to see if that's the cause of all of the failures | 16:18 |
| *** dosaboy has quit IRC | 16:18 | |
| fungi | dhellmann: it shouldn't impact the release process, but it does imply that we're not building any new centos 7 wheels in our wheel mirrors (unrelated to publishing and mirroring wheels of our releases on pypi) | 16:19 |
| dhellmann | fungi : yeah, I was more worried about the release folks ignoring failure emails because "oh, it's always that wheel build failure" | 16:19 |
| *** raunak has joined #openstack-infra | 16:19 | |
| *** raunak has quit IRC | 16:20 | |
| dhellmann | I mean, I do also want it to work right for its own sake. :-) | 16:20 |
| fungi | dhellmann: interestingly, we've got tons of free disk space on that server at the moment, so presumably the build itself eats a lot of space or is relying on a very small filesystem for something | 16:20 |
| dhellmann | well, it's a "quota exceeded" error, not a disk full error | 16:20 |
| dhellmann | so I wonder if there's a misaligned config somewhere? | 16:20 |
| clarkb | check the afs volume maybe | 16:20 |
| fungi | oh! | 16:21 |
| fungi | this is afs quota | 16:21 |
| fungi | "k5start: unable to create PAG: Disk quota exceeded" | 16:21 |
| *** sdake_ is now known as sdake | 16:21 | |
| fungi | not disk _space_ exceeded | 16:21 |
| fungi | i glossed over the word quota earlier | 16:21 |
| fungi | so we presumably just need to increase our quota for the wheel mirrors? | 16:21 |
| fungi | checking our afs docs now to see if that's something we have details on | 16:22 |
| jeblair | that console log file is large. | 16:23 |
| * jeblair kills firefox | 16:23 | |
| fungi | looks like we do `fs setquota /afs/.openstack.org/mirror/something <some number of bytes>` | 16:23 |
| *** maishsk has joined #openstack-infra | 16:24 | |
| *** amotoki has joined #openstack-infra | 16:24 | |
| jeblair | fungi: i'm a little confused about getting that error from k5start | 16:24 |
| *** jamesmcarthur has quit IRC | 16:24 | |
| pabelanger | our AFS quotas look to be okay | 16:24 |
| pabelanger | http://paste.openstack.org/show/521705/ | 16:24 |
| pabelanger | I may have run the incorrect command | 16:25 |
| fungi | is there a special meta-volume k5start is using? | 16:25 |
| *** ifarkas has quit IRC | 16:25 | |
| jeblair | fungi: k5start's job is to get a kerberos tgt, and afs service ticket, and create a linux PAG group to hold them | 16:25 |
| fungi | yeah, that quota looks sane | 16:25 |
| fungi | so it's when trying to write the pag that it's dying on this | 16:26 |
| *** raunak has joined #openstack-infra | 16:26 | |
| jeblair | where do those get written? (tmp?) | 16:26 |
| fungi | that's what i'm looking for | 16:26 |
| *** Apoorva has joined #openstack-infra | 16:27 | |
| *** jamesmcarthur has joined #openstack-infra | 16:27 | |
| *** pilgrimstack has quit IRC | 16:27 | |
| *** Nikolay_St has quit IRC | 16:28 | |
| *** esberglu has quit IRC | 16:28 | |
| fungi | looks like the error is generated here... https://github.com/rra/kstart/blob/master/framework.c#L313 | 16:29 |
| jeblair | sudo su - jenkins; k5start -t -f /etc/wheel.keytab service/wheel -- bash | 16:29 |
| jeblair | that will reproduce the error ^ | 16:29 |
| jeblair | when i run it as root, it succeeds | 16:29 |
| fungi | only readable by jenkins, not writable | 16:30 |
| fungi | oh, the keytab is, but shouldn't be written to anyway | 16:30 |
| fungi | so nevermind | 16:30 |
| *** kushal has joined #openstack-infra | 16:31 | |
| fungi | seeing if i can figure out how k_setpag() decides where to write | 16:32 |
| *** esberglu has joined #openstack-infra | 16:32 | |
| jeblair | weirdly, /var/cache/openafs is empty | 16:33 |
| jeblair | i don't know if that's where it writes, but that's where i expect the afs client cache data to be | 16:33 |
| fungi | only writable by root, fwiw | 16:33 |
| fungi | setuid? | 16:34 |
| jeblair | fungi: that should be fine -- it's only written to by the cache manager which runs as root | 16:34 |
| fungi | k5start doesn't look setuid | 16:34 |
| fungi | ahh | 16:34 |
| jeblair | fungi: and k5start is doing an ioctl to do the pag thing | 16:34 |
| fungi | thought you were suggesting that's where it might try to write the pag | 16:34 |
| fungi | are you stracing it? | 16:35 |
| jeblair | fungi: i do expect the cache manager to be writing things (cached files) to that directory -- i don't know if it stores pag data there too. | 16:35 |
| *** matrohon has quit IRC | 16:35 | |
| jeblair | fungi: i did to see the ioctl | 16:35 |
| jeblair | fungi: it's the ioctl that returns the error | 16:35 |
| fungi | k | 16:36 |
| jeblair | /afs:/var/cache/openafs:500000 | 16:36 |
| jeblair | that's the cacheinfo file | 16:36 |
| jeblair | i would like to restart the cache manager | 16:37 |
| fungi | no objection from me | 16:37 |
| * jeblair looks for something vaguely resembling an init script | 16:37 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update sonar plugin https://review.openstack.org/333482 | 16:38 |
| jeblair | where do the unit files or whatever live on centos7? | 16:38 |
| mordred | jeblair: you do systemctl something something | 16:38 |
| fungi | they're in /usr/share/systemd/something i think | 16:38 |
| jeblair | huh | 16:39 |
| *** dosaboy has joined #openstack-infra | 16:39 | |
| jeblair | what i actually did was tab-complete | 16:39 |
| mordred | nice | 16:39 |
| fungi | if you want to modify them you copy them to /etc/systemd/something and modify that to override | 16:39 |
| jeblair | i just wanted to know the name :) | 16:39 |
| jeblair | service open<TAB> | 16:39 |
| fungi | yeah, happens to me all the time | 16:39 |
| *** pvaneck has joined #openstack-infra | 16:40 | |
| *** mriedem1 has joined #openstack-infra | 16:40 | |
| jeblair | systemd says: | 16:40 |
| jeblair | Jun 23 16:40:27 wheel-mirror-centos-7-amd64.slave.openstack.org afsd[27561]: afsd: Error calling AFSOP_CACHEINODE: not configured | 16:40 |
| *** pilgrimstack has joined #openstack-infra | 16:40 | |
| mordred | jeblair: is the config file on centos somewhere else? | 16:41 |
| jeblair | mordred: yeah, i think our puppet accounts for it... | 16:41 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update sonar plugin https://review.openstack.org/333482 | 16:41 |
| *** jistr|mtg is now known as jistr | 16:42 | |
| mordred | /usr/vice/etc/afs | 16:42 |
| mordred | or, rather /usr/vice/etc | 16:42 |
| jeblair | yeah, that dir looks okay so far | 16:43 |
| jeblair | wha? | 16:43 |
| *** ddieterly is now known as ddieterly[away] | 16:44 | |
| jeblair | there are 4 afsd's running | 16:44 |
| *** mriedem has quit IRC | 16:44 | |
| *** jpich has quit IRC | 16:44 | |
| anteaya | review-dev is 500'ing for me | 16:44 |
| jeblair | systemd said the word 'failed' literally 6 times | 16:44 |
| jeblair | does that mean success? | 16:44 |
| anteaya | maybe just that one patch, how odd | 16:44 |
| jeblair | mordred, fungi: i would like to reboot the machine | 16:45 |
| mordred | jeblair: ok | 16:45 |
| jeblair | pabelanger: ^? | 16:45 |
| pabelanger | jeblair: okay | 16:45 |
| fungi | jeblair: the wheel builder node? fine with me. it should be nondisruptive | 16:46 |
| fungi | afaik | 16:46 |
| jeblair | there is a job running. we'll see what happens :) | 16:46 |
| mordred | I have so many snarky systemd comments in my brain I don't even know how to start | 16:46 |
| pabelanger | wee! | 16:46 |
| fungi | there's just about always a job running there anyway since they take an hour-ish and get rerun for every single release tag | 16:46 |
| fungi | so unless nobody's tagged anything for a while, the wheel builders are pretty much always backlogged | 16:47 |
| nibalizer | /usr/vice seems like where I would keep my etc files | 16:47 |
| fungi | mordred: i've long since burned out the systemd snark centers of my brain | 16:47 |
| fungi | nibalizer: /miami/vice | 16:47 |
| *** ddieterly[away] is now known as ddieterly | 16:48 | |
| jeblair | how long till we start to wonder when it's going to come back up? | 16:48 |
| pabelanger | Ansible exit code: 3 | 16:49 |
| nibalizer | oh AFS was originally named 'vice' | 16:49 |
| jeblair | right on :) | 16:49 |
| pabelanger | seem see what z-l does :) | 16:49 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View https://review.openstack.org/272667 | 16:49 |
| pabelanger | oh neat, it is looping | 16:49 |
| pabelanger | nice | 16:49 |
| mordred | nibalizer: http://people.sju.edu/~jhodgson/netw/andrew.html | 16:49 |
| jeblair | nibalizer: vice and venus | 16:49 |
| jeblair | so, uh, i guess ill dig out some rax creds and see if i can see a console | 16:50 |
| *** akshai has quit IRC | 16:51 | |
| mordred | jeblair: yay! | 16:51 |
| *** dizquierdo is now known as dizquierdo_afk | 16:53 | |
| *** mriedem1 is now known as mriedem | 16:54 | |
| *** pilgrimstack has quit IRC | 16:55 | |
| jeblair | um. when i go to that page i get "this plugin is disabled". | 16:55 |
| jeblair | i don't know what "this plugin" is, unfortunately. | 16:55 |
| jeblair | oh, if i click on the lego, i see it's java | 16:56 |
| jeblair | i didn't think they required java | 16:56 |
| *** arxcruz has quit IRC | 16:56 | |
| mordred | for the consone | 16:56 |
| mordred | it's a java applet | 16:56 |
| mordred | jeblair: we could cloud and just create a new centos7 wheel builder ... it doesn't have an precious data | 16:57 |
| jeblair | mordred: we could spend a lot of time doing that to get to this point :( | 16:57 |
| mordred | jeblair: that's a very good point | 16:57 |
| *** akshai has joined #openstack-infra | 16:58 | |
| jeblair | okay, even enabling java is a no go | 16:59 |
| jeblair | i'll press the reboot button harder | 16:59 |
| jeblair | (this is strange; i am certain i have used the rax console many times) | 17:00 |
| jeblair | pressing reboot harder worked | 17:00 |
| *** Srinu_ has joined #openstack-infra | 17:01 | |
| *** haaj has quit IRC | 17:01 | |
| jeblair | i have starte openafs (systemd did not do it automatically -- maybe because it failed? maybe because i did not sacrifice?) | 17:01 |
| jeblair | /var/cache/openafs/ has file in it | 17:01 |
| *** haaj has joined #openstack-infra | 17:02 | |
| mordred | jeblair: we have to do systemctl enable to tell systemd to automatically start a service on boot | 17:02 |
| jeblair | and k5start bash as jenkins worked | 17:02 |
| mordred | jeblair: woot! | 17:02 |
| jeblair | mordred: okay, i have run systemctl enable openafs-client.service | 17:02 |
| *** pilgrimstack has joined #openstack-infra | 17:02 | |
| jeblair | i would have expected puppet to have done that, but who knows | 17:02 |
| jeblair | it said it created symlinks when i ran that, so i think that confirms it was not enabled | 17:03 |
| jeblair | (i ran it again and it said nothing) | 17:03 |
| mordred | jeblair: maybe there is more magic we need to tell puppet to tell systemd | 17:03 |
| Srinu_ | Jeblair:hi | 17:03 |
| mordred | crinkle, EmilienM: do you happen to know what we need to do to tell systemd to enable a service to start on boot in puppet? | 17:03 |
| *** azvyagintsev has joined #openstack-infra | 17:04 | |
| *** esberglu_ has joined #openstack-infra | 17:04 | |
| EmilienM | mordred: https://github.com/openstack/puppet-cinder/blob/master/manifests/api.pp#L268 | 17:04 |
| clarkb | double check we have a service ensure enabled? | 17:04 |
| EmilienM | mordred: https://docs.puppet.com/puppet/latest/reference/type.html#service-attribute-enable | 17:04 |
| Srinu_ | Jeblair: i am running jenkins job when the build in progress the slave went offline and did you face this issue can you check this logs please | 17:04 |
| Srinu_ | http://paste.openstack.org/show/521666/ | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-openafs: Enable the openafs-client service https://review.openstack.org/333496 | 17:05 |
| *** isaacb has quit IRC | 17:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-openafs: Enable the openafs-client service https://review.openstack.org/333496 | 17:06 |
| mordred | EmilienM: thanks!~ | 17:06 |
| *** daemontool_ has quit IRC | 17:06 | |
| *** haaj has quit IRC | 17:06 | |
| *** pilgrimstack has quit IRC | 17:06 | |
| EmilienM | mordred: yw! | 17:06 |
| *** dizquierdo_afk has quit IRC | 17:07 | |
| *** esberglu has quit IRC | 17:07 | |
| asselin | rcarrillocruz, how do you run https://review.openstack.org/#/c/320466/13/playbooks/launch_node.yml ? All the parts at the top are supposed to be modified before using? | 17:09 |
| *** esberglu_ has quit IRC | 17:10 | |
| rcarrillocruz | At the airport | 17:10 |
| Srinu_ | Asselin: hi | 17:10 |
| rcarrillocruz | -e cloud blah | 17:10 |
| rcarrillocruz | Same with other values | 17:10 |
| *** Jeffrey4l has quit IRC | 17:11 | |
| *** coolsvap has quit IRC | 17:11 | |
| *** shashank_hegde has joined #openstack-infra | 17:11 | |
| *** nmagnezi has quit IRC | 17:11 | |
| Srinu_ | Asselin: did you face that issue | 17:12 |
| *** esberglu has joined #openstack-infra | 17:12 | |
| *** rdx_matt has quit IRC | 17:12 | |
| asselin | rcarrillocruz, ok....I'm still ansible newbie...so not following what you mean by -e cloud blah. | 17:12 |
| pabelanger | jeblair: do you think we still need cgroups on our long lived zuul workers? Here is a ML post with more details: http://lists.openstack.org/pipermail/openstack-infra/2016-June/004449.html | 17:13 |
| asselin | Srinu_, ?? which issue? | 17:13 |
| *** tqtran has joined #openstack-infra | 17:13 | |
| rcarrillocruz | You can pass param valued from CLI with -e | 17:13 |
| Srinu_ | Slave went offline while build in progress | 17:14 |
| rcarrillocruz | -e cloud=devstack-admin | 17:14 |
| jeblair | pabelanger: good question, we should ask clarkb to help refresh our memories on that | 17:14 |
| Srinu_ | Asselin: can you check this logs | 17:14 |
| *** signed8bit is now known as signed8bit_Zzz | 17:14 | |
| Srinu_ | http://paste.openstack.org/show/521666/ | 17:14 |
| rcarrillocruz | That as an ansible-playbook argument | 17:14 |
| *** markmcd1 has quit IRC | 17:14 | |
| *** markmcd has quit IRC | 17:14 | |
| pabelanger | Srinu_: I am unsure what else you are looking for. Multiple people have commented about OFFLINE_NODE_WHEN_COMPLETE not being properly setup. You should look to be fixing that | 17:15 |
| pabelanger | jeblair: ack | 17:15 |
| *** vhosakot has quit IRC | 17:16 | |
| Srinu_ | Pabelanger: i fixed that issue | 17:16 |
| *** signed8bit_Zzz is now known as signed8bit | 17:16 | |
| *** markmcd has joined #openstack-infra | 17:17 | |
| pabelanger | Srinu_: you are still saying your slave went offline during a build, that is because nodepool is likely deleting it still | 17:17 |
| pabelanger | look in your nodepool debug.log and see what is happening to the node | 17:17 |
| *** devananda has quit IRC | 17:17 | |
| Srinu_ | Pabelanger: check this logs | 17:17 |
| Srinu_ | http://paste.openstack.org/show/521666/ | 17:17 |
| pabelanger | Srinu_: right, the slave was deleted | 17:18 |
| pabelanger | likely by nodepool | 17:18 |
| asselin | rcarrillocruz, I get an ansible yaml sytax error: /home/asselin/git/openstack-system-config/playbooks/launch_node.yml': line 46, column 53, | 17:18 |
| Srinu_ | Pabelanger: because of ssh connection closed | 17:19 |
| jeblair | pabelanger, mordred, Shrews: okay, so i think the next step is to delete the previously held nodes, enable verbose logging in all the launchers, then wait for nodepool to catch another one, yeah? | 17:19 |
| pabelanger | Srinu_: the ssh connection likely closed because the server we deleted. Does the node still exist? | 17:19 |
| asselin | Srinu_, yeah, what pabelanger said, or network failed. Check the timestamps in the nodepool log & jenkins console log files to see which happened first. | 17:19 |
| pabelanger | jeblair: +1 | 17:20 |
| *** coolsvap has joined #openstack-infra | 17:20 | |
| Srinu_ | Pabelanger: when slave went offline the slave will be deleted by nodepool | 17:20 |
| *** ddieterly is now known as ddieterly[away] | 17:20 | |
| jeblair | pabelanger, mordred, Shrews: verbose enabled on all launchers | 17:21 |
| clarkb | pabelanger: jeblair the cgroups are there to restrict memory for the most part on the jenkins user processes. This means OOMkiller targets those processes and not eg systemd | 17:21 |
| jeblair | pabelanger, mordred, Shrews: we'll need to find a job started *after* 17:20 for that to take effect | 17:21 |
| clarkb | the risk is far lower now that we really only run truated things on them | 17:22 |
| clarkb | but could be a good safety net for wheel builds and the like | 17:22 |
| Srinu_ | Pabelanger: some tests are passed but some are failed due to slave went offline | 17:22 |
| *** raildo is now known as raildo-afk | 17:22 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View https://review.openstack.org/272667 | 17:22 |
| *** sdake has quit IRC | 17:22 | |
| jeblair | EmilienM: we're now set up to capture lots of debug output during the next failure | 17:23 |
| EmilienM | jeblair: ok thx, do you want me to kick off lot of CI jobs? | 17:23 |
| EmilienM | (like 2 or 3 patches) | 17:23 |
| pabelanger | Srinu_: right, when nodepool is finished with a nodepool, part of the process is to make the node offline in jenkins then delete the node from the cloud. If a job is still running when that happens, then OFFLINE_NODE_WHEN_COMPLETE is not configured properly. Only after a job is SUCCESS or FAILURE, does nodepool delete a node | 17:23 |
| *** BobBall is now known as BobBall_AWOL | 17:23 | |
| jeblair | EmilienM: that would be great | 17:23 |
| EmilienM | k | 17:23 |
| Srinu_ | Pabelanger: some jobs are passed and some jobs are failing due to slave went offline | 17:24 |
| *** sarob has joined #openstack-infra | 17:24 | |
| pabelanger | Srinu_: if you look at your jobs in jenkins, you case see which node they run on. each job should be limited to a single node | 17:24 |
| EmilienM | jeblair: https://review.openstack.org/#/q/project:openstack/puppet-openstack-integration+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22++topic:ci-test+status:open | 17:24 |
| *** kushal has quit IRC | 17:24 | |
| pabelanger | Srinu_: so, if 2 jobs are running on the same slave, then you have a configuration error | 17:24 |
| *** vhosakot has joined #openstack-infra | 17:25 | |
| *** PollyZ has quit IRC | 17:25 | |
| pabelanger | clarkb: okay, so it should like we need to move cgroups out from jenkins into a generic puppet module | 17:25 |
| *** trown is now known as trown|lunch | 17:25 | |
| pabelanger | clarkb: or set it up in our DIB element | 17:25 |
| pabelanger | actually, not DIB, since this is long lived workers | 17:26 |
| EmilienM | jeblair: we currently have 12 xenial jobs (that are supposed to work, and 12 others that were -nv before), that are running now | 17:26 |
| *** shashank_hegde has quit IRC | 17:26 | |
| EmilienM | jeblair: /me afk lunch -- will monitor that | 17:27 |
| *** ayoung has quit IRC | 17:28 | |
| Srinu_ | Pabelanger: i enabled single slave use node which will work same as offline node when complete. The job that are in queue are not taking the same node and then i run the jenkins jobs but 4 passed and remaining failed due to slave went ofline | 17:28 |
| asselin | rcarrillocruz, where are the dependent roles defined? ERROR! the role 'bootstrap_puppet' was not found | 17:29 |
| pabelanger | Srinu_: you need to figure out why the slave went offline. Use the node id and look into nodepool debug logs | 17:30 |
| pabelanger | Srinu_: either nodepool deleted it, or you have a networking issue | 17:30 |
| Srinu_ | Pabelanger: i got this error in logs check this log pleae | 17:30 |
| Srinu_ | http://paste.openstack.org/show/521666/ | 17:30 |
| pabelanger | Srinu_: you have posted that log 3 times now, it doesn't provide the right data you need. | 17:31 |
| pabelanger | Srinu_: look at nodepool, not jenkins | 17:31 |
| Srinu_ | Pabelanger: when slave went offline and node is deleting automatocally | 17:31 |
| asselin | Srinu_, /var/log/nodepool/nodepool.log check the timestamps for actions on that node | 17:32 |
| *** PollyZ has joined #openstack-infra | 17:32 | |
| pabelanger | what asselin said | 17:32 |
| *** vhosakot has quit IRC | 17:33 | |
| Srinu_ | Pabelanger: in nodepool log it is deleting and creating a new slave | 17:33 |
| pabelanger | Srinu_: show us your nodepool debug.log using paste.openstack.org | 17:34 |
| pabelanger | fungi: do you mind reviewing https://review.openstack.org/#/c/332921/ before I patch bomb our puppet modules? | 17:34 |
| *** jamesmcarthur has quit IRC | 17:34 | |
| *** tosky has quit IRC | 17:34 | |
| pabelanger | fungi: adding other-requirements.txt for bindep | 17:35 |
| *** Srinu_ has quit IRC | 17:36 | |
| *** thiagop has quit IRC | 17:36 | |
| *** jcooley has quit IRC | 17:37 | |
| *** gabriel-bezerra has quit IRC | 17:37 | |
| *** thiagop has joined #openstack-infra | 17:37 | |
| fungi | pabelanger: do you happen to have any numbers on how much speedup that gave on that project? | 17:38 |
| *** rcernin has quit IRC | 17:38 | |
| *** electrofelix has quit IRC | 17:38 | |
| fungi | pabelanger: which of our jobs there are using install-distro-packages, out of curiosity? | 17:38 |
| *** gabriel-bezerra has joined #openstack-infra | 17:39 | |
| *** armax has joined #openstack-infra | 17:39 | |
| pabelanger | fungi: in rax we shave about 2mins off our puppet-lint jobs | 17:41 |
| mordred | pabelanger: I look forward to your patch bomb | 17:42 |
| *** salv-orlando has joined #openstack-infra | 17:42 | |
| pabelanger | fungi: right now, only puppet-lint and syntax and beaker-rspec use it | 17:42 |
| *** mriedem1 has joined #openstack-infra | 17:43 | |
| fungi | pabelanger: awesome, make sure you use a consistent topic on the patchbomb and i'll approve whichever ones are passing all their jobs | 17:43 |
| *** mriedem has quit IRC | 17:44 | |
| pabelanger | fungi: ack | 17:44 |
| esberglu | I’m hitting this error when trying to spawning a new instance | 17:44 |
| esberglu | http://paste.openstack.org/show/521713/ | 17:44 |
| *** vhosakot has joined #openstack-infra | 17:45 | |
| *** abregman has quit IRC | 17:45 | |
| pabelanger | esberglu: looks like you need to migrate your database | 17:45 |
| pabelanger | let me find the ML post | 17:45 |
| *** pvaneck has quit IRC | 17:45 | |
| pabelanger | esberglu: http://lists.openstack.org/pipermail/openstack-infra/2016-April/004204.html | 17:45 |
| *** SumitNaiksatam has joined #openstack-infra | 17:46 | |
| fungi | pabelanger: also, i think we might have a cookiecutter template for infra puppet modules? if we do, make sure to update that too | 17:46 |
| esberglu | pabelanger: thanks! | 17:46 |
| fungi | nibalizer: ^ probably same goes for the gem patch | 17:46 |
| pabelanger | fungi: good idea | 17:46 |
| *** sarob_ has joined #openstack-infra | 17:48 | |
| pabelanger | jeblair: http://logs.openstack.org/06/333506/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial-nv/4ea8e22/ is a failure, but surprisingly the extra verbosity is not logged in ansible_log.txt. | 17:49 |
| *** salv-orl_ has joined #openstack-infra | 17:49 | |
| *** salv-orlando has quit IRC | 17:49 | |
| nibalizer | we do | 17:50 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add other-requirements.txt for bindep support https://review.openstack.org/332921 | 17:50 |
| pabelanger | jeblair: it was captured by zuul-launcher | 17:50 |
| nibalizer | jeblair: whomever else is hacking on nodepool, our ibm team got caught by the schema change today | 17:51 |
| *** sarob has quit IRC | 17:51 | |
| *** shashank_hegde has joined #openstack-infra | 17:51 | |
| jeblair | nibalizer: yep, i sent an email: http://lists.openstack.org/pipermail/openstack-infra/2016-June/004469.html | 17:53 |
| nibalizer | yah... i know | 17:53 |
| *** e0ne has joined #openstack-infra | 17:54 | |
| *** tosky has joined #openstack-infra | 17:54 | |
| nibalizer | but i thought the reccomendation is to continously deploy, then we threw out something that isnt continously deployable | 17:54 |
| nibalizer | so that seems incongruent | 17:55 |
| jeblair | nibalizer: yep. it's unfortunate. there is a patch to add migrations that could use some love. or hopefully we'll be on zookeeper before too much longer. | 17:55 |
| *** jlanoux has quit IRC | 17:57 | |
| fungi | wow... a hunt to track down an error with my current artifact signing implementation change took me all the way back to https://review.openstack.org/7 (yes, just 7, that's not a paste error) | 17:58 |
| *** _nadya_ has joined #openstack-infra | 17:58 | |
| nibalizer | jeblair: more than unfortunate... its downtime | 17:58 |
| nibalizer | seems wrong to me to break downstreams just to add a comment field to an operation that is very rarely run | 17:58 |
| jeblair | nibalizer: are you venting or do you have a concrete suggestion? | 17:59 |
| *** mriedem1 is now known as mriedem | 17:59 | |
| fungi | concrete suggestion of mine would be that downstreams interested in maintaining continuous deployments of nodepool without vetting them would do well to help with one of the existing proposed fixes for that issue | 18:00 |
| nibalizer | fungi: continuous deployment is something we coded into puppet-openstackci, its not an expert-use-case, its the reccomended use case | 18:01 |
| jeblair | the whole system is, currently, an expert use case | 18:01 |
| jeblair | i'm working on fixing that, but it's going to be a while | 18:02 |
| fungi | nibalizer: well, to be fair, it is something that was extracted from our config repo. it is also not free from needing improvement | 18:02 |
| jeblair | puppet-openstackci is "openstack's ci system the way openstack runs it" | 18:02 |
| *** kzaitsev_mb has quit IRC | 18:02 | |
| *** haaj has joined #openstack-infra | 18:02 | |
| fungi | mordred: i know this is an insane longshot, but do you remember what you wanted to use that pubring.gpg for? or should i try to import it and find out? | 18:02 |
| jeblair | even if we add migrations, or some other fix, there will almost certainly be breaking changes of other types and other components in the future | 18:03 |
| nibalizer | as a downstream you have two options, fall behind or continously deploy | 18:03 |
| *** PollyZ has quit IRC | 18:03 | |
| jeblair | so it would be great for folks to be susbscribed to the mailing list for things like that | 18:03 |
| *** PollyZ has joined #openstack-infra | 18:03 | |
| mordred | fungi: uhm | 18:04 |
| mordred | fungi: which pubring.gpg ? | 18:04 |
| fungi | mordred: in https://review.openstack.org/7 | 18:05 |
| fungi | mordred: here's the key it injects... http://paste.openstack.org/show/521721/ | 18:05 |
| *** PollyZ has quit IRC | 18:05 | |
| mordred | fungi: wow. you ... you just linked to change 7 | 18:05 |
| fungi | mordred: yes, i was marvelling over that earlier | 18:06 |
| mordred | fungi: I now see that marvel | 18:06 |
| nibalizer | jeblair: hrm, im trying to think of ways to make things better | 18:06 |
| fungi | mordred: pretty certain this is actually used in zero places at this point. i'm going to propose deleting that from the puppet-jenkins module (which is where it was eventually shuffled over to) | 18:06 |
| mordred | fungi: I believe those are there from when we used to make debs | 18:06 |
| *** akshai has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Make dib_cache dir with sudo in dib jobs https://review.openstack.org/333415 | 18:07 |
| jeblair | nibalizer: i think the alembic change would go a long way | 18:07 |
| AJaeger | jeblair, pabelanger: With EmilienM's change to rename the xenial jobs to -nv, you might need to change the name of the jobs to hold... | 18:07 |
| mordred | fungi: because, you know, it used to be the workflow for us to make source packages and upload them to a PPA on launchpad and then use apt-get build-dep on the slaves to make sure that the depends were installed | 18:07 |
| jeblair | AJaeger: very good point | 18:07 |
| nibalizer | jeblair: would you be willing to send an email like the one you did, but send it a few days ahead of time? that way folk can be ready or turn off their CD if they rae not | 18:07 |
| fungi | mordred: neat! the failures on the change i have proposed which led me to that nugget of history could eventually be used to sign secure apt repos ;) | 18:07 |
| *** e0ne has quit IRC | 18:07 | |
| nibalizer | jeblair: i don't know what alembic means | 18:07 |
| pabelanger | AJaeger: Oh, that might explain why I haven't seen a held node | 18:07 |
| jeblair | AJaeger: that explains why we didn't catch it | 18:07 |
| mordred | nibalizer, jeblair: I agree, I think the alembic change will help with most of these | 18:07 |
| *** haaj has quit IRC | 18:07 | |
| mordred | nibalizer: alembic is the database-migratoins patch | 18:08 |
| jeblair | nibalizer: alembic == database migrations | 18:08 |
| mordred | nibalizer: https://review.openstack.org/#/c/136589/ | 18:08 |
| jeblair | nibalizer: in this instance, no. that patch series was an emergency fix to help us debug an ongoing operational problem that has caused a project to switch half their jobs to non-voting | 18:08 |
| mordred | nibalizer: if we get that patch fixed up and merged, then nodepool schema changes should be safe for CD | 18:09 |
| jeblair | nibalizer: i think on the desperation scale, this is pretty close to the far end of things. | 18:09 |
| *** PollyZ has joined #openstack-infra | 18:09 | |
| jeblair | nodepool job-create gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial-nv --hold-on-failure 3 | 18:10 |
| jeblair | pabelanger, AJaeger:^ look right? | 18:10 |
| *** yamahata has quit IRC | 18:10 | |
| nibalizer | jeblair: we're talkign about commenting a nodepool hold? | 18:10 |
| nibalizer | or something else? | 18:10 |
| mordred | jeblair: ++ | 18:10 |
| jeblair | nibalizer: yes, it was a pre-requisite for the auto-hold failed node patch | 18:10 |
| *** e0ne has joined #openstack-infra | 18:11 | |
| openstackgerrit | Rick Aulino proposed openstack-infra/project-config: Upgrade experimental searchlight ES 2.3.2 https://review.openstack.org/333525 | 18:11 |
| jeblair | nibalizer: which i think is, frankly, something nodepool operators have been screaming for for a while | 18:11 |
| *** sdake has joined #openstack-infra | 18:11 | |
| *** csomerville has joined #openstack-infra | 18:11 | |
| mordred | I do agree that a longer notice period on the mailng list when we start landing things like zookeeper and friends is totally fair | 18:11 |
| jeblair | i do not | 18:12 |
| mordred | no? | 18:12 |
| jeblair | nope | 18:12 |
| mordred | ok | 18:12 |
| nibalizer | jeblair: also 'emergency operational problem' is what was created for more organizations | 18:12 |
| *** PollyZ has quit IRC | 18:12 | |
| pabelanger | jeblair: it looks correct | 18:12 |
| jeblair | nibalizer: are you still trying to tell me something i already know? | 18:12 |
| pabelanger | jeblair: we can also add gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial-nv for additional coverage if needed, I've seen that fail too | 18:13 |
| jeblair | nibalizer: again, we *clearly* understood the potential impact which is why i sent the email | 18:13 |
| *** e0ne has quit IRC | 18:13 | |
| *** rdx_matt has joined #openstack-infra | 18:13 | |
| nibalizer | jeblair: im not sure im being clear | 18:13 |
| *** rdx_matt has quit IRC | 18:13 | |
| *** rdx_matt has joined #openstack-infra | 18:14 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:14 | |
| pabelanger | jeblair: regarding alembic and nodepool, we'd only support upgrade right? Not sure I'd want to deal with the downgrade path and testing | 18:15 |
| *** cody-somerville has quit IRC | 18:15 | |
| clarkb | pabelanger: yes, at least thats how the existing patch was written iirc | 18:15 |
| jeblair | pabelanger: yes, only upgrade | 18:15 |
| mordred | pabelanger: ++ | 18:16 |
| clarkb | pabelanger: fungi why doesn't https://review.openstack.org/#/c/332921/1 have any content? even if we don't need to install the deps becaues they are already there we should list them | 18:16 |
| clarkb | then apt/yum will nop and say all is well | 18:16 |
| *** degorenko is now known as _degorenko|afk | 18:16 | |
| clarkb | or is the issue more that we install puppet from not the distro and all that? | 18:17 |
| fungi | clarkb: the implication is that it has no dependencies worth mentioning, but if there are some i'm game for having them listed there | 18:17 |
| fungi | _prerequisite_ dependencies that is | 18:17 |
| clarkb | fungi: it depends on puppet for one | 18:17 |
| clarkb | but also beaker? | 18:17 |
| fungi | yeah, seems reasonable to list puppet in there, and whatever else we're preinstalling if the jobs don't install them explicitly already | 18:18 |
| nibalizer | jeblair: i guess i expected to hear 'yea that was our bad, we're gonna not do that in the future' | 18:18 |
| *** armax has quit IRC | 18:18 | |
| fungi | nibalizer: we likely will do that in the future | 18:18 |
| nibalizer | or at the least 'this is how we can work together on some agreements to make it so when we do break you, yuo have a chance to not be broken' | 18:18 |
| nibalizer | instead the attitude is 'yep, lol' | 18:18 |
| pabelanger | clarkb: we could add puppet | 18:18 |
| fungi | nibalizer: the attitude is "there are changes which could be implemented to make that smoother, let's work on it | 18:19 |
| jeblair | nibalizer: i won't lie to you. it was not a mistake. and we *might* do it again in the future. i will say that we understand the problem and the significance, and we try to avoid doing it, and we try to mitigate it as best we can. | 18:19 |
| pabelanger | clarkb: but, because our jobs passed when the file was empty, I stopped there | 18:19 |
| jeblair | nibalizer: the response is definitely not LOL | 18:19 |
| jeblair | nibalizer: i am not laughing | 18:19 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/elastic-recheck: Revert "Add e-r query for 1590139" https://review.openstack.org/333527 | 18:19 |
| jeblair | nibalizer: i am, in fact, quite upset that you think i might be | 18:19 |
| nibalizer | so when mordred said 'i agree longer notice for thing slike nodepool is fair' and you said 'i do not, nope' | 18:20 |
| clarkb | pabelanger: they pass because as part of the image build we preinstall all the deps, but I think we should still try to be explicit (though that may lead to bitrot and won't necessarily communicate we use puppet's packages not distro packages) | 18:20 |
| jeblair | nibalizer: that is far too speculative to discuss | 18:20 |
| *** ajmiller has joined #openstack-infra | 18:20 | |
| jeblair | nibalizer: i wish that mordred did not bring up what kind of notice or what we might do for zookeeper | 18:20 |
| jeblair | nibalizer: it is an entirely different problem | 18:20 |
| *** zul has quit IRC | 18:20 | |
| nibalizer | ok | 18:20 |
| jeblair | nibalizer: to that, i will say, of course we don't want to break people and we want to make transitions as easy as possible | 18:20 |
| nibalizer | well i will go back to ibm and try to convince them to get on the infra list | 18:21 |
| *** ddieterly[away] has quit IRC | 18:21 | |
| *** amoralej is now known as amoralej|lunch | 18:21 | |
| *** amoralej|lunch is now known as amoralej|off | 18:21 | |
| fungi | nibalizer: i agree with jeblair (and disagree with mordred) on that. we shouldn't slow the pace of development just to warn users that there are potentially breaking changes coming. nodepool is still a 0.x release, there are no stability guarantees for its interfaces | 18:21 |
| nibalizer | we'll try to be vigilant, and when possible if you could give us the heads up that something is in the pipe (even if it is already landing) that would be really great | 18:21 |
| pabelanger | clarkb: I agree with that, if somebody tried to run the job out of the gate, it would likely fail. Let me update the patch | 18:21 |
| jeblair | nibalizer: i do not know what that will entail, which is why i don't want to say something like "we will give you a weeks notice before we zk" because i don't know what it's really going to look like | 18:21 |
| *** yamamoto has quit IRC | 18:22 | |
| *** PollyZ has joined #openstack-infra | 18:22 | |
| clarkb | pabelanger: if we can put comments in bindep (I think we can) we could add a comment saying "puppet installed from puppet's repos" or something too to try and communicate that | 18:22 |
| pabelanger | clarkb: sure | 18:22 |
| fungi | yeah, bindep totally supports comments | 18:22 |
| *** yamamoto has joined #openstack-infra | 18:22 | |
| nibalizer | fungi: the 0.x forever isn't a good experience either though, projects that have existed for years and have users and are sitll 0.x just create conversations like this one | 18:22 |
| jeblair | nibalizer: yes, i will try to give as much notice as i can | 18:23 |
| notmorgan | jeblair: this is related to the schema change, right? | 18:23 |
| *** ddieterly has joined #openstack-infra | 18:24 | |
| *** akshai has joined #openstack-infra | 18:24 | |
| mordred | notmorgan: yah | 18:24 |
| jeblair | nibalizer: my full-time job is to make these things better. they will get better, but it will take time. some of them might even get a little worse before they do get better. | 18:24 |
| fungi | pabelanger: clarkb: i recommend something along the lines of the header comment block i have proposed in https://review.openstack.org/200672 too | 18:24 |
| nibalizer | jeblair: sure, and we want to be on upstream and on matser | 18:24 |
| nibalizer | becuase I've seen the perma-nodepool-fork and its not pretty | 18:24 |
| fungi | pabelanger: clarkb: so that people know what that file is and why it's there and how they can use it | 18:24 |
| *** zul has joined #openstack-infra | 18:24 | |
| pabelanger | fungi: okay, let me get a new patch up for that | 18:24 |
| pabelanger | and I'll pull in the tox.ini change too | 18:25 |
| erlon | guys, does anyone have see this failure in tempest gate? | 18:25 |
| erlon | https://review.openstack.org/#/c/310061/ -> gate-tempest-dsvm-neutron-full-liberty | 18:25 |
| *** Sukhdev has joined #openstack-infra | 18:27 | |
| *** jcooley has joined #openstack-infra | 18:27 | |
| *** jamesmcarthur has joined #openstack-infra | 18:27 | |
| nibalizer | jeblair: so what about cuting a 1.0? that way we can use version numbers to know about upgrade impact? | 18:27 |
| nibalizer | just as a spitball idea, im sure there are reasons not to | 18:28 |
| *** salv-orl_ has quit IRC | 18:28 | |
| jeblair | nibalizer: well, i mean, we didn't even make a release here, so it doesn't address the CD issue | 18:28 |
| jeblair | nibalizer: honestly, the biggest gain we can do here is the alembic change | 18:28 |
| mordred | yah. I'm going to rebase that change real quick | 18:28 |
| nibalizer | we could stop CDing if there were frequent releases | 18:28 |
| jeblair | it really will take care of 90% of the problem | 18:28 |
| nibalizer | ok | 18:28 |
| mordred | I should have followed through on it like a year ago | 18:29 |
| nibalizer | so if mordred rebases that, i will use my rusty python eyes and review | 18:29 |
| notmorgan | i was going to offer to resurrect it | 18:29 |
| notmorgan | for the short term. | 18:29 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr: generate the reno cache during sdist if reno is installed https://review.openstack.org/330814 | 18:29 |
| nibalizer | yah lets just push on that | 18:29 |
| mordred | notmorgan: woot. actually - if you want to take it, that would be great | 18:29 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-jenkins: Remove unused GnuPG keyring https://review.openstack.org/333529 | 18:29 |
| notmorgan | mordred: yeah happy to do that. i need a break from staring at gear | 18:29 |
| *** yamamoto has quit IRC | 18:30 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs https://review.openstack.org/333065 | 18:30 |
| mordred | notmorgan: the thing jeblair does in gertty I think is even better than what I have in this patch so far | 18:30 |
| notmorgan | mordred: automatic migration on startup? | 18:30 |
| mordred | notmorgan: which is gertty noticing that a migration needs to be run and running it for you, rather than needing a db upgrade command to be run | 18:30 |
| mordred | yah | 18:30 |
| mordred | jeblair: ^^ thoughts? | 18:31 |
| jeblair | oh yeah, lets do that | 18:31 |
| notmorgan | mordred: yeah that was my plan actually, was typing that | 18:31 |
| notmorgan | ;) | 18:31 |
| mordred | woot | 18:31 |
| jeblair | cause otherwise it's sort of a puppet nightmare | 18:31 |
| mordred | yup | 18:31 |
| notmorgan | i mean, fail forward is the right answer here | 18:31 |
| mordred | and will introduce a breaking change to downstream deployments in order to avoid breaking changes :) | 18:31 |
| notmorgan | esp. in 0.XX | 18:31 |
| clarkb | the trickiest bit is really going to be making sure you don't miss any of the db changes in the last year and a half | 18:31 |
| jeblair | you can create a new base migration | 18:31 |
| mordred | clarkb: I think just making that initial schema file froam sctach | 18:32 |
| notmorgan | jeblair: ++ | 18:32 |
| mordred | yah. that | 18:32 |
| jeblair | just start from the current place | 18:32 |
| notmorgan | exactly | 18:32 |
| clarkb | that works too | 18:32 |
| nibalizer | notmorgan: so whatever puppet-nodepool does | 18:32 |
| notmorgan | we start from *today* and go forward. sorry if that breaks you. | 18:32 |
| nibalizer | i think it just reloads? | 18:32 |
| notmorgan | but that is unwinding this specifically forward | 18:32 |
| *** markusry has quit IRC | 18:32 | |
| pabelanger | +1 to start from today | 18:32 |
| notmorgan | chances are you were going to break otherwise. | 18:32 |
| fungi | alternative is to make a single migration that starts from the most recent tag? not sure how easy that is though but might help people who upgrade from release to release (or is the first one a catch-22 there?) | 18:32 |
| nibalizer | oh hahah puppet-nodepool doesn't restart nodepool, neat | 18:33 |
| jeblair | yeah, that's intentional | 18:33 |
| *** pvaneck has joined #openstack-infra | 18:33 | |
| jeblair | it's a bit disruptive | 18:33 |
| jeblair | (we can revisit that after builders are in zk) | 18:33 |
| clarkb | erlon: you might have better luck in the qa or stable maintenance channel? | 18:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackci: Add puppet dependency for bindep https://review.openstack.org/333532 | 18:33 |
| nibalizer | jeblair: ah ok | 18:33 |
| clarkb | erlon: have you at least identified the failure and can link directly to that? | 18:33 |
| *** _nadya_ has quit IRC | 18:34 | |
| jeblair | pabelanger: so even though we didn't hold that node, did you get anything useful from the launcher logs? | 18:35 |
| *** akshai has quit IRC | 18:35 | |
| notmorgan | mordred: i'll get on rebasing and making that thing all workable shortly | 18:35 |
| *** salv-orlando has joined #openstack-infra | 18:36 | |
| pabelanger | jeblair: I didn't see too much actually. http://paste.openstack.org/show/521725/ confirms we are still SSHing into the system. | 18:36 |
| *** lvdongbing has quit IRC | 18:37 | |
| *** th3rt has joined #openstack-infra | 18:37 | |
| pabelanger | jeblair: need to step away for 30mins, when I get back I'll recheck some jobs in an attempt to catch a new node | 18:37 |
| *** vhosakot has quit IRC | 18:37 | |
| *** trown|lunch is now known as trown | 18:37 | |
| jeblair | mordred, Shrews, pabelanger, notmorgan: yeah, '-vvv' still isn't telling us why async_status failed. i think we need serious ansible debugging help here. | 18:38 |
| *** PollyZ has quit IRC | 18:39 | |
| pabelanger | So, apparently we can do stragegy: debug to launcher the ansible debugger. I've never tried it before | 18:39 |
| Shrews | jeblair: my recommendation would be either abadger or bcoca. jimi is in the act of moving | 18:40 |
| *** tongli has quit IRC | 18:40 | |
| * fungi wonders how intensely verbose that gets | 18:40 | |
| *** rodrigods has quit IRC | 18:40 | |
| *** rodrigods has joined #openstack-infra | 18:40 | |
| clarkb | pabelanger: jeblair in this case you probably want to debug the status process on the remote end, not the launcher itself? | 18:41 |
| *** coolsvap has quit IRC | 18:41 | |
| jeblair | clarkb: probably so (maybe there's a small chance the problem is launcher-side) | 18:42 |
| *** coolsvap has joined #openstack-infra | 18:42 | |
| jeblair | clarkb: i want to see what async_status run on the worker output | 18:42 |
| pabelanger | maybe have the discussion about using poll: 0 and calling the async_status module directly as a possible solution to moving away from https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/task_executor.py#L553 | 18:44 |
| *** r-mibu has quit IRC | 18:45 | |
| *** r-mibu has joined #openstack-infra | 18:45 | |
| Shrews | pabelanger: i think that's a discussion worth having, but i think we need to bring in one of the ansible core guys to give their opinion, too | 18:46 |
| *** vhosakot has joined #openstack-infra | 18:47 | |
| *** vhosakot has quit IRC | 18:47 | |
| *** vhosakot has joined #openstack-infra | 18:48 | |
| *** rdx_matt has quit IRC | 18:48 | |
| mordred | Shrews: yah. it seems like we might need more help from our friends over there - maybe we shoudl go chat with them in #ansible-devel ? | 18:49 |
| Shrews | ++ | 18:49 |
| *** PollyZ has joined #openstack-infra | 18:49 | |
| *** tonytan4ever has joined #openstack-infra | 18:49 | |
| *** hongbin has joined #openstack-infra | 18:51 | |
| *** Sukhdev has quit IRC | 18:51 | |
| pabelanger | I have to run my errand now, I'll catch up on the discussions once I get back | 18:51 |
| *** HeOS has quit IRC | 18:52 | |
| *** ccamacho is now known as ccamacho|Zzz | 18:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:54 | |
| mordred | jeblair: do we have a good link to the ansible logs of a failed run handy? | 18:54 |
| *** maishsk_ has joined #openstack-infra | 18:54 | |
| jeblair | mordred: yeah http://logs.openstack.org/06/333506/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial-nv/4ea8e22/_zuul_ansible/ansible_log.txt and http://paste.openstack.org/show/521725/ | 18:54 |
| jeblair | mordred: the second is the -vvv output | 18:54 |
| *** Sukhdev has joined #openstack-infra | 18:54 | |
| fungi | has anybody else noticed how much smoother our node utilization graphs are at peak now that the jenkins api is out of the picture? | 18:55 |
| *** thorst is now known as thorst_away | 18:55 | |
| *** maishsk has quit IRC | 18:55 | |
| clarkb | we also don't seem to leak ready nodes | 18:55 |
| *** vhosakot has quit IRC | 18:55 | |
| fungi | it's like... "wham, yep here the quota" | 18:56 |
| *** maishsk has joined #openstack-infra | 18:56 | |
| jeblair | neat! | 18:56 |
| *** vhosakot has joined #openstack-infra | 18:56 | |
| pleia2 | how nice :) | 18:56 |
| * clarkb gets back to unpacking the office | 18:56 | |
| fungi | the zuul job queue "running" line at peak is the most impressive part i think | 18:56 |
| *** ajmiller has quit IRC | 18:57 | |
| *** gordc has quit IRC | 18:57 | |
| jeblair | that reminds me, i think geard isn't recording disconnects properly in some cases, which is why the number of workers is so high | 18:57 |
| jeblair | i have not been able to figure that one out | 18:57 |
| jeblair | (can not repro locally) | 18:57 |
| jeblair | but the other 3 lines should be accurate | 18:58 |
| fungi | my take on the graphs is that our node deployment and cleanup times are way more consistent (which would make sense given how long jenkins would take to respond under load, and how often it would fail to act on api calls) | 18:59 |
| *** maishsk_ has quit IRC | 18:59 | |
| *** akshai has joined #openstack-infra | 18:59 | |
| *** amit213 has joined #openstack-infra | 19:00 | |
| *** shamail has joined #openstack-infra | 19:00 | |
| *** raunak has quit IRC | 19:01 | |
| *** sambetts is now known as sambetts|afk | 19:01 | |
| *** mdrabe has quit IRC | 19:03 | |
| *** haaj has joined #openstack-infra | 19:03 | |
| *** esikachev has quit IRC | 19:04 | |
| *** csomerville has quit IRC | 19:04 | |
| erlon | clarkb: I'v tried the qa, but they are hard to get | 19:06 |
| erlon | clarkb: http://logs.openstack.org/61/310061/8/check/gate-tempest-dsvm-neutron-full-liberty/23aef06/logs/screen-q-agt.txt.gz?level=ERROR | 19:06 |
| *** ajmiller has joined #openstack-infra | 19:06 | |
| erlon | clarkb: but, this is one of the errors in q-*, there are others, but they does not see to be related to cinder | 19:07 |
| clarkb | erlon: http://logs.openstack.org/61/310061/8/check/gate-tempest-dsvm-neutron-full-liberty/23aef06/console.html#_2016-06-22_14_25_12_748925 that gives yo umore info | 19:07 |
| *** ajmiller has quit IRC | 19:07 | |
| clarkb | erlon: that seems like an issue either with the test asking for the wrong IP or with neutron/dhcp/etc | 19:07 |
| *** jcooley has quit IRC | 19:08 | |
| *** ajmiller_ has joined #openstack-infra | 19:08 | |
| *** hashar has joined #openstack-infra | 19:09 | |
| *** haaj has quit IRC | 19:09 | |
| erlon | clarkb: if were a scenario test I could understand why some IP weren't working, but is a create_snapshot | 19:09 |
| *** mdrabe has joined #openstack-infra | 19:09 | |
| erlon | clarkb: anyway, Ill still try to reach folks on QA | 19:10 |
| clarkb | erlon: create_snapshot requires a VM to snapshot... | 19:10 |
| *** gouthamr has quit IRC | 19:10 | |
| erlon | clarkb: hmm | 19:10 |
| *** mat128 is now known as mat128|away | 19:12 | |
| *** flepied1 has joined #openstack-infra | 19:19 | |
| *** ddieterly is now known as ddieterly[away] | 19:20 | |
| *** Rockyg has quit IRC | 19:20 | |
| EmilienM | mordred: have you seen the async issue anywhere else than Ubuntu Xenial? | 19:20 |
| mordred | EmilienM: we haven't - but given conversation in #ansible-devel I think that might not be the actual cause | 19:21 |
| mordred | EmilienM: there are a couple of issues we may have found in the ansible side, looking in to patches now | 19:21 |
| *** e0ne has joined #openstack-infra | 19:21 | |
| *** jamesmcarthur has quit IRC | 19:21 | |
| clarkb | mordred: you saw my PR ya? | 19:22 |
| mordred | clarkb: yah. bcoca had a comment about removing the first sleep - so looks like putting that back and accounting for it in the loop may be more betterer | 19:22 |
| *** flepied has quit IRC | 19:22 | |
| mordred | clarkb: we've got two other potential bugs identified now too | 19:22 |
| *** htruta is now known as henrique | 19:22 | |
| clarkb | uh I need to see this comment, the first sleep is still there, it just happens in the loop? | 19:23 |
| jeblair | pabelanger, mordred, Shrews: i'm turning off verbose on all the launchers since we have what we need for this phase | 19:23 |
| *** henrique is now known as Guest78419 | 19:23 | |
| clarkb | mordred: that comment is wat worthy, I will ahve to think on it | 19:23 |
| *** Guest78419 is now known as htruta | 19:23 | |
| *** cody-somerville has joined #openstack-infra | 19:24 | |
| *** jamesmcarthur has joined #openstack-infra | 19:24 | |
| jeblair | Shrews, mordred: when we have those patches ready, we can manually install a patched ansible on all the workers. since we exec, we won't need to restart or anything, so we can iterate on this pretty quickly. | 19:24 |
| jeblair | Shrews, mordred: regardless of whether bcoca wants the async_status debug output in ansible, i think we should still patch that in on ours to save an extra debug cycle if there's still something else amiss | 19:25 |
| *** jamesmcarthur has quit IRC | 19:26 | |
| *** PollyZ has quit IRC | 19:27 | |
| mordred | jeblair: agree | 19:28 |
| *** jamesmcarthur has joined #openstack-infra | 19:28 | |
| *** PollyZ has joined #openstack-infra | 19:29 | |
| *** salv-orlando has quit IRC | 19:30 | |
| *** yamamoto has joined #openstack-infra | 19:30 | |
| pabelanger | jeblair: ack | 19:31 |
| *** ayoung has joined #openstack-infra | 19:31 | |
| *** e0ne has quit IRC | 19:31 | |
| *** jamesmcarthur has quit IRC | 19:32 | |
| *** akshai has quit IRC | 19:32 | |
| EmilienM | jeblair: can I abandon the ci jobs patches? | 19:32 |
| pabelanger | sounds like we have some things to try now | 19:34 |
| *** jamesmcarthur has joined #openstack-infra | 19:34 | |
| *** e0ne has joined #openstack-infra | 19:34 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop) https://review.openstack.org/319591 | 19:34 |
| wznoinsk | is ci-watch broken? I can see at least one recent Jenkins comment that should cause the http://ci-watch.tintri.com/project?project=nova&time=24+hours to generate | 19:34 |
| mmedvede | wznoinsk: it appears so | 19:35 |
| *** gouthamr has joined #openstack-infra | 19:35 | |
| *** PollyZ has quit IRC | 19:35 | |
| mmedvede | Apoorva: are you aware of the problems with ci-watch deployed at tintri? | 19:37 |
| *** PollyZ has joined #openstack-infra | 19:37 | |
| clarkb | mordred: responded | 19:38 |
| clarkb | you can't comment on lines outside of the default diff context in github PRs | 19:38 |
| clarkb | its sort of annoying | 19:38 |
| *** akshai has joined #openstack-infra | 19:39 | |
| *** yamamoto has quit IRC | 19:40 | |
| *** ddieterly[away] is now known as ddieterly | 19:44 | |
| *** tphummel has quit IRC | 19:44 | |
| *** rdx_matt has joined #openstack-infra | 19:44 | |
| fungi | that seems like a shortcoming | 19:45 |
| fungi | though gerrit doesn't let you comment on lines in files not touched by a change too, which i've wanted at times | 19:46 |
| *** acoles is now known as acoles_ | 19:47 | |
| *** esikachev has joined #openstack-infra | 19:48 | |
| *** th3rt has quit IRC | 19:49 | |
| *** e0ne has quit IRC | 19:50 | |
| *** derekh has joined #openstack-infra | 19:50 | |
| *** salv-orlando has joined #openstack-infra | 19:52 | |
| *** e0ne has joined #openstack-infra | 19:54 | |
| derekh | I see the patch to add rh2 to nodepool has landed, I also have a experimental job to start using the cloud, https://review.openstack.org/#/c/330515/5 | 19:55 |
| jeblair | EmilienM: i'm hoping we'll be ready for another attempt within a couple of hours | 19:55 |
| jeblair | EmilienM: so, whatever you think best :) | 19:56 |
| derekh | does that have to wait until nodepool is restarted or something so that nodes are available | 19:56 |
| mmedvede | nibalizer: do you know if your https://github.com/puppetlabs/puppet/pull/5024 (use https when talking ot pypi) shoud also be ported to infra/puppet-pip? | 19:56 |
| *** dtardivel has quit IRC | 19:57 | |
| pabelanger | derekh: yes, we need to restart nodepool-builder | 19:57 |
| pabelanger | which will happen once our DIB uploads are finished for the day | 19:57 |
| EmilienM | jeblair: ok, /me nothing on the patch now | 19:57 |
| derekh | pabelanger: ok thanks | 19:57 |
| *** jamesmcarthur has quit IRC | 19:58 | |
| nibalizer | mmedvede: we moved all the infra stuff to the openstack_pip provider in the infra pip module | 19:58 |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| dmsimard | Where do I start looking if nodepool sees that there are queued jobs but doesn't create any VMs ? The node list is clean, the quotas are fine and the max-servers is not reached. There's no obvious errors in nodepool.log or debug.log. | 19:59 |
| dmsimard | It'll create, like, at most 2 VMs for a job, but no more than that | 20:00 |
| dmsimard | (this is outside of openstack-infra) | 20:00 |
| *** vhosakot has quit IRC | 20:02 | |
| *** vhosakot has joined #openstack-infra | 20:02 | |
| mmedvede | nibalizer: ah, I see now. thanks | 20:03 |
| fungi | dmsimard: one common cause for that is if nodepool has lost contact with zuul's gearman, since it won't see the demand any longer | 20:03 |
| *** esikachev has quit IRC | 20:04 | |
| *** ayoung has quit IRC | 20:04 | |
| pabelanger | Ya, zuul should be reporting what the current demand is, and reacts according to that | 20:05 |
| fungi | jroll: looking at the virtual sprint writeup, i see that "June 23 2016 00:00 to 04:00 UTC ... Audio quality issues inhibited communication. Session ended at 00:18 UTC." any details on that? | 20:05 |
| *** haaj has joined #openstack-infra | 20:05 | |
| *** jamesmcarthur has quit IRC | 20:05 | |
| jroll | fungi: hrm, I wasn't there, mat128|away would know but just took off for the weekend, maybe TheJulia | 20:05 |
| fungi | thanks! | 20:05 |
| *** jamesmcarthur has joined #openstack-infra | 20:05 | |
| fungi | TheJulia: any recollection whether it was all participants or just some? sip clients, dial-in, both? no discernable pattern? | 20:06 |
| *** abregman has joined #openstack-infra | 20:07 | |
| notmorgan | mordred: is it safe to assume: DB is empty -> create_all(), if db exists, assume we are "today" on nodepool? | 20:08 |
| dmsimard | fungi: thanks I'll look | 20:08 |
| *** gouthamr has quit IRC | 20:08 | |
| notmorgan | jeblair: ^ cc. | 20:08 |
| *** rajinir has joined #openstack-infra | 20:08 | |
| notmorgan | or do we want to say "if db isn't empty, do a drop/recreate and let nodepool *then* own it"? | 20:09 |
| *** jcoufal_ has quit IRC | 20:09 | |
| fungi | neat, rdo is using a gerrit instance for their repos too? change count is low so must be a recent development. also... they have it setup for openid from github not the fedoraproject ipsilon instance. i wonder if that's something they're interested in working out | 20:10 |
| *** gouthamr has joined #openstack-infra | 20:10 | |
| *** e0ne has quit IRC | 20:10 | |
| fungi | https://review.rdoproject.org/ | 20:10 |
| jeblair | notmorgan: i think assume today is okay | 20:10 |
| fungi | dmsimard: ^ any idea about that? | 20:10 |
| *** haaj has quit IRC | 20:11 | |
| dmsimard | fungi: review.rdoproject.org is a software factory instance ( http://softwarefactory-project.io/docs/ ) | 20:11 |
| dmsimard | tl;dr openstack-infra in a box | 20:11 |
| fungi | dmsimard: neat-o | 20:11 |
| dmsimard | It's a project started by Enovance a while back, it's a Red Hat project now | 20:12 |
| notmorgan | jeblair: ok wfm, just checking before making that kind of assumption :) | 20:12 |
| dmsimard | tristanC is one of the engineers that works on it :) | 20:12 |
| pabelanger | fungi: look at asterisk logs, I see a codec mismatch during that time. Basically starts at 2016-06-23 00:07:55.723 could be related | 20:12 |
| fungi | dmsimard: yeah, i'm aware of software factory, just didn't know rdo was using it | 20:12 |
| dmsimard | It's fairly recent, like maybe 2 months top | 20:13 |
| fungi | dmsimard: that's roughly what it looks like from mining some old change ids out of it | 20:13 |
| *** banix has joined #openstack-infra | 20:13 | |
| dmsimard | fungi: as you can tell from my question, we're still working out some kinks :) | 20:13 |
| fungi | dmsimard: curious if there's any interest from the rdo devs in using https://id.fedoraproject.org/ as an openid provider with it, and whether anyone's looked into trying it for that and whether there are any gerrit-side gotchas there | 20:14 |
| dmsimard | I don't know, you guys want to switch from launchpad to fedora account system ? | 20:16 |
| *** gyee has joined #openstack-infra | 20:16 | |
| pleia2 | well, the software driving the fedora account system (ipsilon) | 20:17 |
| dmsimard | I'm a user of software factory, not a developer so I don't know if they have something like that in the pipe | 20:17 |
| dmsimard | pleia2: aaaaaah, understood | 20:17 |
| *** ayoung has joined #openstack-infra | 20:18 | |
| *** rdx_matt has quit IRC | 20:18 | |
| *** armax has joined #openstack-infra | 20:18 | |
| *** gordc has joined #openstack-infra | 20:20 | |
| mtreinish | jeblair: have you ever hit this: http://paste.openstack.org/show/521737/ | 20:21 |
| mtreinish | jeblair: it looks like nova blows up sqlite | 20:21 |
| mtreinish | I truncated the output there it was very large | 20:22 |
| mtreinish | and just listed every nova change | 20:22 |
| fungi | dmsimard: right, it's "just another openid/oidc" provider implementation, mainly curious since it's a red hat thing and software factory is a red hat thing and rdo is a red hat thing whether they were looking at switching from github login to that (or having both available) | 20:24 |
| fungi | but i also have no idea how much crossover there is between the rdo devs and the fedora project | 20:24 |
| fungi | could be the number of users on review.rdoproject.org who also have fedora project logins is small | 20:25 |
| jeblair | mtreinish: well, drat | 20:25 |
| *** PollyZ has quit IRC | 20:25 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid https://review.openstack.org/333567 | 20:25 |
| dmsimard | fungi: I'll ask and get back to you | 20:26 |
| *** vhosakot has quit IRC | 20:26 | |
| *** kgiusti has left #openstack-infra | 20:26 | |
| fungi | dmsimard: not super important, but if you happen to find out i'd be interested to know | 20:26 |
| fungi | thanks! | 20:26 |
| *** e0ne has joined #openstack-infra | 20:27 | |
| *** tonytan4ever has quit IRC | 20:27 | |
| *** zul has quit IRC | 20:27 | |
| *** rbradfor has quit IRC | 20:27 | |
| *** itisha has quit IRC | 20:27 | |
| *** rlandy has quit IRC | 20:27 | |
| *** TaylorHuang has quit IRC | 20:27 | |
| *** linuxmaniac_ has quit IRC | 20:27 | |
| *** fbo has quit IRC | 20:27 | |
| *** mkoderer__ has quit IRC | 20:27 | |
| *** sheeprine has quit IRC | 20:27 | |
| *** AJaeger has quit IRC | 20:27 | |
| *** notmorgan has quit IRC | 20:27 | |
| *** melwitt has quit IRC | 20:27 | |
| *** timsim has quit IRC | 20:27 | |
| *** erikwilson has quit IRC | 20:27 | |
| *** lewo has quit IRC | 20:27 | |
| *** patricku has quit IRC | 20:27 | |
| *** NobodyCam has quit IRC | 20:27 | |
| *** Daviey has quit IRC | 20:27 | |
| *** Ng has quit IRC | 20:27 | |
| *** freerunner has quit IRC | 20:27 | |
| *** timrc_ has quit IRC | 20:27 | |
| *** mika has quit IRC | 20:27 | |
| *** Anticimex has quit IRC | 20:27 | |
| *** igorbelikov has quit IRC | 20:27 | |
| *** ericksonsantos has quit IRC | 20:27 | |
| *** smcginnis has quit IRC | 20:27 | |
| *** trown has quit IRC | 20:27 | |
| *** peterlisak has quit IRC | 20:27 | |
| *** liusheng has quit IRC | 20:27 | |
| *** dimtruck has quit IRC | 20:27 | |
| *** Kiall has quit IRC | 20:27 | |
| *** ZZelle has quit IRC | 20:27 | |
| *** zigo has quit IRC | 20:27 | |
| *** zaro has quit IRC | 20:27 | |
| *** warpc has quit IRC | 20:27 | |
| *** lindsayk has quit IRC | 20:27 | |
| *** patrickeast has quit IRC | 20:27 | |
| *** ogelbukh has quit IRC | 20:27 | |
| *** zxiiro has quit IRC | 20:27 | |
| *** bryan_att has quit IRC | 20:27 | |
| *** alexm2 has quit IRC | 20:27 | |
| *** DuncanT has quit IRC | 20:27 | |
| *** jbryce has quit IRC | 20:27 | |
| *** mgagne has quit IRC | 20:27 | |
| *** mnencia has quit IRC | 20:27 | |
| *** BobBall_AWOL has quit IRC | 20:27 | |
| *** cburgess has quit IRC | 20:27 | |
| *** onovy has quit IRC | 20:27 | |
| *** morgabra has quit IRC | 20:27 | |
| *** bastelfreak has quit IRC | 20:27 | |
| *** senk_ has quit IRC | 20:27 | |
| *** phschwartz has quit IRC | 20:27 | |
| *** Shrews has quit IRC | 20:27 | |
| *** mat128|away has quit IRC | 20:27 | |
| *** _fortis has quit IRC | 20:27 | |
| *** jvrbanac has quit IRC | 20:27 | |
| *** dutsmoc has quit IRC | 20:27 | |
| *** odyssey4me has quit IRC | 20:27 | |
| *** lbragstad has quit IRC | 20:27 | |
| *** isviridoz has quit IRC | 20:27 | |
| *** j_king has quit IRC | 20:27 | |
| *** jamielennox has quit IRC | 20:27 | |
| *** mattoliverau has quit IRC | 20:27 | |
| *** mcarden has quit IRC | 20:27 | |
| *** puiterwijk has quit IRC | 20:27 | |
| *** mat128 has joined #openstack-infra | 20:27 | |
| *** j_king has joined #openstack-infra | 20:27 | |
| *** matt6434 has joined #openstack-infra | 20:27 | |
| *** lbragstad_ has joined #openstack-infra | 20:27 | |
| *** mika has joined #openstack-infra | 20:27 | |
| *** Kiall has joined #openstack-infra | 20:27 | |
| *** BobBall_AWOL has joined #openstack-infra | 20:27 | |
| *** mika has quit IRC | 20:27 | |
| *** mika has joined #openstack-infra | 20:27 | |
| *** onovy has joined #openstack-infra | 20:27 | |
| *** erikmwilson has joined #openstack-infra | 20:27 | |
| *** jvrbanac has joined #openstack-infra | 20:27 | |
| *** phschwartz has joined #openstack-infra | 20:27 | |
| *** Anticimex has joined #openstack-infra | 20:27 | |
| *** sheeprine has joined #openstack-infra | 20:27 | |
| *** smcginnis has joined #openstack-infra | 20:27 | |
| *** peterlisak has joined #openstack-infra | 20:27 | |
| *** notmorgan has joined #openstack-infra | 20:27 | |
| *** notmorgan has quit IRC | 20:27 | |
| *** notmorgan has joined #openstack-infra | 20:27 | |
| *** TaylorHuang has joined #openstack-infra | 20:27 | |
| *** alexm2 has joined #openstack-infra | 20:27 | |
| *** timrc has joined #openstack-infra | 20:27 | |
| *** rlandy has joined #openstack-infra | 20:27 | |
| *** AJaeger has joined #openstack-infra | 20:27 | |
| *** cburgess has joined #openstack-infra | 20:27 | |
| *** ZZelle has joined #openstack-infra | 20:27 | |
| *** ericksonsantos has joined #openstack-infra | 20:28 | |
| *** liusheng has joined #openstack-infra | 20:28 | |
| *** HeOS has joined #openstack-infra | 20:28 | |
| *** morgabra has joined #openstack-infra | 20:28 | |
| *** tonytan4ever has joined #openstack-infra | 20:28 | |
| *** rbradfor has joined #openstack-infra | 20:28 | |
| *** melwitt has joined #openstack-infra | 20:28 | |
| *** senk has joined #openstack-infra | 20:28 | |
| *** freerunner has joined #openstack-infra | 20:28 | |
| *** mnencia has joined #openstack-infra | 20:28 | |
| *** bastelfreak has joined #openstack-infra | 20:28 | |
| *** zul has joined #openstack-infra | 20:28 | |
| *** mkoderer__ has joined #openstack-infra | 20:28 | |
| *** odyssey4me has joined #openstack-infra | 20:28 | |
| *** melwitt is now known as Guest13191 | 20:28 | |
| *** puiterwijk has joined #openstack-infra | 20:28 | |
| *** mcarden has joined #openstack-infra | 20:28 | |
| *** trown has joined #openstack-infra | 20:28 | |
| jeblair | mtreinish: that was added in Iae6d789409b88f4e3cb35ac9e4f839f8e4d79a97 | 20:28 |
| jeblair | mtreinish: i think we can take a different approach there | 20:28 |
| jeblair | mtreinish: i'll hack it up real quick | 20:28 |
| *** _fortis has joined #openstack-infra | 20:29 | |
| *** zigo has joined #openstack-infra | 20:29 | |
| *** Ng has joined #openstack-infra | 20:29 | |
| *** lindsayk has joined #openstack-infra | 20:29 | |
| *** Shrews has joined #openstack-infra | 20:29 | |
| *** linuxmaniac has joined #openstack-infra | 20:29 | |
| *** woodster_ has quit IRC | 20:29 | |
| *** warpc has joined #openstack-infra | 20:30 | |
| *** ewindisch has quit IRC | 20:30 | |
| *** electrical has quit IRC | 20:30 | |
| *** simonmcc has quit IRC | 20:30 | |
| *** fkautz has quit IRC | 20:30 | |
| *** ogelbukh has joined #openstack-infra | 20:30 | |
| *** zxiiro has joined #openstack-infra | 20:30 | |
| *** NobodyCam has joined #openstack-infra | 20:30 | |
| *** mgagne has joined #openstack-infra | 20:30 | |
| *** igorbelikov has joined #openstack-infra | 20:30 | |
| *** mgagne is now known as Guest20454 | 20:30 | |
| *** bryan_att has joined #openstack-infra | 20:30 | |
| mtreinish | jeblair: ok cool, thanks | 20:30 |
| *** fbo has joined #openstack-infra | 20:30 | |
| *** dimtruck has joined #openstack-infra | 20:30 | |
| *** timsim has joined #openstack-infra | 20:30 | |
| *** tonytan4ever has quit IRC | 20:30 | |
| *** mwhahaha has quit IRC | 20:30 | |
| *** dougwig has quit IRC | 20:30 | |
| jeblair | mtreinish: apparently the limit is 999 :) | 20:31 |
| *** isviridov_away has joined #openstack-infra | 20:31 | |
| *** tpsilva has quit IRC | 20:31 | |
| *** mattfarina has quit IRC | 20:31 | |
| *** patrickeast has joined #openstack-infra | 20:31 | |
| mtreinish | jeblair: haha, I was expecting a power of 2 | 20:31 |
| *** itisha has joined #openstack-infra | 20:31 | |
| *** Guest13191 is now known as melwitt | 20:31 | |
| jeblair | right? | 20:31 |
| *** DuncanT has joined #openstack-infra | 20:31 | |
| *** comstud has joined #openstack-infra | 20:32 | |
| *** zaro has joined #openstack-infra | 20:32 | |
| *** patricku has joined #openstack-infra | 20:32 | |
| *** mwhahaha has joined #openstack-infra | 20:33 | |
| fungi | hello arbitrary limit? | 20:33 |
| *** tpsilva has joined #openstack-infra | 20:34 | |
| *** Daviey has joined #openstack-infra | 20:34 | |
| *** lbragstad_ is now known as lbragstad | 20:35 | |
| *** rfolco has quit IRC | 20:35 | |
| *** teran has quit IRC | 20:35 | |
| mordred | fungi: 999 is all you should ever need for anything | 20:35 |
| *** rfolco has joined #openstack-infra | 20:35 | |
| *** _fortis has quit IRC | 20:35 | |
| *** tonytan4ever has joined #openstack-infra | 20:36 | |
| *** simonmcc has joined #openstack-infra | 20:36 | |
| *** electrical has joined #openstack-infra | 20:36 | |
| rbrndt | found an error in openstack/requirements that is blocking CI jobs for at least monasca and magnum | 20:37 |
| fungi | rbrndt: ooh, an error? | 20:38 |
| rbrndt | fix has been posted, could someone review it please? https://review.openstack.org/#/c/333566/1 | 20:38 |
| *** jamielennox has joined #openstack-infra | 20:38 | |
| fungi | rbrndt: also the newly formed requirements team hangs out in #openstack-requirements now according to https://wiki.openstack.org/wiki/IRC | 20:39 |
| rbrndt | oh, thanks. didn't know about that channel | 20:39 |
| fungi | yeah, we can finally stop telling people *shrug* when they ask how to get attention from requirements reviewers | 20:39 |
| *** lewo has joined #openstack-infra | 20:40 | |
| rbrndt | :) | 20:40 |
| *** jamesmca_ has joined #openstack-infra | 20:40 | |
| mordred | fungi: what if we want to tell them *shrug* anyway? | 20:41 |
| fungi | i think that's still entirely within the realm of possibility | 20:41 |
| mordred | \o/ | 20:41 |
| *** ewindisch has joined #openstack-infra | 20:41 | |
| *** fkautz has joined #openstack-infra | 20:42 | |
| fungi | pabelanger: i incorporated your thin=>true request into 333065 and i have it passing all its jobs now as well, if you're interested in taking another peek | 20:42 |
| *** mattfarina has joined #openstack-infra | 20:42 | |
| pabelanger | fungi: sure! and thanks | 20:42 |
| fungi | also it comes with a bonus history lesson in the form of a depends-on link | 20:43 |
| *** teran has joined #openstack-infra | 20:43 | |
| *** gluytium has joined #openstack-infra | 20:43 | |
| *** gouthamr has quit IRC | 20:44 | |
| *** jamesmcarthur has quit IRC | 20:44 | |
| *** vhosakot has joined #openstack-infra | 20:44 | |
| *** kzaitsev_mb has quit IRC | 20:45 | |
| Shrews | mordred: jeblair: fyi, looking at adding use of atomic_move to the async core modules now | 20:46 |
| jeblair | Shrews: woot! i'll work on prepping things so we can deploy that | 20:47 |
| *** weshay has quit IRC | 20:47 | |
| *** dougwig has joined #openstack-infra | 20:47 | |
| *** jbryce has joined #openstack-infra | 20:48 | |
| *** ayoung has quit IRC | 20:51 | |
| *** e0ne has quit IRC | 20:51 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email https://review.openstack.org/333567 | 20:52 |
| *** ajmiller has joined #openstack-infra | 20:52 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email https://review.openstack.org/333567 | 20:53 |
| *** hashar has quit IRC | 20:53 | |
| *** _ari_ has quit IRC | 20:54 | |
| *** ajmiller_ has quit IRC | 20:55 | |
| *** ffrank has joined #openstack-infra | 20:55 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix "too many SQL variables" error https://review.openstack.org/333583 | 20:56 |
| jeblair | mtreinish: your choice of that change or https://review.openstack.org/310089 should fix the problem | 20:56 |
| mtreinish | jeblair: ok, I'll give them both a try | 20:57 |
| jeblair | mtreinish: (they are mutually exclusive, but i don't know which is better) | 20:58 |
| jeblair | (i can get worst-case timing for mine easily, the other one is harder) | 20:58 |
| *** kzaitsev_mb has joined #openstack-infra | 20:58 | |
| mtreinish | jeblair: sure, I meant try them independently | 20:59 |
| *** ajmiller has quit IRC | 20:59 | |
| *** eharney has quit IRC | 21:00 | |
| *** akshai has quit IRC | 21:01 | |
| *** woodster_ has joined #openstack-infra | 21:01 | |
| jeblair | mordred: what do i need to do if i'm pip installing ansible from a working directory? | 21:01 |
| jeblair | mordred: do i need to install multiple repos, or just 'ansible'? | 21:01 |
| mordred | jeblair: if you submodule update to get the submodules, it should just work | 21:02 |
| clarkb | jeblair: you have to init the subrepos | 21:02 |
| jeblair | k | 21:02 |
| *** rfolco has quit IRC | 21:03 | |
| *** inc0 has quit IRC | 21:04 | |
| TheJulia | fungi: it largely seemed like sip clients, I don't think anyone who dialed in actually mentioned any issues. :( | 21:05 |
| *** amitgandhinz has quit IRC | 21:06 | |
| *** haaj has joined #openstack-infra | 21:07 | |
| ianw | jeblair: tbh i don't see why the "zuul" user is any different to all the sysadmin users, and the jenkins user really. fundamentally it depends on info in heira (the pub key coming from pupept) so we're just making work for ourselves communicating that out-of-band | 21:07 |
| ianw | i don't really care if someone wants to go and re-write all the system-config puppet in ansible, or whatever | 21:07 |
| jeblair | ianw: the jenkins user will be removed. and perhaps we should stop installing our own keys as well. | 21:08 |
| ianw | but just doing that for the sake of doing that seems like there's more fun things to do | 21:08 |
| *** gouthamr has joined #openstack-infra | 21:08 | |
| *** rhallisey has quit IRC | 21:08 | |
| *** akshai has joined #openstack-infra | 21:08 | |
| jeblair | ianw: it's not for the sake of doing it, it's for the sake of extricating the disk images we use for running tests from our long running systems, which at this point, have very little in common, yet the fact that they are so closely tied makes things very difficult | 21:08 |
| fungi | TheJulia: thanks. pabelanger noticed a codec mismatch in the logs right around that time too | 21:09 |
| fungi | so that would potentially make sense | 21:09 |
| jeblair | ianw: most of all of this dates from when our test machines *were* long running systems | 21:10 |
| ianw | jeblair: so how do you want dib to get the public key for the zuul user? | 21:10 |
| *** thorst_away has quit IRC | 21:10 | |
| njohnston | So in discussions yesterday I think it was ascertained that accessbot is not approaching the max-channels limit. If so, can https://review.openstack.org/#/c/329570 be approved so I can get #openstack-fwaas logged? | 21:10 |
| jeblair | ianw: i do not have an immediate solution, i merely argue that we should proceed in developing one :) | 21:10 |
| *** trown is now known as trown|outtypewww | 21:10 | |
| *** psilvad_ has quit IRC | 21:11 | |
| *** gouthamr_ has joined #openstack-infra | 21:11 | |
| fungi | ianw: we can put that just about anywhere. public keys are public. we can bake it into public configuration/parameters if it helps | 21:11 |
| clarkb | ianw: I would have devuser write it and an easy way to do that is put the public key in nodepool.yaml with the env vars for the images | 21:11 |
| *** ajmiller has joined #openstack-infra | 21:11 | |
| *** haaj has quit IRC | 21:12 | |
| fungi | njohnston: right, accessbot will actually never hit that limit because it doesn't join any channels, just queries chanserv | 21:12 |
| ianw | clarkb: devuser is the wrong thing really, it was designed to stick one key in for a developer -- comments in https://review.openstack.org/#/c/333080/ | 21:12 |
| ianw | clarkb: the env var into nodepool.yaml was what i suggested in the comment to. i guess i can do that | 21:12 |
| njohnston | fungi: Is my information correct, then, that accessbot is the mechanism by which channels get logged to http://eavesdrop.openstack.org/irclogs/ ? | 21:12 |
| clarkb | I am not sure I agree about the devuser element | 21:12 |
| clarkb | it does exactly what we need, create a user and give it a key | 21:13 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement gate-puppet-openstack-spec-helper-lint https://review.openstack.org/333589 | 21:13 |
| fungi | njohnston: no, accessbot checks whether we have control over channels. meetbot does teh logging (i agree, teh name is confusing) | 21:13 |
| *** thorst_away has joined #openstack-infra | 21:13 | |
| *** ajmiller_ has joined #openstack-infra | 21:13 | |
| ianw | clarkb: well, it creates one user, and overwrites the developer if they're trying to use it (e.g. me) and it grabs the keys from their ~/.ssh/authorized_keys ... i mean why can "jenkins" log in as "nodepool" on nodepool now... | 21:13 |
| *** gouthamr has quit IRC | 21:14 | |
| *** yaume has quit IRC | 21:14 | |
| clarkb | ianw: what do you mean by overwrites the developer? you tell it "make a zuul user and give it this public key" | 21:14 |
| njohnston | ok, so it sounds like accessbot should still be there, but meetbot is the important thing. All right, I'll put in a change to add the channel to that config as well. | 21:14 |
| clarkb | ianw: I use it all the time | 21:14 |
| *** akshai has quit IRC | 21:14 | |
| clarkb | works fine | 21:14 |
| *** thorst_a_ has joined #openstack-infra | 21:15 | |
| *** shamail has left #openstack-infra | 21:15 | |
| fungi | njohnston: yeah, and you have it in the list for statusbot in 329570, you can just update that proposed change. the meetbot list is in the same file | 21:15 |
| *** rdx_matt has joined #openstack-infra | 21:16 | |
| *** maishsk_ has joined #openstack-infra | 21:16 | |
| ianw | clarkb: heh, you have your keys in the image from puppet and probably log in to the final image as yourself. i build and use a devuser to login with my key, but the openstack element overwrites my devuser to "zuul" | 21:16 |
| jeblair | hrm. i just installed ansible devel on zl01 and it seems to be unable to connect to any host | 21:16 |
| clarkb | ianw: I am saying I build dib images all the time using devuser, I don't use our puppet stuff | 21:16 |
| fungi | njohnston: statusbot is the thing that echoes urgent infrastructure announcements into the channel, and also implements the #success logging | 21:16 |
| clarkb | ianw: I do not have any issues using devuser at all | 21:16 |
| njohnston | fungi: Is there anything that needs to change in project-config? | 21:16 |
| *** ajmiller has quit IRC | 21:17 | |
| *** ldnunes has quit IRC | 21:17 | |
| ianw | clarkb: i am not saying devuser doesn't work. i am saying that using devuser to deploy the zuul user isn't the right thing, because it *overrides* any devuser a developer might have setup | 21:17 |
| fungi | njohnston: nope, looks like you already added your channel to http://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml#n114 so that's sufficient for project-config | 21:17 |
| clarkb | ianw: you are saying that you can only have a single devuser? | 21:17 |
| *** thorst_away has quit IRC | 21:17 | |
| ianw | clarkb: yes | 21:17 |
| *** maishsk has quit IRC | 21:17 | |
| *** maishsk_ is now known as maishsk | 21:17 | |
| clarkb | ianw: seems like that would be easy to address in dib rather than deciding not to do it at all? | 21:17 |
| fungi | njohnston: so just update 329570 to also add your channel to the meetbot list and you're all set | 21:17 |
| clarkb | ianw: anyways what jeblair is saying is to have a single user, remove any othre special users that map to humans | 21:18 |
| clarkb | ianw: then for you or I working locally we would just tell zuul user to auth with a different public key | 21:18 |
| ianw | clarkb: where does dib get the keys? | 21:18 |
| clarkb | then the only difference between production and dev is what key is in place | 21:18 |
| openstackgerrit | Nate Johnston proposed openstack-infra/system-config: Add #openstack-fwaas to accessbot channels https://review.openstack.org/329570 | 21:18 |
| clarkb | ianw: so this shouldn't be an issue for our use case | 21:18 |
| njohnston | fungi: Thanks, updated! ^^ | 21:18 |
| jeblair | i have downgraded ansible to 2.1.0.0 on zl01 | 21:19 |
| clarkb | ianw: you could enumerate per user vars if you wanted to keep a similar interface | 21:19 |
| jeblair | it seems to be doing things again | 21:19 |
| clarkb | ianw: I don't have the answers off the top of my head but it is certainly doable | 21:19 |
| *** thorst_a_ has quit IRC | 21:19 | |
| fungi | njohnston: also your commit message is a little off. you're adding to statusbot's and meetbot's channel lists, not accessbot's (you already took care of that in 326821). | 21:20 |
| *** aysyd has quit IRC | 21:20 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/system-config: Add #openstack-fwaas to statusbot/meetbot https://review.openstack.org/329570 | 21:21 |
| *** vhosakot has quit IRC | 21:21 | |
| njohnston | fungi: Thanks, fixed. | 21:21 |
| *** robot has quit IRC | 21:22 | |
| openstackgerrit | Rick Aulino proposed openstack-infra/project-config: Upgrade experimental searchlight ES 2.3.2 https://review.openstack.org/333525 | 21:22 |
| clarkb | another possible option would be to have the user element work like the pkgmap stuff and colate a bunch of user spec files | 21:22 |
| ianw | clarkb: well it's software, anything is doable. but it seems like the solution is to take data from puppet, write it into the nodepool config file, pass it through environment variables to a hand-written dib element that adds users ... to avoid using puppet directly which has the key info and someone else has already debugged the orchestration parts of adding users instead of my crappy shell script | 21:22 |
| Shrews | jeblair: oh? what version was it at? | 21:23 |
| jeblair | Shrews: tip of devel | 21:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement gate-puppet-openstack-spec-helper-syntax https://review.openstack.org/333596 | 21:23 |
| Shrews | jeblair: and what b0rked with the new version? | 21:23 |
| jeblair | Shrews: i'm seeing if i can reproduce locally | 21:23 |
| jeblair | Shrews: it just said unable to connect for every host | 21:23 |
| *** ajmiller_ has quit IRC | 21:23 | |
| Shrews | jeblair: oh, that's a new feature that's coming in 2.2. ansible will only run locally | 21:23 |
| clarkb | ianw: I think what I am suggesting is rather than having multiple user elemnts have one that works | 21:24 |
| Shrews | because networking is hard | 21:24 |
| Shrews | :) | 21:24 |
| clarkb | the current existing element is called devuser | 21:24 |
| clarkb | anyways I don't relaly have a dog in this fight, just wanted to point out that devuser works well, dib users know to use devuser, so seems like a good fit | 21:25 |
| jeblair | Shrews: :) | 21:25 |
| fungi | hah | 21:25 |
| TheJulia | fungi: I started trying other codecs to lower bandwidth utilization on my end, it was still choppy for mat128 last night regardless of the codec. My client showed 0-4% loss on inbound from the sip server, but nothing else | 21:25 |
| pabelanger | ianw: I'm working on an ansible-playbook for the zuul-worker element. I should have something ready for tomorrow | 21:25 |
| Shrews | jeblair: so, here is a possibly atomic-move-enabled version of async_wrapper: https://github.com/Shrews/ansible-modules-core/blob/async_atomic_move/utilities/logic/async_wrapper.py | 21:25 |
| fungi | TheJulia: oh, trying additional codecs might account for the error pabelanger saw. if so, not a result of the issue but rather just a result of you trying workarounds for the issue | 21:26 |
| Shrews | jeblair: i'm not sure how to test that other than just try it | 21:26 |
| jeblair | Shrews, pabelanger: i think i can reproduce locally -- but there's a slight difference -- i get an ssh-askpass dialog asking if i want to trust the worker's host key. i say 'yes' and it fails with fatal: [node]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true} | 21:26 |
| fungi | TheJulia: in that case, could have been network issues in rackspace or something | 21:26 |
| pabelanger | fungi: TheJulia: Yes, I see seen gsm as a codec for sure. In fact, I am not sure which low-bandwidth codec we have enabled | 21:26 |
| ianw | pabelanger: ok, well that's fine ... still how is it going to get the keys? | 21:26 |
| jeblair | Shrews, pabelanger: so i assume there's something up with our known_hosts enforcement | 21:26 |
| Shrews | jeblair: i'm going to try that with a simple playbook first | 21:27 |
| *** esberglu has quit IRC | 21:27 | |
| pabelanger | ianw: in this case, key should be stored in a playbook variable | 21:27 |
| TheJulia | fungi: that or between rackspace and centurylink last night, I know on monday there were a few people reporting choppy audio on the stream at times and I think there were two major carriers with significant packet loss | 21:27 |
| fungi | TheJulia: makes sense. internet gonna internet | 21:27 |
| pabelanger | ianw: something like: https://github.com/openstack/windmill/blob/master/playbooks/group_vars/nodepool-server.yaml#L23 | 21:27 |
| *** jamesmca_ has quit IRC | 21:27 | |
| TheJulia | fungi: exactly | 21:28 |
| *** ajmiller has joined #openstack-infra | 21:28 | |
| pabelanger | ianw: or, we can read it in from a file | 21:28 |
| *** ddieterly is now known as ddieterly[away] | 21:28 | |
| clarkb | sip is udp too right? | 21:28 |
| pabelanger | yes | 21:28 |
| clarkb | ya so any unreliability in the path will be very noticeable | 21:29 |
| pabelanger | we can do TCP | 21:29 |
| ianw | pabelanger: well storing the public key in the element seems wrong, so yeah, it has to read from an external file ... which is provided by puppet? | 21:29 |
| pabelanger | but RTP will still be over udp | 21:29 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow users to be filtered by openid and email https://review.openstack.org/333567 | 21:29 |
| pabelanger | ianw: where would you like the key stored, that's likely the easy thing for me to ask | 21:30 |
| ianw | pabelanger: well, as https://review.openstack.org/#/c/333080/ suggests I think that leaving it in puppet is fine :) | 21:31 |
| ianw | it seems like the right tool to be adding at user at this point in time | 21:32 |
| pabelanger | jeblair: Just catching up, there is problems with ssh_known_host? | 21:32 |
| jeblair | ianw: did this conversation just go in circles? | 21:32 |
| fungi | ianw: i agree that leaving it in puppet for now is fine, though we also want to stop running puppet in our image setup at some point so we can stop preinstalling it | 21:32 |
| jeblair | we agreed like 2 years ago to stop doing this | 21:33 |
| pabelanger | ianw: I think the main sticking point, is we want to stop using puppet in DIBs. That was my initial reason for using devuser elemtn | 21:33 |
| pabelanger | element* | 21:33 |
| fungi | and continuing to put the new account in puppet seems like prolonging the situation | 21:33 |
| fungi | since we still want to move it out of puppet | 21:34 |
| pabelanger | at the same token, I wouldn't like to see us using DIB elements to do puppet things or bash scripts. Which leave an ansible-playbook, which seems to be okay atm | 21:34 |
| jeblair | elements can inherit other elements, right? | 21:34 |
| ianw | jeblair: not really? they can require | 21:34 |
| ianw | yes, i think my main issue is rewriting stuff like "add a user" in crap shell scripts in dib | 21:35 |
| openstackgerrit | Stephanie Miller proposed openstack/diskimage-builder: Make svc-map support overridden elements https://review.openstack.org/333597 | 21:35 |
| ianw | stuff that puppet/ansible/etc has had figured out just fine | 21:35 |
| jeblair | i can't keep repeating myself | 21:36 |
| jeblair | i want to help find a solution here | 21:36 |
| jeblair | there's a world of possibilities that could satisfy all of us | 21:37 |
| roaet | is there a command for recheck but for the gate? | 21:37 |
| clarkb | roaet: its the same command, our zuul config requires that check pass before going into the gate so yo urecheck, it goes through check, if it has approval it then goes to gate if check apsses | 21:37 |
| jeblair | ianw: but we need to at least agree on the premise -- that we should not run system-config puppet in dib | 21:38 |
| *** thorst_away has joined #openstack-infra | 21:38 | |
| *** sdake has quit IRC | 21:38 | |
| roaet | clarkb: having a weird thing where the recheck passes but the gate fails | 21:38 |
| roaet | clarkb: https://review.openstack.org/#/c/333546/1 | 21:38 |
| roaet | but it is queued | 21:39 |
| roaet | so maybe it'll fail again | 21:39 |
| ianw | jeblair: i feel like we might be at the point that a user-deployment spec might be useful ... cause it's going to touch nodepool, system-config and dib | 21:39 |
| clarkb | roaet: python27 job faile,d you can look at the logs to try and determine why | 21:39 |
| jeblair | ianw: how is this possibly that hard? | 21:39 |
| roaet | clarkb: aye, but it didn't fail on the validate. just weird | 21:39 |
| *** abregman has quit IRC | 21:40 | |
| *** rbrndt has quit IRC | 21:40 | |
| clarkb | roaet: you likely have flaky tests then? | 21:40 |
| roaet | clarkb: concur | 21:40 |
| *** rbrndt_ has joined #openstack-infra | 21:41 | |
| jeblair | ianw: do you have any suggestions about how someone who is not us -- so they have no system-config puppet -- should configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use? | 21:41 |
| *** vhosakot has joined #openstack-infra | 21:41 | |
| *** maishsk_ has joined #openstack-infra | 21:41 | |
| fungi | roaet: seems to have worked that time. it's merged now | 21:42 |
| roaet | fungi: aye. I'm telling the dev to make sure his test is deterministic | 21:42 |
| * clarkb wonders where that jira lives but probably doesn't want to know | 21:42 | |
| *** sdake has joined #openstack-infra | 21:42 | |
| *** thorst_away has quit IRC | 21:43 | |
| *** claudiub has quit IRC | 21:43 | |
| *** maishsk has quit IRC | 21:43 | |
| *** maishsk_ is now known as maishsk | 21:43 | |
| *** thorst_away has joined #openstack-infra | 21:43 | |
| roaet | clarkb: heh, ya | 21:43 |
| ianw | jeblair: i would think they'd get the key from meta-data (configdrive in the glean case) | 21:43 |
| ianw | rather than baking it in | 21:43 |
| jeblair | ianw: how does zuul use it? | 21:43 |
| *** armax has left #openstack-infra | 21:43 | |
| jeblair | ianw: and does glean create users? | 21:44 |
| *** matt-borland has quit IRC | 21:44 | |
| fungi | glean does not add users afaik | 21:44 |
| fungi | that's a cloud-init-ism | 21:44 |
| ianw | glean does not, cloud-init does | 21:44 |
| *** njohnston is now known as njohnston|afk | 21:44 | |
| fungi | and part of why we bake credentials into our image builds | 21:44 |
| clarkb | the other part is that metadata services have been unreliable | 21:45 |
| jeblair | right, so perhaps we can agree that the goal here is still to bake a user with associated information into our image builds? | 21:45 |
| clarkb | config drive seems to be much more reliable | 21:45 |
| pabelanger | jeblair: I think so, it seems to have worked well for us so far | 21:45 |
| jeblair | pabelanger: the converse is also true -- not baking things into our images has worked very poorly for is. | 21:46 |
| jeblair | us | 21:46 |
| zigo | What is the ready-script: configure_mirror.sh that I can see at https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml ? | 21:46 |
| *** mriedem has quit IRC | 21:46 | |
| clarkb | zigo: it is a script that nodepool runs on every test node in order to make it "ready" | 21:46 |
| jeblair | zigo: it is found here: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh | 21:47 |
| pabelanger | jeblair: right | 21:47 |
| clarkb | zigo: once that script completes successfully nodepool assigns the instance to a "target" so that target can run jobs | 21:47 |
| *** thorst_away has quit IRC | 21:47 | |
| *** esberglu has joined #openstack-infra | 21:48 | |
| zigo | I was hoping to find some convenient stuff to configure mirrors depending on the provider ... | 21:48 |
| *** rdx_matt has quit IRC | 21:48 | |
| zigo | I know that at least OVH has a Debian mirror. | 21:48 |
| zigo | Would it make sense to make that script decide one or another hostname for the mrrors, depending on the hostname? | 21:48 |
| jeblair | ianw: you seem to have strong opinions on this, and you know the space. so i'm hoping if we can agree on the premise, you can help come up with a solution. so what i wrote above is what we should be developing for: | 21:49 |
| pabelanger | zigo: we've had issues with provider mirrors in the past. Thats the reason we configure our own AFS mirrors | 21:49 |
| clarkb | zigo: you can actually query the provider and region directly, they are in /etc/nodepool somewhere. That said not sure how people feel about that, is the debian mirror we are using unreliable? | 21:49 |
| zigo | Ok. | 21:49 |
| clarkb | also what pabelanger said | 21:50 |
| ianw | jeblair: my opinion is mostly around avoiding turning dib into a bespoke and completely crap version of puppet | 21:50 |
| jeblair | ianw, pabelanger: the problem statement is basically: someone with no system-config puppet repo should be able to configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use. | 21:50 |
| zigo | clarkb: The Debian mirror is the httpredir.debian.org, so it's just a 302 redirect type of mirror. | 21:50 |
| zigo | That works well, it's better than a DNS redirector, because more dynamic. | 21:51 |
| clarkb | oh I thought we stopped doing that and hard coded it to a specific one | 21:51 |
| zigo | But it adds the cost of the redirection. | 21:51 |
| clarkb | since httpredir was failing us | 21:51 |
| zigo | Ah? | 21:51 |
| fungi | clarkb: i think zigo is saying the mirror he _wants_ to use is that way | 21:51 |
| fungi | not ours | 21:51 |
| clarkb | zigo: ya for a time httpredir would either fail to redirect or redirect to mirrors that were down | 21:51 |
| zigo | Well, https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh has the redirector. | 21:51 |
| jeblair | ian, pabelanger: i don't really care how that happens. if the dib best practice is to have an element that runs puppet or ansible, that's fine*. it needs to be completely self-contained though, and not depend on any external infrastructure -- it especially should not assume anything about the configuration management of the nodepool server. | 21:51 |
| jeblair | ianw: ^ sorry dropped the w | 21:51 |
| clarkb | and the plan I thought was to point at a specific mirror. But then I moved and stopped computering for a while and am still catching up | 21:52 |
| fungi | clarkb: zigo: oh, weird. i guess we didn't set up an actual debian package mirror yet? | 21:52 |
| clarkb | jeblair: ianw pabelanger keep in mind that we also need to configure dns and firewalls which puppet is doing for those instances | 21:53 |
| jeblair | ianw, pabelanger: * (i will say that if dib best practice to create a user is to invoke puppet, i may be underwhelmed) | 21:53 |
| zigo | fungi: Right. | 21:53 |
| clarkb | (this may or may not influence the decision here) | 21:53 |
| zigo | fungi: That's fine for the moment though. | 21:53 |
| jeblair | clarkb: yes, there is more work to be done. | 21:53 |
| zigo | I had no issue with the redirector yet. | 21:53 |
| jeblair | clarkb: however, pabelanger took a small step in the direction we have wanted to move for a long time | 21:53 |
| clarkb | zigo: we had a lot of issues with it doing image builds | 21:53 |
| zigo | Oh ok. | 21:54 |
| jeblair | clarkb: and i would like him to be able to continue | 21:54 |
| *** tonytan4ever has quit IRC | 21:54 | |
| *** dingyichen has joined #openstack-infra | 21:54 | |
| jeblair | clarkb: moreover *we* need to configure dns and firewalls, but not everyone using nodepool and zuul do. *everyone* using nodepool and zuul needs a user. | 21:55 |
| ianw | clarkb: yeah, which is why i brought this up because turning these from puppet modules into bespoke scripts seems like a regression ... i mean puppet was supposed to be the end of bespoke scripts | 21:55 |
| clarkb | jeblair: ya | 21:55 |
| pabelanger | Right, I think having a DIB element to run ansible (or puppet) could be the first step to address some of these issue. The location of say roles or manifests would then not live in system-config | 21:55 |
| jeblair | pabelanger: that is an option | 21:56 |
| jeblair | pabelanger: i will look upon dib very poorly if that is the best option | 21:56 |
| greghaynes | jeblair: I assure you that there are no dib best practices which involve invoking puppet, although some people do that | 21:56 |
| pabelanger | ianw: right, personally I don't want to write bash scripts for any of this. | 21:56 |
| ianw | pabelanger / jeblair : using ansible to do things like deploy users and setup firewalls seems like a step in the right direction, over making it up by hand | 21:56 |
| fungi | but i also don't want us dragging in the openstack-infra/system-config openstack_project puppet module to do our image builds | 21:57 |
| ianw | i do feel like we've *already* solved this in puppet, and have to re-write it in ansible, but ... ok | 21:57 |
| greghaynes | I get the impression that the argument against using dib devuser and friends is really a tool preference argument... | 21:57 |
| *** baoli has quit IRC | 21:57 | |
| jeblair | ianw: i will be quite happy for openstack to congfigure its firewalls via either ansible or puppet if it wants (as long as it isn't system-config). the user thing has to be solved without thinking about openstack's peculiarities. | 21:57 |
| fungi | the argument against using seems to primarily be that someone might also want to have their own devuser element and use nodepool | 21:57 |
| pabelanger | greghaynes: I think that is part of it | 21:57 |
| fungi | greghaynes: ^ | 21:57 |
| *** roxanaghe has joined #openstack-infra | 21:58 | |
| clarkb | fungi: yes and we can have devuser support that | 21:58 |
| clarkb | would not be hard | 21:58 |
| fungi | and per dib there can be only one devuser | 21:58 |
| ianw | fungi: and that it's grabbing the users ~/.ssh/authorized_keys, rather than deploying from a given list | 21:58 |
| pabelanger | greghaynes: For me, the line is grey. Where do image builds stop and your cfgmgt tool of choice starts | 21:58 |
| greghaynes | fungi: ah. So for that case I'd just make your own element. We could extend devuser to be smart, but makng an element that makes a speical user is 1 dir and 1 file in that dir | 21:58 |
| *** pilgrimstack has joined #openstack-infra | 21:58 | |
| greghaynes | so its hard to argue that we need to make some fancy system when that is the workaround | 21:58 |
| ianw | greghaynes: well, this "add user" element really needs to take key data from puppet | 21:59 |
| ianw | and handle adding multiple users | 21:59 |
| ianw | so it probably wants to parse a json type config file | 21:59 |
| fungi | can't we just put the key in a variable? why does it need to come "from puppet"? | 21:59 |
| jeblair | ianw: it does not need to take key data from puppet | 21:59 |
| fungi | it's just a blob of entirely public data we want put on disk | 21:59 |
| greghaynes | also, adding multiple users is just adduser foo; adduser bar | 21:59 |
| jeblair | ianw: the key can even live *in* the element | 22:00 |
| ianw | fungi / jeblair: but then when we want to change it, it's baked in all these different places | 22:00 |
| jeblair | ianw: *we* will put it in puppet certainly | 22:00 |
| ianw | this seems like a regression from having the data kept in once place and handed out | 22:00 |
| *** ddieterly[away] is now known as ddieterly | 22:00 | |
| fungi | that's more of an argument for ldap then | 22:01 |
| ianw | jeblair: right -- *we* need to get it from puppet into a format that dib understands. so we need to write the dib element with some interface ... say it reads a json file with a list of users and keys | 22:01 |
| greghaynes | I don't know all the context, but you can use an env var (as mentioned) or extra-data.d to run commands outside the chroot that grab the data. I hope you can either input the data or retrieve it with command though otherwise I'm not sure its a dib design issue. | 22:02 |
| pabelanger | I don't know, talking about updating devuser element to add more support for user management doesn't seem to be the best use of time, IMO | 22:02 |
| fungi | seems like the concern over that is having the public key stored in more than one git repo | 22:02 |
| jeblair | graghaynes: the problem statement is basically: someone with no system-config puppet repo should be able to configure nodepool to create an image with dib that has a user whose authorized_keys corresponds to the key that nodepool and zuul will use. | 22:03 |
| fungi | i agree needing to replace that key in multiple repos or even multiple files in one repo is an annoyance. not sure how much of one and whether it's significant enough to avoid simplicity | 22:03 |
| pabelanger | jeblair: right, but duplicated data isn't really a new problem for us today | 22:03 |
| jeblair | fungi: that's an annoyance to be solved at a higher level with configuration management. for us, that's puppet (for now). | 22:04 |
| *** dimtruck is now known as zz_dimtruck | 22:04 | |
| ianw | well i'm also thinking about the keys of all the sysadmin users we put in there today | 22:04 |
| jeblair | ianw: discard them | 22:04 |
| greghaynes | jeblair: ah. So we template out our nodepool config, correct? We could inject the key via env var that way. | 22:04 |
| clarkb | greghaynes: I don't even think we need to template it, just hard code it in there | 22:04 |
| greghaynes | or that | 22:04 |
| jeblair | ianw: that's "extra stuff that openstack does" and we do not need to worry about it here. we can continue doing that in puppet as long as we want. and some day we can stop. | 22:05 |
| fungi | the keys we put on those images today are also keys which have access to the zuul launchers. we can hop through to add our keys as needed | 22:05 |
| clarkb | since the reast of the config is pretty us specific and so on | 22:05 |
| greghaynes | that was my way to work around duplication | 22:05 |
| *** e0ne has joined #openstack-infra | 22:05 | |
| *** gluytium has quit IRC | 22:05 | |
| pabelanger | this is how we are doing it today, with our zuul-worker element: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker | 22:05 |
| *** e0ne has quit IRC | 22:05 | |
| *** robot has joined #openstack-infra | 22:05 | |
| pabelanger | we're getting the data from the local nodepool.o.o filesystem with devuser | 22:06 |
| *** ajmiller has quit IRC | 22:07 | |
| *** xyang1 has quit IRC | 22:07 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Remove gate-storyboard-js-integration to experimental https://review.openstack.org/333604 | 22:08 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Move gate-storyboard-js-integration to experimental https://review.openstack.org/333604 | 22:08 |
| ianw | pabelanger/jeblair: well, discarding a more generic deployment mechanism, we might as well just hack in a one-off environment variable for zuul-worker to consume and do an "adduser" and forget about this | 22:08 |
| *** denisra has joined #openstack-infra | 22:09 | |
| ianw | hack the variable into nodepool.conf ... populated by puppet (for us) | 22:09 |
| pabelanger | isn't that basically what devuser element does today? | 22:10 |
| *** signed8bit is now known as signed8bit_Zzz | 22:11 | |
| *** derekh has quit IRC | 22:11 | |
| ianw | pabelanger: almost ... let's make something that doesn't conflict with existing users of devuser, and only deploys the one key, instead of grabbing anything in /home/nodepool/.ssh/authorized_keys | 22:11 |
| jeblair | okay. we'd probably actually need to have the nodepool config point to a file on disk instead of template it in (because we don't and can't template stuff into that file since it's in project-config) | 22:11 |
| jeblair | but the actual key can be put into place by system-config via puppet+hiera | 22:12 |
| *** davidlenwell has quit IRC | 22:12 | |
| ianw | right | 22:12 |
| Shrews | jeblair: question... is it possible that the repeatedly failing jobs have more output than other jobs? | 22:12 |
| clarkb | Shrews: in the case of the puppet job at least I think that is the case, since it spews a lot of verbose puppet things then a lot of verbose tempest things whereas our normal teempest just are relatively short devstack related stuff then tempest | 22:13 |
| *** wznoinsk is now known as wznoinsk_ | 22:13 | |
| clarkb | Shrews: but I would have to double check against a sample to be sure | 22:13 |
| ianw | jeblair / pabelanger : ok, i can get something going in that direction. pabelanger -- your ansible idea is much more the type of thing if we're going to deploy all of the sysadmin users, and do firewall setup, etc etc, so worth fleshing out | 22:13 |
| jeblair | Shrews: but ansible module output? no -- zuul_runner returns a very simple fixed size structure | 22:13 |
| *** davidlenwell has joined #openstack-infra | 22:13 | |
| jeblair | ianw: ++ | 22:13 |
| jeblair | clarkb: ^^ | 22:13 |
| clarkb | I still say devuser is what I would do... | 22:14 |
| Shrews | jeblair: not ansible module output, but the job itself (the stdout/stderr) | 22:14 |
| *** cody-somerville has quit IRC | 22:14 | |
| clarkb | but don't feel so stronly to demand it | 22:14 |
| jeblair | Shrews: okay, but ansible doesn't see that, it goes to a file in /tmp | 22:14 |
| pabelanger | jeblair: right. And that goes back to my comment about the grey line | 22:14 |
| pabelanger | I think it would be good to list out our minimal things we need to do to get nodepool / DIB launched. | 22:15 |
| pabelanger | As I understand it now, it is create zuul user add SSH key | 22:15 |
| jeblair | Shrews: (remember, we're not running 'shell' which does return stdout; we're running zuul_runner which essentially only returns success/failure) | 22:15 |
| greghaynes | re: devuser, if you only need 1 user that works. Its basically the 'hello, world!' of element's which is why I mentioned making another - if it isnt going to just work then its probably just as easy to make your own. | 22:15 |
| Shrews | jeblair: right. k | 22:15 |
| ianw | greghaynes: yep, just do the adduser by hand in zuul-worker. devuser is fine for exactly what it says, a developer adding a user to just quickly log in and test | 22:16 |
| *** ajmiller has joined #openstack-infra | 22:16 | |
| jeblair | Shrews, pabelanger: also, i hand-reverted the known_hosts stuff, and i'm still getting the connection errors. -vvv and debug=1 are giving me nothing useful to go on | 22:16 |
| *** edmondsw has quit IRC | 22:16 | |
| pabelanger | jeblair: sure, I am fine with the revert. When you have spare time I would be interested in looking at the logs if possible | 22:17 |
| Shrews | jeblair: well, i'd never suggest running from HEAD anyway | 22:17 |
| clarkb | greghaynes: ya we onlt need one user "zuul" | 22:17 |
| *** matt6434 is now known as mattoliverau | 22:18 | |
| jeblair | pabelanger: no -- i'm not reverting it -- i'm trying to see what's causing ansible HEAD to fail | 22:18 |
| jeblair | Shrews: okay, should we abandon this and just apply your patch to 2.1.0.0? | 22:18 |
| Shrews | jeblair: my patch does not work | 22:18 |
| *** ddieterly has quit IRC | 22:18 | |
| Shrews | jeblair: still trying to figure out wth is going on | 22:18 |
| *** mdrabe has quit IRC | 22:19 | |
| jeblair | (i was trying to be a good ansible citizen and help test, but if it's more like "it's usually broke don't bother" that's cool) | 22:19 |
| pabelanger | jeblair: oh, confused, sorry. reverted locally | 22:19 |
| jeblair | pabelanger: right. and it still failed, so that seems unlikely to be the problem | 22:19 |
| Shrews | jeblair: we have shade tests against ansible HEAD to help them catch their breaking changes :) | 22:19 |
| *** gluytium has joined #openstack-infra | 22:20 | |
| clarkb | Shrews: those probably all delegate to localhost and use a local connection so don't catch ssh problems. :) | 22:20 |
| *** pradk has quit IRC | 22:21 | |
| Shrews | clarkb: yup | 22:21 |
| pabelanger | 127.0.0.2 :) | 22:21 |
| *** wznoinsk has joined #openstack-infra | 22:21 | |
| Shrews | clarkb: point being, HEAD breaks a lot | 22:21 |
| *** wznoinsk_ has quit IRC | 22:22 | |
| Shrews | jeblair: ah, ok. i think i may have a working something. gimme a bit | 22:23 |
| *** roxanaghe has quit IRC | 22:24 | |
| *** acoles_ is now known as acoles | 22:24 | |
| *** rbrndt_ has quit IRC | 22:24 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist https://review.openstack.org/319337 | 22:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 22:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat https://review.openstack.org/317730 | 22:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job https://review.openstack.org/296765 | 22:24 |
| *** pilgrimstack has quit IRC | 22:25 | |
| *** rlandy has quit IRC | 22:25 | |
| *** erlon has quit IRC | 22:26 | |
| Shrews | jeblair: https://github.com/Shrews/ansible-modules-core/blob/async_atomic_move/utilities/logic/async_wrapper.py is updated | 22:26 |
| Shrews | jeblair: it uses os.rename() instead of the atomic_move() func b/c we cannot import that without breaking async | 22:27 |
| Shrews | jeblair: bcoca suggested that | 22:27 |
| *** cody-somerville has joined #openstack-infra | 22:27 | |
| clarkb | Shrews: does async_wrapper run on windows too? if so rename isn't a thing iirc | 22:28 |
| clarkb | Shrews: so you might need to special case windows | 22:28 |
| *** tosky has quit IRC | 22:28 | |
| Shrews | clarkb: i have no idea | 22:29 |
| *** Sukhdev has quit IRC | 22:29 | |
| Shrews | clarkb: if this solves the immediate problem, we can worry about it then | 22:29 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist https://review.openstack.org/319337 | 22:30 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 22:30 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat https://review.openstack.org/317730 | 22:30 |
| clarkb | Shrews: ++ | 22:30 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job https://review.openstack.org/296765 | 22:30 |
| *** Sukhdev has joined #openstack-infra | 22:30 | |
| *** roxanaghe has joined #openstack-infra | 22:34 | |
| Shrews | jeblair: you got a patch for the suggested debug line in the executor? | 22:34 |
| *** irdkehn_ has joined #openstack-infra | 22:34 | |
| *** Sukhdev has quit IRC | 22:35 | |
| *** irdkehn_ is now known as dkehn | 22:35 | |
| *** Sukhdev has joined #openstack-infra | 22:35 | |
| *** javeriak has joined #openstack-infra | 22:35 | |
| jeblair | Shrews: no -- but i think we should just interpolate it into the failure message for our purposes. that's not what they want upstream, but i don't want to run with debug=1 -- it's *way* too much data. so since we're local-patching anyway, let's just do that. | 22:36 |
| jeblair | clarkb: after i checkout v2.1.0.0, how do i make the submodules be in the correct state? | 22:37 |
| Shrews | jeblair: git submodule update --init --recursive | 22:37 |
| Shrews | i think | 22:37 |
| jeblair | oh ok, --init doesn't do what i thought it does then :) | 22:37 |
| *** tpsilva has quit IRC | 22:37 | |
| clarkb | jeblair: git submodule update oh ya shrews has it | 22:38 |
| jeblair | yep that did it, thx | 22:39 |
| *** vhosakot has quit IRC | 22:39 | |
| *** javeriak_ has joined #openstack-infra | 22:39 | |
| *** javeriak has quit IRC | 22:40 | |
| *** Sukhdev has quit IRC | 22:40 | |
| *** dkehn has quit IRC | 22:41 | |
| *** irdkehn has joined #openstack-infra | 22:41 | |
| *** yamahata has joined #openstack-infra | 22:41 | |
| jeblair | Shrews: your patch is on the devel branch, right? | 22:41 |
| Shrews | jeblair: no | 22:41 |
| Shrews | jeblair: async_atomic_move | 22:42 |
| jeblair | okay, let me try harder to github | 22:42 |
| Shrews | jeblair: easier to copy pasta | 22:42 |
| jeblair | Shrews: no i mean it was *based* on devel? | 22:42 |
| Shrews | jeblair: oh, yeah. up-to-date devel | 22:42 |
| clarkb | jeblair: be careful doing that | 22:43 |
| jeblair | right, i tried to pull it onto 2100 and it conflicted | 22:43 |
| jeblair | i will just pull it into a devel branch then cherry pick | 22:43 |
| Shrews | gah. must have been changes b/w 2.1 and HEAD | 22:43 |
| Shrews | jeblair: sorry, didn't consider that | 22:43 |
| jeblair | no prob, it cherry-picks cleanly | 22:43 |
| Shrews | it's a small diff | 22:43 |
| *** dkehn_ has joined #openstack-infra | 22:44 | |
| *** rdx_matt has joined #openstack-infra | 22:45 | |
| *** roxanaghe has quit IRC | 22:46 | |
| jeblair | btw, this would be much easier if we had afs on all the hosts | 22:48 |
| clarkb | jeblair: this? | 22:48 |
| jeblair | pushing a git commit onto a repo on all the launchers | 22:48 |
| jeblair | pulling is easy, pushing is harder | 22:49 |
| clarkb | ah | 22:49 |
| jeblair | but pulling from file:///afs/openstack.org is easy | 22:49 |
| mordred | jeblair: rsync? | 22:50 |
| clarkb | I tend to use patch files but there are proably better ways | 22:51 |
| jeblair | i can easily use ansible to run a command on all the hosts | 22:51 |
| jeblair | but if i want to push something to them, i'm into playbook territory | 22:51 |
| *** Sukhdev has joined #openstack-infra | 22:54 | |
| Shrews | let's hope we don't have to revert | 22:55 |
| jeblair | no kidding :) | 22:55 |
| jhesketh | Morning | 22:56 |
| Shrews | i'm pretty sure it doesn't BREAK anything, it's the fixing aspect where i have doubts | 22:56 |
| jeblair | okay, i made temporary branches in both ansible and the core submodule; i pushed those in a bash for-loop to each host, then switched back to ansible to do the checkouts and submodule inits | 22:57 |
| jeblair | the repo in zl01 looks to be in the correct state | 22:58 |
| clarkb | jeblair: double check the submodules as those can be twitchy | 22:58 |
| jeblair | i will manually install it now only on that host and see if it fails to break things | 22:58 |
| clarkb | its one of the ways you can lose data with git | 22:58 |
| jeblair | clarkb: yeah, git log in the core submodule looks correct | 22:58 |
| clarkb | cool | 22:58 |
| jeblair | i see Shrews commit at the top | 22:58 |
| *** Sukhdev has quit IRC | 22:59 | |
| *** gordc has quit IRC | 22:59 | |
| anteaya | morning jhesketh | 22:59 |
| *** salv-orl_ has joined #openstack-infra | 22:59 | |
| jeblair | oh, pip install . isn't going to do anything | 22:59 |
| *** sdague has quit IRC | 22:59 | |
| jeblair | pip install -U . is going to break because it can't compile something | 22:59 |
| jeblair | hrm. | 22:59 |
| jeblair | maybe i should pip uninstall ansible; pip install .; and hope/assume that jobs launched in the interim will get kicked back for rescheduling | 23:00 |
| jeblair | no one has screamed, so i will do that. | 23:01 |
| clarkb | I am quickly running out of stema. was up at 3am to do an airport run today | 23:01 |
| *** salv-orlando has quit IRC | 23:02 | |
| jeblair | nope, turns out that was impactful | 23:02 |
| jeblair | lots of POST_FAILURES | 23:02 |
| * Shrews hands clarkb more stema | 23:02 | |
| jeblair | next time we do this we may need to do a zuul-launcher stop to abort all the jobs | 23:02 |
| clarkb | jeblair: stop, update ansible, start? | 23:03 |
| jeblair | or does anyone know of a way to force pip to "re-install" a version? | 23:03 |
| *** Sukhdev has joined #openstack-infra | 23:03 | |
| jeblair | clarkb: yeah; failing that ^ | 23:03 |
| *** ams__ has quit IRC | 23:03 | |
| jeblair | http://stackoverflow.com/questions/19548957/can-i-force-pip-to-reinstall-the-current-version | 23:04 |
| clarkb | --force-reinstall but that seems to be a type of --upgrade | 23:04 |
| jeblair | that's clear as mud | 23:04 |
| jeblair | yeah, and i suspect upgrade will break | 23:04 |
| clarkb | you could try --no-deps | 23:04 |
| clarkb | to keep those fixed but update ansible | 23:04 |
| jeblair | aha | 23:05 |
| jeblair | i will try that on zl02 | 23:05 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-mysql_backup: Use puppetlabs-mysql to install mysql client package https://review.openstack.org/333619 | 23:05 |
| *** tphummel has joined #openstack-infra | 23:07 | |
| *** Sukhdev has quit IRC | 23:07 | |
| *** Sukhdev has joined #openstack-infra | 23:11 | |
| jeblair | okay, that worked, but basically did the same thing as on zl01. just slightly faster | 23:12 |
| jeblair | still killed a bunch of jobs though. | 23:12 |
| jeblair | i'm hard stopping zl03 | 23:13 |
| jeblair | reinstalled and started | 23:15 |
| jeblair | i'll repeat that on 04-07 | 23:15 |
| pabelanger | the hope was to keep current ansible-playbook jobs working? | 23:15 |
| *** bhavik has quit IRC | 23:16 | |
| jeblair | pabelanger: yeah, but the reinstall is too slow + too many jobs running | 23:16 |
| pabelanger | ya, too bad. would have been nice | 23:16 |
| openstackgerrit | Stephanie Miller proposed openstack/diskimage-builder: Make svc-map support overridden elements https://review.openstack.org/333597 | 23:16 |
| *** rcernin has joined #openstack-infra | 23:16 | |
| *** zqfan has joined #openstack-infra | 23:16 | |
| rcernin | Got d/c anyone for Klingon? | 23:18 |
| *** rdx_matt has quit IRC | 23:19 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: UPDATE REPO FOR DELOREAN_REPO_URL https://review.openstack.org/333198 | 23:19 |
| *** dprince has quit IRC | 23:19 | |
| jeblair | all launchers updated | 23:20 |
| jeblair | i'm going to delete the held nodes now so we get a new batch on the next failures | 23:20 |
| *** sdague has joined #openstack-infra | 23:20 | |
| *** hongbin has quit IRC | 23:21 | |
| *** vhosakot has joined #openstack-infra | 23:22 | |
| jeblair | #status log zuul launchers are all running locally patched ansible (source in ~root/ansible) to correct and/or further debug async timeout issue | 23:22 |
| openstackstatus | jeblair: finished logging | 23:22 |
| jeblair | infra-root, EmilienM, Shrews: ^ | 23:22 |
| clarkb | jeblair: maybe link to the patch you are testing too? | 23:22 |
| *** aeng has joined #openstack-infra | 23:22 | |
| jeblair | #status log https://github.com/Shrews/ansible-modules-core/commit/d11cb0d9a1c768735d9cb4b7acc32b971b524f13 | 23:23 |
| openstackstatus | jeblair: finished logging | 23:23 |
| jeblair | (plus also a local-only debug info patch) | 23:23 |
| Shrews | oh good | 23:24 |
| EmilienM | Woot | 23:24 |
| clarkb | Shrews: doing quick code review on that change, does the rename at the end of your diff want to be in a finally block? | 23:26 |
| anteaya | clarkb: your parents went home today? | 23:26 |
| clarkb | anteaya: yes | 23:26 |
| anteaya | nice | 23:26 |
| anteaya | I'm glad they were able to help you with the move | 23:26 |
| Shrews | clarkb: hrm, i think both the close and the rename should be, yes | 23:27 |
| anteaya | I'll add to Shrews offer of stema | 23:27 |
| Shrews | clarkb: good spot | 23:27 |
| Shrews | jeblair: so.... | 23:27 |
| pabelanger | jeblair: Shrews: great work | 23:27 |
| Shrews | clarkb: ideally, that's not the code path we're hitting anyway | 23:28 |
| clarkb | Shrews: ya the old code seemed to lack it as well | 23:28 |
| Shrews | clarkb: jeblair: i'm going to update the patch to include the finally clause. likely not needed right away, but should update it if this seems to fix things | 23:30 |
| *** SumitNaiksatam has quit IRC | 23:30 | |
| *** acoles is now known as acoles_ | 23:30 | |
| clarkb | sounds good | 23:30 |
| jeblair | Shrews: sounds good. i won't apply it now since it doesn't seem to be critical, but if we have to update again, we'll fold it in | 23:30 |
| *** sdake_ has joined #openstack-infra | 23:31 | |
| *** banix has quit IRC | 23:32 | |
| *** bswartz has quit IRC | 23:32 | |
| *** zz_dimtruck is now known as dimtruck | 23:33 | |
| *** sdake has quit IRC | 23:35 | |
| Shrews | jeblair: clarkb: https://github.com/Shrews/ansible-modules-core/commit/e35d116657ec674e588efe3824804ad09a6bd64e | 23:35 |
| Shrews | confirmed that patch doesn't break a simple playbook either | 23:40 |
| jeblair | if only we could run simple playbooks ;) | 23:42 |
| Shrews | srsly | 23:42 |
| Shrews | jeblair: finally going to EOD. ping me if needed | 23:42 |
| Shrews | moar scotch time | 23:43 |
| *** hichihara has quit IRC | 23:46 | |
| *** dmsimard is now known as dmsimard|pto | 23:48 | |
| *** sarob_ has quit IRC | 23:48 | |
| *** amotoki has quit IRC | 23:48 | |
| *** ajmiller has quit IRC | 23:48 | |
| *** sdake_ has quit IRC | 23:53 | |
| *** daemontool has joined #openstack-infra | 23:53 | |
| *** javeriak_ has quit IRC | 23:54 | |
| *** mriedem has joined #openstack-infra | 23:55 | |
| *** gluytium has quit IRC | 23:55 | |
| *** sdake has joined #openstack-infra | 23:58 | |
| *** xarses has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!