| *** mudpuppy has quit IRC | 00:00 | |
| *** dimtruck has quit IRC | 00:00 | |
| *** mancdaz has quit IRC | 00:00 | |
| clarkb | ianychoi: ok all the pre steps except for cacti are done now | 00:00 |
|---|---|---|
| *** rackertom has quit IRC | 00:00 | |
| *** scottda has quit IRC | 00:00 | |
| *** sindhu has quit IRC | 00:00 | |
| *** betherly has quit IRC | 00:00 | |
| *** edwarnicke has quit IRC | 00:00 | |
| *** spotz has quit IRC | 00:00 | |
| *** lewo has quit IRC | 00:00 | |
| clarkb | ianychoi: jhesketh should we still plan for wednesday afternnon PST? That will be late night UTC time wednesday and early thursday your time | 00:00 |
| *** madhuvishy has quit IRC | 00:01 | |
| clarkb | ianw: ^ is rough plan we came up with at PTG as for timing. Let us know if that doens't work for you though I think between ianychoi jhesketh and myself we should have plenty of helpt | 00:01 |
| *** Adri2000 has quit IRC | 00:01 | |
| *** aeng has quit IRC | 00:01 | |
| *** aglarendil has quit IRC | 00:01 | |
| *** ashtokolov has quit IRC | 00:01 | |
| *** geekinutah has quit IRC | 00:01 | |
| *** amit213 has quit IRC | 00:01 | |
| *** comstud has quit IRC | 00:01 | |
| *** mjb has quit IRC | 00:01 | |
| clarkb | ianychoi: jhesketh what is an appropriate UTC time so you don't have to wake up early? 2300? | 00:01 |
| *** patrickeast has quit IRC | 00:02 | |
| *** johndperkins has quit IRC | 00:02 | |
| *** berendt has quit IRC | 00:02 | |
| *** esheffield has quit IRC | 00:02 | |
| *** melwitt has quit IRC | 00:02 | |
| *** zeus has quit IRC | 00:02 | |
| clarkb | thats 3pm PST, I'm happy with earlier too if it owrks but don't want to wake you up too early :) | 00:02 |
| *** mhayden has quit IRC | 00:02 | |
| *** wendar has quit IRC | 00:02 | |
| *** kuromagi has quit IRC | 00:02 | |
| *** odyssey4me has quit IRC | 00:02 | |
| *** jmccrory has quit IRC | 00:02 | |
| *** esheffield has joined #openstack-infra | 00:02 | |
| ianychoi | clarkb, I see. I am quite fine to adjust with the timezone - with clarkb jhesketh and ianw . 23:00 UTC is 8:00 AM here and fine :) | 00:02 |
| *** kuromagi has joined #openstack-infra | 00:03 | |
| ianychoi | Interestingly during PTG I slept before 10 pm and woke up at 4 am.. For my usual life I sleep around 2-3am and get up around 8:a0 am | 00:03 |
| clarkb | jhesketh: if 2300UTC works for you lets go for that. I think an hour downtime should be plenty too, just based on how long the rsync and db backup took | 00:03 |
| ianychoi | s/a0/30 | 00:03 |
| *** sindhu has joined #openstack-infra | 00:04 | |
| *** madhuvishy has joined #openstack-infra | 00:04 | |
| *** aglarendil has joined #openstack-infra | 00:05 | |
| *** ashtokolov has joined #openstack-infra | 00:05 | |
| *** scottda has joined #openstack-infra | 00:05 | |
| *** hongbin has quit IRC | 00:05 | |
| *** geekinutah has joined #openstack-infra | 00:06 | |
| *** patrickeast has joined #openstack-infra | 00:06 | |
| *** amit213 has joined #openstack-infra | 00:06 | |
| *** berendt has joined #openstack-infra | 00:06 | |
| *** ewindisch has quit IRC | 00:06 | |
| *** cody-somerville has quit IRC | 00:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: gnocchi: only run upgrade-from-2.2 from stable/3.1 https://review.openstack.org/438428 | 00:06 |
| *** wendar has joined #openstack-infra | 00:06 | |
| *** ewindisch has joined #openstack-infra | 00:06 | |
| *** mancdaz has joined #openstack-infra | 00:07 | |
| *** zeus has joined #openstack-infra | 00:07 | |
| *** mjb has joined #openstack-infra | 00:08 | |
| *** zeus is now known as Guest68724 | 00:08 | |
| *** jamespage has quit IRC | 00:08 | |
| *** dimtruck has joined #openstack-infra | 00:08 | |
| *** jcoufal has quit IRC | 00:09 | |
| *** melwitt has joined #openstack-infra | 00:09 | |
| *** jmccrory has joined #openstack-infra | 00:09 | |
| *** comstud has joined #openstack-infra | 00:09 | |
| *** spotz has joined #openstack-infra | 00:09 | |
| *** melwitt is now known as Guest68076 | 00:09 | |
| *** mudpuppy has joined #openstack-infra | 00:10 | |
| *** jamespage has joined #openstack-infra | 00:10 | |
| *** betherly has joined #openstack-infra | 00:11 | |
| *** mhayden has joined #openstack-infra | 00:11 | |
| *** odyssey4me has joined #openstack-infra | 00:11 | |
| *** jamesdenton has joined #openstack-infra | 00:11 | |
| *** edwarnicke_ has joined #openstack-infra | 00:12 | |
| *** stakeda has joined #openstack-infra | 00:12 | |
| *** Guest68076 is now known as melwitt | 00:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Changing kolla-kubernetes ironic jobs to source type https://review.openstack.org/438581 | 00:13 |
| *** lewo has joined #openstack-infra | 00:14 | |
| *** aeng has joined #openstack-infra | 00:15 | |
| *** johndperkins has joined #openstack-infra | 00:15 | |
| *** EricGonczer_ has joined #openstack-infra | 00:16 | |
| *** darrenc has quit IRC | 00:16 | |
| *** ihrachys has joined #openstack-infra | 00:16 | |
| *** jamielennox|away is now known as jamielennox | 00:16 | |
| *** darrenc has joined #openstack-infra | 00:17 | |
| *** asselin__ has quit IRC | 00:17 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job https://review.openstack.org/431777 | 00:17 |
| *** asselin has joined #openstack-infra | 00:19 | |
| *** sdague has quit IRC | 00:20 | |
| *** rajinir_ has quit IRC | 00:21 | |
| ianw | oh that's like 10am sydney time, so definitely here | 00:22 |
| *** ZZelle has quit IRC | 00:23 | |
| *** esikachev has quit IRC | 00:23 | |
| *** aeng has quit IRC | 00:25 | |
| *** ihrachys has quit IRC | 00:26 | |
| *** s-shiono has joined #openstack-infra | 00:27 | |
| *** gildub has joined #openstack-infra | 00:27 | |
| *** anthonyper has quit IRC | 00:28 | |
| *** yuanying has quit IRC | 00:28 | |
| *** ijw has quit IRC | 00:29 | |
| *** mtanino_ has joined #openstack-infra | 00:29 | |
| *** ijw has joined #openstack-infra | 00:30 | |
| *** ijw has quit IRC | 00:30 | |
| *** mtanino has quit IRC | 00:30 | |
| *** anthonyper has joined #openstack-infra | 00:32 | |
| *** Julien-zte has joined #openstack-infra | 00:32 | |
| *** esikachev has joined #openstack-infra | 00:35 | |
| *** jamesdenton has quit IRC | 00:37 | |
| *** esikachev has quit IRC | 00:40 | |
| *** beslemon has quit IRC | 00:40 | |
| *** aeng has joined #openstack-infra | 00:43 | |
| *** dhinesh has quit IRC | 00:46 | |
| *** Adri2000 has joined #openstack-infra | 00:46 | |
| *** dhinesh has joined #openstack-infra | 00:46 | |
| *** s-shiono has quit IRC | 00:47 | |
| *** yujunz has joined #openstack-infra | 00:52 | |
| *** esikachev has joined #openstack-infra | 00:53 | |
| *** dave-mccowan has joined #openstack-infra | 00:55 | |
| *** esikachev has quit IRC | 00:57 | |
| *** pahuang has quit IRC | 00:57 | |
| *** Guest68724 is now known as zeus` | 01:02 | |
| *** cody-somerville has joined #openstack-infra | 01:02 | |
| *** cody-somerville has joined #openstack-infra | 01:02 | |
| *** zeus` is now known as zeus | 01:03 | |
| *** zeus has joined #openstack-infra | 01:03 | |
| *** sambetts|afk has quit IRC | 01:04 | |
| *** tphummel has quit IRC | 01:05 | |
| *** EricGonczer_ has quit IRC | 01:06 | |
| *** EricGonczer_ has joined #openstack-infra | 01:06 | |
| *** sambetts_ has joined #openstack-infra | 01:07 | |
| *** yujunz is now known as yujunz[away] | 01:08 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Undercloud containers job with tripleo.sh https://review.openstack.org/437411 | 01:10 |
| *** pahuang has joined #openstack-infra | 01:14 | |
| *** pramodrj07 has quit IRC | 01:14 | |
| *** MasterOfBugs has quit IRC | 01:14 | |
| openstackgerrit | Clay Gerrard proposed openstack-infra/project-config master: Add experimental python34 job to pyeclib https://review.openstack.org/438756 | 01:15 |
| *** esikachev has joined #openstack-infra | 01:15 | |
| *** Swami_ has quit IRC | 01:16 | |
| *** esikachev has quit IRC | 01:21 | |
| *** yuanying has joined #openstack-infra | 01:24 | |
| *** thorst has joined #openstack-infra | 01:26 | |
| *** baoli has quit IRC | 01:28 | |
| *** Apoorva_ has joined #openstack-infra | 01:30 | |
| *** dave-mccowan has quit IRC | 01:30 | |
| *** sdake_ has joined #openstack-infra | 01:30 | |
| openstackgerrit | Merged openstack/diskimage-builder master: pip-and-virtualenv: also handle rhel distros https://review.openstack.org/433693 | 01:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: gentoo: do not manually clean /tmp https://review.openstack.org/436101 | 01:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: [suse] remove --no-confirm from zypper invocation https://review.openstack.org/436167 | 01:31 |
| *** sdake has quit IRC | 01:33 | |
| *** Apoorva has quit IRC | 01:33 | |
| *** pvaneck has quit IRC | 01:34 | |
| *** Apoorva_ has quit IRC | 01:34 | |
| *** dave-mccowan has joined #openstack-infra | 01:37 | |
| *** yujunz[away] is now known as yujunz | 01:38 | |
| *** sdake_ has quit IRC | 01:40 | |
| *** gyee has quit IRC | 01:45 | |
| *** adrian_otto has quit IRC | 01:51 | |
| *** liujiong has joined #openstack-infra | 01:51 | |
| *** lespaul has quit IRC | 01:58 | |
| *** dave-mccowan has quit IRC | 01:59 | |
| *** rackertom has joined #openstack-infra | 02:00 | |
| *** Goneri has quit IRC | 02:00 | |
| *** esikachev has joined #openstack-infra | 02:00 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Enable peakmem_tracker in gate https://review.openstack.org/434511 | 02:02 |
| *** baoli has joined #openstack-infra | 02:02 | |
| *** yujunz has quit IRC | 02:03 | |
| *** esikachev has quit IRC | 02:05 | |
| openstackgerrit | Jeremy Zhang proposed openstack-infra/project-config master: Use local_conf for tempest tests on test_unmanage_manage_volume https://review.openstack.org/438347 | 02:05 |
| *** EricGonczer_ has quit IRC | 02:06 | |
| *** eliqiao has quit IRC | 02:07 | |
| *** esikachev has joined #openstack-infra | 02:13 | |
| *** esikachev has quit IRC | 02:17 | |
| *** mtanino_ has quit IRC | 02:17 | |
| *** eliqiao has joined #openstack-infra | 02:19 | |
| *** yujunz has joined #openstack-infra | 02:19 | |
| *** gildub_ has joined #openstack-infra | 02:20 | |
| *** gildub has quit IRC | 02:22 | |
| *** rlandy has quit IRC | 02:24 | |
| *** yujunz is now known as yujunz[away] | 02:27 | |
| *** yujunz[away] is now known as yujunz | 02:27 | |
| *** baoli has quit IRC | 02:32 | |
| tdasilva | does Ian Wienand hang out here? | 02:33 |
| *** aeng has quit IRC | 02:41 | |
| ianw | tdasilva: he does | 02:42 |
| *** thorst has quit IRC | 02:44 | |
| *** thorst has joined #openstack-infra | 02:44 | |
| *** mixos has quit IRC | 02:44 | |
| *** aeng has joined #openstack-infra | 02:45 | |
| *** yanyanhu has joined #openstack-infra | 02:46 | |
| tdasilva | ianw: hi! i guess I will need to take you up on your offer for help on patch 436632 :) | 02:48 |
| tdasilva | ianw: trying to figure out what's wrong with this patch now: https://review.openstack.org/#/c/438758/2 | 02:48 |
| tdasilva | mkfs.xfs fails on centos ?? | 02:49 |
| *** thorst has quit IRC | 02:49 | |
| tdasilva | ianw: http://logs.openstack.org/58/438758/2/experimental/gate-swift-tox-xfs-tmp-func-centos-7/c702bf9/console.html | 02:50 |
| ianw | tdasilva: ok, on the first one, we'll need the epel python3 i guess | 02:50 |
| tdasilva | ianw: it seems like even with epel, only py34 is avaiable?? | 02:51 |
| tdasilva | does that seem right to you or am i missing something? | 02:51 |
| *** Rockyg has quit IRC | 02:51 | |
| ianw | yes, the epel python3 for centos is 3.4 currently | 02:53 |
| ianw | as for the mkfs.xfs ... that will need the xfsprogs | 02:54 |
| ianw | package | 02:54 |
| tdasilva | ianw: xfsprogs was installed, couple lines up in that log | 02:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove rdo-qemu-ev workaround https://review.openstack.org/425753 | 02:55 |
| ianw | tdasilva: ahhh, ok. i wonder, i bet that lives in /sbin and might need to be called directly? | 02:56 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Drop ceph workaround for mitaka https://review.openstack.org/425754 | 02:56 |
| ianw | /usr/sbin/mkfs.xfs i guess | 02:56 |
| *** gongysh has joined #openstack-infra | 02:56 | |
| ianw | that should work since it's on a file, not a block device | 02:56 |
| ianw | it's not running as root | 02:57 |
| *** yujunz is now known as yujunz[away] | 02:57 | |
| *** yujunz[away] is now known as yujunz | 02:57 | |
| bkero | yarp, should work fine | 02:58 |
| tdasilva | ianw: i can try changing that, but where is that line coming from??? | 02:58 |
| tdasilva | what script is trying to execute mkfs.xfs ? | 02:58 |
| *** hongbin has joined #openstack-infra | 02:59 | |
| bkero | tdasilva, ianw: /usr/sbin likely isn't in the path | 02:59 |
| bkero | Maybe PATH=${PATH}:/usr/sbin; mkfs.xfs? | 02:59 |
| bkero | with OLDPATH=$PATH; ... ; PATH=$OLDPATH | 03:00 |
| *** tphummel has joined #openstack-infra | 03:00 | |
| ianw | bkero: i think just /sbin/mkfs.xfs | 03:00 |
| ianw | tdasilva: that's a macro defined in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n986 | 03:00 |
| *** hongbin has quit IRC | 03:00 | |
| *** hongbin has joined #openstack-infra | 03:01 | |
| bkero | ianw: on my f25 host it's /usr/sbin/mkfs.xfs | 03:01 |
| bkero | CentOS7 might be different though | 03:01 |
| ianw | bkero: well /sbin -> /usr/sbin anyway? | 03:01 |
| tdasilva | ianw, bkero: what about ubuntu? | 03:02 |
| *** gildub_ has quit IRC | 03:02 | |
| bkero | ianw: On F25 yeah, /sbin -> /usr/sbin/ | 03:02 |
| ianw | tdasilva: you want to just propose changing that to /sbin/mkfs.xfs? that should be portable | 03:02 |
| tdasilva | ianw: sure, will do that | 03:02 |
| bkero | /sbin/mkfs.xfs is what I see on trusty | 03:03 |
| *** EricGonczer_ has joined #openstack-infra | 03:03 | |
| *** tphummel has quit IRC | 03:04 | |
| ianw | tdasilva: btw i kind of remember that from ages ago, but http://codesearch.openstack.org/?q=mkfs.xfs&i=nope&files=&repos= also is a good way to find stuff like that, then a bit of contextual searching often helps | 03:04 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: update the mkfs.xfs path https://review.openstack.org/438776 | 03:04 |
| tdasilva | ianw: that is super neat! thanks for sharing | 03:05 |
| ianw | tdasilva: can you just put in a comment that it isn't on the path for centos and that's why we want to do it. just to give context for others who see the change going by | 03:06 |
| ianw | in the change comment i mean | 03:06 |
| tdasilva | ianw: heh, good point...trying to go fast, my bad... | 03:06 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: update the mkfs.xfs path https://review.openstack.org/438776 | 03:07 |
| ianw | bkero: ^ happy with that? I think we can merge | 03:09 |
| *** knangia has quit IRC | 03:10 | |
| bkero | ianw: lgtm | 03:11 |
| bkero | I haven't checked my gentoo hosts, but I'm sure it can be fixed later if anybody encounters it | 03:11 |
| bkero | ;) | 03:11 |
| *** dimtruck is now known as zz_dimtruck | 03:12 | |
| ianw | tdasilva: so with the erasurecode thing, we should have RDO repos on the centos hosts | 03:13 |
| *** zz_dimtruck is now known as dimtruck | 03:13 | |
| ianw | installing the rdo package would not work anyway from bindep, i don't think. because the repos won't be around unitl the *next* run of yum | 03:13 |
| tdasilva | ianw: oh i see, so no need to add centos-release-openstack-newton | 03:14 |
| tdasilva | nice! | 03:15 |
| tdasilva | just saw your comment | 03:15 |
| bkero | Can watch on https://trunk.rdoproject.org/centos7-master-head/report.html | 03:15 |
| *** pahuang has quit IRC | 03:16 | |
| tdasilva | ianw, bkero: changing subjects a bit, what's the deal with py35 testing on centos? My understanding is that py34 was restricted to <= Mitaka ? | 03:16 |
| *** esikachev has joined #openstack-infra | 03:17 | |
| ianw | tdasilva: this is an interesting question i hadn't thought too much about | 03:17 |
| ianw | but should | 03:17 |
| *** EricGonczer_ has quit IRC | 03:18 | |
| tdasilva | bkero: can't find liberasurecode on https://trunk.rdoproject.org/centos7-master-head/report.html should it be there? | 03:18 |
| *** tqtran has quit IRC | 03:19 | |
| bkero | tdasilva: probably not, it's not in here: https://trunk.rdoproject.org/centos7/51/c9/51c922e6ff55c81d8d71a5630fa0d4778664d3f2_047e9ec4/ | 03:22 |
| bkero | So it's likely in another repo | 03:22 |
| openstackgerrit | Merged openstack-infra/project-config master: update the mkfs.xfs path https://review.openstack.org/438776 | 03:22 |
| *** esikachev has quit IRC | 03:23 | |
| bkero | tdasilva: http://cbs.centos.org/koji/packageinfo?packageID=643 | 03:23 |
| ianw | i guess the 3.5 solution is in software collections ... | 03:25 |
| bkero | Is OpenStack submitting for GSoC 2017? | 03:25 |
| tdasilva | bkero: thanks | 03:27 |
| tdasilva | ianw: thanks for all the help | 03:27 |
| *** pahuang has joined #openstack-infra | 03:29 | |
| *** gildub has joined #openstack-infra | 03:30 | |
| tdasilva | ianw, bkero: neat, got centos func tests passing: http://logs.openstack.org/58/438758/3/experimental/gate-swift-tox-xfs-tmp-func-centos-7/cd2467a/console.html | 03:31 |
| tdasilva | thanks a bunch! | 03:31 |
| ianw | tdasilva: np | 03:35 |
| ianw | http://logs.openstack.org/58/438758/3/experimental/gate-swift-tox-xfs-tmp-func-centos-7/cd2467a/console.html#_2017-02-28_03_28_55_365380 | 03:35 |
| ianw | heh, we override TMPDIR but don't unmount it before we try to rm it | 03:35 |
| *** vikrant has joined #openstack-infra | 03:35 | |
| bkero | jhesketh: do you know about gsoc17? | 03:35 |
| *** zhenguo has joined #openstack-infra | 03:37 | |
| *** thorst has joined #openstack-infra | 03:45 | |
| *** thorst has quit IRC | 03:50 | |
| *** cshastri has joined #openstack-infra | 03:50 | |
| *** dhinesh has quit IRC | 03:50 | |
| *** Julien-zte has quit IRC | 03:50 | |
| *** Julien-zte has joined #openstack-infra | 03:51 | |
| *** baoli has joined #openstack-infra | 03:51 | |
| *** Julien-zte has quit IRC | 03:51 | |
| *** Julien-zte has joined #openstack-infra | 03:52 | |
| *** tphummel has joined #openstack-infra | 03:54 | |
| *** gouthamr has quit IRC | 03:54 | |
| *** nmathew has joined #openstack-infra | 03:56 | |
| *** yamahata_ has quit IRC | 03:57 | |
| *** Julien-zte has quit IRC | 03:58 | |
| *** Julien-zte has joined #openstack-infra | 03:59 | |
| *** aeng has quit IRC | 04:01 | |
| *** yujunz is now known as yujunz[away] | 04:02 | |
| *** yujunz[away] is now known as yujunz | 04:02 | |
| *** liusheng has quit IRC | 04:05 | |
| *** liusheng has joined #openstack-infra | 04:07 | |
| *** rraja has joined #openstack-infra | 04:10 | |
| *** gongysh has quit IRC | 04:12 | |
| *** yujunz has quit IRC | 04:15 | |
| *** hongbin has quit IRC | 04:17 | |
| *** aeng has joined #openstack-infra | 04:19 | |
| *** psachin has joined #openstack-infra | 04:23 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 04:29 | |
| *** tlian has quit IRC | 04:29 | |
| openstackgerrit | toan.t.pham proposed openstack-infra/jenkins-job-builder master: Add folder create for cloudbee folder jobs https://review.openstack.org/438795 | 04:31 |
| *** EricGonczer_ has joined #openstack-infra | 04:33 | |
| *** EricGonczer_ has quit IRC | 04:35 | |
| *** jasondotstar has quit IRC | 04:35 | |
| *** esikachev has joined #openstack-infra | 04:35 | |
| *** raissa has quit IRC | 04:36 | |
| *** raissa has joined #openstack-infra | 04:36 | |
| *** esikachev has quit IRC | 04:40 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: temprevert Swift Global EC Cluster Support https://review.openstack.org/438797 | 04:45 |
| *** jasondotstar has joined #openstack-infra | 04:48 | |
| *** Jeffrey4l__ has quit IRC | 04:49 | |
| *** Jeffrey4l__ has joined #openstack-infra | 04:52 | |
| *** baoli has quit IRC | 04:54 | |
| *** sreeram_v has joined #openstack-infra | 04:54 | |
| *** _ody has quit IRC | 04:57 | |
| *** _ody has joined #openstack-infra | 05:00 | |
| *** sdake has joined #openstack-infra | 05:03 | |
| *** sdake has quit IRC | 05:08 | |
| *** sdake_ has joined #openstack-infra | 05:08 | |
| *** amitkuhikar has joined #openstack-infra | 05:09 | |
| amitkuhikar | Hi I am getting below error with "dsvm-tempest-cinder-driver" CI run | 05:10 |
| amitkuhikar | 10:21:28 + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L692: /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/dsvm-tempest-cinder-driver/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh executable=/bin/bash' | 05:10 |
| amitkuhikar | 10:21:31 ERROR: the main setup script run by this job failed - exit code: 2 | 05:10 |
| amitkuhikar | 10:21:40 Build step 'Execute shell' marked build as failure | 05:11 |
| amitkuhikar | Log : http://54.169.104.103/92/336092/28/check/dsvm-tempest-cinder-driver/83c26f3/ | 05:11 |
| amitkuhikar | can someone please help on that | 05:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo/layout: run scenarios on haproxy manifests https://review.openstack.org/438803 | 05:12 |
| *** esikachev has joined #openstack-infra | 05:14 | |
| *** esikachev has quit IRC | 05:18 | |
| *** tphummel has quit IRC | 05:20 | |
| *** Nilesh has joined #openstack-infra | 05:24 | |
| *** yamamoto has quit IRC | 05:24 | |
| *** adriant has quit IRC | 05:30 | |
| *** amitkuhikar has quit IRC | 05:32 | |
| *** rraja has quit IRC | 05:33 | |
| *** armax has quit IRC | 05:36 | |
| *** john-davidge has joined #openstack-infra | 05:38 | |
| *** jaosorior has joined #openstack-infra | 05:41 | |
| *** aeng has quit IRC | 05:42 | |
| *** larainema has quit IRC | 05:42 | |
| *** john-davidge has quit IRC | 05:43 | |
| *** dhinesh has joined #openstack-infra | 05:43 | |
| *** sdake_ has quit IRC | 05:47 | |
| *** sdake has joined #openstack-infra | 05:47 | |
| *** shu-mutou has quit IRC | 05:48 | |
| *** yujunz has joined #openstack-infra | 05:51 | |
| *** amitkuhikar has joined #openstack-infra | 05:58 | |
| *** aeng has joined #openstack-infra | 05:59 | |
| amitkuhikar | hi I am getting below issue while running the "dsvm-tempest-cinder-driver" | 06:00 |
| amitkuhikar | 10:21:28 + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L692: /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/dsvm-tempest-cinder-driver/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh executable=/bin/bash' | 06:00 |
| amitkuhikar | 10:21:31 ERROR: the main setup script run by this job failed - exit code: 2 | 06:00 |
| amitkuhikar | 10:21:40 Build step 'Execute shell' marked build as failure | 06:00 |
| amitkuhikar | log : http://54.169.104.103/92/336092/28/check/dsvm-tempest-cinder-driver/83c26f3/ | 06:00 |
| amitkuhikar | can someone please help on the same | 06:00 |
| *** Douhet has quit IRC | 06:00 | |
| *** Douhet has joined #openstack-infra | 06:01 | |
| *** dhajare has joined #openstack-infra | 06:03 | |
| *** salv-orlando has joined #openstack-infra | 06:03 | |
| *** yamamoto has joined #openstack-infra | 06:04 | |
| *** pgadiya has joined #openstack-infra | 06:05 | |
| *** esikachev has joined #openstack-infra | 06:06 | |
| *** mhickey has joined #openstack-infra | 06:08 | |
| *** sdake_ has joined #openstack-infra | 06:10 | |
| *** Jeffrey4l__ is now known as Jeffrey4l | 06:10 | |
| *** yamamoto has quit IRC | 06:10 | |
| amitkuhikar | has anyone encounter below issue | 06:10 |
| amitkuhikar | 10:21:28 + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L692: /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/dsvm-tempest-cinder-driver/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh executable=/bin/bash' | 06:10 |
| amitkuhikar | 10:21:31 ERROR: the main setup script run by this job failed - exit code: 2 | 06:10 |
| amitkuhikar | 10:21:40 Build step 'Execute shell' marked build as failure | 06:10 |
| amitkuhikar | with "dsvm-tempest-cinder-driver" | 06:10 |
| amitkuhikar | CI run | 06:10 |
| *** abregman has joined #openstack-infra | 06:10 | |
| *** yamamoto has joined #openstack-infra | 06:11 | |
| *** esikachev has quit IRC | 06:11 | |
| *** sdake has quit IRC | 06:13 | |
| *** tuanluong has joined #openstack-infra | 06:13 | |
| *** tojuvone has joined #openstack-infra | 06:14 | |
| *** tojuvone has left #openstack-infra | 06:14 | |
| *** salv-orl_ has joined #openstack-infra | 06:18 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ https://review.openstack.org/426307 | 06:20 |
| *** salv-orl_ has quit IRC | 06:20 | |
| *** salv-orl_ has joined #openstack-infra | 06:20 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** salv-orlando has quit IRC | 06:21 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ https://review.openstack.org/426307 | 06:21 |
| *** masuberu has quit IRC | 06:27 | |
| *** yamamoto has joined #openstack-infra | 06:29 | |
| *** yamamoto has quit IRC | 06:30 | |
| *** alexchadin has joined #openstack-infra | 06:33 | |
| *** yamahata_ has joined #openstack-infra | 06:36 | |
| *** dhajare has quit IRC | 06:38 | |
| Nilesh | hello, can some one help how to update http://lists.openstack.org/pipermail/openstack-dev/2017-February/112822.html | 06:41 |
| Nilesh | for CI | 06:41 |
| *** kaisers has joined #openstack-infra | 06:41 | |
| Nilesh | any smaple example for local.conf please share | 06:41 |
| *** dhajare has joined #openstack-infra | 06:45 | |
| *** thorst has joined #openstack-infra | 06:46 | |
| *** yamamoto has joined #openstack-infra | 06:47 | |
| *** yamamoto has quit IRC | 06:47 | |
| *** alexchadin has quit IRC | 06:48 | |
| *** thorst has quit IRC | 06:51 | |
| *** alexchadin has joined #openstack-infra | 06:51 | |
| *** rraja has joined #openstack-infra | 06:53 | |
| *** rossella_s has quit IRC | 06:53 | |
| *** rossella_s has joined #openstack-infra | 06:54 | |
| *** apetrich has quit IRC | 06:55 | |
| *** amitkuhikar has quit IRC | 06:57 | |
| *** esikachev has joined #openstack-infra | 06:59 | |
| *** Nilesh has quit IRC | 07:02 | |
| *** rraja has quit IRC | 07:02 | |
| *** tqtran has joined #openstack-infra | 07:03 | |
| *** csomerville has joined #openstack-infra | 07:03 | |
| *** esikachev has quit IRC | 07:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Support deploy args in OVB jobs https://review.openstack.org/437531 | 07:04 |
| *** cody-somerville has quit IRC | 07:06 | |
| *** ijw has joined #openstack-infra | 07:07 | |
| *** tqtran has quit IRC | 07:07 | |
| *** esikachev has joined #openstack-infra | 07:08 | |
| *** mhickey has quit IRC | 07:09 | |
| *** yamamoto has joined #openstack-infra | 07:10 | |
| *** rcernin has joined #openstack-infra | 07:11 | |
| *** ijw has quit IRC | 07:12 | |
| *** esikachev has quit IRC | 07:12 | |
| *** e0ne has joined #openstack-infra | 07:12 | |
| *** yamamoto has quit IRC | 07:13 | |
| *** andreas_s has joined #openstack-infra | 07:13 | |
| *** Kevin_Zheng_ has joined #openstack-infra | 07:14 | |
| *** apetrich has joined #openstack-infra | 07:16 | |
| *** Kevin_Zheng_ is now known as Kevin_Zheng | 07:17 | |
| *** yuanying_ has joined #openstack-infra | 07:25 | |
| *** yuanying has quit IRC | 07:27 | |
| *** makowals_ has quit IRC | 07:27 | |
| *** makowals has joined #openstack-infra | 07:28 | |
| *** vsaienk0 has joined #openstack-infra | 07:32 | |
| *** tesseract has joined #openstack-infra | 07:34 | |
| *** e0ne has quit IRC | 07:34 | |
| *** markus_z has joined #openstack-infra | 07:34 | |
| *** gmann has quit IRC | 07:35 | |
| *** gmann has joined #openstack-infra | 07:37 | |
| ianw | amitkuhikar: see http://54.169.104.103/92/336092/28/check/dsvm-tempest-cinder-driver/83c26f3/logs/devstack-early.txt.gz | 07:38 |
| ianw | it's failing to run on trusty | 07:38 |
| ianw | trusty has been removed from devstack master since it does not have a recent enough libvirt | 07:38 |
| ianw | you'll need to test on xenial | 07:38 |
| *** dtroyer has quit IRC | 07:40 | |
| *** dtroyer has joined #openstack-infra | 07:40 | |
| *** isaacb has joined #openstack-infra | 07:43 | |
| *** pgadiya has quit IRC | 07:48 | |
| *** matrohon has joined #openstack-infra | 07:49 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/git-review master: Support usernames that contain '@' and ssh Git URLs https://review.openstack.org/428700 | 07:52 |
| *** esikachev has joined #openstack-infra | 07:53 | |
| *** shardy has joined #openstack-infra | 07:56 | |
| *** shu-mutou has joined #openstack-infra | 07:56 | |
| *** askb has quit IRC | 07:56 | |
| *** sdake_ is now known as sdake | 07:56 | |
| *** esikachev has quit IRC | 07:58 | |
| *** Hal has joined #openstack-infra | 07:58 | |
| *** bhavik1 has joined #openstack-infra | 07:59 | |
| *** yanyanhu has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run containers-oooq jobs on every patch https://review.openstack.org/437944 | 08:01 |
| *** esikachev has joined #openstack-infra | 08:01 | |
| *** gk-1wm-su has joined #openstack-infra | 08:01 | |
| *** gk-1wm-su has quit IRC | 08:02 | |
| *** vsaienk0 has quit IRC | 08:02 | |
| *** pgadiya has joined #openstack-infra | 08:02 | |
| *** ramishra has quit IRC | 08:03 | |
| *** acabot has joined #openstack-infra | 08:03 | |
| *** ramishra has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: switch redis legacy tests to non-voting https://review.openstack.org/438706 | 08:05 |
| *** esikachev has quit IRC | 08:07 | |
| *** ccamacho has joined #openstack-infra | 08:07 | |
| *** gongysh has joined #openstack-infra | 08:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Install python-barbicanclient from git for castellan gate https://review.openstack.org/438712 | 08:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove mysql setup https://review.openstack.org/435569 | 08:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove final db setup https://review.openstack.org/427895 | 08:10 |
| *** pgadiya has quit IRC | 08:10 | |
| *** yamamoto has joined #openstack-infra | 08:13 | |
| *** pcaruana has joined #openstack-infra | 08:17 | |
| *** markus_z has quit IRC | 08:17 | |
| *** aviau has quit IRC | 08:19 | |
| *** yamamoto has quit IRC | 08:19 | |
| *** florianf has joined #openstack-infra | 08:19 | |
| *** aviau has joined #openstack-infra | 08:20 | |
| *** sreeram__ has joined #openstack-infra | 08:21 | |
| *** sreera___ has joined #openstack-infra | 08:22 | |
| *** esikachev has joined #openstack-infra | 08:22 | |
| *** tmorin has joined #openstack-infra | 08:23 | |
| *** flepied has quit IRC | 08:24 | |
| *** masber has joined #openstack-infra | 08:24 | |
| *** sreeram_v has quit IRC | 08:25 | |
| tmorin | hi AJaeger: about https://review.openstack.org/#/c/438573/ => got your point that asking an admin is the way, next question is: who would be such an admin ? | 08:25 |
| *** shardy has quit IRC | 08:25 | |
| *** sreeram__ has quit IRC | 08:26 | |
| *** pgadiya has joined #openstack-infra | 08:26 | |
| *** bhavik1 has quit IRC | 08:26 | |
| *** esikachev has quit IRC | 08:26 | |
| *** esikachev has joined #openstack-infra | 08:30 | |
| *** hashar has joined #openstack-infra | 08:30 | |
| *** dingyichen has quit IRC | 08:31 | |
| *** sreera___ has quit IRC | 08:31 | |
| *** rraja has joined #openstack-infra | 08:32 | |
| *** markus_z has joined #openstack-infra | 08:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Detect misaligned ':' in zuul/layout.yaml https://review.openstack.org/438735 | 08:34 |
| *** esikachev has quit IRC | 08:35 | |
| *** sreeram_v has joined #openstack-infra | 08:36 | |
| *** esikachev has joined #openstack-infra | 08:37 | |
| *** jordanP has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo/layout: run scenarios on haproxy manifests https://review.openstack.org/438803 | 08:39 |
| *** yujunz is now known as yujunz[away] | 08:43 | |
| *** priteau has joined #openstack-infra | 08:44 | |
| *** kornica has quit IRC | 08:46 | |
| *** rraja has quit IRC | 08:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove trove tempest plugin job https://review.openstack.org/437019 | 08:48 |
| *** yujunz[away] is now known as yujunz | 08:48 | |
| *** yujunz is now known as yujunz[away] | 08:49 | |
| *** esikachev has quit IRC | 08:50 | |
| *** bhavik1 has joined #openstack-infra | 08:50 | |
| *** bhavik1 has quit IRC | 08:50 | |
| *** nadya has joined #openstack-infra | 08:52 | |
| openstackgerrit | Steven Dake proposed openstack-infra/system-config master: Add #openstack-deployment channel to logging https://review.openstack.org/438857 | 08:53 |
| *** esikachev has joined #openstack-infra | 08:53 | |
| *** tosky has joined #openstack-infra | 08:53 | |
| *** yuanying_ has quit IRC | 08:53 | |
| *** dhinesh has quit IRC | 08:56 | |
| *** yujunz[away] is now known as yujunz | 08:57 | |
| *** esikachev has quit IRC | 08:58 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Add option for scenarios jobs promotion https://review.openstack.org/425819 | 08:59 |
| *** zzzeek has quit IRC | 09:00 | |
| *** Jeffrey4l has quit IRC | 09:00 | |
| *** Jeffrey4l has joined #openstack-infra | 09:00 | |
| *** esikachev has joined #openstack-infra | 09:00 | |
| *** pgadiya has quit IRC | 09:01 | |
| *** aarefiev_afk is now known as aarefiev | 09:02 | |
| *** yujunz has quit IRC | 09:03 | |
| *** yujunz has joined #openstack-infra | 09:04 | |
| *** flepied has joined #openstack-infra | 09:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ceilometer patch with periodic https://review.openstack.org/437008 | 09:06 |
| *** pgadiya has joined #openstack-infra | 09:09 | |
| *** masayukig[m] has quit IRC | 09:09 | |
| *** csmart has quit IRC | 09:09 | |
| frickler | infra-root: can you take a look at https://docs.openstack.org please? requests are taking 8-10s for me currently | 09:09 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Move containers-oooq to the check pipeline https://review.openstack.org/437945 | 09:10 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Move containers-oooq to the check pipeline https://review.openstack.org/437945 | 09:10 |
| *** gildub has quit IRC | 09:10 | |
| *** esikachev has quit IRC | 09:12 | |
| *** zzzeek has joined #openstack-infra | 09:12 | |
| *** martinkopec has joined #openstack-infra | 09:13 | |
| *** sdake has quit IRC | 09:13 | |
| *** tinwood_swap is now known as tinwood | 09:14 | |
| *** yamamoto has joined #openstack-infra | 09:15 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:16 | |
| openstackgerrit | Zara proposed openstack-infra/project-config master: Create the pynotedb project https://review.openstack.org/437528 | 09:17 |
| *** fandi has joined #openstack-infra | 09:20 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: removed open option from npm serve https://review.openstack.org/437130 | 09:21 |
| openstackgerrit | Chaoyi Huang proposed openstack-infra/project-config master: Add Tricircle multi-region job https://review.openstack.org/414909 | 09:23 |
| *** ramishra has quit IRC | 09:23 | |
| *** ramishra has joined #openstack-infra | 09:24 | |
| berendt | docs.o.o response times are getting higher at the moment | 09:25 |
| *** pgadiya has left #openstack-infra | 09:26 | |
| *** dimtruck is now known as zz_dimtruck | 09:28 | |
| *** apetrich has quit IRC | 09:29 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate master: TEST for multinode grenade devstack https://review.openstack.org/396679 | 09:34 |
| *** e0ne has joined #openstack-infra | 09:35 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** apetrich has joined #openstack-infra | 09:38 | |
| openstackgerrit | Zara proposed openstack-infra/project-config master: Create the pynotedb project https://review.openstack.org/437528 | 09:39 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/puppet-jenkins master: Add Suse family support to jenkins puppet module https://review.openstack.org/438877 | 09:39 |
| *** dtardivel has joined #openstack-infra | 09:39 | |
| *** jaosorior has quit IRC | 09:39 | |
| *** jaosorior has joined #openstack-infra | 09:40 | |
| *** dizquierdo has joined #openstack-infra | 09:40 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:41 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/puppet-unbound master: Add Suse support for unbound https://review.openstack.org/438878 | 09:41 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/puppet-pip master: Add support for Suse family https://review.openstack.org/438881 | 09:42 |
| *** electrofelix has joined #openstack-infra | 09:43 | |
| *** florianf has quit IRC | 09:44 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Implement openSUSE slaves configuration https://review.openstack.org/438882 | 09:45 |
| *** esikachev has joined #openstack-infra | 09:45 | |
| *** yamahata_ has quit IRC | 09:45 | |
| *** gongysh has quit IRC | 09:46 | |
| Zara | I think the pynotedb project-creation patch is ready for review | 09:47 |
| *** esikachev has quit IRC | 09:49 | |
| *** esikachev has joined #openstack-infra | 09:50 | |
| *** lucas-afk is now known as lucasagomes | 09:50 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config master: oslo.messaging: Make dsvm-full-zmq-ubuntu-xenial non-voting https://review.openstack.org/438883 | 09:52 |
| *** e0ne has quit IRC | 09:53 | |
| *** e0ne has joined #openstack-infra | 09:54 | |
| robcresswell | Anyone struggling with gerrit? Every button click is followed by about a 15 second delay. | 09:57 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Update nodepool elements for openSUSE Leap 42.2 https://review.openstack.org/438888 | 09:58 |
| neiljerram | robcresswell, yes, gerrit seems painfully slow for me this morning, too | 09:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config master: oslo.messaging: Make dsvm-full-zmq-ubuntu-xenial non-voting https://review.openstack.org/438883 | 10:01 |
| *** ramishra has quit IRC | 10:01 | |
| *** nadya has quit IRC | 10:02 | |
| *** GnomeKing has joined #openstack-infra | 10:02 | |
| *** nadya has joined #openstack-infra | 10:03 | |
| *** ramishra has joined #openstack-infra | 10:03 | |
| *** sdake has joined #openstack-infra | 10:05 | |
| *** isaacb has quit IRC | 10:06 | |
| *** nadya has quit IRC | 10:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config master: oslo.messaging: Make dsvm-full-zmq-ubuntu-xenial non-voting https://review.openstack.org/438883 | 10:09 |
| *** alexchadin has quit IRC | 10:10 | |
| *** liujiong has quit IRC | 10:11 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:12 | |
| *** yujunz has quit IRC | 10:13 | |
| *** Julien-zte has quit IRC | 10:13 | |
| *** fandi has quit IRC | 10:13 | |
| *** yujunz has joined #openstack-infra | 10:17 | |
| *** sambetts_ is now known as sambetts | 10:17 | |
| *** tosky has quit IRC | 10:21 | |
| robcresswell | neiljerram: I'm about ready to give up, can't get anything done like this :/ | 10:21 |
| *** tosky has joined #openstack-infra | 10:22 | |
| *** isviridov_away has joined #openstack-infra | 10:22 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config master: oslo.messaging: Make dsvm-full-zmq-ubuntu-xenial non-voting https://review.openstack.org/438883 | 10:23 |
| berendt | can anybody restart the apache of docs.o.o? it's getting slower and slower :( | 10:24 |
| asettle | berendt: I"m not having any issues with it, but Gauvain mentioned the same thing | 10:25 |
| *** sandanar has joined #openstack-infra | 10:26 | |
| *** masayukig[m] has joined #openstack-infra | 10:26 | |
| *** isaacb has joined #openstack-infra | 10:33 | |
| *** sreeram_v has quit IRC | 10:35 | |
| *** yamamoto has quit IRC | 10:35 | |
| *** Julien-zte has joined #openstack-infra | 10:35 | |
| *** dfflanders has quit IRC | 10:37 | |
| *** yuanying has joined #openstack-infra | 10:40 | |
| *** asselin has quit IRC | 10:40 | |
| *** markus_z has quit IRC | 10:41 | |
| *** asselin has joined #openstack-infra | 10:41 | |
| *** sdake has quit IRC | 10:42 | |
| *** nadya has joined #openstack-infra | 10:43 | |
| *** shardy has joined #openstack-infra | 10:45 | |
| *** ijw has joined #openstack-infra | 10:47 | |
| *** dtantsur|afk is now known as dtantsur | 10:47 | |
| openstackgerrit | Sorin Sbârnea proposed openstack-infra/git-review master: Support usernames that contain '@' and ssh Git URLs https://review.openstack.org/428700 | 10:48 |
| *** thorst has joined #openstack-infra | 10:48 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** sreeram_v has joined #openstack-infra | 10:50 | |
| *** thorst has quit IRC | 10:53 | |
| *** rossella_s has quit IRC | 10:53 | |
| *** rossella_s has joined #openstack-infra | 10:54 | |
| bauzas | looks like review.o.o is having a bad day, could someone from -infra make it happier ? | 10:55 |
| *** sreeram_v has quit IRC | 10:55 | |
| bauzas | lots of HTTP502 I'm getting | 10:55 |
| bauzas | a recycle could be nice | 10:55 |
| *** yamamoto has quit IRC | 10:55 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-baremetal project https://review.openstack.org/438907 | 10:56 |
| *** ssbarnea has quit IRC | 10:56 | |
| *** sreeram_v has joined #openstack-infra | 10:57 | |
| *** salv-orl_ has quit IRC | 10:57 | |
| *** dpaterson has joined #openstack-infra | 10:59 | |
| *** isaacb has quit IRC | 11:02 | |
| *** sreeram_v has quit IRC | 11:03 | |
| *** sreeram_v has joined #openstack-infra | 11:04 | |
| *** salv-orlando has joined #openstack-infra | 11:05 | |
| *** sdague has joined #openstack-infra | 11:06 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 11:08 |
| *** sreeram_v has quit IRC | 11:08 | |
| *** nicolasbock has joined #openstack-infra | 11:09 | |
| *** sreeram_v has joined #openstack-infra | 11:15 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 11:16 |
| *** HeOS has joined #openstack-infra | 11:17 | |
| *** sreeram_v has quit IRC | 11:19 | |
| *** yujunz is now known as yujunz[away] | 11:25 | |
| *** yujunz[away] is now known as yujunz | 11:26 | |
| *** mhickey has joined #openstack-infra | 11:26 | |
| jordanP | yes, gerrit is super slow | 11:26 |
| odyssey4me | worse than a snail :p | 11:29 |
| *** john-davidge has joined #openstack-infra | 11:31 | |
| *** madhuvishy has quit IRC | 11:31 | |
| *** madhuvishy has joined #openstack-infra | 11:31 | |
| *** Julien-zte has quit IRC | 11:32 | |
| *** Julien-z_ has joined #openstack-infra | 11:32 | |
| *** uberjay has quit IRC | 11:32 | |
| *** uberjay has joined #openstack-infra | 11:37 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 11:37 |
| *** yamamoto has joined #openstack-infra | 11:38 | |
| *** sreeram_v has joined #openstack-infra | 11:38 | |
| *** nmathew has quit IRC | 11:41 | |
| *** jkilpatr has quit IRC | 11:41 | |
| *** hashar is now known as hasharLunch | 11:45 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config master: TripleO: Move fakeha-caserver job to periodic https://review.openstack.org/436381 | 11:47 |
| *** yamamoto has quit IRC | 11:49 | |
| *** csmart has joined #openstack-infra | 11:51 | |
| *** zeus has quit IRC | 11:52 | |
| *** betherly has quit IRC | 11:53 | |
| *** wolverineav has joined #openstack-infra | 11:53 | |
| *** zeus has joined #openstack-infra | 11:54 | |
| *** zeus is now known as Guest91249 | 11:54 | |
| *** markus_z has joined #openstack-infra | 11:54 | |
| *** esikachev has quit IRC | 11:54 | |
| *** betherly has joined #openstack-infra | 11:54 | |
| *** tosky_ has joined #openstack-infra | 11:57 | |
| *** isaacb has joined #openstack-infra | 11:57 | |
| *** wolverineav has quit IRC | 11:58 | |
| *** dtardivel_ has joined #openstack-infra | 11:58 | |
| *** alexchadin has joined #openstack-infra | 11:59 | |
| *** madhuvishy has quit IRC | 11:59 | |
| *** tosky has quit IRC | 11:59 | |
| *** dtardivel has quit IRC | 11:59 | |
| *** dtardivel_ is now known as dtardivel | 11:59 | |
| *** madhuvishy has joined #openstack-infra | 12:01 | |
| *** salv-orlando has quit IRC | 12:01 | |
| *** igorbelikov_ has quit IRC | 12:01 | |
| *** ildikov has quit IRC | 12:01 | |
| *** ildikov has joined #openstack-infra | 12:03 | |
| *** igorbelikov_ has joined #openstack-infra | 12:03 | |
| *** johnthetubaguy has quit IRC | 12:04 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 12:06 |
| *** tosky_ is now known as tosky | 12:06 | |
| *** pblaho has quit IRC | 12:07 | |
| *** fungi has quit IRC | 12:11 | |
| *** tuanluong has quit IRC | 12:12 | |
| *** jkilpatr has joined #openstack-infra | 12:13 | |
| *** esikachev has joined #openstack-infra | 12:13 | |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** johnthetubaguy has joined #openstack-infra | 12:14 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-stackalytics master: Add redis-server https://review.openstack.org/438939 | 12:14 |
| *** aglarendil has quit IRC | 12:16 | |
| *** dtardivel_ has joined #openstack-infra | 12:16 | |
| *** fungi has joined #openstack-infra | 12:17 | |
| *** dtardivel has quit IRC | 12:17 | |
| *** Guest91249 has quit IRC | 12:17 | |
| *** dtardivel_ is now known as dtardivel | 12:17 | |
| *** esikachev has quit IRC | 12:17 | |
| *** aglarendil has joined #openstack-infra | 12:18 | |
| *** Julien-z_ has quit IRC | 12:20 | |
| *** wolverineav has joined #openstack-infra | 12:20 | |
| *** yujunz has quit IRC | 12:20 | |
| *** wolverin_ has joined #openstack-infra | 12:22 | |
| *** EricGonczer_ has joined #openstack-infra | 12:22 | |
| *** rhallisey has joined #openstack-infra | 12:23 | |
| *** pblaho has joined #openstack-infra | 12:23 | |
| johnthetubaguy | +1 for the need for a gerrit restart, if anyone is around that can trigger that | 12:23 |
| johnthetubaguy | its getting really slow again | 12:23 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 12:24 |
| *** zeus- has joined #openstack-infra | 12:24 | |
| *** wolverineav has quit IRC | 12:25 | |
| *** ewindisch has quit IRC | 12:26 | |
| *** sreeram_v has quit IRC | 12:26 | |
| *** ewindisch has joined #openstack-infra | 12:28 | |
| *** pblaho has quit IRC | 12:30 | |
| *** ijw has joined #openstack-infra | 12:31 | |
| *** ccamacho has quit IRC | 12:34 | |
| *** yamamoto has joined #openstack-infra | 12:34 | |
| EmilienM | +1 with johnthetubaguy | 12:34 |
| *** yamamoto has quit IRC | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:35 | |
| *** amotoki has joined #openstack-infra | 12:35 | |
| *** larainema has joined #openstack-infra | 12:35 | |
| *** ijw has quit IRC | 12:36 | |
| *** masayukig[m] has quit IRC | 12:37 | |
| *** dtardivel has quit IRC | 12:38 | |
| *** masayukig[m] has joined #openstack-infra | 12:39 | |
| *** kong has quit IRC | 12:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 12:39 |
| *** esikachev has joined #openstack-infra | 12:39 | |
| *** dtardivel has joined #openstack-infra | 12:39 | |
| *** csmart has quit IRC | 12:39 | |
| *** csmart has joined #openstack-infra | 12:40 | |
| amrith | hiya infra ... echoing jordanP, review.o.o is really slow. | 12:40 |
| amrith | and odyssey4me and johnthetubaguy and EmilienM (hi!) | 12:40 |
| *** zeus- is now known as zeus | 12:41 | |
| *** zeus is now known as Guest83991 | 12:41 | |
| *** salv-orlando has joined #openstack-infra | 12:42 | |
| *** pblaho has joined #openstack-infra | 12:43 | |
| *** esikachev has quit IRC | 12:43 | |
| amrith | thanks infra, something made it a lot faster, I'm assuming someone cycled it. | 12:43 |
| robcresswell | amrith: Really? Mine is still painfully slow | 12:44 |
| amrith | yes, I spoke too soon | 12:44 |
| amrith | I got one server error | 12:44 |
| amrith | and after that very quick response | 12:44 |
| amrith | now the spinning wheel again | 12:45 |
| amrith | I think I'll fix it and change that to the hourglass and the problem (spinning wheel) will go away | 12:45 |
| robcresswell | I guess there are no infra peeps in the EU tz right now :( | 12:45 |
| amrith | well robcresswell you are correct; 502 now | 12:45 |
| robcresswell | yeah | 12:46 |
| robcresswell | I give up on OSS for today :( | 12:46 |
| *** thorst has joined #openstack-infra | 12:47 | |
| *** ccamacho has joined #openstack-infra | 12:50 | |
| *** ggnel_t has quit IRC | 12:51 | |
| *** vikrant has quit IRC | 12:51 | |
| EmilienM | pabelanger, fungi, yolanda: if you around, gerrit is super slow | 12:51 |
| *** esikachev has joined #openstack-infra | 12:51 | |
| *** markmcclain has quit IRC | 12:53 | |
| *** yamamoto has quit IRC | 12:54 | |
| *** thingee has quit IRC | 12:54 | |
| *** esikachev has quit IRC | 12:56 | |
| *** esikachev has joined #openstack-infra | 12:59 | |
| *** markmcclain has joined #openstack-infra | 13:00 | |
| *** thingee has joined #openstack-infra | 13:00 | |
| *** dprince has joined #openstack-infra | 13:02 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 13:02 |
| *** cshastri has quit IRC | 13:03 | |
| mordred | #status alert restarting gerrit to address performance problems | 13:03 |
| openstackstatus | mordred: sending alert | 13:03 |
| odyssey4me | mordred kerblooey | 13:04 |
| *** ssbarnea has joined #openstack-infra | 13:04 | |
| mordred | odyssey4me: as best we can tell, the issue is that sometimes gerrit gets tired | 13:04 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Support setting a variable to get local.conf merged late https://review.openstack.org/438957 | 13:05 |
| odyssey4me | mordred it appears that gerrit needs coffee | 13:05 |
| mordred | odyssey4me: so say we all | 13:05 |
| -openstackstatus- NOTICE: restarting gerrit to address performance problems | 13:06 | |
| *** ChanServ changes topic to "restarting gerrit to address performance problems" | 13:06 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-stackalytics master: Add redis-server https://review.openstack.org/438939 | 13:06 |
| asettle | Gerrit is supposed to live forever and do our bidding, not get tired | 13:06 |
| openstackstatus | mordred: finished sending alert | 13:10 |
| odyssey4me | asettle spoken like a queen :) | 13:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Updates to Chef OpenStack project https://review.openstack.org/438501 | 13:11 |
| asettle | odyssey4me: hhahahahahah | 13:11 |
| asettle | I really need to get a crown | 13:11 |
| asettle | And a cape | 13:11 |
| odyssey4me | obviously | 13:11 |
| asettle | For all necessary events (erryday) | 13:11 |
| *** yamamoto has joined #openstack-infra | 13:11 | |
| *** apetrich has quit IRC | 13:16 | |
| *** yamamoto has quit IRC | 13:17 | |
| *** Julien-zte has joined #openstack-infra | 13:17 | |
| *** sslypushenko_ has quit IRC | 13:18 | |
| mordred | asettle: I didn't realize it before just now, but I clearly also need a crown and cape | 13:19 |
| asettle | mordred: happy to bulk buy \o/ | 13:19 |
| *** sslypushenko_ has joined #openstack-infra | 13:20 | |
| *** trown|outtypewww is now known as trown | 13:20 | |
| frickler | does anyone know Eric Young? an account of that name is leaving broken-looking CI votes, see https://review.openstack.org/438501 | 13:20 |
| *** lucasagomes is now known as lucas-hungry | 13:21 | |
| *** hasharLunch is now known as hashar | 13:22 | |
| *** Julien-zte has quit IRC | 13:22 | |
| frickler | mordred: seems gerrit is doing fine again, so you could revert the alert? | 13:22 |
| *** dave-mccowan has joined #openstack-infra | 13:26 | |
| *** apetrich has joined #openstack-infra | 13:26 | |
| *** Julien-zte has joined #openstack-infra | 13:26 | |
| *** pblaho has quit IRC | 13:29 | |
| *** pblaho has joined #openstack-infra | 13:30 | |
| mordred | #status alert ok gerrit is back to normal | 13:32 |
| openstackstatus | mordred: sending alert | 13:32 |
| mordred | frickler: thanks for the reminder. I believe I should do notice next time for a gerrit restart :) | 13:33 |
| fungi | i usually just #status notice those because the gerrit restart takes less time than the broadcast anyway | 13:33 |
| mordred | yah | 13:34 |
| asettle | All hail mordred and his fixing abilities (cape and crown on its way!) | 13:34 |
| *** esikachev has quit IRC | 13:34 | |
| frickler | mordred: that didn't resolve the alert, still, the topic for other channels now is "ok gerrit is back to normal" | 13:34 |
| mordred | gah. I really suck | 13:35 |
| *** esikachev has joined #openstack-infra | 13:35 | |
| fungi | wait until the second alert comes back | 13:35 |
| mordred | yah | 13:35 |
| asettle | hahahha mordred I take it back. Although mine did get faster for a sec | 13:35 |
| -openstackstatus- NOTICE: ok gerrit is back to normal | 13:35 | |
| *** ChanServ changes topic to "ok gerrit is back to normal" | 13:35 | |
| *** gordc has joined #openstack-infra | 13:35 | |
| fungi | i can't remember whether we have a bug interrupting it or if it will queue properly | 13:35 |
| * mordred is considering making today long-form-performance-art-poetry day | 13:35 | |
| fungi | but in the past we've done similar things and left ourselves a mess to clean up | 13:36 |
| mordred | msg help openstackstatus would be nice if it doesn't do it already | 13:36 |
| fungi | yep | 13:36 |
| mordred | of course, getting all of our bots rewritten would be really nice | 13:36 |
| fungi | also you can refer to https://docs.openstack.org/infra/system-config/irc.html#statusbot | 13:37 |
| *** kgiusti has joined #openstack-infra | 13:37 | |
| *** mdrabe has joined #openstack-infra | 13:38 | |
| *** dizquierdo has quit IRC | 13:38 | |
| openstackstatus | mordred: finished sending alert | 13:39 |
| mordred | #status ok gerrit is back to normal and I don't know how to use the openstackstaus bot | 13:39 |
| openstackstatus | mordred: sending ok | 13:39 |
| *** sbezverk has quit IRC | 13:39 | |
| mordred | this morning's status bot interaction seems to be positive proof that this thread https://twitter.com/e_monty/status/836555520995655681 is relevant on the daily | 13:40 |
| asettle | mordred: staus bot? I love it. | 13:40 |
| mordred | asettle: we gots all sorts of bots up in here ... it's like a frickin menagerie | 13:40 |
| asettle | status bots, staus bots (is it the German/European version?) | 13:41 |
| mordred | hahahahaha | 13:41 |
| mordred | we call this bot "Ricard" and enjoy its walzes | 13:41 |
| *** ChanServ changes topic to "PTG: Grand Ballroom C, Floor 3, South Tower | Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 13:42 | |
| -openstackstatus- NOTICE: gerrit is back to normal and I don't know how to use the openstackstaus bot | 13:42 | |
| mordred | (and now I really do want a Strauss bot) | 13:42 |
| *** salv-orlando has quit IRC | 13:42 | |
| *** ggnel_t has joined #openstack-infra | 13:42 | |
| asettle | We need a Strauss bot. I just dunno what it could do. Randomly generate updates based solely in our incompetences? | 13:42 |
| mordred | fungi: did we ever get anywhere with our fedora friends on project bot consolidation? | 13:42 |
| mordred | asettle: zomg. I need that | 13:42 |
| mordred | although you've given me a TERRIBLE idea for a project I'll now probably spend way too much of my off-time working on | 13:44 |
| asettle | mordred: happy doing business with you. We'll naturally delegate the generation work to someone else, because #community | 13:44 |
| asettle | Or *i'll* delegate the work to *you* | 13:44 |
| asettle | Super. | 13:44 |
| fungi | mordred: nope, they expressed interest in collaborating but nobody from our end who had been hacking on bot-related stuff replied | 13:45 |
| *** tojuvone has joined #openstack-infra | 13:45 | |
| mordred | which we'll call strauss bot - but will controll something like a shoutcast server on the backend, and for every channel you ask it to join, it'll maintain a separate streaming channel. in IRC channel, you can enqueue songs, and whenver it switches songs it'll announce in that channel what song it's playing now | 13:45 |
| *** tojuvone has left #openstack-infra | 13:45 | |
| mordred | so it would be like a community operated collection of streaming internet radio stations collaboratively dj'd in chat channels | 13:45 |
| openstackstatus | mordred: finished sending ok | 13:45 |
| *** rlandy has joined #openstack-infra | 13:45 | |
| fungi | and only selections from johann strauss, of course | 13:46 |
| mordred | yay! it's alreayd written: https://www.shoutirc.com/ | 13:46 |
| *** ggnel_t has quit IRC | 13:46 | |
| *** shardy has quit IRC | 13:46 | |
| *** shardy has joined #openstack-infra | 13:47 | |
| mordred | oh. that one is not open source it seems | 13:47 |
| johnthetubaguy | asettle: next time I am in burger king, I should totally pick you up a crown | 13:47 |
| jroll | mordred: we have a bot to shout in irc, it's called mordred | 13:47 |
| asettle | johnthetubaguy: um please do | 13:48 |
| asettle | mordred: might just have to make another OS one yourself ;) | 13:48 |
| jroll | mordred: also if you actually do write a shoutcast collaborative DJ bot, let me know, I'll totally use that thing | 13:48 |
| mordred | jroll: I NEVER SHOUT IN IRC WHAT ARE YOU IMPLYING???? | 13:49 |
| jroll | :D | 13:49 |
| robcresswell | Those status updates made me laugh | 13:49 |
| * mordred should also make a mordred bot that joins channels as mordred_ and/or mordred__ and just from time to time says random stuff I'd say | 13:50 | |
| *** gouthamr has joined #openstack-infra | 13:50 | |
| * mordred may have WAY too much energy this morning - goes to make more coffee | 13:50 | |
| persia | mordred: someone wrote a good markov-chain based imitation persia (based on log data from IRC). If you want such a bot, I'd be happy to try to get you the code. You'll need to get permission from freenode to run it, of course. | 13:51 |
| *** zhurong has joined #openstack-infra | 13:52 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 13:52 |
| *** baoli has joined #openstack-infra | 13:52 | |
| *** pradk has joined #openstack-infra | 13:52 | |
| *** wolverin_ has quit IRC | 13:53 | |
| mordred | persia: :) | 13:54 |
| *** xyang1 has joined #openstack-infra | 13:54 | |
| *** baoli_ has joined #openstack-infra | 13:54 | |
| mordred | persia: it's probably a very _Bad_ idea for me to run such a thing | 13:54 |
| persia | Apparently it was https://megahal.alioth.debian.org/ : training it off one's logs is an exercise for the reader | 13:55 |
| dprince | jeblair: hi, quick question. Is there anything in zuul today (or perhaps coming in zuul v3) that would allow you to get a web hit notification from a specific job failure... fail fast | 13:56 |
| *** esikache1 has joined #openstack-infra | 13:56 | |
| dprince | jeblair: like polling http://zuul.openstack.org/status/change/<change,id> and then if it fails, curling a URL real fast like | 13:57 |
| dprince | our jobs take so long and I often find its just one I care about. Having a faster (more direct) notification channel is what I'm after | 13:57 |
| * dprince wonders if hacking up a script to do this would be useful | 13:58 | |
| *** baoli has quit IRC | 13:58 | |
| dprince | pabelanger: rather cryptic docker failure here https://review.openstack.org/#/c/437411/, wondering if you have seen this before? | 13:59 |
| *** esikachev has quit IRC | 13:59 | |
| *** yujunz has joined #openstack-infra | 14:00 | |
| *** ihrachys has joined #openstack-infra | 14:00 | |
| *** wolverineav has joined #openstack-infra | 14:01 | |
| *** openstackgerrit has quit IRC | 14:03 | |
| *** isaacb has quit IRC | 14:03 | |
| *** wolverineav has quit IRC | 14:05 | |
| *** mriedem has joined #openstack-infra | 14:07 | |
| *** user` has joined #openstack-infra | 14:07 | |
| Zara | following the project creator's guide, I started making the pynotedb project-to-be-imported-into-infra-asap with cookiecutter earlier; a colleague noticed a lot of files are copyright hp and we wondered if those should be copyright 'the openstack foundation' these days. I'm not fussed, but it worried him, so I can see it scaring people off. | 14:10 |
| *** eroux has joined #openstack-infra | 14:11 | |
| *** user` is now known as chem | 14:12 | |
| sdague | mordred / clarkb / fungi - when working with neutron team, I think we're going to need something like this in devstack-gate to support how they are using gate hooks - https://review.openstack.org/#/c/438957/ | 14:12 |
| sdague | which is basically just a way to get a devstack local.conf merged late | 14:12 |
| *** chem has quit IRC | 14:12 | |
| *** chem has joined #openstack-infra | 14:12 | |
| *** esikache1 has quit IRC | 14:13 | |
| *** yamamoto has joined #openstack-infra | 14:13 | |
| fungi | i don't suppose there's any way to move the fancy dynamism in their jobs earlier in the process? | 14:14 |
| *** matt-borland has joined #openstack-infra | 14:15 | |
| *** lucas-hungry is now known as lucasagomes | 14:15 | |
| Zara | I've put a patch up to create the pynotedb project here: https://review.openstack.org/#/c/437528/ , along with a governance patch here: https://review.openstack.org/#/c/438291/ and a package registered in pypi with openstackci as an owner. I don't think there's anything else I need to do, but apologies if I'm wrong on that. | 14:16 |
| fungi | Zara: the files in the cookiecutter were written by people under the employ of hewlett packard. if you're replacing them entirely then there's no need to keep the copyright. otherwise the modified (or even unmodified) copies of those files are still at least partly authored by the original copyright holder | 14:16 |
| fungi | you can of course add your copyright along with it in the files you've altered if you want | 14:17 |
| *** yujunz is now known as yujunz[away] | 14:17 | |
| *** yujunz[away] is now known as yujunz | 14:17 | |
| Zara | fungi: yeah, that's fine, I just wondered since there was a drive to change a bunch of things from individual companies to the foundation a while ago | 14:17 |
| fungi | putting "copyright XXXX, OpenStack Foundation" in files only makes sense if you're an employee or contractor of the openstack foundation | 14:17 |
| *** yamamoto has quit IRC | 14:18 | |
| *** eroux has quit IRC | 14:18 | |
| fungi | Zara: https://wiki.openstack.org/wiki/LegalIssuesFAQ#Copyright_Headers should hopefully explain | 14:18 |
| sdague | fungi: it is earlier in the process, that's the issue | 14:19 |
| Zara | fungi: ah, I think I was getting confused with the Rackspace things that changed, thanks | 14:19 |
| sdague | fungi: https://github.com/openstack/neutron/blob/29d03cdb4c28eac29588b103249186802a8bc29e/neutron/tests/contrib/gate_hook.sh#L85-L106 | 14:20 |
| sdague | they do a ton of dynamic stuff to configure, but then they call devstack-gate-vm.sh | 14:20 |
| *** dpaterson has quit IRC | 14:20 | |
| fungi | i meant early enough in the process to be able to generate their modifications to local.conf before anything uses it, but i'm probably misunderstanding. looking closer at their implementation now | 14:21 |
| *** openstackgerrit has joined #openstack-infra | 14:22 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 14:22 |
| sdague | yeh, I think that the issue really comes down to d-g has been pretty fluid and non contract, so people iterated until a thing worked | 14:22 |
| sdague | and it turns out it used behaviors that at least I didn't understand were a thing on the devstack side | 14:22 |
| sdague | which all got broken by doing first class local.conf in the whole toolchain | 14:23 |
| *** sdake has joined #openstack-infra | 14:23 | |
| sdague | because previously people were using a locally generated local.conf that d-g didn't understand to side channel around the whole process, because devstack would (because of a bug) use both localrc and local.conf | 14:23 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Undercloud containers job with tripleo.sh https://review.openstack.org/437411 | 14:23 |
| fungi | ye olde overheating spacebar problem | 14:24 |
| sdague | but now that is first class, you can't sneak around it | 14:24 |
| sdague | yep | 14:24 |
| sdague | well, and also speaks to that if you aren't super clear about what your contract is (including warning when people touch things they shouldn't), the answer is "whatever it does today" | 14:25 |
| sdague | :) | 14:25 |
| *** jamesdenton has joined #openstack-infra | 14:25 | |
| *** zz_dimtruck is now known as dimtruck | 14:25 | |
| *** isaacb has joined #openstack-infra | 14:26 | |
| *** tlian has joined #openstack-infra | 14:26 | |
| *** rhallisey has quit IRC | 14:26 | |
| Zara | fungi: other thing I said I'd ask about-- pygit2 is GPL with a linking exception; gitpython is BSD. I saw there's an infra exception here: https://governance.openstack.org/tc/reference/licensing.html but I'm wondering if we'd be safer using gitpython (eg: down the line we'd like to integrate with zuul), or if it really doesn't matter. | 14:29 |
| *** cloudtrainme has joined #openstack-infra | 14:30 | |
| *** esikache1 has joined #openstack-infra | 14:30 | |
| *** chlong has joined #openstack-infra | 14:31 | |
| *** pblaho has quit IRC | 14:32 | |
| *** cloudtrainme has quit IRC | 14:32 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: quickstart settings: fix collect-logs settings https://review.openstack.org/438978 | 14:33 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Export UPGRADE_ENV for scenario-upgrades https://review.openstack.org/438567 | 14:33 |
| fungi | Zara: i'm not personally concerned about either license there, but we use gitpython in many places and pygit2 pretty much nowhere | 14:34 |
| *** Hal has quit IRC | 14:34 | |
| fungi | so if you can use gitpython, that provides a little additional consistency | 14:34 |
| Zara | okay, I'll pass that on | 14:36 |
| fungi | Zara: and i think i see the reason why... gitpython is a pure python library while pygit2 links against a binary lib so needs install-time recompilation | 14:37 |
| *** Hal has joined #openstack-infra | 14:37 | |
| fungi | it's basically a python wrapper around https://github.com/libgit2/libgit2 and so inherits its license from there | 14:38 |
| Zara | yeah | 14:38 |
| Zara | the perl implementation uses libgit2, which is why we started looking at pygit2 | 14:39 |
| *** esikache1 has quit IRC | 14:39 | |
| *** Goneri has joined #openstack-infra | 14:39 | |
| Zara | since it's easier to translate it that way.... but if the licence is going to cause problems, now is the time to switch | 14:39 |
| Zara | before we get too involved | 14:39 |
| *** dpaterson has joined #openstack-infra | 14:40 | |
| *** tlian has quit IRC | 14:40 | |
| Zara | so I'm interested to know what others who might want to use a python notedb parser (and, down the road, client) would have to say on this | 14:40 |
| fungi | gplv2 could cause problems if we try to import it in an apache2 license project since the two licenses are considered incompatible... i'm digging up the details of the exception in the libgit license to see whether it addresses that explicitly | 14:41 |
| *** Guest83991 is now known as zeus | 14:42 | |
| *** zeus has quit IRC | 14:42 | |
| *** zeus has joined #openstack-infra | 14:42 | |
| *** annegentle has joined #openstack-infra | 14:43 | |
| fungi | looks like pygit2 is just straight gplv2 from what i can tell. libgit has an added exception allowing you to link it from otherly-licensed software but that doesn't really help if pygit2 is what you're talking about | 14:43 |
| Zara | ah okay, it was probably libgit (I'd heard second-hand) | 14:44 |
| fungi | https://github.com/libgit2/libgit2/blob/master/COPYING#L10 would allow you to, say, make an apache2-licensed replacement for pygit2 and link against libgit | 14:44 |
| fungi | oh, wait, it is there in the pygit2 license too, just not as clearly set apart from the preamble | 14:45 |
| fungi | https://github.com/libgit2/pygit2/blob/master/COPYING#L6 | 14:45 |
| fungi | though "compiled executable" is vague in this case | 14:46 |
| *** thorst is now known as thorst_afk | 14:46 | |
| *** dhajare has quit IRC | 14:46 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Storyboard: Support invalid tasks in update script https://review.openstack.org/438704 | 14:46 |
| fungi | they just lifted it from the libgit2 license (where it _does_ make sense given the language in use there is c) | 14:46 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 14:47 |
| Zara | heh | 14:47 |
| *** annegentle has quit IRC | 14:47 | |
| Zara | he asked if he should join here so that's probably a good idea, I feel weird playing messenger | 14:48 |
| *** Gravious has joined #openstack-infra | 14:49 | |
| * Zara waves at Gravious | 14:50 | |
| Gravious | howdy :) | 14:50 |
| Zara | fungi: Gravious is the person who will likely be architecting pynotedb and is more familiar with libgit2 and pygit2 than I am. | 14:51 |
| * fungi is confused by the applicability (or lack thereof) for the linking exception in pygit2's license, which looks like it's specific to compiled languages | 14:51 | |
| Gravious | hi fungi | 14:52 |
| fungi | which given the text seems to be lifted word for word from the libgit2 license, makes sense | 14:52 |
| Gravious | yes i'm not a legal expert sadly | 14:52 |
| Gravious | i would interpret it as having the same implications for C programs linking to libgit2, but the python equiv, i.e. import instead of link | 14:52 |
| fungi | if we're considering comingling pygit2 and apache license v2 libraries, it could get sticky (lack of pure-python implementation aside) | 14:53 |
| Gravious | if the gplv2 prevented pygit2 consumers from choosing their own licenses that would make it much less useful | 14:53 |
| *** rossella_s has quit IRC | 14:53 | |
| *** rossella_s has joined #openstack-infra | 14:54 | |
| fungi | sure. my guess at the intent is that libgit2 wants to allow other-licensed software to link it, and the authors wanted to convey similar exemptions for their pygit2 python wrapper, but the wording seems at odds with the fact that it's implemented in an interpreted language | 14:54 |
| Gravious | quite possibly, maybe that's an issue worth raising with the pygit2 devs? | 14:55 |
| fungi | they didn't adjust the wording in the pygit2 license to cover the differences in implementation language from libgit2 i guess | 14:55 |
| fungi | well, in addition to the license concerns (for which finding clarification would be good), i'm digging deeper now to find out what system build dependencies pygit2 and libgit will drag in | 14:57 |
| fungi | up side to gitpython is that it's a pure python implementation, so more portable (as long as you have git and a python interpreter you're all set) | 14:58 |
| pabelanger | morning | 14:58 |
| fungi | looks like pygit2 is going to need at least cffi/libffi | 14:59 |
| Gravious | the downside would be performance, and it is an unknown for me. as the author of perl-notedb i am confident that libgit2 has all the functionality i need to implement this, gitpython is an unknown | 14:59 |
| fungi | right, if the goal is to simply translate your current perl to python and reuse the same backing library, i can see that would be a simpler path | 15:00 |
| Gravious | indeed :) | 15:00 |
| fungi | obviously a pure python implementation is of less benefit when it comes to perl | 15:00 |
| Gravious | indeed :) | 15:00 |
| *** dhajare has joined #openstack-infra | 15:01 | |
| *** bh526r has joined #openstack-infra | 15:01 | |
| *** hongbin has joined #openstack-infra | 15:02 | |
| fungi | if we're hosting it in our ci, we'll probably have to forego publishing wheels for this and just stick to sdists | 15:03 |
| *** pblaho has joined #openstack-infra | 15:03 | |
| fungi | due to needing install-time linking to the user's local c libs | 15:04 |
| Gravious | sounds alright | 15:04 |
| *** HeOS has quit IRC | 15:05 | |
| *** abregman has quit IRC | 15:05 | |
| *** esikache1 has joined #openstack-infra | 15:06 | |
| *** yujunz has quit IRC | 15:06 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DON'T MERGE: test tempest all branches https://review.openstack.org/436923 | 15:06 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 15:07 |
| *** andreas_s has quit IRC | 15:07 | |
| fungi | Gravious: i had originally gotten the impression Zara was leading development on this, so assumed a bit more alignment with our other python-based libraries and dependency chain (as well as familiarity with our community's tools and workflow), but if this is more closely aligned with your perl implementation then it may make more sense to host it in the same place you're already using so you can rely | 15:08 |
| fungi | on tighter versioning and integration there? | 15:08 |
| *** hemna is now known as guest_666 | 15:08 | |
| *** guest_999 has joined #openstack-infra | 15:08 | |
| *** saibarspeis has joined #openstack-infra | 15:08 | |
| *** guest_666 is now known as hemna | 15:08 | |
| *** alexchadin has quit IRC | 15:08 | |
| *** xarses has quit IRC | 15:09 | |
| fungi | i wouldn't want to push our community norms on a project for which they may not be a good fit | 15:09 |
| *** esikache1 has quit IRC | 15:10 | |
| *** zhurong has quit IRC | 15:10 | |
| *** chlong has quit IRC | 15:11 | |
| *** esikache1 has joined #openstack-infra | 15:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove job trees from pipelines https://review.openstack.org/435944 | 15:12 |
| electrocucaracha | hey there, the python-tricircleclient repo was recently added to Openstack official repos https://review.openstack.org/#/c/426419/ but there is no account for modifying its gerrit persmission, who can help me to solve that? | 15:13 |
| jaosorior | yolanda: if you have time could you check this out https://review.openstack.org/#/c/438883/ ? | 15:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add nodepool-id to infracloud-chocolate https://review.openstack.org/438998 | 15:14 |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config master: add python 2.7 to devstack-tools tests https://review.openstack.org/439000 | 15:15 |
| pabelanger | AJaeger: morning, have time to talk about https://review.openstack.org/#/c/438573/ | 15:15 |
| jaosorior | yolanda: thanks :) | 15:16 |
| yolanda | glad to help | 15:16 |
| *** EricGonc_ has joined #openstack-infra | 15:17 | |
| *** larainema has quit IRC | 15:18 | |
| electrocucaracha | yolanda: ping? regarding my last question about who can help me to add an account to gerrit permissions? can you do it or who can help me on that? | 15:19 |
| *** EricGonczer_ has quit IRC | 15:19 | |
| yolanda | hi electrocucaracha , sorry, i missed that | 15:19 |
| yolanda | so i can help you, or other cores. Can you paste the link of your change? | 15:19 |
| *** larainema has joined #openstack-infra | 15:19 | |
| *** yamamoto has quit IRC | 15:19 | |
| electrocucaracha | yolanda: well, this is the change for adding that project to infra https://review.openstack.org/#/c/426419/ | 15:20 |
| yolanda | ok adding you | 15:20 |
| *** esikache1 has quit IRC | 15:21 | |
| electrocucaracha | yolanda: thanks, can you also add Chaoyi Huang who is the PTL of Tricircle | 15:21 |
| yolanda | i will add you, then you need to add Chaoyi | 15:21 |
| *** rhallisey has joined #openstack-infra | 15:21 | |
| electrocucaracha | yolanda: perfect, thanks :) | 15:21 |
| yolanda | done | 15:22 |
| *** Julien-zte has quit IRC | 15:24 | |
| electrocucaracha | yolanda: should I see a difference in https://review.openstack.org/#/admin/projects/openstack/python-tricircleclient,access ? | 15:25 |
| yolanda | https://review.openstack.org/#/admin/groups/1706,members | 15:25 |
| *** Julien-zte has joined #openstack-infra | 15:25 | |
| *** marst has joined #openstack-infra | 15:25 | |
| yolanda | and https://review.openstack.org/#/admin/groups/1707,members | 15:25 |
| *** trown is now known as trown|brb | 15:26 | |
| yolanda | you are there | 15:26 |
| yolanda | electrocucaracha, if you don't have perms to add more people, logout and login | 15:26 |
| electrocucaracha | yolanda: I already did to verify if I had +2 but let me try to add more people | 15:26 |
| *** nadya has quit IRC | 15:27 | |
| electrocucaracha | yolanda: ok for adding there is no problem | 15:27 |
| *** erikmwil_ has quit IRC | 15:28 | |
| yolanda | cool | 15:28 |
| *** erikmwilson has joined #openstack-infra | 15:28 | |
| *** trown|brb is now known as trown | 15:28 | |
| *** dizquierdo has joined #openstack-infra | 15:28 | |
| *** jcoufal has joined #openstack-infra | 15:31 | |
| Zara | I think we should investigate gitpython and then revisit it if things look intractable; we'd prefer to match up with other projects if possible. | 15:33 |
| *** mtanino has joined #openstack-infra | 15:33 | |
| Gravious | okay | 15:34 |
| Gravious | i'll check it out, worst case we'll just have to add any missing stuff to gitpython :D | 15:34 |
| Zara | heh, that'd be pretty great actually :) | 15:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Use a more appropriate domain name for FreeIPA https://review.openstack.org/439023 | 15:36 |
| *** sandanar has quit IRC | 15:36 | |
| *** jgriffith has quit IRC | 15:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 15:39 |
| *** jgriffith has joined #openstack-infra | 15:39 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Undercloud containers job with tripleo.sh https://review.openstack.org/437411 | 15:41 |
| *** psachin has quit IRC | 15:41 | |
| *** dhajare has quit IRC | 15:41 | |
| *** yamahata_ has joined #openstack-infra | 15:41 | |
| *** adrian_otto has joined #openstack-infra | 15:42 | |
| *** chlong has joined #openstack-infra | 15:43 | |
| *** ggnel_t has joined #openstack-infra | 15:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Plumb job variables through to ansible https://review.openstack.org/438653 | 15:43 |
| *** ediardo has quit IRC | 15:44 | |
| *** ediardo has joined #openstack-infra | 15:46 | |
| pabelanger | infra-root: I'd like to restart nodepool.o.o this morning to pick up nodepool-id change: https://review.openstack.org/#/c/438677/ | 15:46 |
| fungi | Gravious: Zara: yeah, i'm not immediately finding a gitpython analog of pygit2's create_note(), lookup_note() and notes() methods, nor note class (at least not in the api docs). so depending on effort level it may or may not make sense to try and use gitpython for this. but also the gitpython upstream looks quite active with lots of contributors so probably not hard to get support in there fairly | 15:47 |
| fungi | quickly | 15:47 |
| jeblair | pabelanger: i was just about to ask about that... :) zv3 is starting to leave -1 NODE_FAILUREs, possibly because of that | 15:47 |
| Gravious | fungi, don't worry about absence of note specific methods, they are not required :) | 15:47 |
| fungi | oh, even better | 15:47 |
| *** armax has joined #openstack-infra | 15:47 | |
| fungi | there are definitely low-level object and tree methods in gitpython anyway | 15:47 |
| pabelanger | jeblair: I noticed that last night too. I was going to ping you this morning to look at nl01.o.o logs to see if you can see the reason. | 15:47 |
| pabelanger | or just move forward with nodepool-id restart first | 15:48 |
| pabelanger | Oh, | 15:48 |
| pabelanger | I see an issue | 15:48 |
| pabelanger | http://paste.openstack.org/show/600799/ | 15:48 |
| jeblair | yeah that looks like a bug | 15:49 |
| jeblair | i can try to help track it down in a little while | 15:49 |
| pabelanger | k | 15:49 |
| pabelanger | for now, I am going to stop nl01.o.o, since it is just spamming logs right now | 15:49 |
| *** isaacb has quit IRC | 15:50 | |
| fungi | heh, a tuple consisting of an empty tuple and an empty dictionary does not count as empty! ;) | 15:50 |
| *** ijw has joined #openstack-infra | 15:54 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-baremetal project https://review.openstack.org/438907 | 15:54 |
| *** rajinir_ has joined #openstack-infra | 15:56 | |
| *** guest_999 has quit IRC | 15:57 | |
| *** sdake has quit IRC | 15:58 | |
| *** ociuhandu has joined #openstack-infra | 15:59 | |
| *** dhajare has joined #openstack-infra | 16:01 | |
| *** salv-orlando has joined #openstack-infra | 16:01 | |
| *** xarses has joined #openstack-infra | 16:01 | |
| *** xarses has quit IRC | 16:02 | |
| *** xarses has joined #openstack-infra | 16:02 | |
| *** martinkopec has quit IRC | 16:03 | |
| *** jaosorior has quit IRC | 16:03 | |
| *** ociuhandu has quit IRC | 16:05 | |
| *** vhosakot has joined #openstack-infra | 16:07 | |
| *** sdake has joined #openstack-infra | 16:08 | |
| *** rcernin has quit IRC | 16:08 | |
| *** mugsie__ is now known as mugsie | 16:09 | |
| *** nadya has joined #openstack-infra | 16:09 | |
| *** markus_z has quit IRC | 16:10 | |
| *** esikache1 has joined #openstack-infra | 16:11 | |
| openstackgerrit | Merged openstack-dev/pbr master: doc: Clarify sections in 'setup.cfg' https://review.openstack.org/438509 | 16:12 |
| *** yamamoto has joined #openstack-infra | 16:16 | |
| *** vsaienk0 has joined #openstack-infra | 16:17 | |
| vsaienk0 | infra team please help to figure out why enabled_plugin ironic .... is not set on subnode in ironic multinode + grenade job http://logs.openstack.org/57/436957/6/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial-nv/3bb6256/logs/local.conf.txt.gz | 16:18 |
| *** isaacb has joined #openstack-infra | 16:19 | |
| *** guest_999 has joined #openstack-infra | 16:21 | |
| *** yamamoto has quit IRC | 16:21 | |
| vsaienk0 | sorry valid link is http://logs.openstack.org/57/436957/6/experimental/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial-nv/47ed693/logs/subnode-2/old/local_conf.txt.gz | 16:22 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 16:23 |
| tmorin | clarkb: would you have time to have a look at https://review.openstack.org/#/c/438573/ ? | 16:23 |
| *** vhosakot has quit IRC | 16:23 | |
| fungi | vsaienk0: i want to say we saw similar behaviors from the local.conf change sdague was spearheading in devstack. do you maybe have both a localrc and a local.conf? i gather a lot of the issues which were observed were due to that | 16:23 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 16:24 |
| *** mdrabe_ has joined #openstack-infra | 16:24 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-baremetal project https://review.openstack.org/438907 | 16:24 |
| *** vhosakot has joined #openstack-infra | 16:24 | |
| tmorin | AJaeger, pabelanger: me pinging clarkb above because presented to me as a potential point of contact around git questions, so I thought he may be a git admin able to able in this case | 16:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update favicon for newer OpenStack logo https://review.openstack.org/439045 | 16:24 |
| tmorin | pabelanger: agreeing with you that avoiding submodule is better ; although we've had a rather smooth experience with that until now, it is obviously better if we can do without | 16:25 |
| vsaienk0 | fungi: thanks for quick response. There is no localrc on subnode only local.conf http://logs.openstack.org/57/436957/6/experimental/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial-nv/47ed693/logs/subnode-2/old/ | 16:25 |
| *** mdrabe has quit IRC | 16:27 | |
| fungi | vsaienk0: did you already see http://lists.openstack.org/pipermail/openstack-dev/2017-February/112872.html ? | 16:27 |
| pabelanger | tmorin: that is fine, I can do the 'git admin' things too. I plan on bringing this up in our weekly infra meeting today | 16:27 |
| tmorin | ah ok, cool then | 16:27 |
| clarkb | this is the inject different repo into gerrit repo thing? | 16:27 |
| tmorin | (I'm not sure what was the misunderstanding about submodules, but nevermind) | 16:28 |
| *** wolverineav has joined #openstack-infra | 16:28 | |
| clarkb | I think you probably want to stage what that looks like. A gerrit admin can force push into a branch then you can merge that branch into where you want it to live permanently | 16:28 |
| clarkb | but none of that requires changes project-config | 16:28 |
| *** baoli_ has quit IRC | 16:28 | |
| *** baoli has joined #openstack-infra | 16:29 | |
| vsaienk0 | fungi: no, checking thanks! | 16:30 |
| *** cloudtrainme has joined #openstack-infra | 16:31 | |
| *** salv-orlando has quit IRC | 16:32 | |
| *** wolverineav has quit IRC | 16:33 | |
| sdague | would love to add some additional test jobs to the devstack-tools repo for when we have a chance - https://review.openstack.org/#/c/439000/ | 16:33 |
| *** pcaruana has quit IRC | 16:34 | |
| pabelanger | clarkb: ya, I suggested we could use track-upstream for it, but force push sounds better. I couldn't remember how we did it in the past | 16:35 |
| *** e0ne has quit IRC | 16:36 | |
| clarkb | pabelanger: since its a one off, I think that its easier. If the goal was persistent tracking then definitely do the track upstream thing | 16:36 |
| pabelanger | k | 16:36 |
| *** vhosakot has quit IRC | 16:36 | |
| *** vhosakot has joined #openstack-infra | 16:37 | |
| *** baoli has quit IRC | 16:37 | |
| tmorin | clarkb: not sure this is what you mean as "stage", but I've got a repo ready with everything merged as wanted : https://github.com/tmmorin/networking-bagpipe | 16:38 |
| openstackgerrit | Merged openstack-dev/pbr master: Stop using 'warnerrors' https://review.openstack.org/438510 | 16:38 |
| tmorin | if this gerrit repo can be forced-merged now into networking-bagpipe (master, or another branch if preferred) then that would be great | 16:39 |
| *** xarses has quit IRC | 16:40 | |
| *** xarses has joined #openstack-infra | 16:41 | |
| *** Jeffrey4l has quit IRC | 16:41 | |
| *** csmart has quit IRC | 16:42 | |
| *** xarses has quit IRC | 16:42 | |
| *** Jeffrey4l has joined #openstack-infra | 16:42 | |
| clarkb | tmorin: well we'd have to push a specific ref/branch/tag to another branch | 16:43 |
| clarkb | then you could merge that branch into wherever yo uwant it to live more permanently | 16:43 |
| tmorin | clarkb: you mean a new branch on networking-bagpipe, not master: ok for me | 16:43 |
| *** vhosakot has quit IRC | 16:43 | |
| clarkb | tmorin: yes that way we avoid races and it allows you to merge it when you are ready | 16:43 |
| *** salv-orlando has joined #openstack-infra | 16:44 | |
| tmorin | clarkb: the source is the tip of master on https://github.com/tmmorin/networking-bagpipe | 16:44 |
| clarkb | also allows for it to go thorugh testing first to make sure it won't break you | 16:44 |
| tmorin | clarkb: ok | 16:44 |
| *** guest_999 has quit IRC | 16:44 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 16:45 |
| tmorin | clarkb: I'm not familiar with how we will merge that branch when everything is ready, but I'm sure I'll find help when the time comes | 16:45 |
| tmorin | clarkb/pabelanger: when could the operation be made of force-pushing the content of https://github.com/tmmorin/networking-bagpipe master into a bagpipe_bgp_merge branch in openstack/networking-bagpipe ? | 16:46 |
| clarkb | tmorin: the merging should be the same rough process as used for feature branches. https://docs.openstack.org/infra/manual/drivers.html#feature-branches | 16:46 |
| clarkb | tmorin: maybe you can review that and double check it sounds like it will work for you | 16:46 |
| mugsie | is there issues with the devstack images at the moment? - I am seeing http://paste.openstack.org/show/600804/ | 16:47 |
| clarkb | tmorin: in this case the feature branch is the code you have prepared that we are force pushing then you will sync it to master | 16:47 |
| tmorin | clarkb: understood | 16:47 |
| *** wolverineav has joined #openstack-infra | 16:48 | |
| clarkb | mugsie: I'm not aware of any issues, I do notice that that job is running on trusty. Is it for a master branch? not sure if devstack still support trusty against master | 16:48 |
| *** nadya has quit IRC | 16:48 | |
| mugsie | its a mitaka job | 16:49 |
| tmorin | clarkb: all this sounds good -- next question is: can this be done now ? | 16:49 |
| mugsie | we dropped all our trusty jobs on master | 16:50 |
| clarkb | tmorin: I'm not going to be able to do it now. I have a bunch of other things currently on my plate. Any other gerrit admin can do it if they have time | 16:50 |
| clarkb | mugsie: ok so it shoul dwork there. Possibly just a stable regression? | 16:50 |
| tmorin | clarkb: ok, who can I ask ? do I need to open a bug to track this request ? | 16:50 |
| mugsie | maybe? I am not sure, as i dont know enough about devstack to figure out what it is | 16:50 |
| *** csmart has joined #openstack-infra | 16:51 | |
| clarkb | tmorin: any gerrit admin, roughly the people that should respond to infra-root | 16:51 |
| *** nadya has joined #openstack-infra | 16:51 | |
| clarkb | tmorin: as pabelanger mentioned earlier he is one of them as is fungi and mordred and yolanda, I think that is who I see has been active today | 16:52 |
| *** wolverineav has quit IRC | 16:52 | |
| *** pblaho has quit IRC | 16:52 | |
| tmorin | ah, ok | 16:52 |
| tmorin | I wasn't aware there were that many | 16:52 |
| *** nadya has quit IRC | 16:52 | |
| clarkb | mugsie: looks like tput wants to set things up for shell based on terminfo. And it can't determine what the terminal is. Guessing that could be related to how ansible uses ssh to run devstack-gate | 16:53 |
| clarkb | mugsie: we may not allocate a tty | 16:53 |
| jeblair | did someone say infra-root? | 16:53 |
| mugsie | clarkb: yes - want me to file a bug? | 16:53 |
| mugsie | I got to run off now, but I can check back in the UTC morning | 16:53 |
| tmorin | jeblair: clarkb just did, and gave me a few names of who could help | 16:53 |
| clarkb | mugsie: probably not a bad idea, that said does it cause the job to fial? looks like that may be noisy but not failure related? | 16:53 |
| *** ijw_ has joined #openstack-infra | 16:54 | |
| amrith | hi infra, would some of you please review (and hopefully approve) https://review.openstack.org/#/c/438668/ the change is to put mysql on a diet and reduce the memory consumption | 16:54 |
| amrith | clarkb, fungi ^^ | 16:54 |
| tmorin | pabelanger, yolanda: would any of you have some time to force-pull the content of https://github.com/tmmorin/networking-bagpipe master into a bagpipe_bgp_merge branch in openstack/networking-bagpipe ? (as discussed above with clarkb?) | 16:54 |
| mugsie | clarkb: i assumed it was the root cause, as the required packages were not installed, but I could be wrong | 16:54 |
| clarkb | mugsie: its just the PS4 not being able to print due to the "error" | 16:55 |
| clarkb | mugsie: I don't know that that will affect shell exit codes, its purely informational I htink | 16:55 |
| mugsie | ah | 16:55 |
| mugsie | til | 16:55 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/zuul master: Prepare correct refspec on new Gerrit https://review.openstack.org/439057 | 16:56 |
| *** ijw has quit IRC | 16:57 | |
| mugsie | clarkb: filed https://bugs.launchpad.net/devstack/+bug/1668695 | 16:58 |
| openstack | Launchpad bug 1668695 in devstack "Devstack is failing on stable/mitaka with "cannot find generate-subunit"" [Undecided,New] | 16:58 |
| *** tqtran has joined #openstack-infra | 16:59 | |
| openstackgerrit | Clay Gerrard proposed openstack-infra/project-config master: Add experimental python35 job to pyeclib https://review.openstack.org/438756 | 16:59 |
| *** tqtran has quit IRC | 16:59 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Undercloud containers job with tripleo.sh https://review.openstack.org/437411 | 16:59 |
| clarkb | mugsie: oh the cannot find generate-subunit issue is realted to the local.conf/localrc things. Are you having failures from that today? if so please read sdague's email on the subject from last week | 17:00 |
| clarkb | mugsie: I htink he outlines the way to fix | 17:00 |
| *** saibarspeis has quit IRC | 17:01 | |
| mugsie | clarkb: OK, cool. will dig into that | 17:01 |
| mugsie | thankms for the help | 17:02 |
| sdague | clarkb: https://review.openstack.org/#/c/438957/ to get neutron fully over I think we'll need this - https://review.openstack.org/#/c/438957/ | 17:02 |
| *** Julien-zte has quit IRC | 17:02 | |
| *** tphummel has joined #openstack-infra | 17:02 | |
| *** sdake_ has joined #openstack-infra | 17:03 | |
| clarkb | sdague: ya stuff like that is what made debugging ironic so complicated... | 17:03 |
| sdague | clarkb: yep | 17:03 |
| sdake_ | quick q related to failing gate jobs - since these jobs are non-voting, are they safe to ignore or is more work on my end required? I looked at the logs and it looks like the gate jobs are fundamentally busted: https://review.openstack.org/#/c/438857/ | 17:03 |
| sdague | clarkb: well, it is what people are using | 17:03 |
| *** flepied has quit IRC | 17:04 | |
| clarkb | sdague: those two fails can be ignored, fixing them is work in progress | 17:04 |
| clarkb | er sdake_ ^ | 17:04 |
| sdake_ | clarkb i get that alot :) | 17:04 |
| *** nicolasbock has quit IRC | 17:04 | |
| clarkb | sdague: is that actually late enough though? they are running things in pre_test_hook when is later than setup_localrc iirc | 17:06 |
| *** ramishra has quit IRC | 17:06 | |
| *** ijw has joined #openstack-infra | 17:06 | |
| *** sdake has quit IRC | 17:07 | |
| *** nicolasbock has joined #openstack-infra | 17:08 | |
| clarkb | sdague: or is it sufficient to just be the last set of things happening in setup_localrc? | 17:08 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Add kuryr-libnetwork into $PROJECTS https://review.openstack.org/438731 | 17:08 |
| sdague | clarkb: it worked putting it there | 17:09 |
| *** tesseract has quit IRC | 17:09 | |
| sdague | previously they were using the vars listed above | 17:09 |
| *** asselin_ has joined #openstack-infra | 17:09 | |
| sdague | clarkb: https://review.openstack.org/#/c/438682/ is the neutron patch that consumes it | 17:09 |
| *** ijw_ has quit IRC | 17:09 | |
| sdague | and... to the best of my understanding, is working | 17:10 |
| clarkb | sdague: I guess I am just trying to figure out how DEVSTACK_LOCAL_CONFIG wasn't late enough | 17:10 |
| clarkb | or is it more that that only support localrc things? | 17:10 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Fuxi] Add python-manilaclient into $PROJECTS https://review.openstack.org/438733 | 17:10 |
| *** Swami has joined #openstack-infra | 17:10 | |
| *** tmorin has quit IRC | 17:11 | |
| sdague | clarkb: it's setting post-config things as well | 17:11 |
| sdague | but, the real collision was the local section | 17:11 |
| sdague | because when you add local section items, they always go at the end | 17:11 |
| *** csmart has quit IRC | 17:11 | |
| clarkb | line 80-84 on the old side of the diff is scary... | 17:11 |
| *** coreyob_ is now known as coreyob | 17:11 | |
| *** masayukig[m] has quit IRC | 17:12 | |
| sdague | so order is important, because we never replace things | 17:12 |
| sdague | because it's not a config format, it's litterally shell code | 17:12 |
| clarkb | (thats the same deal that got really confusing with ironic... I really wish we didn't turtle d-g this way) | 17:12 |
| *** sshnaidm is now known as sshnaidm|off | 17:12 | |
| sdague | clarkb: right | 17:12 |
| clarkb | basically your config to d-g/devstack should be at top level | 17:12 |
| clarkb | and when it isn't everything gets crazy complicated, things break, reproduce.sh can't be accurate, etc | 17:13 |
| *** asselin has quit IRC | 17:13 | |
| sdague | yeh, so if you use the project config stanza, you get that, but if you are dynamically generating it, ... all bets are off | 17:13 |
| sdague | I think the real issue was ever letting gate hooks do that :) | 17:13 |
| sdague | but, sins of the past | 17:14 |
| clarkb | well there isn't really any way around it other than to say we don't care if we break you because what you are doing is unsupported | 17:14 |
| *** wolverineav has joined #openstack-infra | 17:14 | |
| clarkb | they then reexec d-g later to make it work | 17:14 |
| sdague | someone else is going to need to dig out of it | 17:14 |
| sdague | this will let neutron use the new system | 17:14 |
| clarkb | so all they ever showed infra was the correct stuff with config in project-config that sets it at top level | 17:14 |
| *** knangia has joined #openstack-infra | 17:14 | |
| clarkb | then they without anyone else knowing subvert that within their gate hook by changing configs and reexecing | 17:14 |
| *** nadya has joined #openstack-infra | 17:14 | |
| clarkb | and thats not something you can prevent if people want to do it | 17:14 |
| clarkb | but we also shouldn't support it imo | 17:15 |
| sdague | clarkb: yeh, well, that's a giant unwind | 17:15 |
| sdague | and I'm not signing up for it | 17:15 |
| *** ihrachys has quit IRC | 17:15 | |
| *** wolverineav has quit IRC | 17:16 | |
| *** wolverineav has joined #openstack-infra | 17:16 | |
| clarkb | ya... | 17:16 |
| clarkb | I'm just grumpy that we have the confusing spaghetti mess | 17:17 |
| *** wolverin_ has joined #openstack-infra | 17:17 | |
| pabelanger | clarkb: jeblair: ready to move forward with nodepool.o.o restart? | 17:17 |
| *** yamamoto has joined #openstack-infra | 17:18 | |
| clarkb | sdague: so wouldn't this also have worked if neutron just did their own merge instead of overwriting local.conf from scratch? | 17:18 |
| clarkb | sdague: wondering why we need to change d-g at all for that and instead just force neutron to fix their breaking of internal interfaces | 17:18 |
| clarkb | sdague: basically line 84 on the old diff is what is breaking things because it uses > and not merge_lc | 17:19 |
| *** jordanP has quit IRC | 17:19 | |
| *** funzo has quit IRC | 17:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Ignore release bot account in owners example https://review.openstack.org/439065 | 17:19 |
| *** funzo has joined #openstack-infra | 17:20 | |
| jeblair | pabelanger: ++ | 17:20 |
| *** wolverineav has quit IRC | 17:20 | |
| *** armax has quit IRC | 17:21 | |
| clarkb | sorry in this case its setlc_raw | 17:21 |
| clarkb | via the existing d-g code | 17:22 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 17:22 |
| *** nadya has quit IRC | 17:23 | |
| *** isaacb has quit IRC | 17:23 | |
| openstackgerrit | Merged openstack-dev/pbr master: tox: Don't set skipsdist=True https://review.openstack.org/438511 | 17:23 |
| *** yamahata_ has quit IRC | 17:23 | |
| pabelanger | okay, nodepool has been restarted | 17:23 |
| *** yamamoto has quit IRC | 17:23 | |
| *** sdake_ is now known as sdake | 17:24 | |
| *** Hal has quit IRC | 17:24 | |
| sdague | clarkb: nope | 17:24 |
| *** nadya has joined #openstack-infra | 17:25 | |
| sdague | clarkb: so, the thing you need to remember is that it's 3 different cases in the run block | 17:25 |
| sdague | clarkb: L51 on the old block | 17:26 |
| sdague | they are switching on $VENV | 17:26 |
| sdague | so, in case #1, they run stack.sh at the end, in which case, merge is fine | 17:26 |
| *** eranrom has joined #openstack-infra | 17:26 | |
| sdague | and that's where L84 is removed | 17:26 |
| sdague | but... | 17:26 |
| *** thorst_afk is now known as thorst | 17:27 | |
| sdague | case #2 - L93 | 17:27 |
| sdague | ends up just running devstack-gate | 17:27 |
| sdague | as does case #3 - L 115 | 17:27 |
| *** eranrom has quit IRC | 17:28 | |
| sdague | in those cases, without this variable, the flow mean that the local.conf is written, then d-g is merging into it, so things like ENABLED_SERVICES end up after all the enable_service stanzas they have in their files | 17:28 |
| *** ijw has quit IRC | 17:28 | |
| *** eranrom has joined #openstack-infra | 17:28 | |
| clarkb | sdague: right so if we stop writing to local.conf entirely and only set the var and let d-g do it then wouldn't it be happy? I guess that doesn't account for the case that runs stack.sh directly | 17:29 |
| sdague | clarkb: right | 17:29 |
| pabelanger | jeblair: clarkb: next step is to reduce infracloud-chocolate to 50nodes for nodepool.o.o, and add nodepool-id: https://review.openstack.org/#/c/438998/ This will give nl01.o.o 50 nodes to work with | 17:29 |
| *** nadya has quit IRC | 17:29 | |
| *** eranrom has quit IRC | 17:29 | |
| *** csomerville has quit IRC | 17:29 | |
| sdague | clarkb: I guess I'm not understanding the suggestion beyond what I did there | 17:29 |
| sdague | because I think I did what you are suggesting | 17:29 |
| clarkb | sdague: my suggestion is to not add a new var that does the appeneding and just use the one that already exists a few lines above | 17:30 |
| sdague | clarkb: that injects into the local section | 17:30 |
| *** trown is now known as trown|lunch | 17:30 | |
| sdague | these files include [[post-config|$NEUTRON_CONF]] sections | 17:30 |
| *** eranrom has joined #openstack-infra | 17:31 | |
| *** eranrom has quit IRC | 17:31 | |
| jeblair | pabelanger, clarkb: oh, we have 100 servers on chocolate? | 17:31 |
| *** ZZelle has joined #openstack-infra | 17:31 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add gate-tripleo-ci-centos-7-ovb-updates into promotion pipeline https://review.openstack.org/433796 | 17:31 |
| clarkb | jeblair: ya | 17:31 |
| clarkb | sdague: gotcha ok that was the missing link for me | 17:31 |
| sdague | you can't reuse that old variable, because people are using it under the assumption that they don't need [[local|localrc]] explicitly in their stream | 17:31 |
| *** tosky has quit IRC | 17:31 | |
| sdague | and we'd break all of them | 17:31 |
| jeblair | pabelanger, clarkb: do we want to reserve that much for v3, or do we want to maybe only reserve 10-20? | 17:31 |
| *** eranrom has joined #openstack-infra | 17:31 | |
| sdague | unless we did hack based on detection for that | 17:31 |
| clarkb | jeblair: I threw out 10 yesterday | 17:31 |
| sdague | but, that makes me nervous | 17:31 |
| clarkb | jeblair: seems like that should be plenty for multinode even | 17:32 |
| sdague | at least here it's explicit | 17:32 |
| clarkb | sdague: ya lets avoid hacks and be explicit here. With this I will post review | 17:32 |
| sdague | clarkb: cool | 17:32 |
| jeblair | pabelanger: okay, let's set it to 90? | 17:32 |
| amrith | yolanda, jeblair, clarkb, fungi, rcarrillocruz, pabelanger, ianw ... when you get a chance please take a look at https://review.openstack.org/#/c/438668/; monty gave it one thumbs up | 17:33 |
| *** ccamacho has left #openstack-infra | 17:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add nodepool-id to infracloud-chocolate https://review.openstack.org/438998 | 17:35 |
| pabelanger | jeblair: done | 17:35 |
| *** claudiub has quit IRC | 17:36 | |
| *** rossella_s has quit IRC | 17:36 | |
| *** ijw has joined #openstack-infra | 17:36 | |
| *** ihrachys has joined #openstack-infra | 17:36 | |
| fungi | amrith: lgtm, but one of the reasons i suggested getting feedback from the devstack maintainers in qa is that the infra team in general doesn't have approval rights on qa repos | 17:36 |
| amrith | ok fungi will ping devstack maintainers | 17:37 |
| clarkb | amrith: was a compraison made between old mem consumption and new with that chane? | 17:38 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 17:39 | |
| amrith | clark, I did it on my local machine | 17:39 |
| jeblair | fungi: i reset channel topic | 17:39 |
| amrith | but didn't in the gate | 17:39 |
| fungi | oh, hah, thanks jeblair! | 17:39 |
| fungi | i completely spaced on the fact it was still set to that | 17:39 |
| amrith | clarkb, I tried | 17:39 |
| clarkb | amrith: would probably be good to collect and publish that data somewhere | 17:39 |
| amrith | clarkb, but I can't depend on this change | 17:39 |
| sdague | amrith: so, at the end of a full gate run, that seems to change the mysql footprint from 530 MB to 480 MB | 17:39 |
| clarkb | oh good sdague will do it really quick :) | 17:39 |
| sdague | which... doesn't seem like a huge win | 17:39 |
| amrith | clarkb and therefore unless this merges, I can't get the real impact on (say) trove | 17:39 |
| sdague | http://logs.openstack.org/57/438957/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa590e1/logs/screen-dstat.txt.gz | 17:39 |
| *** strigazi has quit IRC | 17:39 | |
| sdague | vs | 17:40 |
| sdague | http://logs.openstack.org/68/438668/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c2b21b/logs/screen-dstat.txt.gz | 17:40 |
| jeblair | amrith: why can't you depend on the change? | 17:40 |
| *** masayukig[m] has joined #openstack-infra | 17:40 | |
| amrith | jeblair, it is a change in devstack which means that if I depend, I don't (appear) to get the benefits of the change | 17:40 |
| amrith | I'm not sure how the magic of the gate works | 17:40 |
| *** strigazi has joined #openstack-infra | 17:40 | |
| amrith | but if I push a trove change and have a depends-on, nothing happens | 17:40 |
| *** ijw has quit IRC | 17:41 | |
| sdague | so, if we think that 50 MB is enough gain here, that's cool. I don't know what parts of performance get squeezed in the interim that might cause other fails to emerge | 17:41 |
| amrith | like https://review.openstack.org/#/c/438664/ | 17:41 |
| fungi | you should definitely get devstack modifications applied to devstack-based jobs run on changes depending on that one | 17:41 |
| *** pblaho has joined #openstack-infra | 17:41 | |
| amrith | sdague, this pushes mysql quite to the wall | 17:41 |
| amrith | I didn't drop max connecitons | 17:41 |
| amrith | which someone seems to set (IMHO, too high) | 17:41 |
| amrith | see line 99 in the change | 17:42 |
| sdague | amrith: if we don't have it set that high, we run out of connections iirc | 17:42 |
| amrith | 1024 connections seems much for a gate | 17:42 |
| *** flepied has joined #openstack-infra | 17:42 | |
| fungi | tempest performs a lot of operations in parallel | 17:42 |
| sdague | what is the per connection overhead? | 17:42 |
| amrith | sdague, sounds like someone is orphaning connections or there's a badly tuned connection pool | 17:42 |
| *** mhickey has quit IRC | 17:42 | |
| sdague | connection pools vs. eventlet I thought were an issue | 17:42 |
| amrith | sdague, the per connection overhead is a combination of many of the other things | 17:42 |
| amrith | like caches, don't have the exact number for these configurations | 17:43 |
| amrith | I know that I can run trove with 64 connections | 17:43 |
| amrith | but 1024 seemed high to me, wouldn't mind dropping to 512 to see what happened | 17:43 |
| jeblair | amrith: for change 438664, how did you determine that 438668 was not being used? | 17:43 |
| *** derekh has quit IRC | 17:44 | |
| sdague | amrith: well, probably worth asking SpamapS, he made that change as a performance turning - https://review.openstack.org/#/c/193295/ | 17:44 |
| amrith | jeblair, yes. the setting sprintend in 664 were as if 668 was not taking effect | 17:44 |
| jeblair | amrith: can you point me at that printed output? | 17:45 |
| *** Apoorva has joined #openstack-infra | 17:45 | |
| amrith | one second | 17:45 |
| SpamapS | per-connection overhead in mysql is highly subjective | 17:45 |
| SpamapS | the only fixed cost is the stack, which is configurable and the default varies by mysql version | 17:46 |
| SpamapS | https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_thread_stack | 17:47 |
| *** matrohon has quit IRC | 17:47 | |
| amrith | jeblair, see http://logs.openstack.org/64/438664/2/check/gate-trove-scenario-dsvm-mariadb-single-ubuntu-xenial-nv/5a3bfc2/logs/devstack-early.txt.gz#_2017-02-27_18_52_46_215 look for innodb_buffer_pool_size | 17:47 |
| amrith | I want it at 5mb, it doesn't appear to stick | 17:47 |
| amrith | see: http://logs.openstack.org/64/438664/2/check/gate-trove-scenario-dsvm-mariadb-single-ubuntu-xenial-nv/5a3bfc2/logs/devstack-early.txt.gz#_2017-02-27_18_52_46_228 | 17:47 |
| SpamapS | Beyond that, you have a few buffers which can be per-join, and in-memory tables which can be per-join | 17:47 |
| SpamapS | Idle connections in mysql only use the stack though. | 17:48 |
| *** annegentle has joined #openstack-infra | 17:48 | |
| amrith | SpamapS, the other things which are large are buffer pool sizes | 17:48 |
| amrith | which I've tried to reduce | 17:48 |
| amrith | given that the tables in use are rarely all that big | 17:48 |
| *** nadya has joined #openstack-infra | 17:48 | |
| *** sdake_ has joined #openstack-infra | 17:48 | |
| SpamapS | buffer pool size is global, and you can definitely tune that down to the expected size of the db | 17:49 |
| jeblair | amrith: http://logs.openstack.org/64/438664/2/check/gate-trove-scenario-dsvm-mariadb-single-ubuntu-xenial-nv/5a3bfc2/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-02-27_18_35_04_926 | 17:49 |
| jeblair | amrith: that's your devstack commit getting checked out | 17:49 |
| amrith | hmm, well sdague said it reduced memory usage looking at something; let me scroll back up | 17:49 |
| *** sdake has quit IRC | 17:50 | |
| SpamapS | Note that you can limit the impact of the per-query/join buffers like sort_buffer_size by setting this low: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_thread_concurrency | 17:50 |
| sdague | amrith: I was just looking at dstat | 17:50 |
| sdague | because mysql is the biggest memory process | 17:50 |
| SpamapS | oh derp | 17:50 |
| SpamapS | no, that's solaris only | 17:50 |
| sdague | so the dstat biggest memory, scroll to end | 17:50 |
| sdague | compare the numbers | 17:50 |
| SpamapS | and is gone in 5.7.2 double-derp | 17:50 |
| sdague | 530 MB vs. 480 MB | 17:51 |
| jeblair | amrith: http://logs.openstack.org/64/438664/2/check/gate-trove-scenario-dsvm-mariadb-single-ubuntu-xenial-nv/5a3bfc2/logs/devstack-early.txt.gz#_2017-02-27_18_45_14_907 | 17:51 |
| jeblair | amrith: that's the iniset for 5M running | 17:51 |
| amrith | jeblair, any way we can get a my.cnf off a machine | 17:52 |
| SpamapS | https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_thread_pool_size this one is the thing I was thinking of | 17:52 |
| amrith | jeblair, found the issue | 17:52 |
| amrith | I don't have a section on these configurations | 17:53 |
| amrith | so they are not taking effect | 17:53 |
| amrith | ( i think ) | 17:53 |
| amrith | let me try again | 17:53 |
| SpamapS | The way to right-size mysqld in this case is probably to hold a node, and size innodb_buffer_pool_size to the disk usage in /var/lib/mysql + 5% | 17:53 |
| SpamapS | That will likely yield the biggest gains. | 17:53 |
| sdague | so... before folks focus on this too much | 17:53 |
| sdague | just to put it in perspective | 17:53 |
| * SpamapS can't help but paint this bike shed | 17:54 | |
| sdague | 530 MB memory, is 8% of our memory | 17:54 |
| sdague | not including swap | 17:54 |
| amrith | sdague, the change is currently not taking effect; I don't think | 17:54 |
| sdague | it's not like mysql is massively overrunning everything | 17:54 |
| amrith | let me fix it first | 17:54 |
| SpamapS | so if we can drop it in half (which I think we can) then it will be a nice 4% pad. :) | 17:54 |
| sdague | that the python workers will all grow into in 6 weeks time? | 17:55 |
| SpamapS | exactly | 17:55 |
| SpamapS | unchain the python! | 17:55 |
| sdague | it just seems like the wrong focus point for memory profiling | 17:55 |
| SpamapS | agree with you on that in the grander scheme | 17:55 |
| amrith | thanks jeblair | 17:55 |
| amrith | retrying now | 17:55 |
| SpamapS | at the micro level, 6 weeks of time to calmly memory profile seems useful? | 17:55 |
| SpamapS | (assuming somebody keeps pounding on the larger problem set) | 17:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki master: Optionally alias to a favicon.ico file if provided https://review.openstack.org/439082 | 17:58 |
| amrith | sdague, I've pushed up a new change, let's see what it does now that I got the configuration parameters into the right section. thx jeblair for pointing out what the issue was | 17:58 |
| jeblair | clarkb: can you +3 438998 | 17:58 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Manage the favicon.ico file for the wiki https://review.openstack.org/439083 | 17:58 |
| jeblair | amrith: np | 17:58 |
| clarkb | jeblair: pabelanger do we not want to set the id on both the configs? nodepool.yaml and nodepoolv3.yaml? | 17:59 |
| notmyname | in case we've got any dependencies directly or indirectly, looks like S3 is having issues (at least us-east-1) | 17:59 |
| jeblair | clarkb: i think only nodepool.yaml is required. i don't think v3 will honor it. | 18:00 |
| clarkb | jeblair: and we don't have to worry about v3 deleting htings becaues it doesn't have cleanups written yet right? | 18:00 |
| jeblair | clarkb, pabelanger: which is a problem. | 18:00 |
| fungi | notmyname: thanks for the heads up. i don't think we use it directly, but i seem to recall hearing that pypi does so we might see fallout there | 18:00 |
| notmyname | fungi: we just checked. pypi seems to use us-west-2 (and seems to be working) | 18:01 |
| jeblair | clarkb, pabelanger: we only compare nodepool_id if it is *set* in the metadata, so we actually only need it on the v3 side, which is where it is not implemented. | 18:01 |
| fungi | notmyname: cool, that's a relief at least. thanks! | 18:02 |
| *** sdake_ has quit IRC | 18:02 | |
| jeblair | clarkb, pabelanger: patchset 1 of 438677 would have worked | 18:02 |
| *** lin_yang has joined #openstack-infra | 18:02 | |
| jeblair | clarkb, pabelanger, mordred: but patchset 2 avoided comparing to None, which is, in fact, the value we expect nodepoolv3 to supply. | 18:04 |
| *** lucasagomes is now known as lucas-afk | 18:05 | |
| *** sdake has joined #openstack-infra | 18:05 | |
| *** armax has joined #openstack-infra | 18:05 | |
| pabelanger | yes, that seem right | 18:06 |
| jeblair | clarkb, pabelanger, mordred: i think "if provider.nodepool_id is not None and nodepoo_id != provider.nodepool_id" is how that should read. | 18:06 |
| *** cloudtrainme has quit IRC | 18:06 | |
| *** hashar has quit IRC | 18:06 | |
| jeblair | so let's either update it to that and restart again, or go with clarkb's idea. :) | 18:06 |
| *** jpich has quit IRC | 18:06 | |
| *** chlong has quit IRC | 18:06 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use custom network-isolation-absolute.yaml in ipv4 net-iso jobs https://review.openstack.org/422297 | 18:07 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Add storage-mgmt network to compute nodes https://review.openstack.org/419049 | 18:07 |
| pabelanger | clarkb: jeblair: okay, either choice appears to be a code change. Happy to do either | 18:07 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Expose lbaasv2 log in tempest-logs.html not lbaasv1 https://review.openstack.org/439088 | 18:08 |
| jeblair | pabelanger: i think it's your call | 18:09 |
| pabelanger | k, will update nodepool master | 18:10 |
| jeblair | notmyname: but the checkmark is green! https://status.aws.amazon.com/ | 18:11 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 18:12 |
| notmyname | jeblair: well then in that case... no worries! ;-) | 18:12 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use custom network-isolation-absolute.yaml in ipv4 net-iso jobs https://review.openstack.org/422297 | 18:14 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Add storage-mgmt network to compute nodes https://review.openstack.org/419049 | 18:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Fix logic error with nodepool-id https://review.openstack.org/439090 | 18:15 |
| pabelanger | jeblair: clarkb: ^ | 18:15 |
| clarkb | pabelanger: isn't that the same check? | 18:16 |
| clarkb | actually I see now that is what jeblair suggested above so now I have to udnerstand why/how that is different | 18:16 |
| clarkb | oh its comparing against whether we set it in the config | 18:17 |
| pabelanger | right | 18:17 |
| pabelanger | since we are only changing nodepool.yaml right now | 18:17 |
| *** salv-orl_ has joined #openstack-infra | 18:17 | |
| jeblair | clarkb: if provider.nodepool_id is None, we don't perform the check at all (so we will delete nodes launched from other nodepools with same provider name) | 18:17 |
| clarkb | right | 18:17 |
| jeblair | clarkb: if it is not None, then we perform the check to make sure it was launched by us. | 18:18 |
| *** dhinesh has joined #openstack-infra | 18:18 | |
| jeblair | (and the node's nodepool_id still might be None (it will be in our case from zuulv3). that's fine, it won't match and we won't delete it) | 18:19 |
| *** esikache1 has quit IRC | 18:19 | |
| clarkb | right | 18:19 |
| *** yamamoto has joined #openstack-infra | 18:19 | |
| jeblair | it will also be None for the production nodes launched before the config file change. we won't delete those instances if they leak. that's probably okay. it's a small race. | 18:20 |
| *** salv-orlando has quit IRC | 18:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DON'T MERGE: test tempest all branches https://review.openstack.org/436923 | 18:20 |
| clarkb | and easy enough to do a listing and remove any really old ones tomorrow | 18:20 |
| jeblair | (we should just know that if we see any leaked chocolate instances, it might be because of that race) | 18:20 |
| *** sdake has quit IRC | 18:21 | |
| *** sdake has joined #openstack-infra | 18:23 | |
| *** yamamoto has quit IRC | 18:25 | |
| AJaeger | fungi, https://review.openstack.org/#/c/427895/ and https://review.openstack.org/#/c/435569/ are now ready to move, trove change is merged, amrith worked around the issue (thanks) | 18:27 |
| *** esikache1 has joined #openstack-infra | 18:27 | |
| AJaeger | infra-root, I'm on family vacation this week, don't expect to see much of me... | 18:27 |
| clarkb | AJaeger: enjoy | 18:27 |
| AJaeger | thanks, clarkb | 18:28 |
| pabelanger | AJaeger: enjoy | 18:28 |
| fungi | AJaeger: thanks for the heads up! (also same for me wrt family vacation) | 18:28 |
| *** dizquierdo has quit IRC | 18:28 | |
| AJaeger | thanks, pabelanger and fungi | 18:28 |
| *** dharinic is now known as dharinic|lunch | 18:31 | |
| *** krtaylor has quit IRC | 18:32 | |
| *** esikache1 has quit IRC | 18:32 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 18:33 |
| *** sambetts is now known as sambetts|afk | 18:33 | |
| *** dtantsur is now known as dtantsur|afk | 18:34 | |
| *** armax has quit IRC | 18:38 | |
| ihrachys | anyone has an idea what generates those nice Log detail: links (WARNING, TRACE, ...) in logs? and why not every log file gets them? | 18:38 |
| *** dprince has quit IRC | 18:38 | |
| ihrachys | I see there is os-loganalyze but I fail to find where it's hooked | 18:38 |
| clarkb | ihrachys: os-loganalyze is what does it, there is an explicit whitelist in there that will only do it for files it knows it can do it for | 18:39 |
| clarkb | ihrachys: so if a file is parseable and missing the links you likely just need to whitelist it | 18:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Move containers-oooq to the check pipeline https://review.openstack.org/437945 | 18:40 |
| EmilienM | flaper87: I took over ^ | 18:40 |
| ihrachys | ohhhh, it's in code | 18:40 |
| ihrachys | couldn't expect it there, we usually control that externally through configs | 18:40 |
| *** baoli has joined #openstack-infra | 18:41 | |
| flaper87 | EmilienM: sounds good | 18:43 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/os-loganalyze master: Capture logs for lib/neutron services https://review.openstack.org/439099 | 18:43 |
| ihrachys | clarkb: ^ that will do for neutron- services | 18:43 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 18:44 |
| *** salv-orl_ has quit IRC | 18:45 | |
| *** trown|lunch is now known as trown | 18:47 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Clarify description for q-agt service https://review.openstack.org/439102 | 18:48 |
| clarkb | fungi: I am adding last minute infra meeting topic | 18:49 |
| *** yamahata has joined #openstack-infra | 18:49 | |
| *** chem is now known as chem_gone | 18:49 | |
| fungi | thanks | 18:50 |
| fungi | i am still scrambling to do last-minute meeting prep anyway | 18:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove hardcoded reference to newton for debian-package jobs https://review.openstack.org/439103 | 18:50 |
| *** esikache1 has joined #openstack-infra | 18:50 | |
| *** ijw has joined #openstack-infra | 18:51 | |
| *** savihou has joined #openstack-infra | 18:51 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix logic error with nodepool-id https://review.openstack.org/439090 | 18:54 |
| *** esikache1 has quit IRC | 18:55 | |
| mnaser | https://news.ycombinator.com/item?id=13755673 | 18:56 |
| mnaser | in case anyone sees weird failures | 18:56 |
| mnaser | the internet is melting | 18:56 |
| clarkb | its ok I think our internet is sufficiently detached from the rest of the internet | 18:57 |
| * fungi lives on an alternet | 18:57 | |
| *** ijw has quit IRC | 18:57 | |
| fungi | mnaser: don't forget to stock up on canned goods and bottled water! | 18:58 |
| *** esikache1 has joined #openstack-infra | 18:58 | |
| mnaser | fungi openstack marketing people stocking up on failure stories :D | 18:58 |
| fungi | mnaser: they have more class than that | 18:58 |
| mnaser | fungi more than s3's storage classes? :> | 18:59 |
| fungi | anybody can have outages, after all | 18:59 |
| clarkb | mnaser: that said, I am somewhat surprised my pandora streams haven't died. I guess they don't store all their songs in s3 | 19:00 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| mnaser | clarkb its just weird starting to run into all the weird failure scenarios today :( | 19:00 |
| *** baoli has quit IRC | 19:00 | |
| *** baoli has joined #openstack-infra | 19:01 | |
| *** baoli has quit IRC | 19:01 | |
| *** esikache1 has quit IRC | 19:02 | |
| *** pvaneck has joined #openstack-infra | 19:04 | |
| *** dprince has joined #openstack-infra | 19:08 | |
| clayg | Andreas Jaeger helpfully pointed me toward https://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats from https://review.openstack.org/#/c/438756/ | 19:08 |
| *** csomerville has joined #openstack-infra | 19:09 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 19:11 |
| *** baoli has joined #openstack-infra | 19:11 | |
| clayg | It makes sense. Is is naive of me to think same truth could also be phrased "CI infrastructure doesn't have a staging environment" :) | 19:11 |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/reviewday master: Add a new dashboard https://review.openstack.org/439114 | 19:11 |
| *** salv-orlando has joined #openstack-infra | 19:12 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 19:12 |
| clarkb | clayg: correct, however most things are self testing once you get enough of the base CI configured | 19:12 |
| clarkb | clayg: all changes to devstack-gate and devstack are self testing for example | 19:12 |
| *** rlandy is now known as rlandy|brb | 19:12 | |
| clarkb | clayg: also zuulv3 is path forward to make most changes to the CI itself also self testing. We managed to get the first jobs doing that during the PTG. | 19:13 |
| clayg | clarkb: I assumed the full story was something more like that! | 19:13 |
| clarkb | so hopefully in the not too distant future that will be fixed | 19:13 |
| clayg | clarkb: I even think you had mentioned something like that - where the project-config'y bits are *in* the projects? | 19:13 |
| clarkb | yup thats it | 19:14 |
| *** dharinic|lunch is now known as dharinic | 19:14 | |
| pabelanger | Hmm, nodepool.o.o is dropping | 19:14 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-baremetal project https://review.openstack.org/438907 | 19:15 |
| *** krtaylor has joined #openstack-infra | 19:15 | |
| *** esikache1 has joined #openstack-infra | 19:16 | |
| *** csmart has joined #openstack-infra | 19:17 | |
| *** krtaylor has quit IRC | 19:18 | |
| *** bh526r has quit IRC | 19:18 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Pass ironic microversion through from api_version https://review.openstack.org/439118 | 19:18 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add support for bailing on invalid service versions https://review.openstack.org/439119 | 19:18 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add helper factory functions for use in OpenStack services https://review.openstack.org/439120 | 19:18 |
| *** krtaylor has joined #openstack-infra | 19:18 | |
| *** sdake has quit IRC | 19:19 | |
| pabelanger | ticking upwards again | 19:20 |
| *** yamamoto has joined #openstack-infra | 19:21 | |
| *** armax has joined #openstack-infra | 19:23 | |
| *** krtaylor has quit IRC | 19:23 | |
| SpamapS | jeblair: oh my.. the assignee button in boartty is... not usable | 19:23 |
| *** Apoorva has quit IRC | 19:24 | |
| *** Apoorva has joined #openstack-infra | 19:25 | |
| *** esikache1 has quit IRC | 19:25 | |
| *** chlong has joined #openstack-infra | 19:25 | |
| *** rlandy|brb is now known as rlandy | 19:26 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add helper factory functions for use in OpenStack services https://review.openstack.org/439120 | 19:26 |
| jeblair | SpamapS: no.. it works well in dev, but not in prod. :) i envision an interactive search box at the top of that to filter, but have not written it. | 19:26 |
| *** yamamoto has quit IRC | 19:27 | |
| *** sdake has joined #openstack-infra | 19:27 | |
| openstackgerrit | Melvin Hillsman (mrhillsman) proposed openstack-infra/irc-meetings master: Add OSOps Team meeting https://review.openstack.org/439124 | 19:28 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-baremetal project https://review.openstack.org/438907 | 19:30 |
| SpamapS | jeblair: yeah, the class of the button suggests so :) | 19:30 |
| SpamapS | jeblair: I'm poking at whether I can shoe-horn the ctrl-S search in | 19:31 |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/reviewday master: Add a new dashboard https://review.openstack.org/439114 | 19:32 |
| *** electrofelix has quit IRC | 19:32 | |
| pabelanger | looks like tripleo-test-cloud-rh1 is having problems, which is affecting the ability to launch more nodes in the other clouds? | 19:32 |
| SpamapS | does it depend on AWS S3? ;-) | 19:33 |
| pabelanger | more nodes are coming online now, but tripleo-test-cloud-rh1 is still having some issues | 19:33 |
| *** krtaylor has joined #openstack-infra | 19:36 | |
| *** shardy has quit IRC | 19:38 | |
| *** krtaylor has quit IRC | 19:39 | |
| *** krtaylor has joined #openstack-infra | 19:39 | |
| pabelanger | mudpuppy: we are seeing some delays in launching nodes in osic-cloud1 today too. http://grafana.openstack.org/dashboard/db/nodepool-osic shows some increase in delay from the API calls too. Wanted to check if you are aware | 19:39 |
| pabelanger | mudpuppy: no node failures yet just a slowdown it looks like | 19:40 |
| mudpuppy | pabelanger: Thank you | 19:40 |
| mudpuppy | I hear the internet gods are angry today, with AWS outage and east coast in a tizzy. | 19:41 |
| mudpuppy | I'm looking now | 19:42 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 19:42 |
| pabelanger | mudpuppy: that makes sense | 19:42 |
| *** krtaylor has quit IRC | 19:42 | |
| *** baoli has quit IRC | 19:42 | |
| *** baoli has joined #openstack-infra | 19:43 | |
| *** panda|bbl has joined #openstack-infra | 19:43 | |
| *** krtaylor has joined #openstack-infra | 19:43 | |
| *** Gravious has quit IRC | 19:44 | |
| *** rossella_s has joined #openstack-infra | 19:45 | |
| *** krtaylor has quit IRC | 19:46 | |
| *** krtaylor has joined #openstack-infra | 19:48 | |
| *** krtaylor has quit IRC | 19:50 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/irc-meetings master: Update the QA team meeting chair https://review.openstack.org/439134 | 19:51 |
| *** esikache1 has joined #openstack-infra | 19:52 | |
| *** cloudtrainme has joined #openstack-infra | 19:54 | |
| *** smarcet has joined #openstack-infra | 19:54 | |
| smarcet | fungi: clarkb: ianychoi: https://review.openstack.org/#/c/419667/ thx a lot ! :) | 19:55 |
| *** ijw has joined #openstack-infra | 19:55 | |
| *** baoli has quit IRC | 19:56 | |
| *** jbryce has joined #openstack-infra | 19:56 | |
| fungi | smarcet: sorry for the wait. production zanata upgrade is finally moving ahead tomorrow so they're done testing teh dev server | 19:57 |
| smarcet | fungi: np thx u :) | 19:57 |
| *** rockyg has joined #openstack-infra | 19:57 | |
| smarcet | fungi: could u remind me zanata dev url please? | 19:57 |
| clarkb | smarcet: translate-dev.openstack.org | 19:58 |
| bnemec | pabelanger: Any idea why nodes aren't spawning on rh1? It appears to be largely sitting idle so I don't think it's a load thing. | 19:58 |
| smarcet | clarkb thx | 19:58 |
| pabelanger | bnemec: they are spawning, triple-test-cloud-rh1 is failing to launch them | 19:58 |
| pabelanger | bnemec: Time to ready is over 1.11 hours now | 19:59 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-tripleo-test-cloud | 19:59 |
| *** ijw has quit IRC | 19:59 | |
| pabelanger | mind you | 20:00 |
| pabelanger | our last allocation request for nodepool was | 20:01 |
| pabelanger | 2017-02-28 19:28:06,666 DEBUG nodepool.NodePool: Allocation requests: | 20:01 |
| pabelanger | so, something is going on | 20:01 |
| clarkb | can you manually boot an instance with it? | 20:01 |
| clarkb | er s/with it/in rh1/ ? | 20:01 |
| bnemec | Will try that | 20:02 |
| clarkb | I apologize for all english language butchery in advance :P | 20:02 |
| pabelanger | clarkb: I'm going to restart nodepool.o.o too, to pick up latest nodepool-id changes | 20:02 |
| clarkb | pabelanger: rgr | 20:02 |
| clarkb | pabelanger: once thats in Ican approve the nodepool.yaml change | 20:03 |
| *** baoli has joined #openstack-infra | 20:03 | |
| pabelanger | restarted | 20:03 |
| pabelanger | clarkb: feel free to approve | 20:04 |
| clarkb | kk | 20:04 |
| ihrachys | clarkb: https://review.openstack.org/#/c/433949/ may go in? | 20:04 |
| clarkb | ihrachys: ya I can review that really quick | 20:04 |
| pabelanger | we are launching a ton of nodes now | 20:04 |
| pabelanger | so, something was / is blocking the allocation thread | 20:05 |
| bnemec | Ah, there they go. | 20:07 |
| bnemec | pabelanger: Thanks | 20:07 |
| *** sdake has quit IRC | 20:07 | |
| *** sdake has joined #openstack-infra | 20:07 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 20:08 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 20:09 |
| clarkb | mordred: I am reviewing https://review.openstack.org/#/c/433949/ for ihrachys and noticing that we print gathered facts multiple times at http://logs.openstack.org/49/433949/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/083793b/logs/devstack-gate-setup-host.txt.gz can we print them only once? | 20:10 |
| *** Goneri has quit IRC | 20:10 | |
| clarkb | I don't think ihrachys' change is related to that just noticing it as I review. I will dig into the ansible too | 20:10 |
| *** sdake has quit IRC | 20:13 | |
| *** mrhillsman has quit IRC | 20:13 | |
| clarkb | looks like its triggered off of hostvars[inventory_hostname] ? | 20:13 |
| clarkb | so maybe it is ihrachys's change | 20:14 |
| ihrachys | clarkb: I admit I don't know each detail of it, I copy pasted | 20:14 |
| clarkb | ihrachys: can you push a new pathset without the new debug line and I can fast approve/ | 20:14 |
| ihrachys | thought it's some env passing into the command | 20:14 |
| *** sdake has joined #openstack-infra | 20:14 | |
| clarkb | ihrachys: its clearly not well named :) | 20:14 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Gather /proc/cpuinfo https://review.openstack.org/433949 | 20:14 |
| ihrachys | this? | 20:14 |
| *** baoli has quit IRC | 20:15 | |
| clarkb | ya | 20:15 |
| ihrachys | oh I get it now | 20:15 |
| ihrachys | it's three commands | 20:15 |
| *** baoli has joined #openstack-infra | 20:15 | |
| ihrachys | debug not related to commands | 20:15 |
| ihrachys | :) | 20:15 |
| ihrachys | should we put an empty line to make it more clear? | 20:15 |
| *** harlowja has quit IRC | 20:15 | |
| clarkb | ya and maybe also a comment above the debug saying "this is what prints the facts to the logs" | 20:15 |
| ihrachys | ok sec | 20:15 |
| *** harlowja has joined #openstack-infra | 20:16 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Gather /proc/cpuinfo https://review.openstack.org/433949 | 20:16 |
| clarkb | thanks | 20:16 |
| *** adrian_otto has quit IRC | 20:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add nodepool-id to infracloud-chocolate https://review.openstack.org/438998 | 20:18 |
| amrith | sdague, jeblair, SpamapS, clarkb, re: https://review.openstack.org/#/c/438668/ please see https://etherpad.openstack.org/p/change-438668, the savings are substantial (46% to 49% reduction in memory usage by MySQL). | 20:18 |
| *** panda|bbl is now known as panda | 20:18 | |
| amrith | mordred ^^ | 20:18 |
| amrith | fungi ^^ | 20:18 |
| fungi | wow! | 20:19 |
| bnemec | pabelanger: Hmm, so we have instances now, but no fips are being created. The last new fip request that I see in the logs was around 40 minutes ago. | 20:19 |
| pabelanger | bnemec: yes, that would explain why we fail to launch them | 20:21 |
| *** mrhillsman has joined #openstack-infra | 20:21 | |
| ihrachys | armax: kevinbenton: memory diet for mysql ^ | 20:22 |
| *** nadya has quit IRC | 20:22 | |
| *** nadya has joined #openstack-infra | 20:23 | |
| *** nadya has quit IRC | 20:23 | |
| *** xarses has joined #openstack-infra | 20:23 | |
| *** yamamoto has joined #openstack-infra | 20:23 | |
| *** yamamoto has quit IRC | 20:23 | |
| bnemec | pabelanger: Ah, the fip requests just showed up. I guess I need to be more patient. :-) | 20:24 |
| *** rockyg has quit IRC | 20:24 | |
| *** rockyg has joined #openstack-infra | 20:24 | |
| *** baoli has quit IRC | 20:25 | |
| *** baoli has joined #openstack-infra | 20:25 | |
| mordred | amrith: woot! nicely done! | 20:26 |
| amrith | thx monty, credit to jeblair, clarkb and fungi | 20:26 |
| mordred | bnemec: you're saying there was a big lag in booting instances and requesting fips on the rh1 cloud? | 20:26 |
| mordred | bnemec: at some point when I'm not juggling 100 things I'd love to figure out why that is - I would not expect a large gap | 20:27 |
| bnemec | mordred: Yeah, the instances were launched at about 20:05 server time, the fips weren't requested until 20:19. | 20:28 |
| bnemec | The controller is still basically idle so I still don't think there's a load issue. | 20:29 |
| *** baoli has quit IRC | 20:30 | |
| *** askb has joined #openstack-infra | 20:30 | |
| *** HeOS has joined #openstack-infra | 20:32 | |
| *** salv-orlando has quit IRC | 20:32 | |
| mordred | bnemec: weird. ok. it's not super urgent due to the calls eventually being made - but I'm going to go troll through some logs to see if I can figure out why there was 14 minutes between server launch and fip request | 20:36 |
| *** rockyg has quit IRC | 20:36 | |
| mordred | bnemec: oh - actually - any idea when the servers moved to ACTIVE? | 20:36 |
| mordred | we poll for active before doing next steps | 20:36 |
| *** hashar has joined #openstack-infra | 20:37 | |
| *** esikache1 has quit IRC | 20:37 | |
| bnemec | mordred: Based on the last_updated time of the instance I'm looking at, it went active in about 15 seconds. | 20:38 |
| mordred | bnemec: wild | 20:38 |
| mordred | bnemec: cool - thanks for the info | 20:38 |
| bnemec | np | 20:39 |
| *** rockyg has joined #openstack-infra | 20:40 | |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: add centos-7 jobs to libec, pyeclib, swift https://review.openstack.org/439143 | 20:40 |
| *** esikache1 has joined #openstack-infra | 20:40 | |
| *** dkehn has joined #openstack-infra | 20:41 | |
| *** baoli has joined #openstack-infra | 20:43 | |
| *** esikache1 has quit IRC | 20:44 | |
| *** sdake_ has joined #openstack-infra | 20:47 | |
| *** esikache1 has joined #openstack-infra | 20:48 | |
| *** sdake has quit IRC | 20:50 | |
| *** john-davidge has quit IRC | 20:52 | |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/reviewday master: Add a new dashboard https://review.openstack.org/439114 | 20:54 |
| *** esikache1 has quit IRC | 20:55 | |
| *** ijw has joined #openstack-infra | 20:56 | |
| *** baoli has quit IRC | 20:58 | |
| *** kong has joined #openstack-infra | 20:58 | |
| *** esikache1 has joined #openstack-infra | 20:58 | |
| *** gyee has joined #openstack-infra | 20:59 | |
| *** yamamoto has quit IRC | 20:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Configure regional mirrors for our jobs https://review.openstack.org/439152 | 21:00 |
| *** hashar has quit IRC | 21:00 | |
| *** ijw has quit IRC | 21:01 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 21:01 |
| *** Apoorva_ has joined #openstack-infra | 21:01 | |
| *** baoli has joined #openstack-infra | 21:03 | |
| *** esikache1 has quit IRC | 21:03 | |
| *** sdague has quit IRC | 21:03 | |
| *** matt-borland has quit IRC | 21:04 | |
| *** Apoorva has quit IRC | 21:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add generic tox job https://review.openstack.org/438281 | 21:06 |
| *** rockyg has quit IRC | 21:10 | |
| *** Goneri has joined #openstack-infra | 21:11 | |
| *** cloudtrainme has quit IRC | 21:12 | |
| *** smarcet has quit IRC | 21:12 | |
| *** armax has quit IRC | 21:12 | |
| *** adrian_otto has joined #openstack-infra | 21:13 | |
| *** dave-mccowan has quit IRC | 21:13 | |
| *** cloudtrainme has joined #openstack-infra | 21:14 | |
| *** mugsie|a1t has quit IRC | 21:14 | |
| *** mugsie|alt has joined #openstack-infra | 21:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove unused mergeChangeQueue method https://review.openstack.org/435950 | 21:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use hostname in Nodepool requests https://review.openstack.org/435949 | 21:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Clarify Job/Build/BuildSet docstrings https://review.openstack.org/435948 | 21:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add note about pipelines and dynamic layouts https://review.openstack.org/435947 | 21:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove inscrutable TODO https://review.openstack.org/435946 | 21:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Handle merge mode in repeated project configs https://review.openstack.org/435945 | 21:17 |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| *** hashar has joined #openstack-infra | 21:17 | |
| *** hashar has quit IRC | 21:17 | |
| *** savihou has quit IRC | 21:17 | |
| *** savihou has joined #openstack-infra | 21:20 | |
| *** smarcet has joined #openstack-infra | 21:21 | |
| *** esikache1 has joined #openstack-infra | 21:21 | |
| *** rossella_s has quit IRC | 21:21 | |
| *** yamamoto has quit IRC | 21:21 | |
| openstackgerrit | Brandon B. Jozsa proposed openstack-infra/project-config master: New project - OpenStack-Helm https://review.openstack.org/439159 | 21:22 |
| *** gyee has quit IRC | 21:22 | |
| *** gyee has joined #openstack-infra | 21:23 | |
| *** nadya has joined #openstack-infra | 21:23 | |
| *** dfflanders has joined #openstack-infra | 21:24 | |
| *** savihou has quit IRC | 21:25 | |
| *** smarcet has quit IRC | 21:25 | |
| *** aysyd has quit IRC | 21:25 | |
| *** esikache1 has quit IRC | 21:25 | |
| *** chlong has quit IRC | 21:27 | |
| *** xarses_ has joined #openstack-infra | 21:27 | |
| *** Goneri has quit IRC | 21:28 | |
| *** nadya has quit IRC | 21:28 | |
| *** andreww has joined #openstack-infra | 21:28 | |
| *** andreww has quit IRC | 21:29 | |
| *** andreww has joined #openstack-infra | 21:29 | |
| *** xarses has quit IRC | 21:29 | |
| *** cloudtrainme has quit IRC | 21:29 | |
| *** kgiusti has left #openstack-infra | 21:29 | |
| andreaf | easy review - QA irc meeting chair change after ptl election: https://review.openstack.org/#/c/439134/ | 21:30 |
| andreaf | jeblair: tiny gertty patch for 3.x compatibility https://review.openstack.org/#/c/435506/ | 21:30 |
| *** csomerville has quit IRC | 21:30 | |
| *** ociuhandu has joined #openstack-infra | 21:31 | |
| *** xarses_ has quit IRC | 21:31 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: layout/tripleo: don't run scenario004/upgrade on ocata https://review.openstack.org/439167 | 21:33 |
| *** acabot has quit IRC | 21:33 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs master: Fix nodepool algorithm formatting in zuulv3 spec https://review.openstack.org/439169 | 21:35 |
| jeblair | fungi: can you execute speedy approval please ^? | 21:36 |
| *** dhajare has quit IRC | 21:37 | |
| *** ijw has joined #openstack-infra | 21:39 | |
| *** salv-orlando has joined #openstack-infra | 21:39 | |
| *** chlong has joined #openstack-infra | 21:39 | |
| *** e0ne has quit IRC | 21:41 | |
| *** thorst has quit IRC | 21:41 | |
| *** thorst has joined #openstack-infra | 21:42 | |
| *** gouthamr has quit IRC | 21:44 | |
| openstackgerrit | Brandon B. Jozsa proposed openstack-infra/project-config master: New project - OpenStack-Helm https://review.openstack.org/439159 | 21:44 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 21:44 |
| *** sdake_ has quit IRC | 21:45 | |
| *** csomerville has joined #openstack-infra | 21:45 | |
| dims | fungi : clarkb : jeblair : we are going to ship pbr this evening... is that ok? | 21:46 |
| *** thorst has quit IRC | 21:46 | |
| *** acabot has joined #openstack-infra | 21:46 | |
| clarkb | dims: fine with me | 21:48 |
| dims | thanks clarkb | 21:48 |
| *** Jeffrey4l has joined #openstack-infra | 21:48 | |
| fungi | dims: wfm | 21:50 |
| *** Apoorva_ has quit IRC | 21:50 | |
| *** Apoorva has joined #openstack-infra | 21:50 | |
| *** ijw has quit IRC | 21:52 | |
| *** ijw has joined #openstack-infra | 21:52 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: [suse] Handle pip-and-virtualenv install for opensuse https://review.openstack.org/437071 | 21:52 |
| *** jkilpatr has quit IRC | 21:52 | |
| *** annegentle has quit IRC | 21:54 | |
| *** yamahata has quit IRC | 21:55 | |
| *** pvaneck_ has joined #openstack-infra | 21:55 | |
| *** ijw has quit IRC | 21:56 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate master: DNR: add enable_plugin ironic to subnode local.conf https://review.openstack.org/439044 | 21:56 |
| *** trown is now known as trown|outtypewww | 21:57 | |
| *** chlong has quit IRC | 21:58 | |
| *** pvaneck has quit IRC | 21:58 | |
| jhesketh | Morning | 22:00 |
| clarkb | jhesketh: hello | 22:00 |
| dirk | Are there any guidelines around infra specs? E.g. when they're required and when not? | 22:03 |
| *** Apoorva_ has joined #openstack-infra | 22:05 | |
| *** esikache1 has joined #openstack-infra | 22:06 | |
| *** cloudtrainme has joined #openstack-infra | 22:06 | |
| *** armax has joined #openstack-infra | 22:06 | |
| *** Apoorva has quit IRC | 22:08 | |
| *** krtaylor has joined #openstack-infra | 22:08 | |
| *** ihrachys has quit IRC | 22:09 | |
| dims | check jobs are taking a while in queued... | 22:10 |
| clarkb | dims: we aren't actually booting many nodes it doesn't look like | 22:10 |
| clarkb | pabelanger: any chance your nodepool changes have introduced a regression in our ability to meet demand? | 22:10 |
| *** esikache1 has quit IRC | 22:10 | |
| pabelanger | clarkb: I don't think so, but I am happy to revert and see | 22:11 |
| dims | thanks pabelanger clarkb - no rush per se, just curious | 22:12 |
| *** claudiub has joined #openstack-infra | 22:12 | |
| *** yamamoto has joined #openstack-infra | 22:12 | |
| pabelanger | clarkb: I think we are just seeing slowness from clouds today | 22:13 |
| pabelanger | which is talking launch to boot nodes | 22:13 |
| *** baoli has quit IRC | 22:13 | |
| pabelanger | taking longer* | 22:13 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic for example, if very choppy | 22:13 |
| *** baoli has joined #openstack-infra | 22:13 | |
| pabelanger | started at 1800UTC | 22:14 |
| pabelanger | IIRC, our patch landed after that | 22:14 |
| pabelanger | but I can confirm | 22:14 |
| *** baoli has quit IRC | 22:14 | |
| clarkb | pabelanger: if yo ulook at the graph though it looks like we are fine to start after nodepool is restarted then we tail off | 22:14 |
| pabelanger | clarkb: I don't mind reverting to see. We can put nodepool in emergency file to test | 22:14 |
| clarkb | pabelanger: its possible that its not your code that has regressed but actually shade which we picked up from restart | 22:15 |
| *** HeOS has quit IRC | 22:15 | |
| clarkb | mordred: ^ | 22:15 |
| clarkb | pabelanger: and now that I think about it that seems more likely iven what you were changing | 22:15 |
| pabelanger | Oh, that might be it | 22:15 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Add patrole tempest job. https://review.openstack.org/439189 | 22:19 |
| pabelanger | clarkb: what would you like to do? I have to step out shortly to run an errand, will be back in 90mins | 22:19 |
| clarkb | pabelanger: I'm trying to be in cave mode to do track chair things, but we probably want ot check if new shade has been released and if yes install old version, and restart nodepool | 22:20 |
| pabelanger | okay | 22:20 |
| pabelanger | let me do that | 22:20 |
| *** adriant has joined #openstack-infra | 22:21 | |
| clarkb | looks like 2/9 was most recent shade release so depending on how long nodepool was running before last restart that could be it | 22:21 |
| clarkb | 9/2 for the sane world | 22:21 |
| pabelanger | Ya, we have 1.16.0 installed | 22:21 |
| bkero | 2017/02/09? | 22:21 |
| clarkb | bkero: ya | 22:22 |
| *** Goneri has joined #openstack-infra | 22:23 | |
| *** baoli has joined #openstack-infra | 22:24 | |
| pabelanger | clarkb: 2017-01-25 looks like the last restart | 22:24 |
| pabelanger | based on nodepool.log | 22:24 |
| pabelanger | clarkb: so, I think you are right. Roll back shade, restart nodepool | 22:24 |
| clarkb | sounds good | 22:24 |
| jeblair | do we have any idea what's happening? | 22:24 |
| *** wolverin_ has quit IRC | 22:26 | |
| *** wolverineav has joined #openstack-infra | 22:26 | |
| pabelanger | jeblair: our allocation requests appear to be lacking. According to logs, the last one I see is 2017-02-28 21:03:38,418 | 22:26 |
| jeblair | i don't see the link between that and shade | 22:26 |
| fungi | dirk: no hard and fast rules about requiring or not requiring. generally things like adding a new service we're going to host or significantly altering/replacing functionality in a system we're running gets a spec | 22:27 |
| pabelanger | I do see a lot of NeutronFloatingIPList requests to tripleo-test-cloud-rh1 | 22:27 |
| jeblair | pabelanger: what time was the restart? | 22:27 |
| pabelanger | jeblair: 2017-02-28 17:23:18,098 was first restart | 22:27 |
| pabelanger | 2017-02-28 20:03:38,173 was second | 22:28 |
| *** panda is now known as panda|zZ | 22:28 | |
| jeblair | we have not finished a single node launch calculation since then | 22:28 |
| *** gouthamr has joined #openstack-infra | 22:29 | |
| pabelanger | I count 5 in the logs, if I am reading it correctly | 22:29 |
| jeblair | oh wait, i see them | 22:30 |
| jeblair | yeah, they were fairly regular for the first few minutes | 22:30 |
| jeblair | then they slowed down considerably | 22:31 |
| pabelanger | tripleo-test-cloud-rh1 doesn't look healthly on grafana.o.o, lots of gaps in API requests | 22:31 |
| *** wolverineav has quit IRC | 22:31 | |
| pabelanger | time to ready is pushing 2 hours | 22:31 |
| mordred | oh goodie. that sounds like it's going to be fun to trace | 22:32 |
| pabelanger | jeblair: clarkb: I have to step out for 90mins, can I pass the banton for now? | 22:32 |
| mordred | fwiw, rh1 is the only cloud we can't do create-fip-on-port on if I remember correctly - so we have to do the create/attach dance there | 22:33 |
| jeblair | pabelanger: yes, thanks. | 22:33 |
| *** jkilpatr has joined #openstack-infra | 22:33 | |
| pabelanger | thank you, see you shortly | 22:33 |
| jeblair | i'm going to sigusr2 the launcher to get a thread stack | 22:33 |
| *** wolverineav has joined #openstack-infra | 22:34 | |
| *** annegentle has joined #openstack-infra | 22:35 | |
| jeblair | i don't see any output from that | 22:36 |
| mordred | jeblair: I see a LOT of FloatingIPList calls | 22:37 |
| *** amotoki has quit IRC | 22:37 | |
| *** wolverineav has quit IRC | 22:38 | |
| bnemec | mordred: I'm curious, is that maybe from back when we were running our ancient version of rh1? We've since updated to a pretty vanilla mitaka deployment. | 22:38 |
| mordred | bnemec: oh neat! then that is likely not a problem anymore for you | 22:39 |
| mordred | jeblair: so, I was worried that this was somehow related to the restification work - but I haven't gotten to floating ips yet - and they're reporting quite correctly as running the list command | 22:40 |
| dirk | Fungi: thanks. So does adding opensuse to the distro mirroring require a spec? I don't see one for otjer distros, unless i am missing something | 22:41 |
| jeblair | mordred: there may be multiple issues... | 22:42 |
| mordred | jeblair: yay! | 22:42 |
| *** amotoki has joined #openstack-infra | 22:42 | |
| dirk | fungi: I'd like to maintain the opensuse nodepool images going forward, I have posted a few patches for that in the last few days | 22:42 |
| *** esikache1 has joined #openstack-infra | 22:43 | |
| jeblair | mordred: so, i lack my primary debugging tool since sigusr2 isn't working | 22:43 |
| mordred | jeblair: well, fwiw, the floating ip normalization code hasn't been updated since 13 Dec - which was the next thing I was looking at (to see if I messed up a field that the status is looking for so it's continuing to call list when it doesn't need to) | 22:43 |
| mordred | jeblair: sigusr2 not working is concerning | 22:43 |
| *** baoli has quit IRC | 22:43 | |
| mordred | jeblair: I see this in syslog: | 22:45 |
| mordred | Feb 28 22:37:07 nodepool kernel: [4264569.530937] ptrace of pid 10238 was attempted by: strace (pid 6952) | 22:45 |
| jeblair | that's probably me | 22:46 |
| jeblair | strace is not helpful on a program with 1000 concurrent threads :( | 22:46 |
| mordred | no - almost certainly not | 22:46 |
| jeblair | sigusr2 works for the main nodepool daemon | 22:46 |
| jeblair | but has done nothing yet on the launcher | 22:46 |
| *** amotoki has quit IRC | 22:46 | |
| *** priteau has quit IRC | 22:47 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Undercloud containers job with tripleo.sh https://review.openstack.org/437411 | 22:47 |
| *** esikache1 has quit IRC | 22:47 | |
| mordred | jeblair: I don't see anything in the code that would cause the handler not to be installed with our launcher args | 22:47 |
| jeblair | mordred: strace reports 5518 threads | 22:48 |
| mordred | that's not unimpressive | 22:48 |
| *** gildub has joined #openstack-infra | 22:48 | |
| jeblair | i wonder if it just hasn't context switched to the main thread to execute the stack dump handler; my memory of how python deals with signals is rusty. | 22:48 |
| mordred | oh- I could certainly buy that | 22:49 |
| mordred | jeblair: did strace take a long time to come back? | 22:49 |
| clarkb | yes main thread handles all signals in python | 22:49 |
| jeblair | clarkb: yes, but can it context switch out of a handler? i'm assuming so... | 22:49 |
| *** adrian_otto has quit IRC | 22:50 | |
| *** salv-orlando has quit IRC | 22:50 | |
| jeblair | (i'm assuming they are not special, and our main thread instruction pointer is at line #2 of the handler code right now) | 22:50 |
| *** salv-orlando has joined #openstack-infra | 22:50 | |
| jeblair | mordred: no, strace is fast | 22:50 |
| mordred | jeblair: k. I'm curious as to whether the thread count is trending steadily upwards | 22:51 |
| *** mriedem is now known as mriedem_away | 22:51 | |
| jeblair | mordred: it's increased by about 50 since last report | 22:51 |
| mordred | k. thanks | 22:52 |
| *** ijw has joined #openstack-infra | 22:53 | |
| fungi | dirk: i think the addition of a new distro to our set of nodepool image builds is probably enough of a self-contained effort as to not require a spec. that said, if it helps you in composing your thoughts on the process, feel free to submit one and let me know when you think it's ready for final approval | 22:53 |
| jeblair | mordred: stable since last report | 22:54 |
| *** nadya has joined #openstack-infra | 22:54 | |
| mordred | jeblair: fyi - I'm trolling through shade commits to see if I can find any potential culprits that might narrow our investigation | 22:54 |
| jeblair | mordred: it looked like the time between node launch cycles increased over time (likely exponentially). it may be the case that if we restart the launcher we will have a window of about 10-20 minutes where nodepool is responsive enough to give us info on what all the threads are doing. | 22:56 |
| jeblair | (alternatively, the stack dumps may show up eventually) | 22:56 |
| mordred | jeblair: ok. I have found a code change that touches floating ip handling | 22:57 |
| dansmith | I've seen the check queue at 157 for quite a while and some new jobs have been queued for over an hour and have not started anything | 22:57 |
| dansmith | is something known up with zuul at the moment? | 22:57 |
| dirk | fungi: ok, I'll think about it. On the ine side high level the work is clear to me but there are a lot of nitty gritty details where I'll need help from.someone more experienced/with more privileges going forward. First step is anyhow getting all my current patches in.. | 22:58 |
| *** ijw has quit IRC | 22:58 | |
| dirk | fungi: thanks for your thoughts on this | 22:58 |
| *** Goneri has quit IRC | 22:59 | |
| fungi | dirk: you're welcome. i'm probably not a huge help on the details, but i know people who have done it somewhat recently to point you to if you get stuck | 22:59 |
| clarkb | dansmith: nodepool and yes people are looking at it | 22:59 |
| *** nadya has quit IRC | 22:59 | |
| jeblair | mordred: stack dumps just arrived | 22:59 |
| dansmith | clarkb: ack, cool, just checking | 23:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Partially revert floating ip wait change https://review.openstack.org/439202 | 23:00 |
| *** dtardivel has quit IRC | 23:00 | |
| *** hashar has joined #openstack-infra | 23:00 | |
| mordred | jeblair: that ^^ is a change to FIP processing that I could believe could be responsible | 23:00 |
| *** gordc has quit IRC | 23:01 | |
| jeblair | https://pypi.python.org/pypi/paramiko/2.1.2 | 23:01 |
| jeblair | i suspect ^ | 23:01 |
| mordred | jeblair: I'd be willing to believe that too | 23:01 |
| jeblair | we have 5326 threads in http://paste.openstack.org/show/600857/ | 23:02 |
| mordred | nice | 23:02 |
| jeblair | which is an internal paramiko thread | 23:02 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Plumb job variables through to ansible https://review.openstack.org/438653 | 23:02 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki master: Optionally alias to a favicon.ico file if provided https://review.openstack.org/439082 | 23:03 |
| *** armax has quit IRC | 23:03 | |
| *** john-davidge has joined #openstack-infra | 23:04 | |
| *** hashar has quit IRC | 23:04 | |
| *** remem has joined #openstack-infra | 23:04 | |
| jeblair | we actually have 1.18.2 installed, but that was also released on 2017-02-20 | 23:05 |
| jeblair | https://github.com/paramiko/paramiko/issues/44 looks suspcious | 23:06 |
| jeblair | includes commit "Add back-reference from Transport to the SSHClient that created it" | 23:06 |
| clarkb | I like that the issue is 6 years old | 23:07 |
| clarkb | and fixed 8 days ago | 23:07 |
| *** baoli has joined #openstack-infra | 23:07 | |
| *** remem has quit IRC | 23:07 | |
| mordred | jeblair: that does seem rather suspicious | 23:08 |
| *** jamesdenton has quit IRC | 23:08 | |
| jeblair | the changelog for 1.18.2 does not make it look like a pin to 1.18.1 would have a negative security impact for us. | 23:08 |
| *** john-davidge has quit IRC | 23:08 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update the QA team meeting chair https://review.openstack.org/439134 | 23:08 |
| clarkb | jeblair: and based on the commit message you point out we might be ok as long as we explicitly close things so they can gc | 23:09 |
| *** dhajare has joined #openstack-infra | 23:09 | |
| mordred | jeblair, clarkb: I have abandoned the revert patch I posted earlier - I can re-open it if we don't fix this paramiko-side | 23:09 |
| *** jcoufal has quit IRC | 23:10 | |
| jeblair | clarkb: for our use we can probably turn that into a context manager | 23:10 |
| mordred | jeblair, clarkb: yah to explicit close - I think rolling back to 1.18.1 should give us some indication as to whether or not we need to dig further to solve it properly, yeah? | 23:10 |
| clarkb | mordred: yup | 23:10 |
| jeblair | mordred: yep, sounds like a plan. i will downgrade now | 23:11 |
| jeblair | if it is still working after an hour, we should try the explicit close | 23:11 |
| mordred | jeblair: ++ | 23:11 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: add centos-7 jobs to libec, pyeclib, swift https://review.openstack.org/439143 | 23:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Download promoted images in case of stable branch https://review.openstack.org/439206 | 23:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove final db setup https://review.openstack.org/427895 | 23:12 |
| jeblair | #status log manually installed paramiko 1.18.1 on nodepool.o.o and restarted nodepool (due to suspected bug related to https://github.com/paramiko/paramiko/issues/44 in 1.18.2) | 23:13 |
| openstackstatus | jeblair: finished logging | 23:13 |
| jeblair | it's running now | 23:13 |
| jeblair | and how: http://grafana.openstack.org/dashboard/db/nodepool | 23:14 |
| *** gyee has quit IRC | 23:15 | |
| *** jklare has quit IRC | 23:15 | |
| clarkb | buidling: 1796 | 23:16 |
| *** Apoorva_ has quit IRC | 23:17 | |
| *** jklare has joined #openstack-infra | 23:17 | |
| *** Apoorva has joined #openstack-infra | 23:17 | |
| jeblair | we seem to be overrunning our chocolate core quota | 23:19 |
| clarkb | that probably means we didn't scale up all the proper quota fields when we set to 100 max servers | 23:19 |
| *** baoli has quit IRC | 23:22 | |
| *** dhajare has quit IRC | 23:22 | |
| pabelanger | and back | 23:24 |
| jeblair | pabelanger: nice timing ;) | 23:24 |
| pabelanger | paramiko again? grr, I thought we pinned <2.0 | 23:24 |
| mordred | pabelanger: they also released a 1.18.2 | 23:24 |
| clarkb | pabelanger: we did, they made a 1.x point release | 23:24 |
| clarkb | ya that | 23:24 |
| pabelanger | ah, that explains it | 23:24 |
| jeblair | i'm working on a nodepool patch | 23:25 |
| pabelanger | great | 23:26 |
| *** dprince has quit IRC | 23:29 | |
| *** rossella_s has joined #openstack-infra | 23:31 | |
| *** pradk has quit IRC | 23:32 | |
| openstackgerrit | Merged openstack-infra/infra-specs master: Fix nodepool algorithm formatting in zuulv3 spec https://review.openstack.org/439169 | 23:35 |
| *** erikmwilson has quit IRC | 23:37 | |
| jeblair | clarkb, mordred, pabelanger: i have a small test script which confirms that i get a zombie thread for every paramiko ssh connect attempt. | 23:39 |
| mordred | jeblair: that's awesome | 23:40 |
| *** erikmwilson has joined #openstack-infra | 23:40 | |
| * dims grabs a seat by the fire and pouring myself a drink << (cc @mordred) | 23:40 | |
| mordred | dims: enjoy! | 23:41 |
| *** marst has quit IRC | 23:42 | |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack-infra/reviewday master: Add a new dashboard https://review.openstack.org/439114 | 23:42 |
| dims | whoa! zuul.yaml in own repos :) | 23:42 |
| clarkb | jeblair: mmmm brain | 23:44 |
| clarkb | s | 23:44 |
| mordred | dims: just think of all of the trouble you can cause - I mean solutions you can create!!! | 23:45 |
| dims | hahaha :) | 23:45 |
| dims | bindep + zuulv3 FTW | 23:46 |
| *** yamahata has joined #openstack-infra | 23:47 | |
| clarkb | we need a sign somewhere that says "if you use the zuul footgun to remove your leg it is up to you to glue it back together again" | 23:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add destructor to SSHClient https://review.openstack.org/439243 | 23:48 |
| jeblair | clarkb, mordred, pabelanger: ^ i think that should do it, assuming our hypothesis checks out | 23:48 |
| jeblair | about which, considering our launch interval is still around 10-20 seconds, i'm optimistic. | 23:49 |
| *** cloudtrainme has quit IRC | 23:50 | |
| mordred | dims: speaking of bindep - as soon as my patch lands and we cut a new release, I'm going to make an ansible module upstream that groks bindep | 23:50 |
| mordred | jeblair: given the symptom, I'd expect it to be crawling by now | 23:50 |
| dims | nice mordred | 23:50 |
| clarkb | jeblair: maybe also check thread count? | 23:51 |
| clarkb | jeblair: we launched at least a thousand instances successfully I think that should've leaked if they leaked | 23:51 |
| mordred | jeblair: lgtm - remind me again - __del__ is called on going out of scope? or on being GCd? | 23:51 |
| jeblair | mordred: yeah, we saw symptoms between 6 and 30 minutes last time. | 23:51 |
| jeblair | mordred: when reference count goes to 0 | 23:51 |
| mordred | k. cool | 23:51 |
| clarkb | "when it is about to be destroyed" | 23:52 |
| clarkb | so ya would be ref count 0 leading to GC | 23:52 |
| jeblair | yep | 23:52 |
| *** amotoki has joined #openstack-infra | 23:53 | |
| clarkb | also exceptions will be ignored there and only warning printed to stderr | 23:53 |
| jeblair | i started writing a context manager, but much code would need to be changed. this seems almost as good for now. | 23:53 |
| clarkb | so even if the object gets in a weird state such that close() raises it should be "fine" | 23:53 |
| jeblair | clarkb: yeah. and that's all we would do if we got an exception anyway | 23:53 |
| clarkb | ya not much we could do but warn and move on assuming that GC will be successful eventually | 23:54 |
| *** ijw has joined #openstack-infra | 23:54 | |
| clarkb | I've approved, we may want to double chekc the integration tests first though so feel free to wip if you want to do that | 23:54 |
| *** annegentle has quit IRC | 23:55 | |
| jeblair | mordred, clarkb: we're still backed up enough that stack_dump is not immediate, but strace says 443 threads | 23:56 |
| *** v1k0d3n has joined #openstack-infra | 23:56 | |
| clarkb | jeblair: I would expect far more than 443 if we were still leaking so that looks good to me | 23:56 |
| mordred | agree | 23:56 |
| jeblair | stack dump returned; scanning it visually i don't see errors | 23:56 |
| clarkb | just based on total number of launched nodes being >1k | 23:56 |
| jeblair | i will try to grep, but it's in the same log as the old, so will take a min | 23:56 |
| *** dave-mccowan has joined #openstack-infra | 23:57 | |
| ianw | i wonder if our centos testing images really need epel | 23:57 |
| mordred | if the approach _became_ bad ... moving from __del__ to contextmanager is always an option in the future - but I agree with you, that's a large amount of code churn | 23:57 |
| v1k0d3n | hello infra team. i submitted a patchset earlier today for a new project openstack-helm. have a question for your team. | 23:57 |
| * ianw says to nobody in particular | 23:57 | |
| v1k0d3n | (i have an oppertunity to learn from you all) | 23:57 |
| * mordred waves to ianw | 23:57 | |
| clarkb | ianw: they did when dib required epel | 23:57 |
| mordred | hi v1k0d3n ! | 23:57 |
| clarkb | ianw: but other than that I don't know of any reason for having it | 23:57 |
| v1k0d3n | hey mordred :) | 23:58 |
| v1k0d3n | so we only failed one gate, and that was irc because we already have an IRC channel | 23:58 |
| jeblair | clarkb, mordred: we have about 58 threads in that paramiko function. which seems reasonable under the assumption that in good times, that function actually does work. :) | 23:59 |
| *** csomerville has quit IRC | 23:59 | |
| v1k0d3n | so if i follow this to the "t" would this work for us? | 23:59 |
| v1k0d3n | https://docs.openstack.org/infra/system-config/irc.html#access | 23:59 |
| mordred | v1k0d3n: did you do the channel access modificatoin after that test failed? | 23:59 |
| v1k0d3n | i don't want to lock us out if you know where i'm coming from. | 23:59 |
| clarkb | jeblair: ya | 23:59 |
| *** ijw has quit IRC | 23:59 | |
| mordred | v1k0d3n: and yes - it should totally work - it's just going to add openstackinfra to the access list | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!